165
Cloudflare wants Google to change its AI search crawling. Google likely won’t. (arstechnica.com)
submitted 1 year ago by to c/technology@lemmy.world
26 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 20 points 1 year ago (9 children)

If I had to choose my favourite corporation, it would be Cloudflare. They at least do something good.

  • permalink
  • fedilink
  • source
  • hideshow 9 child comments
    •
  • [–] 11 points 1 year ago (4 children)

    There seems to be a line, so far as I can tell. If everything you need sits on the free tier, they're really good (well tbh their R2 storage is reasonably priced too). But once you stray into needing a paid tier, it apparently (I'm not there) quickly gets expensive as you're lured into every higher tiers.

    But yes, in general I don't mind cloudflare so much and do use their free (and R2 paid) services.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 1 year ago

    I thought I'd reply to this with an update. Because, I saw how far the free tier goes, and it's pretty far.

    I had another huuuuge influx of AI/other bots scraping my instance at top speed. Hundreds of requests per second, and it was putting some load on the postgres server.

    What I found was, that there was a mixture of traffic. Some was coming from a handful of AS numbers (each hosting hundreds of large IP blocks) controlled by a small handful of the same names. So, those I was able to block outright by AS number.

    But then I found a very large number of random requests coming in bursts (and definitely not humans) all on mobile or isp customer blocks.. I assume it's some kind of botnet being used? But they were all valid requests for posts and comments.

    I looked at the custom ruleset on cloudflare and, it's quite powerful. I settled on the following.

    1: Allow known fediverse software by user agent (yes, the bots could eventually spoof these. But right now they are not). 2: Allow known instances by IP blocks 3: Allow access to the fediverse inbox specifically. Which is where most inter-instance traffic goes. 4: Allow access to LOCAL users and well known services/other standard ActivityPub urls 5: Everything else, for everyone else. Managed challenge.

    The traffic just completely stopped dead. The fediverse traffic continued unfettered. But the traffic coming in was legitimate (it's mostly me, a handful of others that is so little traffic).

    All it adds, is the interstitial page with the "are you human?" checkbox that for most people automatically checks. And the user moves on fine and can interact normally with the site. So for people it's a very minor inconvenience and it stops bot traffic completely in its tracks.

    What is annoying. I could make this MUCH better with regex matches. But, not only do they not allow free accounts to use regex (I understand). But "Pro" users cannot either. It's only for business or above... Business accounts are eye wateringly expensive for a hobbyist!

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 1 year ago (1 child)

    It would be fine if it was just "lured", but this made me very sceptical of cloudflare: https://robindev.substack.com/p/cloudflare-took-down-our-website

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    This was actually the story I had in mind when I wrote my comment. In my case, I'm using cloudflare for this mbin instance, another unrelated low traffic site, and R2 for the media on the instance. It's so small that it will never really escape their free tier.

    But yeah, if you're doing something that is scaling up this is definitely something you need to be aware of.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 7 points 1 year ago (3 children)
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 4 points 1 year ago (2 children)

    I still think they are good. Isolated incidents like this are going to happen when you are doing business at such scales.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago* (last edited 1 year ago) (1 child)

    This did not sound like an isolated incident at all. You don't get sales responding to a legal/engineering issue by accident. It may have been unintended by leadership, if they put too much pressure on sales not realizing how it was corrupting the company, or the leadership may have tacitly approved of this. Hard to tell.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •