521
Google’s AI Is Destroying Search, the Internet, and Your Brain (www.404media.co)
submitted 1 month ago by to c/technology@lemmy.world
61 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 46 points 1 month ago* (5 children)

I don't think today's bots take any notice of robots.txt. It's a relic from a more civilized time.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 13 points 1 month ago (3 children)

    I run a few personal sites with some being available to the wider internet, and I have found that they generally do respect it. All the bots which advertise themselves as OpenAI, Anthropic, or Google do obey. However, I do occasionally see a stealth bot appear which uses a normal browser's user agent and those just do whatever they want.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 12 points 1 month ago* (last edited 1 month ago) (2 children)

    Meta absolutely does not respect robots.txt. they are scraping what I have at around 200 hits / minute.

    I also notice there is some data center somewhere (or multiple) proxying all its requests through residential proxies so I can't tell who is scraping. They don't scrape like Meta though. Far slower. These are the stealthy bois using spoofed user agents you mention. Can't tell who they are, though. How do they get so many residential IPs?

    I have no organic traffic (I am literally just running a crawler tarpit. My page has nothing) so its really obvious that Im watching AI scrappers. My tarpit generates random links that all resolve to the same place so its super obvious its not human. Its also super obvious when two distinct IPs follow the same random word salad link milliseconds apart.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 5 points 1 month ago

    It is a business model. People install free apps on their phone. These apps generate their income by acting as residential proxy for scrapers. They form a botnet of millions of ips, actual phones, without the owners knowing it.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 1 month ago

    I'll have to double check my stats later. Haven't looked at it for a while.

    I certainly don't recall them ignoring robots.txt, but I also don't remember seeing Meta in my dash at all, so it's entirely possible they, for wherever reason, never found my site.

  • permalink
  • fedilink
  • source
  • parent