521
Google’s AI Is Destroying Search, the Internet, and Your Brain (www.404media.co)
submitted 1 month ago by to c/technology@lemmy.world
61 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 1 month ago (1 child)

I think it’s cute you think the AI sites are paying attention to robots.txt. 🤣

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 1 month ago

    I don’t; I think googlebot is paying attention to it, and AI sites get a LOT of their initial visibility and site ranking from the Google index.

    Case in point: the sites I spend time on that set robots.txt to deny all bots before the LLM wave started still aren’t getting hammered with AI bot requests, while the ones that didn’t are.

    I can see it on my own websites as well; I’ve got one that’s in Google’s index, and it gets a constant low volume traffic from AI bots. The ones that aren’t listed? They just get traffic from exploit scanners.

  • permalink
  • fedilink
  • source
  • parent