you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 4 months ago

LLM bots absolutely don't care about any stated preferences. Heck, they don't care about robots.txt rules, which try to give hints to search engines what's even sensible to add to the indexes.

However, "this isn't our fault, we told them this shouldn't be scraped, but did this dumb bot listen, well no" is still good policy.

  • source
  • parent