550
The Open-Source Software Saving the Internet From AI Bot Scrapers (www.404media.co)
submitted 1 year ago by to c/opensource@lemmy.ml
98 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 1 year ago (1 child)

That's not true, search indexer bots should be allowed through from what I read here.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 8 points 1 year ago

    If you allow my searchxng search scraper then an AI scraper is indistinguishable.

    If you mean, "google and duckduckgo are whitelisted" then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot.

  • permalink
  • fedilink
  • source
  • parent