Ireland’s government website, gov.ie, has a quite comprehensive robots.txt file, which by excludes almost all crawlers associated with AI training from scraping the site. One missing however is Mistral.

you are viewing a single comment's thread
view the rest of the comments
[–] 22 points 2 months ago (1 child)

jokes on them, LLM companies dont care about robots.txt

  • source
  • hideshow 1 child comment