Ireland’s government website, gov.ie, has a quite comprehensive robots.txt file, which by excludes almost all crawlers associated with AI training from scraping the site. One missing however is Mistral.

all 2 comments

sorted by: hot top controversial new old
[–] 22 points 2 months ago (1 child)

jokes on them, LLM companies dont care about robots.txt

  • source
  • hideshow 1 child comment