you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 1 year ago

I don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling

  • source
  • parent