you are viewing a single comment's thread
view the rest of the comments
[–] 71 points 1 year ago (1 child)

There's no excuse for this crap. Even if they insist on scraping every FOSS repo, there needs to be some logic to it (caches, diffs, longer intervals). These AI scrapers are so poorly thought out they are indistinguishable from DOS attacks.

  • source
  • hideshow 1 child comment
  • [–] 27 points 1 year ago*

    You're not wrong.

    Claudebot took down one of my sites repeatedly, hammering it for the same pages over and over at horrendous rates.

    I ended up spending several days having to convert it to a SSG and hosted it on Cloudflare pages. A lot of work I didn't need to do.

  • source
  • parent