If a request is taking a full minute of user CPU time, it's one hell of a mitigation, and anybody who's not a major corporation or government isn't going to shrug it off.

You're right. AI didn't just triple the traffic to my tiny archive's site. It way more than tripled it. After implementing Anubis, we went from 3000 'unique' visitors down to 20 in a half-day. Twenty is a much more expected number for a small college archive in the summer. That's before I did any fine-tuning to Anubis, just the default settings.

I was getting constant outage reports. Now I'm not.

For us, it's not about protecting our IP. We want folks to get to find out information. That's why we write finding aids, scan it, accession it. But, allowing bots to siphon it all up inefficiently was denying everyone access to it.

And if you think bots aren't inefficient, explain why Facebook requests my robots.txt 10 times a second.

AI does not triple traffic. It’s a completely irrational statement to make.

Multiple testimonials from people who host sites say they do. Multiple Lemmy instances also supported this claim.

I would bet that the number of requests per year of s resource by an AI scrapper is on the dozens at most.

You obviously don't know much about hosting a public server. Try dozens per second.

There is a booming startup industry all over the world training AI, and scraping data to sell to companies training AI. It's not just Microsoft, Facebook and Twitter doing it, but also Chinese companies trying to compete. Also companies not developing public models, but models for internal use. They all use public cloud IPs, so the traffic is coming from all over incessantly.

Using as much energy as a available per scrapping doesn’t even make physical sense. What does that sentence even mean?

It means that Microsoft buys a server for scraping, they are going to be running it 24/7, with the CPU/network maxed out, maximum power use, to get as much data as they can. If the server can scrape 100 sites per minute, it will scrape 100 sites. If it can scrape 1000, it will scrape 1000, and if it can do 10, it will do 10.

It will not stop scraping ever, as it is the equivalent of shutting down a production line. Everyone always uses their scrapers as much as they can. Ironically, increasing the cost of scraping would result in less energy consumed in total, since it would force companies to work more "smart" and less "hard" at scraping and training AI.

Oh, and it's S-C-R-A-P-I-N-G, not scrapping. It comes from the word "scrape", meaning to remove the surface from an object using a sharp instrument, not "scrap", which means to take something apart for its components.