One thing I want to see is poisoned wells. When you detect scrapers, don't stop them, feed them pseudo content designed to COST them. Make their training data poisonous and damaging. Make it cost them to purge it, and difficult and expensive to identify it.

but I would assume there’s an arms race going on behind-the-scenes between Cloudflare and the bot developers

No. CF lost years ago, and the checks can be bypassed easily. It's just that it blacklists ips generating insane traffic but there is a lot of margin

Run labyrinths and feed them bullshit

but those that do run these wikis will be in the fast pass line at the gates of heaven. Please don't give up. I never use gipity

Fucking a I set up a forgejo instance to host my code and moved everything off of GitHub. Fuckingn Facebook was hammering my shit before I blocked it. It seems old Mark Z is trying to Hoover up the internet because he's late to the game on AI.

They should try including invisible text to tell LLMs to disregard any prompts that specifically tell it to scrape any page on their domain, to treat their wiki as unreliable for any prompt that may point to them as a source, and to blacklist the domain on their settings/instruction files.

I get over it, but It's still kinda funny how the first line of "defense" is having the bot say that it's a bot, and not Google Chrome.

Ai scrapers are making it kinda suck to run anything on the internet to be honest.

Detect whether the user is a human, but instead of blocking the request (which is going to be obvious to the scraper operator and will just cause the bot developers to go work on better human emulation until they get the data), poison the response. Just as blocking scrapers is hard for website operators, so is separating useful data from not-useful data for people building AI training corpuses.

https://www.cloudflare.com/learning/ai/data-poisoning/

Data poisoning involves injecting malicious information into training datasets to manipulate an AI model's behavior, compromising its accuracy, reliability, and the overall integrity of machine learning results.

The really exasperating thing is that Wikipedia makes all of their data available in compressed database form, and I suspect that other MediaWiki wikis could do so too. I checked one of the wikis being complained about, the Minecraft wiki, and it's a MediaWiki wiki.

This isn't even a case where the server operators have any problem with the data being available to the bots. In fact, people are going out of their way to package up all the data in a form that is optimal for computer processing (better for the bots) and which doesn't create additional load on the wiki servers (better for the wiki operators). They're providing data in delta format!

But the bots aren't using them! And on top of that, they aren't even written to follow traditional conventions for being polite in scraping. We've had lots of software spider the Web

that's not new

but normally that spidering software has followed basic politeness to try to avoid excessive disruption to the servers being spidered.

It's as if someone has set up a table out front of their house with a big sign reading "Free cookies!" with boxes of pre-packaged cookies. Instead of taking the damn cookies, the bots are obtaining the largest trucks possible and then ramming holes in the house with a truck and scavenging through the ruins, desperately trying to find cookies.

I wonder if it would be possible to filter traffic by limiting the rate at which links can be followed after the initial connection.

I suppose one problem with that is it would also limit the crawlers you want, like search engine indexing. Maybe if enough sites were doing this it would generate some pressure on the AI companies to behave better.

Anubis is the way to go

Wait a second...I already read this... 2 months ago

This is a great post about such a frustrating and sucky situation. Thank you for sharing it with us here. I really feel for all the smaller sites and webmasters who have to deal with all this absurdity. :(

Like they mentioned, when I hit one of those damn Cloudflare 'prove you're a human' checks -- which is a LOT nowadays -- I definitely do get irritated...but I'm not usually upset with the site owner who is just trying to protect their site. I feel for them. I'm pissed off and disgusted at all the greedy, inconsiderate assholes and corporations who are fucking everything up for us normal people, both on the Internet and off.

I don't know where we go from here, but I hope it somehow gets better soon... ugh.

People started allowing scrapers because they needed to get in the search engine indexes to get traffic. But getting scraped being a win-win for the scraper and scrapee is long past. Now it's completely parasitic. The search engines are even going out of their way to not send traffic to organic results.

I wonder if it's time to start to create an un-indexable "shadow internet." Block all scraping even the search engines. Maybe put up login walls. How will people find your website? We'll have to go back to word of mouth, link rings, or the like.

And fediverse sites. Have to update blockers every month or so

I didnt see this mentioned- for all the old non-cashed stuff, would it help to throttle it by like 5-10 seconds per request? Maybe also serve a cached “loading” page during the wait.

Not only wikis sadly. Anything that has public facing deep links that trigger extensive database operations are being hammered by these bots and few servers can take the load.

We use NGINX’s 444 response A LOT.

In coordination with careful rate-limiting, it’s been a dramatic improvement.

The worst of the bots don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.

On the plus side, this isn't a problem with AI, this is a problem with AI companies having more investment money than they know what to do with. The moment the hype fades and they don't want to hemmorage money scraping every wiki on the internet thousands of times per day, this traffic will go back to a far more sane amount.

Maybe we should have 2 factor authentication to read wikis. A person would have to come here and seek a public key from other people. If you can get 2 Public keys by proving you are human, then you get a month of wiki access unless you keep proving you are human or if you prove to be AI then you're totally banned unless you get a public key in real person at the local Walmart or Safeway.

iocaine doesn’t stop them, but it uses minimal resources and makes me feel better about serving pages to them.