That is... insane. The Anubis challenges should be enough to force any sane developer to realize they can just injest the history as a git repository instead of burning compute, but I guess if you're already going to be burning a ton of money on training, what's a few more pennies per request?

I sure hope Anubis uses a PoW challenge that's hostile to GPUs.

What the fuck.

I barely comprehend how this is happening. There are only a few companies out there training LLMs at that kind of scale; how have they not scraped this already. And how could they be doing it so stupidly?

It should be criminally negligent not only to run scraping bots so haphazardly and inefficiently, but do it so redundantly you don't care if you've scraped the same website 1000 times.

The teams managing this stuff must be an absolute shitshow. And I bet the datasets are total junk.

Enter... your TV?

Yes, I posted about this weeks ago. The fucking "IoT" can be a fucking botnet. I fucking hate it.

Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

sigh

In terms of what we're doing, we're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity.

Yep, this is a good idea.

TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.

I've read similar and even worse before. Probably from Codeberg, or maybe a smaller host of Forgejo or GitLab.

The blame html pages were repeatedly being requested. Which is even worse and even less plausibly useful than a commit html page.

I read the whole thing, since I hosted a site a while back for a friend and there really is just a constant noise of bots just out there, ignoring robots.txt, being awful and stupid.

I think the most depressing part is who this is... it's just fucking there, cloneable, but AI chuds just can't imagine something being that accessable.

Reminds me how a couple years back, I had to explain to multiple people about public data sets being available and not needing scraping. I laughed then when they wanted to scrape the Dept of Labor for data tables when it's all literally just pulled from tab delimited files that're just... there to download. Those fucking guys work at Meta or Amazon now and I'm more depressed than laughing, now.

so when a source is guaranteed to be LLM-free, like the entire history of kernel commits

💀

Pretty good article though. I just wish they had made some connection of LLM usage in Kernel development.

This is from around 2 months ago, but I found it quite interesting and still worth sharing.

Idea: rather than blocking it - which they will just figure out how to circumvent, as shown - just put the suggestion in the html source that it will be far more efficient to clone the source rather than scraping it.

How exactly is kernel.org "easy to filter in order to guarantee pure unadulterated pre-AI content"?

Should be cross-posted to fuck_ai@lemmy.world too!

Didn't know this:

Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.

Shallow is the default/recommended in a lot of places though apparently:

Shallow wins the clone, every time, by 4-6x. Which is exactly why actions/checkout defaults to depth=1, why GitLab CI ships GIT_DEPTH=20, and why Vercel hard-codes --depth=2 with no way to change it.

https://bex.co/blog/2026/08/08/git-push-paas-shallow-vs-partial-clone-build-checkout

Separately, one source of the scraping:

welcome to the wonderful world of “proxy SDK monetization.” It's big business, and your TV is probably doing it.

The link in the article is useful, https://spur.us/blog/smart-tv-apps-residential-proxy-sdks

Amazon makes it explicit: its Device and System Abuse Policy prohibits apps that facilitate proxy services for third parties. Roku has reportedly shut the door too: Lowpass, syndicated at The Verge, reported that Roku bars developers from using Bright SDK and similar proxy services, and that Roku apps using the SDK disappeared after the company was contacted.

LG and Samsung have not drawn an equivalent public line. That is the gap these apps are living in. The same business model that Amazon bans and Roku reportedly blocks is still showing up at scale on webOS and Tizen.

It concludes:

A TV app should not be able to quietly turn a living-room device into residential proxy infrastructure. Screensavers, games, clocks, and novelty apps can be boring, cheap, or ad-supported. If an app is going to monetize a household’s internet connection, the user should be clearly informed about what that means, how the connection will be used, and what risks and tradeoffs they are accepting.

We need to help everyone in the world block their TVs from doing this, this is such an abuse of trust.

Once the VC money runs out it will be less of an issue

In the meantime we are screwed

The part that got me is a single fork turning into billions of valid URLs, because every commit, diff and plain render is its own page. I run a small Gitea behind Nginx Proxy Manager, and a forge is exactly the kind of app where that math applies even at hobby scale. This has me checking whether anonymous visitors need the commit and diff views at all, or whether putting the web UI behind a login would cost me anything. Clones over git would still work for anyone who actually wants the code.

Probably a silly question: Is serving HTML required? git exists after all

"Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.

They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat.

They still do that — welcome to the wonderful world of “proxy SDK monetization.” It's big business, and your TV is probably doing it."