8
AI scrapers everywhere Idea (palaver.p3x.de)
submitted 6 months ago by to c/piefed_dev@piefed.social
6 comments fedilink hide all child comments
 

There's a nice article: The Day I Logged 1 In Every 2000 Public IPv4: Visualizing The AI Scraper DDoS which someone kindly dropped in the fuck_ai community. It's worth reading.

After reading it, I decided to have another look at my logfiles and I'm pretty sure I see the same thing. A good while back I had my small 1-3 people instance nearly melt because of Tencent and Alibaba. I managed to address that by blocking some of their IP ranges. Proceedingly rimu implemented a Honeypot, which currently has 14.272 IPs blocked on my instance. And a few database requests got more efficient. But I think the overall situation deteriorated. These days the AI scrapers seem to come from everywhere on the internet.

My logfiles just scroll by again. It's not as bad as when I was hit by the really nasty crawlers. The server keeps up with it, I'm just constantly sending out loads of data. There's a crawler getting caught up in the honeypot almost every minute. But there's also still a near infinite amount of addresses left, querying all kinds of posts, communities and user profiles. And I notice since it's a dysproportionate amount of traffic for a small instance.

I think the honeypot is great. But the AI scrapers have way to many addresses. And in the mid-term we probably need to come up with some more mitigations. At least to cater to smaller servers.

Just wanted to draw some attention to how things change. I've switched my instance to "private" for now, but I'll continue to investigate.

I've made two images myself (for palaver.p3x.de). First is all IPv4 addresses in my nginx log. That includes all Fedi instances and users. But I'm just a small instance and I guess most of the reddish areas are caused by crawlers.

AJvKYRbM1j1amZm.png

Second image is what's in my honeypot, (dots visible after zooming in):

1y7biwvTHxw9Q8q.png

you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 6 months ago (2 children)

For single-user instances, where the only user will be logged in continually, setting it to private is really easy and comes with basically no downside. Even if you want to provide anonymous access, often going private for a day or two is all it takes to shake the scrapers so you can open up again later. I did this on piefed.social last week.

Also I found many many scrapers are using quite old Chrome versions (or their user agents use randomly generated version numbers) so blocking those at the web server level pwns 90% of that family of scrapers. e.g. on nginx you'd put this in your server block:

    # Scrapers often use old user-agents  
    if ($http_user_agent ~* "^(?!.*(Googlebot|bingbot)).*(Chrome/([0-9]{1,2}|10[0-9]|11[0-9])\b|Firefox/([0-9]{1,2}|10[0-9]|11[0-4])\b)") {  
        return 403;  
    }  
    # Block dead browsers / engines  
    if ($http_user_agent ~* "(Opera/[0-9]|Presto/|MSIE|Trident/)") {  
        return 403;  
    }  

    if ($http_user_agent ~* "(Sogou web spider|SERankingBacklinksBot|WF search)") {  
        return 403;  
    }

If using Cloudflare you can have it present a "are you a human" challenge to big chunks of the address space using AS numbers. I have AS45899 blocked, which gets rid of most of Vietnam and I've also got Country = China as a criteria. That helps a lot.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 1 point 6 months ago* (last edited 6 months ago) (1 child)

    Thanks. I already had an older version of that in my config. I'd say from looking at the logs, at times it catches every 10th crawler or so.

    I blocked most nasty address ranges from AS45899 yesterday, as well.

    I'm living the Cloudflare -free lifestyle 😅. I got geoip2 running, though. Not sure if I want to hand out some collective punishment against the Chinese. Looking at my honeypot with Tencent and Alibaba blocked, they don't even make the cut with the single digit amount of addresses blocked. Main offenders are: BR (2120), US (864), IQ (805), IN (555), BD (535)... At least as of right now. I guess in a few weeks we might be looking at a different situation anyway.

    Edit: And I'm still being bombarded 12h after switching the instance to 'private'. They seem to move towards the login page now, with the next argument set as in the 302 forwards. But they seem to also be still working on a large backlog of regular URLs to crawl. And it's still multiple requests a second. I'll look at how they're going to adapt.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 1 point 6 months ago

    Update: It's calm again. I've reverted to public instance. But I'll send a 402 "Payment required" to browser agents who claim to use Windows or Mac OS X. People should be using Linux anyway 🙃

    It's hard to tell what's going on here. I thought maybe it's a good sign the scrapers adapt in some way, we might be able to trigger that behaviour in some other ways. But it's completely unclear to me if something like a more aggressive honeypot will be enough to pull it off.

  • permalink
  • fedilink
  • source
  • parent