I found the same IPs doing the same thing for my server, but one thing I noticed in the access log was that nginx was returning a 499 status code. That code means that the client closed the connection before the server answered the request. So this seems to be a deliberate attack instead of the rash of bots many have been dealing with recently. They just firehose out requests to DoS the server since pagination on services with dynamic data is expensive.

I ended up creating a fail2ban rule to add any IP to my firewall blocklist that makes a bunch of 499 entries.

Edit: I also set a rate limit in nginx for any url that has a "page" query included

Get a blocklist and set it up.

Literally all of the IPs are known bots for up to 3 years:

• https://www.abuseipdb.com/check/134.19.178.167
• https://www.abuseipdb.com/check/213.152.162.5
• https://www.abuseipdb.com/check/134.19.179.211

Oh and maybe also a rate-limiter...

That "jackass" sounds like an AI training set scraper. They're known for being incredibly brutal to the sites they scrape, ignoring robots.txt and other honor-based systems for preventing the site from getting overloaded.

FYI these are all on ASN 49453

The other (lazier) option is to block/challenge the ASN

Thanks for sharing

Sounds like such unscoped requests should not be allowed in the first place? Maybe worth reporting in a Lemmy issue?

For Cloudflare users:
Security Rules:

(http.request.uri.path eq "/api/v3/comment/list" and not http.request.uri.query contains "post_id")

For Caddy users:

  # >>> Specific handler for /api/v3/comment/list with post_id check
  handle_path /api/v3/comment/list {
    # Check if the 'post_id' query parameter is present
    @hasPostId {
      query post_id=*
    }
    # Abort the connection if the parameter is missing
    handle @hasPostId {
      reverse_proxy http://localhost:8536/
    }
    # This handles all requests that did not match @hasPostId
    abort
  }

This is for Lemmy I presume (or also for Piefed or Mbin)? You've modified yours heavily though, I thought, which could complicate matters. I wonder if you are having those bot scraping issues that semi-recently (a month or so ago?) started increasing in frequency. So many instances now have a human detector before letting you in whereas before it was not necessary.

We had this issue on and off for a few weeks at least, causing massive postgres CPU spikes. I ended up blocking large page params with an nginx regex.

Maybe we should introduce a gated API and charge $12 for 50k requests...

Isn't the same endpoint used to list all of a user's comments in their profile?

Things have been slow for me off and on in recent weeks. And today it’s quite slow.

Not sure if there's a legit use for just fetching only comments outside of a post

The ability to see all comments is right there at the Lemmy UI.

To everyone in this thread, if you notice a problem in Lemmy please open an issue. We are only two developers and dont have time to browse the Fediverse all day to come across such things. Only if we know about a problem can we actually fix it and make a new release.

For reference here are the issue and proposed fix:

• https://github.com/LemmyNet/lemmy/issues/6016
• https://github.com/LemmyNet/lemmy/pull/6017

I think this could end up blocking https://lemmy-meter.info/ They make requests without post id

Lulz, imagine being so butthurt over people being upset that you were manipulating their online experience without informing them first.