you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 10 months ago* (last edited 10 months ago) (5 children)

I've got bot detection setup in Nginx on my VPS which used to return 444 (Nginx for "close the connection and waste no more resources processing it), but I recently started piping that traffic to Nepenthes to return gibberish data for them to train on.

I documented a rough guide in the comment here. Of relevance to you are the two .conf files at the bottom. In the deny-disallowed.conf, change the line for return 301 ... to return 444

I also utilize firewall and fail2ban in the VPS to block bad actors, overly-aggressive scrapers, password brute forces, etc and the link between the VPS and my homelab equipment never sees that traffic.

In the case of a DDoS, I've done the following:

  • Enable aggressive rate limits in Nginx (it may be slow for everyone but it's still up)
  • Just stop either Wireguard or Nginx on the VPS until the storm blows over. (Crude but useful to avoid any bandwidth overages if you're charged for inbound traffic).

Granted, I'm not running anything mission-critical, just some services for friends and family, so I can deal with a little downtime.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 4 points 10 months ago (4 children)

    I have something similar with fail2ban + hidden buttons. If the requester goes and clicks on the hidden buttons on the main site, it gets into a rabbit hole. After 3 requests, it gets banned for a bit. Usually stops the worst offenders. OpenAI and some of the scrapers are the worst.

    Google/bing, I do actually see them hit robots.txt then jump off, which is what they should be going.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 10 months ago* (3 children)

    Oooooh. That's smart. I mostly host apps, but in theory, I should be able to dynamically modify the response body and tack on some HTML for a hidden button and do that.

    I used to disallow everything in robots.txt but the worst crawlers just ignored it. Now my robots.txt says all are welcome and every bot gets shunted to the tarpit 😈

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 10 months ago (2 children)

    Nice! Thats another way to do it. 😀

    I know others use Arabis(?) I think thats what it called. The anime girl one that does a calc on top. Ive never had good luck with it. I think bot are using something to get around and it messes with my requests. Might also be my own fiddling.

  • source
  • parent
  • hideshow 2 child comments