you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 10 months ago (4 children)

I have something similar with fail2ban + hidden buttons. If the requester goes and clicks on the hidden buttons on the main site, it gets into a rabbit hole. After 3 requests, it gets banned for a bit. Usually stops the worst offenders. OpenAI and some of the scrapers are the worst.

Google/bing, I do actually see them hit robots.txt then jump off, which is what they should be going.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 10 months ago* (3 children)

    Oooooh. That's smart. I mostly host apps, but in theory, I should be able to dynamically modify the response body and tack on some HTML for a hidden button and do that.

    I used to disallow everything in robots.txt but the worst crawlers just ignored it. Now my robots.txt says all are welcome and every bot gets shunted to the tarpit 😈

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 10 months ago (2 children)

    Nice! Thats another way to do it. 😀

    I know others use Arabis(?) I think thats what it called. The anime girl one that does a calc on top. Ive never had good luck with it. I think bot are using something to get around and it messes with my requests. Might also be my own fiddling.

  • source
  • parent
  • hideshow 2 child comments