everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 year ago

It'll still slow them down and reduce load on your server. I also think many of these crawlers focus on volume; time spent computing the hash is time not spent crawling someone else's site.

  • source
  • parent