you are viewing a single comment's thread
view the rest of the comments
[–] 24 points 1 year ago (1 child)

How is blocking scrapers easy?

This instance receives 500+ IPs with differing user agents all connecting at once but keeping within rate limits by distribution of bots.

The only way I know it's a scraper is if they do something dumb like using "google.com" as the referrer for every request or by eyeballing the logs and noticing multiple entries from the same /12.

  • source
  • parent
  • hideshow 1 child comment