LLM scrapers and bots are a continuing challenge. Lemmy is not behind Cloudflare and custom filtering rules using NGINX only gets you so far.

I'm trying the setting called "Private Instance".

It's my understanding that all this does is require a user to be logged in to use the instance. This will hopefully cut down or stop the constant scraping. Preliminary signs are that it helps a ton.

  • walden
you are viewing a single comment's thread
view the rest of the comments
[โ€“] 2 points 1 week ago

dbzer0 is using haproxy with haphash successfully to mitigate.

https://github.com/dgl/haphash

He might be willing to share the uri/paths to approve in your proxy config to deploy haphash or anubis etc.

  • source