LLM scrapers and bots are a continuing challenge. Lemmy is not behind Cloudflare and custom filtering rules using NGINX only gets you so far.

I'm trying the setting called "Private Instance".

It's my understanding that all this does is require a user to be logged in to use the instance. This will hopefully cut down or stop the constant scraping. Preliminary signs are that it helps a ton.

  • walden
you are viewing a single comment's thread
view the rest of the comments
[โ€“] 3 points 2 weeks ago (1 child)

I had some challenges setting it up which were mostly related to using load balancing for Lemmy-UI. In general there isn't much of a problem letting humans through so long as you only apply it to the front end UI not to the API.

We have a rule to allow anyone with a jwt set which means logged in users are not challenged.

Finding an instance that blocks content from logged out users would likely put off new users, but if you're not looking for new users then requiring login seems like an easy solution to the problem.

  • source
  • parent
  • hideshow 1 child comment