LLM scrapers and bots are a continuing challenge. Lemmy is not behind Cloudflare and custom filtering rules using NGINX only gets you so far.

I'm trying the setting called "Private Instance".

It's my understanding that all this does is require a user to be logged in to use the instance. This will hopefully cut down or stop the constant scraping. Preliminary signs are that it helps a ton.

  • walden
you are viewing a single comment's thread
view the rest of the comments
[โ€“] [S] 4 points 2 weeks ago (2 children)

I've heard mostly good things about Anubis, but I've also heard it can be hard to get it right. I've never looked into it so I'm not sure what people are referring to... maybe the balance between blocking lots of bots but still letting humans through?

This Lemmy instance is mostly sticking around for posterity, so hopefully Private Instance cuts down on the B.S. so I don't have to think about it any further!

  • source
  • parent
  • hideshow 2 child comments
  • [โ€“] 3 points 2 weeks ago (1 child)

    I had some challenges setting it up which were mostly related to using load balancing for Lemmy-UI. In general there isn't much of a problem letting humans through so long as you only apply it to the front end UI not to the API.

    We have a rule to allow anyone with a jwt set which means logged in users are not challenged.

    Finding an instance that blocks content from logged out users would likely put off new users, but if you're not looking for new users then requiring login seems like an easy solution to the problem.

  • source
  • parent
  • hideshow 1 child comment