40
old.lemmy.today is now behind Anubis for bot protection (edit: not anymore) (lemmy.today)
submitted 3 months ago by to c/lemmytoday@lemmy.today
20 comments fedilink hide all child comments
 

Hi guys,

We are now trying out Anubis in front of https://old.lemmy.today/ to see if we can block many of the bots scraping lemmy content.

You will notice an image saying Anubis is checking if you are a bot, and then things should work normally after that.

There are many millions of ips trying to scrape lemmy content so blocking on ip level is difficult. Hopefully this improves the situation somewhat. :)

Thanks to @db0@lemmy.dbzer0.com for sharing info about Anubis!

Edit: It was a lot better to just turn off public access of posts on old.lemmy.today, so we are actually not using Anunis anymore. Just log in to old.lemmy.today and things will work as normal.

all 20 comments

sorted by: hot top controversial new old
[–] 6 points 3 months ago (6 children)

Wonder what it means for Lemmy clients if this practice will be expanded to the main URL.

Will mobile apps support it? Or is only old.lemmy.today planned to be protected?

  • permalink
  • fedilink
  • source
  • hideshow 6 child comments
    •
  • [–] 6 points 3 months ago (5 children)

    Pretty sure the API should be exempt, as well as user agents that don't impersonate browsers. So that should be fine, the only problem is that you need JavaScript to get through the challenge when using a browser (which is a bit unfortunate, as old.lemmy.today doesn't otherwise require JS).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] [S] 7 points 3 months ago (1 child)

    Yeah the API should be exempt. Will look into this a bit more in the coming days but wanted to see how well Anubis works on old.lemmy.today first for a while. We probably end up using it on lemmy.today as well. Just didnt want to make a big bang change everywhere on day 1. :)

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • [–] 6 points 3 months ago (3 children)

    Glad to see it working. Strange though, it seems not all comments here are federating to us. Only this chain federated

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 3 points 3 months ago (1 child)

    Lemmy federation has been known to be buggy in certain releases before, and I really dont know what to do about it except try to keep up to date about bugs. I guess its a complex system underneath. From what ive read, its a queue of federation work that is supposed to take place, and stay in the queue until its successful. But I wish there was a good way to see the status of it.

    I have been looking in db tables and log files before to figure out the current status and it works, but its not so convenient.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • [–] 4 points 3 months ago (1 child)

    Anubis seems to do a good job, most of the GitLab instances I've seen outside of gitlab.com use it and I've never had any problems with it even on lower end/mobile devices. Hopefully it cuts down on the bot issues, I love using old.lemmy.today.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 1 point 3 months ago (5 children)

    never heard of anubis, wth is that

  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] 7 points 3 months ago (3 children)

    You have very probably seen its logo on various websites that are using it, showing up before the content does:

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] [S] 5 points 3 months ago*

    It sits between your web browser and old.lemmy.today and detects if you are a bot. If you are, your request doesn't pass through Anubis. It's Gandalf saying "you shall not pass".

    So it will hopefully cut down on bots scraping the instance. Or it will piss them off even more. :)

  • permalink
  • fedilink
  • source
  • parent
    • [–] -4 points 3 months ago

    Thanks for your hard work!

  • permalink
  • fedilink
  • source
    •