Hi,

We had a DDOS attack against old.lemmy.today which overloaded the entire server for about 20 minutes. I had to turn off old.lemmy.today temporarily and will look through logs and probably block more chinese ip ranges, since thats where it came from.

Always something to do. :) Keeping this as sticky for a while to inform everyone.

Edit: I added more ip blocks for Chinese networks and hopefully it will keep future bots away. Bringing up the old.lemmy.today user interface again...

Edit 2: Had to take it down again due to massive bot scraping. Currently down.

Edit 3: Decided to require login for old.lemmy.today instead of using anubis.

all 61 comments

sorted by: hot top controversial new old
[–] 15 points 3 months ago (1 child)

I don't have an account there but I am always happy when fediverse instance admins are transparent about their problems. This will help other instances too.

  • source
  • hideshow 1 child comment
  • [–] 12 points 3 months ago (3 children)

    When I was running a web server this was a common occurrence. Most of the DDOS attacks I experienced came from China.

  • source
  • hideshow 3 child comments
  • [–] [S] 11 points 3 months ago (2 children)

    Yeah its very common. Some instances block the entire Chinese internet for this reason...

  • source
  • parent
  • hideshow 2 child comments
  • [–] 6 points 3 months ago (1 child)

    This is a genuine question, but why doesn't this instance? Is it because we would miss out on some content? I mean you are much more in touch with the technicalities, but do we have a mandarin speaking user base you're not trying to alienate?

  • source
  • parent
  • hideshow 1 child comment
  • [–] [S] 10 points 3 months ago (6 children)

    Alright, https://old.lemmy.today/ is back up, as well as the other alternative user interfaces mentioned in the sidebar.

    I have blocked more chinese ip ranges found in the log files, so hopefully it helps.

  • source
  • hideshow 6 child comments
  • [–] 6 points 3 months ago (5 children)

    While you're here, I published a small update yesterday (26.3.4) that fixes a few more bugs I've found :)

  • source
  • parent
  • hideshow 5 child comments
  • [–] [S] 7 points 3 months ago

    Very good, I pulled the latest version.

    Thanks for your work on this, and the code you added to forward ip numbers was super useful today to see where these botnets are coming from. Really appreciate it.

  • source
  • parent
  • [–] 2 points 3 months ago (2 children)

    Thanks to both of you!

    Now that you’re both here, I have a question/request. What I love most from the “old Reddit” interface is using RES to navigate through posts with the keyboard: “h” to hide, j/k as up/down, “l” to open in a new tab, etc. Makes browsing super fast and efficient.

    Is this something you’d consider? I’m not asking for a full RES reimplementation, of course. Just the keyboard navigation.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 3 months ago* (last edited 3 months ago) (1 child)

    I know of the lemmy keyboard navigation addon that reimplements this feature for lemmy-ui and mlmym. I haven't tested it myself though, and it's been last updated over a year ago, so I don't know if it still works. I have thought about including the feature directly in mlmym (could actually make use of the keyboard navigation setting that lemmy already has to toggle it), but if I do, that'll only happen after I'm done with the refactor for the v4 API. So it will probably take a while 😅

    edit: opened an issue for it (#27) so I don't forget

  • source
  • parent
  • hideshow 1 child comment
  • [–] 10 points 3 months ago (19 children)

    thanks a lot for keeping this instance running without cloudflare stuff, really appreciate it..

  • source
  • hideshow 19 child comments
  • [–] [S] 11 points 3 months ago (18 children)

    Thank you!

    I dont want cloudflare either. :)

  • source
  • parent
  • hideshow 18 child comments
  • [–] 3 points 3 months ago* (17 children)

    First, thank you for all the hard work you do.

    Second, as someone still struggling to understand lemmy, one of the reasons I'm not a fan of old.lemmy.world is that I have to turn on cloudflare.

    I have noscript running and I have cloudlflare blocked.

    Your response implies otherwise that you don't have cloudflare running either.

    So why am I always greeted with cloudflare when I visit old.lemmy.world?

    Edit: Oh I misread the title. Its old.lemmy.today that you're talkinga bout, not old.lemmy.world.

    My goof. Please ignore my post, but do take my gratitude.

    Edit 2: So I went over to old.lemmy.today. It is tempting to switch. I like the default interface and it has content from a lot of places (on quick glance). One of the reasons I'm not a huge fan of old.slrpnk.net is that I can't figure out how to access content from other instances. I seem tempted to switch.

  • source
  • parent
  • hideshow 17 child comments
  • [–] [S] 4 points 3 months ago (14 children)

    You are very welcome if you do. Its the same content in old.lemmy.today as lemmy.today, just a different look. :)

    I dont like cloudflare so doing my best to avoid it if I can.

  • source
  • parent
  • hideshow 14 child comments
  • [–] 2 points 3 months ago* (2 children)

    seems like it's working again but with anubis.. maybe can we just have something like http basic auth with a public password? (like have the password prompt as "type lemmy")

    i don't think anubis is a good solution. it's very trivially bypassed though maybe good enough to block stupid generic bots..

    see also https://lock.cmpxchg8b.com/anubis.html

  • source
  • parent
  • hideshow 2 child comments
  • [–] [S] 3 points 3 months ago

    Ok now its working and seemingly blocking bots, but we will see over time how well this works. Like you said, im sure its possible to bypass but maybe it improves the situation somewhat still.

  • source
  • parent
  • [–] 2 points 3 months ago (8 children)

    Hi I just wanted to follow up on something

    Since last evening, I can't access old.lemmy.today (I can still access lemmy.today but that's not an interface I like).

    Are you having bot issues again or is there something I messed up on my end?

  • source
  • parent
  • hideshow 8 child comments
  • [–] [S] 3 points 3 months ago (6 children)

    Now it should work well again. I put it behind anubis and we will see if it improves the situation with the bots.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 1 point 3 months ago (4 children)

    I'm so sorry to constantly pester you, but I have another question.

    I've been enjoying old.lemmy.today for the past few days. The biggest praise I can give it: I haven't missed reddit at all, and the smaller community actually makes it feel like an upgrade. Like the reddit of old.

    But I wanted to make an account on Lemmy.today to make a post, however to register it seems I need an email address.

    I don't want to give an email address. I could give a disposable one from one of those sites, but that would be obvious (to what it sounds like is a human curation process).

    I know you said its to ward off bots, I'm wondering if its a temporary pause, or if its a permanent policy?

  • source
  • parent
  • hideshow 4 child comments
  • [–] [S] 2 points 3 months ago (3 children)

    It's fine to not give an email address. It's just for the email to let you know that you were approved or denied. So if you put something random there and check if you can login later, it's all good.

    I think it's even optional to give it... You can try registering without. :)

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 3 months ago (1 child)

    So just as an update, I checked both old.lemmy.today and lemmy.today; both seem to require an email address.

    On lemmy.today is explicitly says email is required. On old.lemmy.today I tried fiddling around with various entries in the email field such as na, not_applicable, etc, but that didn't seem to work either.

    Is there something obvious I'm missing?

  • source
  • parent
  • hideshow 1 child comment
  • [–] [S] 1 point 3 months ago

    No, I checked and we do require email. I wasnt sure what was set there but yep, its required. But just put any email address there if you dont want to use a real one. You just wont get an email with approval or denyal after account creation, so you have to come back later and try to log in at some point.

  • source
  • parent
  • [–] 2 points 3 months ago (1 child)

    Hi,

    I'm sorry to bother you again (I waited a week before messaging you), but I wanted to check in on something. Last weekend, I wasn't able to access old.lemmy.world.

    Around Tuesday or so, I could access the site, but I can't see any comments. I always get a 403 Forbidden error (with the words nginx at the bottom).

    Is this is a problem on my end that I can fix?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 3 months ago (1 child)

    One of the reasons I'm not a huge fan of old.slrpnk.net is that I can't figure out how to access content from other instances.

    old.slrpnk.net should still have posts from other instances, as far as I can tell. They only show local stuff by default, so you have to set “listing” to “all” to see them (the small toggle under the yellow tagline). If you want to primarily interact with posts from other instances, lemmy.today is probably the better choice though either way. slrpnk.net looks like it mostly focuses on local communities.

    I have noscript running and I have cloudlflare blocked.

    The good thing about mlmym (the software behind old.lemmy.today, old.lemmy.world, and so on) is that it's entirely usable without JavaScript :)

    So as long as there isn't something like cloudflare in the way that does require JS, it should work fine to use it that way.

    Looking at it right now though, @mrmanager@lemmy.today it looks like old.lemmy.today is erroring with 502?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 5 points 3 months ago (3 children)
  • [–] 5 points 3 months ago (3 children)

    Thanks for keeping the instance running. I happened to get a error message smth about the instance being broken and remembering that disk upgrade post from a couple days ago I feared the DB got nuked for a sec hehe.

  • source
  • hideshow 3 child comments
  • [–] [S] 7 points 3 months ago (2 children)

    The db is like the holy grail for a lemmy instance. Heavily backed up and fortified. :)

  • source
  • parent
  • hideshow 2 child comments
  • [–] 5 points 3 months ago (8 children)

    Look into pow protection. The Chinese botnets come from a practically infinite range at the moment

  • source
  • hideshow 8 child comments
  • [–] [S] 4 points 3 months ago (7 children)

    Thank you, I was reading about pow protection and finding out more about it. Is there some local software you could recommend? I'm not too hot on cloudflare or other big tech services unless absolutely needed.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 4 points 3 months ago (6 children)

    What is your reverse proxy software?

  • source
  • parent
  • hideshow 6 child comments
  • [–] [S] 3 points 3 months ago (5 children)
  • [–] 4 points 3 months ago (4 children)

    You can look into Anubis then.

  • source
  • parent
  • hideshow 4 child comments
  • [–] [S] 2 points 3 months ago* (2 children)

    I added Anubis for https://old.lemmy.today/ and will see if it helps the situation. Bots are hammering it pretty hard but they are not getting any lemmy content at least. Thank you for your advice. :)

  • source
  • parent
  • hideshow 2 child comments
  • [–] 5 points 3 months ago (1 child)

    are you sure it wasnt the tesseract dev slipping in some javascript to DDOS your instance?

  • source
  • hideshow 1 child comment
  • [–] 4 points 3 months ago* (3 children)

    dang it . interesting.
    I wonder if this is shady reddit operation

  • source
  • hideshow 3 child comments
  • [–] 3 points 3 months ago (1 child)

    Are we running a HIPS like IPBan Pro or are you doing this manually via iptables?

  • source
  • hideshow 1 child comment
  • [–] 1 point 3 months ago (1 child)

    Thanks for the heads up. I got a brief error message a while ago and figured that's what happened. Good job keeping the instance going!

  • source
  • hideshow 1 child comment
  • [–] -1 points 3 months ago*

    Thanks for the heads up and keeping us updated!

  • source