It's fine on thunder this morning.

Intermittent issues are so annoying, thanks for looking

Sometimes the site can get sluggish and API responses fail if the site is being hit particularly hard by scrapers. I'll dig into the logs when I get the chance, but for now I've enabled a DDoS protection setting. Can you try now? This is the best idea I have atm for what's causing this.

Just unreachable. It's fine again now on thunder, wasn't working when I sent that message a couple of hours back 🤷‍♂️

I'm able to log into voyager just fine, does it tell you more about the error or is it just a timeout message?

I just tried downloading voyager on the off chance it was indeed a thunder issue - went to log in and got the same issue, Feddit UK not responding.

Are we up to date on versions on Lemmy etc? Just trying to think why the apps might be failing to log in.

Well, glad it's not just me!

Yeah, I've been having the occasion trouble with Thunder as well and I'm not sure why. It just seems very random.

The trouble with the matrix space is that when the site is down it's pretty much impossible to find the details for it.

Anubis doesn't really do anything if the bot knows about it and doesn't send the bits Anubis looks for. There are a few other heuristics that you can enable in the Anubis config, but it is a principal issue with how Anubis works. The general idea is that if a bot doesn't send the things Anubis looks for, then it should be possible to distinguish it from regular browsers by that, but then you are back at a manual whack a mole game and given the extensive use of residential proxies these days it is very much a losing battle.

Anubis recently added a honeypot with automatic IP logging though that might help in combination with fail2ban. But in the end if scraper bot authors are already circumventing Anubis intentionally, it is probably also only a temporary respite.

Would you know why Anubis isn't as effective?

On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.

Someone did actually reach out to my personal Mastodon account to ask what was happening.

Like florge said, there is a Matrix space that mostly serves this function, but that does require you to have a Matrix account so isn't the best for public notices. It'd be nice to have a page (eg status.feddit.uk) where I could put messages about what's happening. I'll need to look to see if something like that exists.

There's a matrix space here

What's the database underneath lemmy? Is there a transactional option so that it fails safe in the event of unexpected shut down?

Also thank you very much for fixing.

See this is why I hate working in the IT department, because it involves using computers, and they suck.

I did and it was surprisingly pretty good.

Yikes sounds like a nightmare. I hope you managed to get some sleep!

Looks like they're already rotated daily.

Anyway by my understanding it wasn't the logs that caused the problem, they just demonstrate the extent of the sudden increase in requests by bots/scrapers.

Glad to see we are back. Those access logs are crazy.

I’m sure you will fine a lot of Chrome/12X, Chrome/13X and Chrome/14X user agents there. Seems to be the favourites of attackers/scrapers recently.

On another note, do the server has a profile somewhere like mastodon or something we could check in case it goes down in future.

Thank you so much for the work you do!

Thanks Flamingos!

There's a setting in Lemmy that only let's logged in users make API requests. I'm going to activate that over night for now.

Rotate your logs!!!

Do you have anything in place for detecting and mitigating DOS attacks? I wonder if you were to grep the logs by IP address you'd see the offending IP addresses. Though if you're using a load balancer or some other kind of proxy and don't have IP address forwarding configured, all requests might come through as the same IP address.

some data got duplicated in random tables that shouldn’t be

I'd also check that there's plenty of disk space spare, given that the log files are so large, as I've seen environments start to behave unusually once disk space has ran out, including SSH locking up.

Thanks for all your hard work!

Yeah, those gigabyte sized access logs is something we have seen for weeks.

Anubis has also gotten largely ineffective, and yesterday I had some time to sample some of the access log IPs and it is clear that this is a coordinated scraping attack from the Huawei ASN that additionally employs residential proxies mostly in Pakistan, Vietnam and Brazil.

I don't really have a good idea how to deal with it, but it is a complete PITA.

There's a matrix channel here: https://matrix.to/#/#feddit.uk:matrix.org