The magpies were singing for their breakfast, despite the cold rainy weather

In light of the dev silently blacklisting blahaj.zone, many of its users and several other instances, we have removed tesseract from the available lemmy front ends we offer.

Cross-posted from "Seagirl" by @fossilesque@mander.xyz in !science_memes@mander.xyz

Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.

The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn't use postgres). From what we can tell, the attacker did not do anything with that access, so we don't believe any user data was accessed, but we can't be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.

Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.

We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it's as smooth as we think it will be, but we can't be sure!

As stated earlier, our writefreely instance is still up and running as it wasn't impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.

We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don't plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won't be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn't seem worth the cost/effort to run it isolated.

in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked

Edit - Piefed has been updated and returned to service

Thanks to someone posting a piefed security exploit without a disclosure period, we've had to pull blahaj piefed down without warning until a fix is implemented. This could take around 24 hours or so.

Edit - We're back!

We've had an issue with our databases. One of our fast database servers ran out of space, and then the second fast server ran out of space whilst replicating to the first.

As a result, we have fallen over to our backup database server, which runs on spinny disks rather than SSDs. Spinny disks means that it's got plenty of space to spare, but it's not fast. The backup DB server is currently replicating to our two main servers to get things back up and running again, but whilst that's happening, all of our services are running slow.

The good news is, we'll be back up and running as if nothing happened because our backup server saved the day. The bad news is, it may take another 24 hours or so, because the backup server is reliable but not fast!

If you're interested, and use blahaj lemmy as your main account, please reply below!

We've been getting a few of these types of posts recently, so this is just a reminder. There are plenty of places we can find endless bad news. This is not meant to be one of them. Bad news has a place here, but only when it's part of a discussion that helps people move forward despite the negative.

If you're non binary, and have a blahaj lemmy account, ideally with history in this community, and you would like to moderate this community, please reply and let me know.

For those quick off the mark, you may already have seen her name appear in the list of lemmy admins! We've brought Jorunn on board, as she's already staff on our piefed instance, and she has been helping moderate lemmy via our admin bot, which can approve signups, delete spam etc. So, she now has her own keys to the castle to make her life easier!

You can reach out to her for issues around community/instance moderation, regarding our lbz users etc. Kaity and I are still the best contacts for sys-admin related issues however.

Anyway, welcome aboard Jorunn and thank you for taking this on. I certainly appreciate it :)

Cross-posted from "Tawny Frogmouth (Podargus strigoides)" by @ada@piefed.blahaj.zone in !birding@lemmy.world

This was taken in near total darkness, with a hand held, 300mm 1 second exposure. I love this camera and lens!

And thank you to the Tawny, who was happy staying perfectly still for my long exposure :)

#frogmouth #TawnyFrogmouth #brisbane #meanjin #bird #urbanbirdphotography #australianbirds

If you're interested, and use Blåhaj Lemmy as your main instance, let me know and I'll add you as a moderator. You'll then be free to add other off instance mods if you like, but the primary mod needs to be a Blåhaj Lemmy user,