Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.
The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn't use postgres). From what we can tell, the attacker did not do anything with that access, so we don't believe any user data was accessed, but we can't be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.
Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.
We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it's as smooth as we think it will be, but we can't be sure!
As stated earlier, our writefreely instance is still up and running as it wasn't impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.
We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don't plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won't be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn't seem worth the cost/effort to run it isolated.
in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked
all 60 comments