Hello all,
After last month's very small update, I think I am making up for it with what will probably be amongst the longest ones I've written yet. In order to make it somewhat readable, I've used spoiler tags for the general Piefed and Lemmy updates, please uncollapse them to read the juicy details.
General server updates:
A little while ago I made some updates to the "old" frontends (mlmym) at old.lemmy.zip and old.piefed.zip. These included massive performance improvements and fixes. Recently, local legend @Blaze@piefed.zip has also been helping out, adding LOADS of features to the piefed version. If you're looking for a familiar but powerful alternative to the old reddit interface, definitely check out old.piefed.zip.
The Button just finished its 6th season (a Halloween themed season) - you can read more about how it went here.
Finally, we'll be doing some feedback collection soon. It has been quite a while since we did our last one, but given how much things have changed, I think it is well worth doing it again. We will let you know when it is ready.
PS hello to all you ex-redditors who have found your way here. Please do reach out if you need anything!
tl;dr - Piefed.zip was hit by a Cryptominer during an update, account sessions were invalidated and passwords reset. No data known to have been taken.
Piefed
On the 24th October (heading into the 25th) Piefed.zip underwent scheduled maintenance to update to the latest version, including a full server update. At the time, it all appeared to work fine and Piefed.zip came back up as expected. On the 26th, Piefed started reporting cron job errors (A cron job is an automated, time-based task scheduled to run on the server):
Some cron tasks have not been run recently:
daily_maintenance (expected every None, last run at 2026-10-24 03:18 UTC),
send_missed_notifs (expected every None, last run at 2026-10-26 12:01 UTC),
process_email_bounces (expected every None, last run at 2026-10-26 12:01 UTC),
clean_up_old_activities (expected every None, last run at 2026-10-26 12:01 UTC),
send_queue (expected every None, last run at 2026-10-26 12:05 UTC)
At the same time, checking the server performance we saw the CPU was being maxed out:
(please excuse the crappy mobile screenshot)
This was obviously unexpected. At this point my suspicion was a cron job had stuck in some unexpected way and caused the cpu to just shit the bed, but checking docker showed another error when attempting to initialise container security filters. Specifically, libseccomp reported an effective API level of 1 despite the installed library meeting the version requirements for the requested functionality.
This behaviour was inconsistent with the expected capabilities of the server. Further investigation here led to the discovery of unauthorised processes and persistence mechanisms.
(Although interference from malicious software is a possible explanation, the investigation did not establish a direct causal relationship between the malware and the seccomp failures.)
At this point I took the piefed and infrastructure stacks we use offline, and started trying to figure out what was going on.
Based on the hidden artifacts and persistence mechanisms, I was able to tie this to perfctl, which is a stealthy, persistent Linux malware family designed primarily for cryptomining (XMRIG Monero miners).
This then led to looking for the attack vector. It is known for targetting exposed portainer agents (usually port 9001) and sure enough we had an exposed portainer agent.
We (over 12 months ago) used portainer to manage our servers without needing to ssh in to them. It should have been removed from all our servers a long time ago, however on the piefed server the old portainer agent container was stopped but not deleted. At some point during the maintenance this has been restarted by mistake, and it looks like it was brought up in its default state. Regardless of this, the fact this could ever happen is totally my fault and all blame lies at my feet for not tidying up properly.
So now we knew the attack vector and that the attack took place sometime on the 24th leading into the 25th (based on the graphs first showing the cpu maxing out, it was the early hours of the 25th).
At this point both INeedMana and myself were able to start a system restore process. Luckily Hetzner provides a really really handy backup system, and we reimaged the server with a backup from the 22nd, and were able to reapply our encrypted offsite database backup to the server. We also redid the updates, and obviously made sure perfctl was not present on the backup from the 22nd, which it was not.
While we can't see any evidence that anything touched the piefed database that shouldn't have, we have changed every single key on the server and in our infrastructure, and we invalidated everyone's passwords by setting them to null
docker compose exec -T db \
psql -U piefed -d piefed -v ON_ERROR_STOP=1 <<'SQL'
BEGIN;
UPDATE public.user
SET password_hash = NULL
WHERE instance_id = 1
AND password_hash IS NOT NULL;
COMMIT;
SQL
By setting them to null, Piefed very handily forces the user to reset. Our other option was to generate new passwords for everyone, but this felt more intuitive, or at least the least confusing option of two bad options.
We also reset the secret key that validates logged in sessions, so this forced everyone to log out too, hence they would have to reset their password.
We also redid the firewall, beefing it up an awful lot (unclear if this would have helped with this, but worth it anyway), redid the docker networking away from the piefed default to be more compartmentalised and hopefully secure, and reviewed the whole networking stack. There is still management software, but this is restricted to just a VPN connection and not accessible externally, which has been stress tested - it will fail-secure should the VPN not work.
Massive shout of thanks from myself to INeedMana for jumping in and spending hours fixing this with me, guiding me, and putting up with my stress and panic!
We will keep monitoring going forwards, and let everyone know if we have any further concerns, but it does look like it is resolved now.
Oh and after all that we did manage to update Piefed too :)
tl;dr - removal of UK geoblock, application changes, improved ban notice, performance improvements etc.
lemmy.zip
I've been having a good old play with Lemmy to tweak it to how I want it, but not too much as 1.0 is coming up.. soon..
Anyway, some of you may have been around here long enough to remember that back at the beginning of 2025 (almost 2 years ago now!) we had to block the UK to comply with the very draconian and only getting worse Online Safety Act (because I am a UK citizen sadly).
Mainly, this is because there is no way to prevent children from accessing NSFW content in Lemmy. Now, I've been able to add a complete an unremovable block on NSFW content on UK users. Is it ideal? No. Is it something I agree with? Also no. Does it allow us to accept UK users back on Lemmy.zip? Probably..? I need to do more digging, but for now UK users can once again sign up, they just can't turn on NSFW content. Sorry about that. You'll have to get your particularly saucy ankle pictures elsewhere.
Users in the UK attempting to turn on NSFW will instead see this:
And UK users going directly to NSFW content will see this instead:
I have also recently added a minimum character length to the Lemmy.zip registrations for the answer. You've pretty much all filled one in if you're on Lemmy.zip, but what you don't see is the amount of people who type stuff such as "lemme in" or "aldkasdkjas" or stuff like that. So, now users have to enter 50 characters to not be auto-rejected. We still read every single one before approving though, we just turn those away that can't be bothered filling it in.
I have also added an improved ban message when a banned user tries to log in. Normal Lemmy just pops up a little message saying can't log in, now the user gets a full on warning:
Hopefully you never see that screen, but at least you know it exists now :)
I did also fix .avif images not opening in-line, although this has since been backported by the devs into 0.19.21 so I hope to anything almighty I stop getting complaints about avif images (just need some outdated apps to finally support them!).
I also dropped some very specific local performance improvements into lemmy, including a very detailed and overly complex (for me) bug between postgres and lemmy that was affecting schwim (one our admins) and I'm sure others out there.
I will try and explain it to you, while also explaining back to myself.
When you scroll to page 2 of a feed like All/New, lemmy says “give me the next 25 posts after the last one I showed.” The way it phrased that to the database was one big combined comparison, roughly “find posts that come after (this pinned-status, this score, this timestamp, this ID)” all bundled into a single tuple comparison.
Example tuple comparison:
(featured, score, published, id) < (last post's featured, score, published, id)
Here’s where Postgres does the database equivalent of confidently walking into a glass door. When it sees a bundled comparison like that, its query planner (the bit that decides how to actually fetch the rows) only bothers to look at the first column in the tuple to guess how many rows will match. The first column was featured (i.e. is this post pinned?). And the comparison effectively asked for posts where featured < false.
Nothing is less than false. False is the bottom. So Postgres looked at that, went “ah, this will match approximately... one row, maybe zero,” and picked a strategy optimised for returning a tiny handful of results.
Except it was about to return thousands.
So now Postgres is running a plan designed for a teaspoon of data against an ocean of it. To make it worse, lemmy.zip users have community block filters (including ghost-blocks left clinging to communities that were removed, the digital equivalent of a restraining order against someone who already moved away). Those filters nudged the planner toward sorting every single older post in the feed from scratch, every time someone clicked “next page.”
The damage:
40 to 100 seconds per page load. Per request. Page two. ~14 GB of temporary files spewed onto disk, because it couldn’t sort that much in memory, so it wrote the overflow to disk and sorted it there. Repeatedly. For a button click. The fix (anticlimactically small, as the best ones are): instead of cramming featured into the bundled comparison and letting Postgres faceplant, we just say up front, “we’re past the pinned posts now, so filter featured = false,” and then only bundle-compare the columns that actually matter (score, time, ID). Same posts. Same order. Postgres now estimates correctly, picks a sane plan, and the results:
~25 milliseconds. Down from up to 100,000.
That’s roughly a 4,000× speedup from telling the database something it technically already knew but was too proud to figure out on its own. No migration, no schema change, no new features, just one clause moved so the query planner would stop hallucinating.
The downside is if the amount of pinned posts spills over on to page 2, then it wouldn't show them. However we've had at most like 3 pinned posts at once... so that is very unlikely to happen.
For mods, I am also looking at implementing more robust moderation reports for us. Currently admins resolving reports interferes with mods resolving reports, so I am working on a resolution to that, at least until 1.0 is here and changes how reports work.
One more thing I've added, once a user does sign up, they should now get this pop up:
It is an attempt to at least explain how the process works. Hopefully it helps?!
Donations
Lemmy and Piefed are not reddit. We don't collect and sell your information. We don't serve adverts. We dont make you the product. Unlike all big tech, we truly are the product here, not you. This is about as close to free as you can get these days.
However, running servers is getting more and more expensive. You may have heard of this "AI" thing going around, and it keeps driving prices up. The cost of the Piefed server has almost doubled in a matter of months. The lemmy server is already an expensive beast, and that isn't going to get any cheaper either. The cost of hosting images, purchasing domains, creating and storing offsite backups, heck even having reliable email all adds up.
So far, we have been incredibly fortunate to be funded by donations. But the costs do keep increasing. Therefore if you are coming here and enjoying it here, please please please consider a donation. It doesn't have to be a big one. If everyone who read this donated $1 we'd be sorted for months. If everyone set up a recurring monthly $1 we'd be sorted for years.
Please do support independent platforms (who spend their spare time running services for you for free) with a small donation if you can.
Oh, and we're not just a random guy behind a computer screen. Lemmy.zip and Piefed.zip are a non profit organisation - DotZip Ltd. We're here for the long haul, because we care about a free and open internet.
Here are some options if you can donate:
Liberpay (Card, Paypal)
Liberpay is our newest donation method. You can donate with Paypal or with a card. The link is https://liberapay.com/dotzip
OpenCollective (Card)
Our classic donation option, but only supports card payments. The link is https://opencollective.com/lemmyzip
Ko-Fi (Card, Paypal)
We also have our Ko-Fi page if you'd rather use this site, which also supports Paypal and Card payments.
If you're enjoying Lemmy.zip or Piefed.zip, please do consider a donation, we have a selection of one-off or recurring donation options. All funds go directly to hosting the sites and keeping the virtual lights on.
Please Don’t Be a Lurker!
If you’re new here - WELCOME! I hope you’re enjoying your time here :)
Just one small teeny-tiny request. The greatest gift you can give Lemmy.zip and Piefed.zip isn’t money, praise, or an autographed picture of yourself completing a skydive. Its participation.
Upvote the things you like. Start a discussion, debate, or just post the same meme of a disappointed cat over and over and over and over and over and over and over and so on.. Make new communities if you don’t see one that fits your oddly specific niche obsession. (We don’t judge. Well, we try not to judge anyway.)
The fediverse naturally ebbs and flows, tides of people come and go. But if you’ve found yourself oddly attached to this strange little corner of the internet? Wonderful. Help it breathe. Help it grow. Help it be just a tiny bit weirder in the best possible way.
So if you’ve gone to the effort of clicking Sign Up and proving you’re not a robot (unless you are, in which case hello and welcome to our new AI overlords), then please, I beg of you:
Stick around. Add your voice. It really does make this place better.
Graphs
Admit it, you skipped all the stuff above for the shiny shiny pictures, didn't you? I knew it!
Piefed.zip
CPU:
RAM:
Performance right now:
Image storage:
Cloudflare 30 day overview:
Visitors:
Data transfer:
Lemmy.zip
CPU:
RAM:
Performance right now:
Images:
(That's coming up on 3TB of furry porn you guys, hope you're proud of yourselves)
Cloudflare overview:
Requests etc:
Visitors:
Right so that tops off an incredibly busy October. I am sure I have forgotten something (I should really write interesting stuff down) but that covers the main points.
All I have left to say is RIP Sir Patrick Stewart.
Thanks
Demigodrick
1
1
