To all you lazy lemmys,

Just had an alert that the VM running the primary Lemmy daemon has low memory available.

I'll be taking the opportunity to update Lemmy to 0.19.20, as well as all the frontends.

Brief outage incoming....

Edit: Update complete. And I noticed the update is labelled "reduce memory usage".... doh

Following the fediverse drama surrounding this front-end and subsequent abandoning of the project by its maintainer, this front-end will be retired effective immediately.

The retaliatory malicious code deployed recently didn't help. Latest drama post at https://lazysoci.al/post/51452204

Apologies for the inconvenience.

The VM running the Lemmy binary itself is getting close to 90% ram usage. I'm just going to quickly restart and double the RAM.

Update: all done

Hi Lazy Lemmys,

When I first put together Lemmy, I setup Lemmy, nginx etc. as standard apps. I then migrated to Docker sometime after.

I've since then split Lemmy into separate VMs for lemmy-core, database, frontends and pict-rs, all deployed using ansible.

I now want to re-deploy lemmy-core with ansible on a fresh VM.

I'll be performing this migration today (in the next hour or so), but I expect Lemmy to only be offline for about 15 minutes.

Update: 1st outage of ~5 minutes completed. Just cloning this VM so I can re-use the proxmox VMID on the new VM. One more outage to go...

Final Update: All up and running on the new VM

Hi me again, here to ruin your day.

I migrated the S3 storage to RustFS last month, which has been running OK. I have now decided that pict-rs should run on its own dedicated instance of RustFS instead of the current shared one.

Unfortunately the mirror takes about 3 hours to complete. lazysoci.al will be offline until this is completed.

Update: Back online! Took 40 minutes longer that expected as I got bored waiting so went out for a ride :)

Took the opportunity to upgrade Lemmy to 0.19.19

We use an S3 back-end for Pict-RS, which hosts thumbnails and cached images. I am migrating the S3 back-end from minio to rustfs.

Minio went rogue, and then they archived their community/FOS version. So I've had a replacement in the form of RustFS ready but have yet to pull the trigger for Lemmy.

I have been doing a regular mirror of the two buckets, which takes just over 2 1/2 hours to complete.

When this post in 1 hour old, I'll be shutting down Lemmy and performing a final migration, so expected downtime is 2 to 3 hours.

Sorry for the outage!

Update: And we're back!

Just done a quick update of Lemmy to 0.19.18 Also updated front-ends

That is all.

Good day Lemmings!

For the handful of people that use this instance, this is a warning that the service will be unavailable for a few hours tomorrow.

From Friday 2026-03-20 1230 UK time I am recreating the proxmox cluster whilst adding new hardware. I expect the total downtime to be ~3 hours, and may go on and off a few times (mostly off) during that period.

Geek time.

Currently, the Proxmox cluster is made up of a single Dell server and a handful of desktops and mini-PCs.

I am introducing an identical Dell server and reducing to a 2-node Proxmox cluster with only Enterprise hardware.

The 2 systems will then be:

• Dell PowerEdge R640
• 2x Intel Xeon Gold 6234 CPU @ 3.30GHz (32 threads)
• 512G DDR4 ECC
• 6x 1.92T Enterprise U.2 NVMe

I wanted more RAM, but prices are outrageous. The current server has 1TB of RAM (for bragging rights).....

.... so I'll split that between the two systems.

I am also introducing a new Mikrotik CRS312-4C+8XG-RM switch and a second Mikrotik RB5009.

The end result will be better performance, and improved availability. Not like Lemmy has had any problems with performance though.

This system is largely my dev playground with a few business services in the mix.

Hope to see you on the other side!

Update 1820: had a few issues and got Lemmy online a few hours ago but all completed now.

Also updated all frontends to their latest revision.

Let me know if any issues

Good morning Lemmings!

For the handful of people that use this instance, this is a warning that the service will be unavailable for a few hours tomorrow.

From Friday 2026-03-20 1230 UK time I am recreating the proxmox cluster whilst adding new hardware. I expect the total downtime to be ~3 hours, and may go on and off a few times (mostly off) during that period.

Geek time.

Currently, the Proxmox cluster is made up of a single Dell server and a handful of desktops and mini-PCs.

I am introducing an identical Dell server and reducing to a 2-node Proxmox cluster with only Enterprise hardware.

The 2 systems will then be:

• Dell PowerEdge R640
• 2x Intel Xeon Gold 6234 CPU @ 3.30GHz (32 threads)
• 512G DDR4 ECC
• 6x 1.92T Enterprise U.2 NVMe

I wanted more RAM, but prices are outrageous. The current server has 1TB of RAM (for bragging rights).....

.... so I'll split that between the two systems.

I am also introducing a new Mikrotik CRS312-4C+8XG-RM switch and a second Mikrotik RB5009.

The end result will be better performance, and improved availability. Not like Lemmy has had any problems with performance though.

This system is largely my dev playground with a few business services in the mix.

Hope to see you the other side!

I've encountered something broken with the comment button on my feed. Normally (when using my lemmy.world account) it would take me in to the thread so I can read comments but on this account it doesn't and sends me to an error page that constantly refreshes. It happens on all subs, so both lazysoci.al and external ones.

@lazynooblet@lazysoci.al

Edit: forgot to mention that this in on Photon UI :/

Updated Lemmy to 0.19.5 Also updated all frontends

I gave the new VM for all the frontends only 2G ram, doh.

Backend continued functioning.

lazysoci.al has been updated to 0.19.14

Let me know if any issues.

As announced earlier this week, some maintenance will be performed causing several hours of downtime on Sunday 2025-12-07.

Edit: This maintenance shall begin at 1730 UK time (2 hours from this edit)

The following changes will be made:

• All front-ends will be migrated to a separate VM.
• pict-rs S3 storage will be migrated from minio to rustfs (minio is no longer being updated).
• Anubis will be implemented and placed in front of all front-ends.
• Cloudflare tunnel will be decommissioned.

The site will be offline for an extended period of at least 2 hours, perhaps longer. Backups are in-place so should there be major issues following these changes, they will be rolled back.

I'll pin this post on Sunday morning a couple of hours before the work starts. If real-life gets in the way, it'll be postponed to the following weekend.

Update 2025-12-07: Maintenance has ended, however somewhat failed.

I want to wait until this rustfs issue is resolved prior to migrating. I tried to perform some pruning of the data post migration, but noticed directories not being removed. I didn't see this until I had already started the sync. At least now I know a re-sync between the two S3 buckets takes about 3 hours.

Implementing Anubis is going to require some additional work to create a good policy configuration file and I ran out of time this evening.

The lemmy UI's are now on a new VM, and Cloudflare has been removed, so there is that, but more work needed.

A short update as things have been quiet lately.

Image proxy

A few weeks ago I made what was meant to be a temporary configuration change to fix imgur.com images. I set pict-rs proxy cache lifetime to 1h in an attempt to quickly have the system re-download failed images. I forgot to remove this :(

The implications were that images were slow to load on the site as pict-rs had to make its own fetch. This has now been resolved and at least for me the site is running smoother for it.

Bots and Cloudflare

You may have noticed that the Cloudflare outage the other week affected the site. I have been using Cloudflare as an attempt to reduce bot-spam, however it isn't very effective.

Whilst the site is on good hardware, I do get regular performance alerts due to scrapers hitting the site. At times there are ~5000 concurrent connections. It usually only lasts a few minutes and whilst the site does slow down a little, it isn't unbearable.

In any case, I plan on exploring the use of Anubis https://anubis.techaro.lol/ as an alternative to Cloudflare. This will (1) reduce reliance on large corporations (2) increase privacy as Cloudflare is a privacy risk (3) combat bot-scraping. It may also effect mobile clients, so I'll be testing all clients as part of the roll-out.

I'm expecting to work on this at the weekend, however I'll post another update beforehand.

That's all for now.

Apologies after the update this morning the docker host running the lemmy frontends (seperate from lemmy itself) was having real issues and took a while to resolve.

All back up and running now :)

Weird issue though - there was an unused container running squid (the forward proxy service) that was causing the system to freeze. The proxmox host node was getting a load average of 600+. Took a while to find it as had to start each container one, wait 2 minutes for it to settle, then try the next.

There shall be a 30 minute downtime after this post is 30 minutes old. ~ 12:40 UK time

The Proxmox cluster Lemmy is running on is being updated to version 9, and I'm doing all nodes at pretty much the same time, so all VMs are going off for the duration.

Update: All back up and working

Imgur have made the decision to block the UK following being threatened by the UK IPO due to the Online Safety Act.

Unfortunately this Lemmy server resides in the UK and is not unable to access imgur as a result. The image failures you may be expieriencing is due to this instance fetching images on your behalf to protect your internet identity (image_mode: ProxyAllImages)

I am working on a work-around which is almost ready to deploy. I expect images to stop functioning completely for a short while whilst I migrate pict-rs to another system (thank god I used S3 for pict-rs storage, else this migration would take days).

Update: pict-rs wasn't migrated, instead its outbound traffic has been routed out of another region. Unfortunately attempting to purge the cache of imgur images so it'll redownload is too much of a headache. The cache is set to 7-days but it seems setting it to 1-hour doesn't affect existing images, only new ones.

If you are in the UK, then new posts that use imgur.com images shall be viewable. However this will depend on your client. For example Boost for Android gets the image URL direct from the original server, so I see the the thumbnail OK (cached via lazysoci.al) and then the "imgur has blocked you" full image (due to Boost going direct).

I expect this will only get worse.

Typing this from my phone as I'm out cruising on the bike :) it's gonna rain soon so planning to do some updates when I get home later today.

I'll upgrade to latest Lemmy, I think we are 2 versions behind.

I'll also relax the image dimensions and movie size restrictions for the image proxy which I've noticed have prevented some image posts from working.

I don't know exactly what time it'll be but sometime shortly after 1300 UK

Update 2025-09-14 10:01: Real-life got in the way of the update yesterday. Updates have now been applied and we are now on 0.9.13. Also updated frontends and added blorp at https://b.lazysoci.al/ (thanks @nord@lazysoci.al)

Update 2025-09-14 10:29: Removed additional restrictions on dimensions of video uploads for the image proxy, this has resolved at least one example of the "too wide" error received for some images. It is now using the defaults, will see if there are any other examples and adjust if needed.

When this post is ~90 minutes old, lazysoci.al will have a downtime of ~1 hour.

I am upgrading the cluster that hosts lemmy, upgrading both RAM and Proxmox on all nodes.

Update: 1230 GMT+1; All back up and running.