First up.. this is not a call for donations.

Without going into the personal details as to why: I don't have the energy to give this place the attention it needs, and I'm no longer willing to cover the majority of running costs.

I don't have a fully fledged solution to this, and don't have the energy to come up with one. The best I can come up with is I hand the reins over to Nath, and some sort of community funding is set up to cover costs.

If anyone knows of above board methods to establish a crowd funded piggy bank for something like this, I'm all ears.

Yes, its been a while since I did one of these... over a year.. oops!.

For something different, I'll post the graphs for the last 12 months, with some explanatory comments.
Feel free to ask question, will do my best to answer.

Finances:
Short version, AZ is costing ~$90 AUD per month to run on current hardware and services. That is partially covered by a small handful of monthly supporters, around $25 per month. The rest is covered by me personally. If you'd like to help cover server costs, any amount is appreciated: https://ko-fi.com/lodion

CPU:

The decrease in August across multiple graphs is due to the AI scraping blocking we put in place. At the time they were effectively DDoSing AZ... since then, I've not heard of any major issues.

Memory:

We're overdue for a lemmy software upgrade to 0.19.13, which should clear out the heavy swap usage. I'll do that tonight... if I remember. Edit: post upgrade, swap freed up.

Network:
As with other graphs, a drop in egress traffic when we blocked AI bots.

Storage:

Cloudflare (30 days):

Cloudflare saving us loads of egress traffic every month, and protecting the site from AI bots. If we weren't using Cloudflare, I don't think AZ would exist as it does today.

Seeing what appears to be a distributed scrape or AZ coming from multiple network from multiple overseas locations. From what I can tell predominantly bon English speaking countries.

I've had to be pretty heavy handed with the catpcha challenges for those countries for now. It seems to have helped, will keep an eye on things. Hopefully not too much collateral damage to legitimate users....

Seems to have been more malicious traffic today, causing issues for the site. I've put a block in place, which seems to have helped.
I'll keep an eye on it.

We've seen some issues with AZ again today. On investigating it looks like more DDOS/abuse traffic from overseas. I've applied the same sort of block as last time we saw this, and things appear to have returned to normal..

The sharp drop is when the "attack" started, the jump is when it was blocked... the red line is blocked requests.

Lemmy 0.19.11 (which I've just upgraded AZ to) has a new feature to allow regular federation, but require users be logged in to view content.

I'd like to gauge feedback from users on this. It will not add privacy, or limit the propagation of posts/comments etc. But it will limit AZ server resource consumption by bots or users that are not logged in.

Thoughts/concerns on enabling this feature?

Update: thank you all for your thoughts and feedback on this. We'll leave AZ as it is, though may use this feature in future if we need to mitigate attacks or other malicious traffic.

Not entirely clear to me what is going on, but we've seen a large influx in traffic from oversea today. This has lead to high CPU and performance issues.

I've put in place a block to what seems to be the source of the traffic, but its not perfect and may cause other issues. If you see/hear of any please let me know here.

I'm about to restart services for this upgrade. Shouldn't be down longer than a few minutes.

I'll be working on upgrading aussie.zone to lemmy 0.19.6 today. All going well disruption will be brief, but there may be some performance issues related to back end DB changes required as part of the upgrade.

I'll unpin this once complete.

I've spun this up for fun, to see how it compares to the base lemmy UI. Give it a whirl, and post any feedback in this thread. Enjoy!

It could go down at any time, as it looks as though the dev is no longer maintining it...

edit: using this https://github.com/rystaf/mlmym

UPDATE Tuesday 12/11: I've killed this off for now. Unclear of why, but was seeing a huge number of requests from this frontend to the lemmy server back end. Today it alone sent ~40% more requests than all clients and federation messages combined.

Its been 6 months or so... figure its time for another of these. Keep in mind there have been some major config changes in the last week, which has resulted in the oddities below.

Graphs below cover 2 months, except Cloudflare which only goes to 30 days on free accounts.

CPU:

Memory:

Network:

Storage:

Cloudflare caching:

Comments: The server is still happily chugging along. Looking even happier now that I've properly migrated pict-rs to its integrated object storage config, rather than the bodged up setup.

RAM/CPU are all fine. Storage use is growing slowly as various databases grow. Still a long way from needing to purge old posts, if ever.

Cloudflare is saving less traffic these days, since Lemmy added support to proxy all images. Not a concern, well under the bandwidth cap for the server.

As usual feel free to ask any questions.

I'm in the process of migrating images to a properly configured object storage setup. This involves an offline migration of files. Once complete, I'll start up pict-rs again. Until then, most images will be broken.

All going well this will finish by morning Perth time, and once up and running again may help with the ongoing issues we've had with images.

After some users have had issues recently, I've finally gotten around to putting in place a better solution for outbound email from this instance. It now sends out via Amazon SES, rather than directly from our OVH VPS.

The result is emails should actually get to more people now, rather than being blocked by over-enthusiastic spam filters... looking at you Outlook and Gmail.

About to reboot the server, hold onto your hats.

Hey all, following the work over the weekend we're now running Lemmy 0.19.4. please post any comments, questions, feedback or issues in this thread.

One of the major features added has been the ability to proxy third party images, which I've enabled. I'll be keeping a closer eye on our server utilisation to see how this goes...

This weekend I'll be working to upgrade AZ to lemmy 0.19.4, which requires changes to some other back end supporting systems.

Expect occasional errors/slowdowns, broken images etc.

Once complete, I'll be making further changes to enable/tweak some of the new features.

UPDATE: one of the back end component upgrades requires dumping and reimporting the entire lemmy database. This will require ~1 hour of total downtime for the site. I expect this to kick off tonight ~9pm Perth time.

UPDATE2: DB dump/re-import going to happen ~6pm Perth time, ie about 10 minutes from this edit.

UPDATGE3: we're back after the postgres upgrade. Next will be a brief outage for the lemmy upgrade itself... after I've had dinner 🙂

UPDATE34: We're on lemmy 0.19.4 now. I'll be looking at new features/settings and playing around with them.

Its been a little while since I posted stuff :)

CPU:

Memory:

Network:

Storage:

Cloudflare caching:

Comments:
Not much has changed in quite a while. I still have a cron-job running to restart Lemmy every day due to memory leaks, hopefully this improves with future updates. Outside of that, CPU, memory and network usage are fine.
Object storage usage is growing steadily, but we're a long way from paying more than the monthly minimum Wasabi fee.

Available here

If you'd like to volunteer as a moderator for this community, please reply here saying so.