When programming.dev became practically unusable most of the time in the last several days I was considering moving elsewhere. No post or announcement even acknowledging the issue here in meta didn't make me hopeful in the issues subsiding at all. (I've experienced instance death before on feddit.de.)
Great to see this development.
Thank you for your continued work and efforts! π
Hetzner is a good provider. Congrats on the migration and good luck.
and thank you for being a great contributor to the community! this site would be nothing without all of you!
it is helping, thank you for the sponsorship. I should have migrated a long time ago because the costs really were adding up. I'll update my sponsor page after I have a fresh month of data for the bucket costs (which are still on Vultr) and the new server costs (which hopefully should be static). Thanks for the suggestion!
Thank you for all the work that goes into maintaining this instance!
Very happy to hear this. I was noticing frequent slowness recently. I never really got to the point of considering leaving because I'm not on here that much anyway. But I do sponsor your github in a small way. I hope the sponsorships cover at least the hosting cost.
You might want to update your Github sponsor page as it sounds like some of that info is out of date after this migration.
Thanks, boss. You are a good man. πΊ
programming.dev is the 9th largest lemmy server. https://join-lemmy.org/instances
That stat was probably that low due to the server being down for around 90% of the last two weeks. If you look now it's at 220 and it will continue to go up.
On top of that, every action on every server that is federated is relayed to every instance. So all of lemmy.world's activity is still relayed to us and we have to handle it. Same for the other servers.
On top of that we also operate many other services:
But really it was mostly just postgres thrashing on all the requests. Here's a look at our Cloudflare dashboard for number of requests:
Yes this should be handle-able by a server that small (think actor paradigm), but I was unable to tune postgres to get it to that point as I'm not great at database stuff. I'm sure a DBA would have done a better job. I will note that some of the queries being used in the lemmy code are very badly optimized and were taking 20+ seconds to run each time, locking up the instance. With that on top of some other badly optimized selects for things like reading comments (which would take like 7s mean), there wasn't much I could do.
With the cost difference it was well worth it to just upgrade to a cheaper better server all around.
Oh we are getting attacked constantly, that definitely didn't help, but a large portion of it was just thrashing from postgres not getting enough memory.
sorry for all the trouble. I would understand it if you still do, as I haven't been the best operator.
Hetzner. Honestly every provider was cheaper. I literally didn't find a single provider that was even close to as expensive as Vultr. You can look at Vultr's deploy page here (might need to be logged in for that). For 16GB of RAM on any product, the minimum cost is $80 a month. We were paying $120+.
It's honestly crazy how expensive Vultr is. The servers might have better processors, didn't really check that, but all our performance depends on RAM and cores, so none of that really matters.
Also was able to get 64GB of ECC RAM on Hetzner. No clue if Vultr provided that, but they don't list it anywhere.
Providers I looked at:
Sidebar stats say this instance gets 23 users/day, which seems absolutely tiny and within the capabilities of 4c/16gb cloud instance.
We were maxing out every part of the server whenever any even slightly significant number of users were on the fediverse.
Idk anything about how lemmy/fediverse works, but does that mean tiny instances like this get hit when the rest of the network is experiencing high load? Seems problematic.
EDIT: btw, thanks for the free service and the effort you put in to keep it running!
Thanks, it seems a lot faster!
Fantastic, thanks! I might be spending more time on Lemmy after all.
Out of curiosity what provider did you guys move to that gives better bang for the buck than vultr?
Oh, cool, to be honest I was actually in the process of transitioning to another instance due to the instability, but it sounds like I may not need to!
Thanks a lot!
Good news. Thank you.
And it's good to hear that it wasn't something nefarious messing with the instance.
For the last ~2 weeks the infrastructure team have been actively investigating performance with our server and actions are currently being taken. You should expect us to post an announcement with more detailed information soonβ’.
It was being used to ddos the site (and wasnt being maintained) so was turned off
not the only one, i had used it too
Wow I never even knew there was an old.programming.dev TIL!
thanks!
From what I understand, that frontend is no longer being developed and has accumulated a number of issues. Some instances may be using a custom fork of it, such as lemmy.world.
The admin team are aware, and are planning on upgrading to the latest lemmy version in the coming days.
Hm. Pulsetic seems to have removed our status monitors. I have gone ahead and added the monitor back, but pulsetic seems to not want to show our status anymore. I'll have to investigate other options for a service monitor.