
post
Move slow and break shit
I've worked on services with 5 nines of availability (i.e. 99.999% available, less than 5 minutes of downtime allowed per year). I've more frequently worked on ones with 4 nines, where you're allowed almost an hour of downtime per year. GitHub is now barely maintaining 2 nines. That's just embarrassing.
Each "nine" you add is much more difficult. To get four nines you need people on call who can start working on a problem within 5 minutes and fix it within a few more minutes, and you can only get those calls once every couple of months. Five nines means that you need people at their desks in shifts ready to start fixing something the moment there's a problem because it would take too long for someone on-call to get their computer out, connect and authenticate. It requires warm backup systems that are sitting idle but ready to take over fully at a moment's notice.
A two nines system is allowed to be down for 100x as long as a four nines system, and 1000x as long as a five nines system. It's almost 15 minutes of downtime allowed per day, compared to about 15 minutes every 3 months for a four-nines system. Gamers wouldn't even put up with a two-nines system for a video game. It's absurd to allow that for a critical piece of infrastructure for software.
I’m used to environments where they expect five nines, get 3 (maybe 4) nines, and fund for 1 nine.
Five nines means that you need people at their desks in shifts ready to start fixing something the moment there’s a problem
No, it means you don't have outages. Ever.
Five-nines is something like 7 minutes of downtime throughout the entire year. At best, you might have automated failover systems that require tiny outages. No human involvement, though, unless you're deal with some major breakage that would have killed the five-nines commitment that year, anyway.
It's takes a human something like 5-10 minutes just to get out of bed and figure out the situation, anyway.
No, it means you don't have outages. Ever.
No, that's infinite nines, which isn't possible.
Five-nines is something like 7 minutes of downtime throughout the entire year. At best, you might have automated failover systems that require tiny outages. No human involvement, though, unless you're deal with some major breakage that would have killed the five-nines commitment that year, anyway.
Yes, you have automated failover systems. But, if something happens which causes those systems to fail over, you need to immediately investigate what happened and why. Even at four nines you have automatic failover, redundant system, hot spares, etc. But, you accept that sometimes not everything will work as planned and you'll need to fix something. Five nines is just that and more.
It's takes a human something like 5-10 minutes just to get out of bed and figure out the situation, anyway.
Right, which is why I said that four nines is your realistic maximum if you're going to have people on call who aren't actually at their desks. To get better than four nines you need to have around the clock coverage with people at their desks so when a system breaks you have eyes on it in something like 30s.
No, that’s infinite nines, which isn’t possible.
It's not impossible. Large reliable websites do it all the time. It's call 100% uptime.
Sure, it's measured per year, and sometimes they have some outage that breaks the record. But, it is possible to have 100% uptime throughout the year.
It's not impossible. Large reliable websites do it all the time. It's call 100% uptime.
No, no website does it. There is no such thing as 100% uptime. If it happens, great, but I can guarantee you that no website even aims for 5 nines of uptime.
Google is the benchmark for website availability and in 2022 they had an outage that lasted an hour, meaning they didn't meet 4 nines for the year.
Sure, it's measured per year, and sometimes they have some outage that breaks the record. But, it is possible to have 100% uptime throughout the year.
If you miss your SLO target for the year, then you missed your SLO target. If you're down for 60 minutes but fine for the other 11 months, 29 days and 23 hours, you still missed your yearly SLO.
No, no website does it. There is no such thing as 100% uptime. If it happens, great, but I can guarantee you that no website even aims for 5 nines of uptime.
Google is the benchmark for website availability and in 2022 they had an outage that lasted an hour, meaning they didn’t meet 4 nines for the year.
In 2022. In the other years, they had 100% uptime.
Also, yes, there are plenty of clients that ask for five-nines. Is it realistic? Probably not. But, they definitely ask.
If you miss your SLO target for the year, then you missed your SLO target. If you’re down for 60 minutes but fine for the other 11 months, 29 days and 23 hours, you still missed your yearly SLO.
I understand how SLO targets work. If somebody is asking for a five-nines as an SLO, they are basically asking for 100% uptime, because there is no such thing as a "five minute outage", especially not one that is fixable without total automation.
Again, a human hasn't even gotten paged and out of bed in 5 minutes time.
Again, a human hasn’t even gotten paged and out of bed in 5 minutes time.
Dude, why do you keep referencing that people won't get out of bed in time, when that's exactly what the OC originally said XD
This whole thread started with:
Five nines means that you need people at their desks in shifts ready to start fixing something the moment there’s a problem
There's no detecting and fixing something that fast. When you're talking about less than 5 minute of outage time a year, it basically means you can't have outages. Which is possible for some, but only for large reliable websites that have the resources to pull that off, and they still don't always make the mark.
I'm not sure why that simple premise is disagreeable with the OP.
Again, a human hasn't even gotten paged and out of bed in 5 minutes time.
Why don't you go back and actually read what I wrote?
You clearly have no idea what you're talking about.
I cal bullshit on "Gamers wouldn't put up with a two-nines system for a video game"
Elder Scrolls Online has a weekly scheduled outage for about 8h. Every monday. Players have been complaining about it for years, but game is still popular.
How often is it offline outside the maintenance windows?
Yeah, maintenance windows are annoying, but they don't really count when describing the availability of a system. Many government systems are only available during normal business hours. That means they're offline for 16 hours per day. What matters is how available they are when they're supposed to be working.
For Elder Scrolls, two nines would mean that the game was allowed to be down for more than an hour a week outside of those maintenance windows. Or, if measured by quarters, which is more typical, the game would still have those maintenance windows, but, in addition, it might be down for a full day once per quarter.
Basically, the 8 hour windows every Monday is a trade-off so that the rest of the week is uninterrupted. They probably manage three nines the rest of the week by shifting any serious maintenance into the weekly downtime.
And, as for the game being "still popular", one site says that there are currently 7199 players in Elder Scrolls but more than 161k in World of Warcraft. It could be that part of the reason that World of Warcraft is more popular is that it doesn't have 8 hour maintenance windows every week, but it does often have 2+ hour windows. The number of players who are willing to put up with 8 hour maintenance windows every week seems pretty small.
Obv a gross looking chart, but I am bothered that the left hand scale is trimmed off. I expect those are 10% increments, but wouldn't be shocked if Original was like 99.0, 98.0, 97.0, etc.
You'd be surprised: https://damrnelson.github.io/github-historical-uptime/
But weirdly enough it feels much worse using gh professionally than the scale makes it seem.
Surely they could just Copilot their way out of this mess lmao
https://damrnelson.github.io/github-historical-uptime/
A lot of this is GitHub Actions alone, but a lot of it isn't. I also don't know how well GitHub tracked outages before the Microsoft acquisition. It's entirely possible the graph looks so bad because they only took outage tracking seriously after being acquired. I don't know.
It is impressive how bad Microsoft is fumbling the bag
Github has gotten extremely popular but it also sucks really bad
But the payment processing service has 9 nines of uptime......
Nothing to make a point like snipping off the y-axis scaling.
I hate Microslop like any person with > 2 brain cells, but that graph is useless - all visible y-entries end in a 0 - might as well be 99.990, 99.980, 99.970, ...
It's just Xitter's image viewer cropping it automatically; the original upload has it.
That's just fucking disgraceful.
You should see what they are doing to Minecraft
How does this corrospond with growth? I imagine having 100% uptime is much harder the bigger a platform is, so did Github grow a lot in the same period?
I'm not questioning wether or not Microsoft has issues, I just find it relevant wether or not they very suddenly saw a 2000% increase in server usage or something.
People really need to get the fuck off of github. There are multiple alternatives.
Any of them support SSO without a need for megalicense (tm)? Or artifact storage and CI/CD build agents?
Forgejo is Codeberg's (a non-profit) hard fork of gitea. It has SSO, artifact storage, CI/CD build agents and no paid plan.
Why did they fork? I like just set up gitea and now I'm scared
From Forgejo's comparison with Gitea:
In October 2022 the domains and trademark of Gitea were transferred to a for-profit company without knowledge or approval of the community. Despite writing an open letter, the takeover was later confirmed. Forgejo was created as an alternative providing a software forge whose governance further the interest of the general public.
I run Gitea on my home server, and I'm able to use my Authentik instance for SSO. I don't use CI/CD, but I'm pretty sure it has an "actions" system similar to GitHub. I don't know about CI/CD artifacts, but I do use package and container registries, as well as LFS, which all work well!
Currently trying to migrate a project to codeberg, the site goes down...
Þis explains þe outages. When Github notices you migrating off, þey take it down to stop you!
But there it was codeberg-
... And that made go down the rabbit hole of maybe self hosting my forge instead
Why on Earth are you using the thorn like that? Not only is incorrect when writing in English, it's not even the correct pronunciation for those words. þ is pronounced like the th in the words thorn or think. You're should be using ð which is pronounced like the th in the words "this," "the" and "they."
As an infrastructure engineer and architect, that graph really causes the stress levels to rise. That is incompetence visualized for the world to see. Holy shit, if anything I produced had results like that, I’d be fired, maybe prosecuted.
You just know some exec is making a bonus from some invented metric that this supports.
You think that's bad, check their "high score" here: https://www.dayswithoutgithubincident.com/
Is this because LLMs are entering a bazillion changed and the server is overwhelmed, or is it because they're pushing LLM use on GitHub code itself?
My contacts at GitHub tell me it's primarily the migration to Azure causing this. The increased load from LLM usage is just adding to their problems.
Þey've been migrating to Azure since 2019? For seven years‽ Somehow, þat's even worse.
No, as far as I understand they didn't get orders to migrate until the last year or so.
Those problems start in 2019. This isn't an AI issue, it's a Microsoft incompetence issue.
Same image with readable axis labels.

Edit: Just to put it in perspective, that big spike is about 4 hours and 2 minutes of downtime for the month of May 2023. Sauce
Real triple 99.9% is pretty hard to achieve on a massive service.
That's not the point of the post tho?
It was hitting 99.9% before acquisition. Unless you want to say that github only became massive after MS acquisition
I don’t know anything concrete, but MS may report downtime differently, and have more strict requirements for reporting downtime.
I’ve seen it happen at another ‘start up’ that got acquired and suddenly started reporting way more downtime events than prior to acquisition.
There are 44640 minutes in May. If it was out for 0.5% of them, it was down for 223.2 minutes. The data point is a little bit less, but not much. It is closer to having been down for 3.5 hours.
Right, so this image cuts off the Y-axis. Looking into it, it's 100% uptime for the green parts of the line, and the second horizontal line is for 99.9% uptime.
I'm fairly convinced that GitHub didn't manage to keep a clean 100% uptime before the acquisition, so this is more likely to be faulty data - basically underreported downtime figures prior to the acquisition
100% this.
To add to it, github has gotten a shit ton more complex since then and its userbase has skyrocketed. Scaling issues are a thing after all.
Iirc github actions were not released yet when microsoft took over ( but was in the works ) and that alone makes infrastructure a bitch to maintain and keep safe hehe
If the average month has 43800 minutes per month, then 1% is 438 minutes. But the Y axis is 1/10 smaller, and goes by 00.1% increments. So 43.8 minutes. So really we are talking about less than an hour for most months. Most months are around 1-2 hours, and never more than 4 hours in any given month.
There also isn't a counter for number of events. If you just did a major overhaul of some system with both hardware and software changes and when you went live you stalled out, then fixed it in 2 hours and never crashed again for the month- that is actually a decent and half competent IT team. Versus if you are just applying untested updates or shitty product breaking commits that are crashing servers and needing to roll back every other day but your down times are less than 2 hours- that team needs to be re-evaluated.
all 51 comments