It worse than you think. A cyber attack would probably be better than the alternative, which is that the systems are more complex than ever, but also lack the expertise to maintain properly, and lack the funding to perform that maintenance. Over time more and more infrastructure will crumble.
post
it’s not reassuring, but it’s probably due to greed-based incompetence
Sorry, I've worked in places of large scale an order of magnitude smaller than MS - the layers of control, redundancy, oversight, etc - I don't buy it was a simple mistake.
greed-based incompetence
I don’t buy it was a simple mistake.
Incompetence scales well, though. It wasn't simple, and it wasn't a mistake so much as an un-hoped-for consequence of a series of stupid decisions based, probably, on greed. As in: it could have been better, for only a few percentage cost more, but that was rejected as a course of action.
Would you buy that it’s a series of complicated mistakes?
Back when I worked at Microsoft, they were converting all of the testers into telemetry and data engineers because everyone can test their own code and we can test shit in prod with AB testing and telemetry. More recently they pushed the engineering teams to use AI to write as much code as possible. They have also pushed out their most experienced engineers with early retirement offers.
All of these strategic decisions could plausibly add up to a perfect storm of fuck ups, big and small, complicated and simple, that form like Voltron into the mother of all fuck ups.
Back when I worked at Microsoft, they were converting all of the testers into telemetry and data engineers because everyone can test their own code and we can test shit in prod with AB testing and telemetry.
As in one set of Windows users got the "A" updates and one set got the "B" updates?
Yeah. It’s a common term in software development. Basically for any software that gets updated over the network you can setup a system to give the update to some percentage of users and then gather data on their usage and if it looks good, roll out the changes to more users. It is a very useful tool but should not be a replacement for common sense local testing.
Cool cool cool cool cool.
So if there are two windows computers hooked up to, I don't know, some commercial freezers, and the only requirement is that they have the latest updates installed, one computer might have the "A" updates and one has the "B"?
Doesn't have to be freezers in the example; it could be industrial sawmills, or computers running Labview.
Yeah. In practice with Windows the machines would all get the same update but Windows Update might say it’s availalable for each machine at a different time, depending on the roll out schedule. I think most of the experimentation happens on machines where the user opts into the Windows Insider program. But I haven’t paid close attention to Windows updates in years so take Windows specific details with a huge grain of salt.
In general though, for some random software they might release an A version witha ad at the top of the screen and a B version with the ad at the bottom and then use telemetry to figure out which version is the most effective.
Do you remember when all the train derailments happened?
They happened before. Then a big one making news happened. Then ALL of them made the news for a while. Now they continue to happen but no longer make news.
Events happen.
The same thing happened with aviation. The Max crashes happened and suddenly every minor aviation incident became national news. There are many such examples. Stories that would only have been covered in specialist publications break containment after a big story.
Big news companies cover what they think will get them clicks. Something being talked about more in the press doesn't mean its more common, just that mainstream media thinks its profitable to cover at the moment.
100%, although to be fair, profits are part of it, and just sheer interest is the other - interest in the news drives eyeballs to watch the ads that make the money. So not to disagree, but we also do it to ourselves. :)
A nationwide grounding event is not just farming clicks. I'm guessing the cause is a vibe coded software update that only hit a bug when Thanksgiving traffic ramped up.
Thank you, a reminder to check our perceptual biases is always relevant. I don't think that it applies in this case, though. Shark attacks, train derailments, minor aviation incidents, et cetera, do indeed happen with regularity. But an email service outage that affects 450 million people, and a ground-stop order that affects every commercial flight in the United States, well, those are highly unusual events. Further, the very fact that they not proximately related seems to indicate (along with dozens of other indicators) that the foundations of our society are getting shaky, for whatever reason.
Thus, the reason I'd feel better if the two events were related, and the reason was an attack by malicious actors. Then, it would have a cause that was a one-off, rather than be part of a pattern.
My local news said the aviation administration had a statement saying the old system was overwhelmed. I find this very sus as I doubt yesterday eclipsed the highes of air traffic in the last few years.
At least when there is a major cyberattack it won't feel so different from the usual.
[The Machine Stops](https://web.cs.ucdavis.edu/~rogaway/classes/188/materials/the machine stops.pdf)
As airplanes fall from the sky in the background: "all praise our glorious overlords!"
all 20 comments