So the people running the systems that underlie our lives are incompetent, and don't know how to keep them running properly. I'm supposed to be reassured by that?

all 20 comments

sorted by: hot top controversial new old
[–] 12 points 10 hours ago (1 child)

It worse than you think. A cyber attack would probably be better than the alternative, which is that the systems are more complex than ever, but also lack the expertise to maintain properly, and lack the funding to perform that maintenance. Over time more and more infrastructure will crumble.

  • source
  • hideshow 1 child comment
  • [–] 23 points 11 hours ago (7 children)

    it’s not reassuring, but it’s probably due to greed-based incompetence

  • source
  • hideshow 7 child comments
  • [–] 3 points 11 hours ago (6 children)

    Sorry, I've worked in places of large scale an order of magnitude smaller than MS - the layers of control, redundancy, oversight, etc - I don't buy it was a simple mistake.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 12 points 11 hours ago

    greed-based incompetence

    I don’t buy it was a simple mistake.

    Incompetence scales well, though. It wasn't simple, and it wasn't a mistake so much as an un-hoped-for consequence of a series of stupid decisions based, probably, on greed. As in: it could have been better, for only a few percentage cost more, but that was rejected as a course of action.

  • source
  • parent
  • [–] 10 points 10 hours ago (4 children)

    Would you buy that it’s a series of complicated mistakes?

    Back when I worked at Microsoft, they were converting all of the testers into telemetry and data engineers because everyone can test their own code and we can test shit in prod with AB testing and telemetry. More recently they pushed the engineering teams to use AI to write as much code as possible. They have also pushed out their most experienced engineers with early retirement offers.

    All of these strategic decisions could plausibly add up to a perfect storm of fuck ups, big and small, complicated and simple, that form like Voltron into the mother of all fuck ups.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 2 points 5 hours ago (3 children)

    Back when I worked at Microsoft, they were converting all of the testers into telemetry and data engineers because everyone can test their own code and we can test shit in prod with AB testing and telemetry.

    As in one set of Windows users got the "A" updates and one set got the "B" updates?

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 5 hours ago (2 children)

    Yeah. It’s a common term in software development. Basically for any software that gets updated over the network you can setup a system to give the update to some percentage of users and then gather data on their usage and if it looks good, roll out the changes to more users. It is a very useful tool but should not be a replacement for common sense local testing.

    https://en.wikipedia.org/wiki/A/B_testing

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 hours ago (1 child)

    Cool cool cool cool cool.

    So if there are two windows computers hooked up to, I don't know, some commercial freezers, and the only requirement is that they have the latest updates installed, one computer might have the "A" updates and one has the "B"?

    Doesn't have to be freezers in the example; it could be industrial sawmills, or computers running Labview.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 5 minutes ago

    Yeah. In practice with Windows the machines would all get the same update but Windows Update might say it’s availalable for each machine at a different time, depending on the roll out schedule. I think most of the experimentation happens on machines where the user opts into the Windows Insider program. But I haven’t paid close attention to Windows updates in years so take Windows specific details with a huge grain of salt.

    In general though, for some random software they might release an A version witha ad at the top of the screen and a B version with the ad at the bottom and then use telemetry to figure out which version is the most effective.

  • source
  • parent
  • [–] 7 points 4 hours ago (5 children)

    Do you remember when all the train derailments happened?

    They happened before. Then a big one making news happened. Then ALL of them made the news for a while. Now they continue to happen but no longer make news.

    Events happen.

  • source
  • hideshow 5 child comments
  • [–] 5 points 4 hours ago* (2 children)

    The same thing happened with aviation. The Max crashes happened and suddenly every minor aviation incident became national news. There are many such examples. Stories that would only have been covered in specialist publications break containment after a big story.

    Big news companies cover what they think will get them clicks. Something being talked about more in the press doesn't mean its more common, just that mainstream media thinks its profitable to cover at the moment.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 4 hours ago

    100%, although to be fair, profits are part of it, and just sheer interest is the other - interest in the news drives eyeballs to watch the ads that make the money. So not to disagree, but we also do it to ourselves. :)

  • source
  • parent
  • [–] [S] 2 points 1 hour ago

    Thank you, a reminder to check our perceptual biases is always relevant. I don't think that it applies in this case, though. Shark attacks, train derailments, minor aviation incidents, et cetera, do indeed happen with regularity. But an email service outage that affects 450 million people, and a ground-stop order that affects every commercial flight in the United States, well, those are highly unusual events. Further, the very fact that they not proximately related seems to indicate (along with dozens of other indicators) that the foundations of our society are getting shaky, for whatever reason.

    Thus, the reason I'd feel better if the two events were related, and the reason was an attack by malicious actors. Then, it would have a cause that was a one-off, rather than be part of a pattern.

  • source
  • parent
  • [–] 6 points 8 hours ago

    My local news said the aviation administration had a statement saying the old system was overwhelmed. I find this very sus as I doubt yesterday eclipsed the highes of air traffic in the last few years.

  • source
  • [–] 3 points 10 hours ago* (last edited 2 hours ago)

    At least when there is a major cyberattack it won't feel so different from the usual.

  • source
  • [–] 2 points 4 hours ago
    [–] 1 point 11 hours ago

    As airplanes fall from the sky in the background: "all praise our glorious overlords!"

  • source