you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 1 day ago (8 children)

Regardless, what followed was the newly installed router “flooding” other routers in the Air Traffic Control system. Duffy said that there were “over 1,000 connection attempts every 4 to 5 seconds,”

I'm not a networking person. Is that a lot?

  • source
  • hideshow 8 child comments
  • [–] 20 points 1 day ago (5 children)

    I was a network person before i changed disciplines.

    A direct answer is no, that is not a lot of connections, in fact, it is trivial. What we don't know is any level of meaningful detail. If there were thousands of endpoints each making thousands of connections all at once, that changes the scale of the problem. Furthermore, most of the time, we assume that one host is trying to connect to another host. However, if those connection attempts were network broadcasts, that's a much bigger problem. A network broadcast is sent from one host to ALL hosts on a given network, and then they ALL respond. If all hosts on a network are sending broadcast frames endlessly, that will flood a network. It's the same concept as a denial of service attack; one host can initiate traffic that will exhaust resources, but takes very little resource to do it. Thusly it can cripple a network with very little need for power or bandwidth.

    My guess is that they fucked up the config on the router and was passing traffic that it should not have or initiated traffic that it couldn't handle the responses from. It started using up all resources on itself and probably other network equipment and the whole thing shits the bed. Techs go around trying to reboot network gear, not sure where the problem is, the gear is overwhelmed and won't respond. They have to dick around with out-of-band management (does not use the clogged network for management, there is a separate network for management), try to reboot or do some diagnostics, but as soon as the network device comes back up, it's saturated again making diagnosis slow and painful. People panic, bosses start freaking out, more and more people are called in adding to confusion and now the first people to respond are not working, they're answering endless questions about what's going on, even though they don't really know yet. Vendors are called, contractors are called, calls spiral and fracture, bosses demand updates while people are guessing at what to do next because they aren't allowed to say "I don't know" when planes are in the air. Someone eventually looks at the change management records (you have to document any configuration changes before you do them) and finds the last thing that was touched before everything shit the bed. They go look at the config, see the fuckup, back it out to the last known good config and the shitstorm dies.

    this is 100% speculation treat is as such.

  • source
  • parent
  • hideshow 5 child comments
  • this is 100% speculation treat is as such.

    Will do.

    Someone eventually looks at the change management records (you have to document any configuration changes before you do them) and finds the last thing that was touched

    I'd hope that those records exist. I think there's value in treating this as a dry run response to a cyber attack, and so far I'm not inspired (mostly with the immediate call for money).

  • source
  • parent
  • [–] 8 points 1 day ago* (3 children)

    I’m above average familiar with networking, but likely not as much as you. Currently working on going into that field tho.

    I’m also semi-familiar with how ATC works from touring their local facilities as part of a course on aerodynamics.

    I’m fairly certain they're lying to cover their ass.

    Everything in the ATC system has redundancy. They’re saying they clipped one copper wire and the whole system went down, but there would be another established path for that connection, severing one copper wore wouldn’t stop any communications, they would transmit through a fail safe.

    Also, the last time they had to ground all flights was 09/11/2001, and let’s just say it was not because of issues with the extremely reliable technology used in ATC.

    They installed their AI slop bullshit a few days ago.

    They think we’re all stupid, so they didn’t bother asking anyone who understands the systems before formulating their low-quality lie (in a similar fashion to how they came up with their slop ATC system - without speaking to anyone who knows how the shit works).

  • source
  • parent
  • hideshow 3 child comments
  • [–] 3 points 21 hours ago

    You are correct. One physical disconnect of a copper cable isn't going to kill off an entire system. As you noted, they have redundant network paths, equipment, etc. Those systems were designed not to go down hard, and until this week, never did.

    Copper maxes out at 10gbps in a production environment and if they were still using copper Ethernet, they were trunking multiple cables together for higher throughput. The trunking also had to be redundant, so it would take a lot of cutting or unplugging to take it all down as an oopsy.

    Again, all speculation, but I'm still betting on a human-grade fuckup. When in doubt, blame DNS.

  • source
  • parent
  • [–] 2 points 1 day ago (1 child)

    With network-loop type shit, it's less likely they clipped the copper wire, and more likely they brought up the fiber before taking down the copper link and things sent over the copper were coming back on the fiber, looping and exploding everything.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 19 hours ago

    It is possible but the old routing protocols that used to do this (spanning tree) have protections in place to prevent it.

    At one point, routing protocols were a little more promiscuous and you could create a loop by plugging a single Ethernet cable into two ports on the same switch. Cue Benny hill music as everyone went running around trying to figure out what happened.

    Now there are mechanisms like BPDU guard and portfast configs that should prevent this. In most cases, they would be enabled by default. However, I'm sure there are some folks out there who have no idea how to handle an old port config done at the command line. They could have been tinkering to try and make something work, turned off some of the protective features and now they're back to the 90s with routing loops and broadcast storms. The song is Yakety Sax fwiw, always have it at the ready.

    If you ever wonder why the networking people are grouchy, this is why. Networks were and remain complex, highly interdependent, challenging to build and maintain. It takes only a few ignorant people to send something tumbling.

  • source
  • parent