288
Cloudflare blames massive internet outage on 'latent bug' (techcrunch.com)
submitted 11 months ago by to c/technology@lemmy.world
42 comments fedilink hide all child comments
 

Around the same time, Cloudflare’s chief technology officer Dane Knecht explained that a latent bug was responsible in an apologetic X post.

“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack,” Knecht wrote, referring to a bug that went undetected in testing and has not caused a failure.

you are viewing a single comment's thread
view the rest of the comments
[–] 0 points 11 months ago (4 children)

So the actual outage comes down to pre-allocating memory, but not actually having error handling to gracefully fail if that limit is or will be exceeded... Bad day for whoever shows up on the git blame for that function

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 14 points 11 months ago (3 children)

    This is the wrong take. Git blame only show who wrote the line. What about the people who reviewed the code?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 10 points 11 months ago* (1 child)

    Plus the guys who are hired to ensure that systems don't fail even under inexperienced or malicious employees, management who designs and enforces the whole system, etc... "one guy fucked up and needs to be fired" is just a toxic mentality that doesn't actually address the chain of conditions that led to the situation

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •