Around the same time, Cloudflare’s chief technology officer Dane Knecht explained that a latent bug was responsible in an apologetic X post.

“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack,” Knecht wrote, referring to a bug that went undetected in testing and has not caused a failure.

all 43 comments

sorted by: hot top controversial new old
[–] 135 points 11 months ago* (6 children)
  • [–] 62 points 11 months ago* (5 children)

    thanks for illustrating the corpo speak

    I hope the bug is fine

  • source
  • parent
  • hideshow 5 child comments
  • [–] 49 points 11 months ago* (4 children)

    Nobody ever asks if the bug is ok

  • source
  • parent
  • hideshow 4 child comments
  • [–] 13 points 11 months ago (1 child)

    Fun fact time:

    That’s why they’re called computer bugs.

    In 1947, the Harvard Mark II computer was malfunctioning. Engineers eventually found a dead moth wedged between two relay points, causing a short. Removing it fixed the problem. They saved the moth and it’s on display at a museum to this day.

    The moth was not okay.

    And to be fair, the word bug had been used to describe little problems and glitches before that incident, but this was the first case of a computer bug.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 41 points 11 months ago (7 children)

    If you want a technical breakdown that isn't "lol AI bad":

    https://blog.cloudflare.com/18-november-2025-outage/

    Basically, a permission change cause an automated query to return more data than was planned for. The query resulted in a configuration file with a large amount of duplicate entries which was pushed to production. The size of the file went over the prealloctaed memory limit for a downstream system which died due to an unhandled error state resulting from the large configuration file. This caused a thread panic leading to the 5xx errors.

    It seems that Crowdstrike isn't alone this year in the 'A bad config file nearly kills the Internet' club.

  • source
  • hideshow 7 child comments
  • [–] 2 points 11 months ago (1 child)

    ‘A bad config file nearly kills the Internet’ club

    There's no such thing as bad data, only shitty code to create it or ingest it, and bad testing that failed to detect the shitty code. The overflow of the magic config-file size threw an exception, and there was no handler for that? Jeez Louise.

    And as for unhandled exceptions, you'd think static analysis would have detected that.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 0 points 11 months ago (4 children)

    So the actual outage comes down to pre-allocating memory, but not actually having error handling to gracefully fail if that limit is or will be exceeded... Bad day for whoever shows up on the git blame for that function

  • source
  • parent
  • hideshow 4 child comments
  • [–] 14 points 11 months ago (3 children)

    This is the wrong take. Git blame only show who wrote the line. What about the people who reviewed the code?

  • source
  • parent
  • hideshow 3 child comments
  • [–] 10 points 11 months ago* (1 child)

    Plus the guys who are hired to ensure that systems don't fail even under inexperienced or malicious employees, management who designs and enforces the whole system, etc... "one guy fucked up and needs to be fired" is just a toxic mentality that doesn't actually address the chain of conditions that led to the situation

  • source
  • parent
  • hideshow 1 child comment
  • [+] 11 points 11 months ago (1 child)
  • [+] 53 points 11 months ago (15 children)
  • [–] 103 points 11 months ago (9 children)

    Shitty code has been around far longer than AI. I should know, I wrote plenty of it.

  • source
  • parent
  • hideshow 9 child comments
  • [–] 30 points 11 months ago (1 child)

    They trained it on the work of people like you.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 28 points 11 months ago

    Shame on them. I mark my career by how long it takes me to regret the code I write. When I was a junior, it was often just a month or two. As I seasoned it became maybe as long as two years. Until finally i don't regret my code, only the exigencies that prevented me from writing better.

  • source
  • parent
  • [–] 6 points 11 months ago (3 children)

    I too have looked at my earliest repos in dispair

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 11 months ago (2 children)

    It's always depressing when you ask the AI to explain your code and then you get banned from OpenAI

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 11 months ago (1 child)

    Who didn't get hit by the fork bug the professor explicitly asked you to watch out for since it would (back then with windows systems being required to use the campus resources) require an admin with Linux access to eliminate.

    It was kind of fun walking in to the tech support area and them asking your login name with no context knowing what the issue was. Must have been a common occurrence that week of the course.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 11 months ago

    It was kind of fun walking in to the tech support area and them asking your login name with no context knowing what the issue was.

    I see this zip bomb was owned by user icpenis, someone track that guy down.

  • source
  • parent
  • [–] 3 points 11 months ago

    But, AI can do the work of 10 of you humans, so it can write 10 times the bugs and deploy them to production 10 times faster. Especially if pesky testers stay out the way instead of finding some of the bugs.

  • source
  • parent
  • Indirectly, this was. He said this was a bug in their recent tool that allows sites to block AI crawlers that caused the outages. It’s a relatively new tool released in the last few months, so it makes sense it might be buggy as the rush to stop the AI DoS attacks has been pertinent.

  • source
  • parent
  • [–] 21 points 11 months ago (2 children)

    Humans are plenty capable of writing crappy code without needing to blame AI.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 15 points 11 months ago (2 children)

    Blame it on the massive tech sector layoffs

  • source
  • hideshow 2 child comments
  • [–] 3 points 11 months ago (1 child)
  • [–] 0 points 11 months ago

    Obviousness? If you mass layoff your tech staff, you take the risk of more technical failures.

    A smaller staff cannot do the same work as a larger one, and I guarantee you they're being asked to progress at the same speed. So, the tradeoff is on the quality of the product and the testing, not on the speed of development.

  • source
  • parent
  • [–] 14 points 11 months ago* (1 child)

    If your website or service was disrupted by the Cloudflare outage, then you only have yourself to blame.

    The Crowdstrike incident from last year was a wakeup call to abandon centralized services like Cloudflare. If you missed that lesson, then it is on you.

  • source
  • hideshow 1 child comment
  • [–] 4 points 11 months ago

    Did they though? Aside from the "every outage is a latent bug" angle, from their postmortem it doesn't seem to me like they tried to blame it on anything but their failure to contain the spread of (and timely diagnose) the issue

  • source
  • [–] 8 points 11 months ago (3 children)

    Why's he saying it's not an attack? Sounds like he's protesting too much.

  • source
  • hideshow 3 child comments