Someone always chimes into these discussions with the experience of being DDOSed and Cloudflare being the only option to prevent it.

Sounds a lot like a protection racket to me.

I hate it but there really isn't much in the way of an alternative. Which is why they're dominant, they're the only game in town

Except, if you chose the wrong 1 of that 10 and your company is the only one down for a day, you get fire-bombed. If "TEH INTERNETS ARE DOWN" and your website is down for a day, no one even calls you.

Note that this outage by itself, based on their chart, was kicking out errors over the span of about 8 hours. This one outage would have almost entirely blown their downtown allowance under 99.9% availability criteria.

If one big provider actually provided 99.9999%, that would be 30 seconds of all outages over a typical year. Not even long enough for people to generally be sure there was an 'outage' as a user. That wouldn't be bad at all.

Their motivation is that that file has to change rapidly to respond to threats. If a new botnet pops up and starts generating a lot of malicious traffic, they can't just let it run for a week

You would do well to go read up on the 1990 AT&T long distance network collapse. A single line of changed code, rolled out months earlier, ultimately triggered what you might call these days a DDoS attack that took down all 114 long distance telephone switches in their global network. Over 50 million long distance calls were blocked in the 9 hours it took them to identify the cause and roll out a fix.

AT&T prided itself on the thoroughness of their testing & rollout strategy for any code changes. The bug that took them down was both timing-dependent and load-dependent, making it extremely difficult to test for, and required fairly specific real world conditions to trigger. That’s how it went unnoticed for months before it triggered.

My assumption is that the pattern you describe is possible/doable on certain scales and in certain combinations of technologies. But doing this across a distributed system with as many nodes and as many different nodes as CloudFlare has, and still have a system that can be updated quickly (responding to DDOS attacks for example) is a lot harder.

If you really feel like you have a better solution please contact them and consult for them, the internet would thank you for it.

Wasn’t it crowdstrike? Close enough though

Sadly, in my case, even after almost destroying a production cluster, they still decided a test cluster is to expensive and they'll just live with the risk.

"Claude said it was fine, ship it."

Yep. This was the difference between a silent, recoverable error and a loud failure.

It seems like they're planning to remove all potential panics based on the end of their article. This would be a good idea considering the scale of the service's usage.

(Also, for anyone who's not reading the article, the unwrap caused the service to crash, but wasn't the source of the issues to begin with. It was just what toppled over first.)

I've read a lot of people saying "oh what a horror, they used unwrap in production, they should NEVER do that". But I don't think that's true at all.

1. The rust type system is not 100% flawless. For example you have a mathematically proven never-errors algorithm, but inside some steps have provisional Nones. At the end you just unwrap, since there's no point in changing the return type to an option to handle a non-existent error. You can't represent "this is None but I promise that it won't by the end" in rust's type system.
2. Sometimes it's not worth handling the error. For example a simple CLI tool, just throw expects everywhere that shows why it failed, since most CLI tools just close anyway on failure, might as well panic.

That being said, this was clearly a case where ? should have been used, since the function has a clear error case. And it also is a critical application that can't afford panicking as an error handling method.

EDIT: Just to clarify, the way you would do this is by having a type Features that can error on creation/parsing which has a limit of 200 entries. And afterwards you can just use that type knowing that it will always have <= 200 entries. And you gracefully handle that one error case on creation/parsing.

EDIT2: Another point for why I disagree with "production should never have unwrap":

3. Unwrap is basically the same as assert(my_variable.is_some). I don't know a single person that would say "you should never have asserts in production code", the only reason I disable some assertions for release builds is for performance. Asserts are statements that should never do anything, by that logic they pose no harm by being there, so no need to remove them, their only effect should be wasting a few CPU cycles. Since asserts are only put when the programmer assures that a certain precondition will always be met. And if it is not, it's because the code changes and automatic tests should catch that.

They also apparently didn't do any input validation. This is why a faulty config was able to even trigger all of that.