Mine was replacing a failed hard drive in array.

• Check array health, see one failed member
• popped out the hot swappable old drive , popped in the new one
• Check array health to make sure the array rebuild is underway
• See array now has TWO failed member, and realize I feel the drive in my hand still spinning down

shit.

I pushed a $1 bln test trade through production instead of my test environment.... that was a sweaty 30 minutes

Not IT but data analyst. Missed a 2% salary increase for our union members when projecting next year's budget. $12 million mistake that was only caught once it was too late to fix.

Yep. Ran a config as code migration on prod instead of dev. We introduced new safeguards for running against prod after that. And changed the expectations for primary on call to do dev work with down time. Shifted to improving ops tooling or making pretty charts from all the metrics. Actually ended up reducing toil substantially over the next couple quarters.

10/10 will absolutely still do something dumb again.

I once deleted the whole production kubernetes environment trying to fix an update to prod give awry, at11pm. My saving grace was that our systems are barely used between 10pm-8am, and I managed to teach myself by reading enough docs and stack overflow comments to rebuild it and fix the initial mistake before 5am. Never learned how to correctly use a piece of stack that quickly before or since.

I deleted all of our DNS records. As it turns out, you can't make money when you can't resolve dns records :P