We had an employee break procedure, make a dumb mistake, and cause ~$160k worth of damage to a mission-critical piece of infrastructure. It happened due to her own inattention and disregarding her “here’s how to shut down at the end of the night” checklist, at like 8PM. Basically, instead of doing steps A, B, C, and D, she went “eh I know what I’m doing,” jumped straight to step D, and suddenly heard very expensive noises. It required me and her supervisor to pull an overnight shift to get a bodged workaround in place, just to be ready for the next morning at 8AM. And even then, the gear was out of commission for about a month until we could get it fixed.

All in all, it was about $80k worth of equipment repairs, $40k in equipment rentals (to keep things running in the meantime), and about $40k in additional labor (we had to hire specialized contractors to fix the gear).

The employee 100% thought she was going to get fired when it happened. We were obviously angry and disappointed that she made such a dumb mistake, but we didn’t yell or chastise her. We simply told her to go ahead and clock out for the evening, and we’d deal with fixing things overnight. She tried to say she could stick around to help… But this was already at the end of her shift, she was obviously not in the right headspace to pull an overnight shift, and we were both too frustrated to have her around at the time. She was crying on her way out the door.

The supervisor decided to keep her on instead of firing her, for this exact reason. She didn’t get a raise, but she didn’t get fired either. She got reprimanded, but her supervisor was confident that she would never make the same dumb mistake again. And now her story is used as a cautionary tale to drive home the importance of following procedure when we’re training new hires.

Back when I worked at the foundation there was a special permission that you couldn't have unless you'd broke stuff super badly. To push new code you have to have one of those folks present. They were responsible for making sure there was a back out plan and ran the actual deploy script. Their ssh keys did the deploy.

I got this bit set after I took out search for Italian Wikipedia. And all none Wikipedias.

You want even more Management?? 😨