Literally who ever said "backups are overrated"?
post
So I have lived in South Korea for 6 years now. The fact that this fire has had such a major impact is quite typical of Korean bureaucracy and tech administration. Very few backups, infrastructure held together with scotch tape and bubblegum, overworked devs and maintainers. It's a bit sad, especially for a country that exports so many tech products.
You met me at a very strange time in my life.
If I had a nickel for every time someone didn't backup their datacenter, I'd have two nickels.
Which isn't a lot, but it's weird that it happened at least twice.
Last time we lost disks at work, there were full backups.
They were just in the same disks as the data. And because everything is abstracted two times into virtual disks on virtual machines, and containers and volumes, the people responsible for the backups didn't even know it.
But won't you like...check? That the backups are own their own drive? The whole 3-2-1 rule kinda make you want to check this, no?
Or was it like they knew where the drives of the backups were, but they didn't know those drives were being virtualized away and were in like production use?
I dunno what possibilities they actually had. But knowing the place, I can fully believe both that they weren't allowed to check and that they never bothered.
The most likely scenario in my head was that they sent a request to the provisioning team asking for the volume to be in a different disk, and that detail never made into the technician actually doing the work (that sits on the next chair, but the requests have to come from the system).
(And the long term backups were fine. We lost 3 days of data.)
I collect stories like this for when I need to make a case for purchasing new gear or services.
You can't lose computerised services if you don't have computerised services. Checkmate SysAdmins.
In Lithuania, healthcare e-services went down after the basement where the servers were kept got flooded in a rainstorm. They went down for a couple of weeks.
top 24 comments