However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost.
Why were there no backups? "Because there was a lot of data" isn't really an answer.
However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost.
Why were there no backups? "Because there was a lot of data" isn't really an answer.
The G-Drive has been in use since 2018, requiring government officials to store all work documents in the cloud instead of on personal computers.
It is currently working to recover alternative data using any files saved locally on personal computers within the past month, along with emails, official documents and printed records.
Imagine being the employee who voilated work policy and saved stuff locally.
Imagine being the employee who voilated work policy and saved stuff locally.
there's a fascinating history along these lines, both good and sad:
https://screenrant.com/toy-story-2-movie-deleted-accident-recovered/
Sucks for most users, but i bet there's at least one person who is like "holy fuck I'm saved!" Because now he can blame the server destruction for why he doesn't have that big project done tomorrow
That's gonna be an expensive lesson about the 3-2-1 strategy...
Damn isn't it a shame we lost the election results? Damn shame anyways the ballots from my district are fine.
I find this hard to believe honestly. how can so many people involved in the administration of this system not have some offsite backups.
dunno about this segment of the korean gov, but it just... seems implausible.
As a sysadmin that's pretty much my worst nightmare. I really feel for those trying to pick the pieces back up again after that.
I feel your pain, but meanwhile I'm laughing. No backups, no fire suppression system, what the fuck are these clowns doing.
Suppressing evidence? Looks like this was a deliberate oversight to later erase data.
Probably data retention safety was overruled earlier by one person.
So, I think the designed erase worked as designed .
Pretty big conspiracy theory for a situation where incompetence doesn't seem out of the ordinary.
SysAdmins saying "We need X, we need Y", someone who signs the cheques saying "But it works fine! Why would there be a fire? We already spent so much on those UPSs"
I don’t know a lot at this time. The cool thing is this will be discussed for years , and any who follow this will know a lot more.
I’m just saying if it looks like a duck, quacks like a duck it may not be a squirrel
What you described isn't "looks like a duck, quacks like a duck".
This is a fire in a government (publicly-funded) location in an industry that's often overlooked and underfunded... in a traditionally conservative country (with a Liberal leader for the last few months) that is part of a very top-down hierarchical structure. Which would suggest that the complaints of a systems administrator are likely to be ignored. And that's assuming that the systems administrator is competent, which isn't a guarantee in and of itself.
I'm seeing what looks like a duck, I'm hearing the quacks. The conspiracy theory is that it's a squirrel in a duck costume that learned how to quack so people would toss it bread.
Here we have a glass of water, it’s half full.
Did someone forget to fill it up, or did someone drink it.
Is it institutional incompetence or criminal conspiracy?
I don't think changing the analogy makes your argument stronger
Hanlon's razor mate. I know it's entirely possible that it's a conspiracy, but it'd entirely possible (and more likely) that it's just incompetence and lack of communication due to inefficient and lackluster processes as well as too many levels of leadership.
You know for a fact that the people doing the largest share of the recovery effort have nothing to do with the decision to have no backups.
...but with the way social/work hierarchies work in SK, it was probably never brought up.
Wow. this is one of the biggest instances of IT incompetence that I've heard of in recent years. Hosting a server farm without remote backups? Sound like the London Magnetic Tape Incident.
The LMTI: One employee was sent to the other end of London with the magnetic backup tapes every day. He got money to take a taxi, but saved it and took the tube. His favourite seat was right above one of the motors, where he sat the bag with the tapes on the floor for the journey. The tapes were just stored at the destination, and not checked in any way. Guess what they learned the first time they had to rely on those tapes?
Hmm a system that stored government documents caught fire and they have no backups? Hmm, this carries the same energy of registry offices catching fire "spontaneously" in the past.
Always have two backups in different places than the original. If not, the least you can do is have one backup copy. How does the government don't have such thing?
Government people get jobs by schmoozing and making deals, not by merit or skill.
Pournelle's Law always seems relevant.
I have one backup but keep legacy things so that in a massive disaster I still have archives that have some of my important long term type documents. So figure one up to date backup and in a disaster I have stuff from last year.
Perhaps contained evidence against the former PM who tried to do a military coup and this is the way of getting rid of it? Perhaps protecting co-conspirators if not the top guy.
At least they saved the Internet box. Otherwise, we'll have to bother the elders of the Internet again.
This is a clear example where 'the cloud' really is a physical place.
What else would it be? What do people think the cloud is?
For a long while, I had hoped it was at least 6 physical places, with various redundancies. A few billion small-ish servers at internet network hubs.
That or the magical floating bits that go over hackers heads in the movies. Those also look like the cloud. Not very secure, but quite convenient.
Disaster Recovery Concepts:
Get disaster insurance, wait for failure, collect a big c-suite payout and move to the next company.
Everyone else gets to find new exciting jobs.
My favorite is places that don't want to allocate time or resources to testing disaster recovery.
Even though they historically used to have a great backup system for their historians and government records.
see link: https://en.wikipedia.org/wiki/Veritable_Records_of_the_Joseon_Dynasty
Weird how there is a rising MAGA party in KR and an aligned former pm who attempted a coup. What a weird coincidence.
saw a sticker recently that said "there is no cloud, just other people's computers"
It's too bad it wasn't a rain cloud or it would have destroyed the fire and not the other way around
Great talk that touches on this from CCC
https://www.youtube.com/watch?v=0dMzlw-qSZI
TLDR it was to destroy evidence the day before a government investigation.
You do know that there's two korean countries, right?
What are you talking about? There's one Korea and half of it is occupied by an imperialist army.
Am I on .world? Why are people downvoting objective reality?
Oh, so they share their passport, their borders, their economy and so forth? Do you also believe that all countries that speak English, is the UK? or US?
Why do I have to put up with this kind of feigned ignorance after being made to look at maps that include Crimea in Ukraine for like a decade now?
You can accept or reject the claim that South Korea is an imperialist outpost of the US. I don't care about that right now.
But you do know that nothing in OPs comment implies that the two korea's share passports or anything like that, right? Like, come on, these are basic reading comprehension skills.
They really had only 1 location?? Really???
due to the system’s large-capacity, low-performance storage structure, no external backups were maintained
Oh man, thats terrifying. Critical systems running without a backup
The G-Drive has been in use since 2018, requiring government officials to store all work documents in the cloud instead of on personal computers
However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained
I'm equal parts exasperated and unsurprised because "automatic backups" were 100% part of the sales pitch to move everything to the cloud
This is actually kinda crazy. It just goes to show how a lot of infrastructure we take for granted is a lot more fragile than one would believe. Kinda concerning. At least this isn't technically critical infrastructure, I guess?
it should be considered 'critical' infrastructure tho not too much. all sorts of gov services were down because of this lol, including texting/video calls (voice calls ok) to 119 (911 equivalent).
top 18 comments