top 24 comments

sorted by: hot top controversial new old
[–] 49 points 1 year ago (12 children)

Literally who ever said "backups are overrated"?

  • source
  • hideshow 12 child comments
  • [–] 21 points 1 year ago (3 children)

    My ex company had for more than 10 years keept all the data customers shared with us. Structured and standardized, should have been easy peasy.

    Somehow they were "appending wrong" in some way and data was useless. In think they were trying to reduce the size by aggregating a bit, but they did in a way that rendered the data useless.

    Of course the CEO wanted to train models with it anyway...

  • source
  • parent
  • hideshow 3 child comments
  • [–] 6 points 1 year ago (1 child)

    10 years and no one bothered to pull some information at random? I mean generally companies have a schedule of assessments to ensure records. Even if it's as simple as checksum.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 13 points 1 year ago (1 child)

    Backups are always considered to be too expensive up until the point that not having backups becomes more expensive. This applies to redundancy of all kinds except the one that means firing employees for not setting up the other kinds.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 1 year ago

    Even at home. I do one usb backup and one internal backup of photos, home videos and documents. I would love to make backups of other stuff, but I can replace a lot of the other crap if need be, because hard drives kinda stalled in price drops.

  • source
  • parent
  • [–] 28 points 1 year ago

    So I have lived in South Korea for 6 years now. The fact that this fire has had such a major impact is quite typical of Korean bureaucracy and tech administration. Very few backups, infrastructure held together with scotch tape and bubblegum, overworked devs and maintainers. It's a bit sad, especially for a country that exports so many tech products.

  • source
  • [–] 16 points 1 year ago (1 child)

    You met me at a very strange time in my life.

  • source
  • hideshow 1 child comment
  • [–] 12 points 1 year ago

    dies after fall

    Well, if that ain't a whitewashed headline.

  • source
  • [–] [S] 23 points 1 year ago (4 children)

    If I had a nickel for every time someone didn't backup their datacenter, I'd have two nickels.

    Which isn't a lot, but it's weird that it happened at least twice.

  • source
  • hideshow 4 child comments
  • [–] 23 points 1 year ago (2 children)

    Last time we lost disks at work, there were full backups.

    They were just in the same disks as the data. And because everything is abstracted two times into virtual disks on virtual machines, and containers and volumes, the people responsible for the backups didn't even know it.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 16 points 1 year ago* (1 child)

    But won't you like...check? That the backups are own their own drive? The whole 3-2-1 rule kinda make you want to check this, no?

    Or was it like they knew where the drives of the backups were, but they didn't know those drives were being virtualized away and were in like production use?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 15 points 1 year ago*

    I dunno what possibilities they actually had. But knowing the place, I can fully believe both that they weren't allowed to check and that they never bothered.

    The most likely scenario in my head was that they sent a request to the provisioning team asking for the volume to be in a different disk, and that detail never made into the technician actually doing the work (that sits on the next chair, but the requests have to come from the system).

    (And the long term backups were fine. We lost 3 days of data.)

  • source
  • parent
  • load more comments (1 reply)
    [–] 11 points 1 year ago

    I collect stories like this for when I need to make a case for purchasing new gear or services.

  • source
  • [–] 10 points 1 year ago

    You can't lose computerised services if you don't have computerised services. Checkmate SysAdmins.

  • source
  • [–] 3 points 1 year ago

    In Lithuania, healthcare e-services went down after the basement where the servers were kept got flooded in a rainstorm. They went down for a couple of weeks.

  • source