Nine PBS is afraid that it will lose over 70 years of archival materials and programming after its contracted cloud storage vendor has apparently gone out of business. According to Current, the channel used Open Source Storage, or OSS, for storing over 50TB of data — but it suddenly lost access to its data earlier this year. Nine PBS has been working with OSS and its predecessor since 2019 and intended to renew its contract on March 6 of this year. The company never responded and abruptly cut off the station’s data access even though it still had 30 days to retrieve its data after the contract ended.

The station dug a little bit deeper when it discovered that the OSS website no longer existed and was listed as delinquent under the Colorado Secretary of State. It also discovered that OSS utilized Iron Mountain’s services, so it sent a demand letter to the latter to preserve its data and to "pay any reasonable costs associated with its demand." At the time, Iron Mountain did not confirm or deny that it possessed Nine PBS’ data.

all 27 comments

sorted by: hot top controversial new old
[–] 85 points 2 months ago (14 children)

But the cloud is safe secure and forever how could this happen

  • source
  • hideshow 14 child comments
  • [–] 42 points 2 months ago (12 children)

    It's safe if you're following proper backup procedures.

    3 backups in 2 different forms of media with 1 being off-site

    A single point of failure is always a bad thing.

  • source
  • parent
  • hideshow 12 child comments
  • [–] 14 points 2 months ago (5 children)

    Yeah, the cloud makes proper 3-2-1 backups easier, *but you still need to take care of the rest of the 3-2-1. The cloud takes care of a third of the 3, half of the 2, and satisfies the 1. But it’s not a complete 3-2-1 backup by itself, and anyone selling it as such is blowing smoke up your ass to get a sale.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 9 points 2 months ago (4 children)

    If done right, a third party could do 3-2-1 for you.

    But unlike in this case, to do it right your clients need to know where the data is stored, and any 3rd party you contract with should have a clause that establishes your client as the data owner (as well as requirements for data return if the business closes).

    This would have been easily avoidable, most likely if IT was allowed to be part of the contracting discussion.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 2 months ago (3 children)

    A single place can't do 3-2-1. That's not "3" (just a single company = single point of failure, as demonstrated here)

  • source
  • parent
  • hideshow 3 child comments
  • [–] 0 points 2 months ago (2 children)
  • [–] 1 point 2 months ago (1 child)

    I did. Did you not read mine? If a single 3rd party entity is setting up all your backups, then that single entity still controls your destiny. That's still a bad idea.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 2 months ago (2 children)

    no it's not... you're thinking of stuff being restored but that doesn't account for neglect and straight up theft of your information from these unsecure criminal companies like microslop

  • source
  • parent
  • hideshow 2 child comments
  • [–] 35 points 2 months ago (2 children)

    50 TB isn't even all that much for your own NAS.

  • source
  • hideshow 2 child comments
  • [–] 32 points 2 months ago*

    This story is in regards to a single PBS station in Missouri and not the national organization, in case anyone missed that.

  • source
  • [–] 40 points 2 months ago*

    IMO if a cloud storage company is gonna shut down, they should legally be required to send you the storage device(s) your data is on.

  • source
  • [–] 15 points 2 months ago (2 children)
  • [–] 11 points 2 months ago

    Me: I wonder how much Trump is related to... Oh, hi Blackrock...

    https://finance.yahoo.com/quote/IRM/holders/?guccounter=1

  • source
  • [–] 11 points 2 months ago

    So the company pulled some shady shit and now the new company that bought out the old one doesn't want to do anything. PBS needs to get a court order in to keep Iron Mountain from deleting that data while the court case is ongoing. They didn't follow the data backup guidelines but they did take the appropriate steps to retrieve their data.

  • source
  • [–] 8 points 2 months ago

    Cloud storage is just someone elses computer.

  • source
  • [–] 5 points 2 months ago

    Do they really not have more than one backup?

  • source
  • 25 comments in c/pcmasterrace@lemmy.world view that post
    [–] 71 points 2 months ago (8 children)

    And that folks, is why you follow the 3 2 1 rule

  • source
  • hideshow 8 child comments
  • [–] 16 points 2 months ago* (5 children)

    3- THREE CDRWs should be used to record the daily backups

    2- TWO of them should be kept at home. Drive the discs home nightly, preferably in a station wagon because they have higher bandwidth

    1- ONE CDRW is burning data in real time. Make sure mirroring is set up so the CDRW is mirroring your live hard drive

  • source
  • parent
  • hideshow 5 child comments
  • [–] 8 points 2 months ago (4 children)

    What do I do if all I have are PS1 memory cards?

  • source
  • parent
  • hideshow 4 child comments
  • [–] 11 points 2 months ago* (3 children)

    Make a PS1 Beowulf cluster and turn it into a NAS.

    If you run out of PS1s, consider furbies.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 55 points 2 months ago (1 child)

    The data is all still there. They were using a subcontractor (who went under) that stored the data at Iron Mountain. Since the subcontractor was the customer and not PBS, PBS needs a judge to declare that they own the data so they can retrieve it from Iron Mountain.

    Still, 3,2,1, folks.

  • source
  • hideshow 1 child comment
  • [–] 48 points 2 months ago (4 children)

    50TB that they didn't have their own backup of? I could realistically get drives to store that if I really wanted to.

  • source
  • hideshow 4 child comments
  • [–] 26 points 2 months ago (3 children)

    I administer a Plex server with more storage space than that. It’s totally doable.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 19 points 2 months ago (2 children)

    Well, before 2026 it was. Now prices are bonkers and PBS is being defunded.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 25 points 2 months ago

    Imagine paying a contractor instead of funding the Internet Archive!

  • source
  • [–] 13 points 2 months ago

    Your friendly local data hoarder can easily fit that much data in a small server of theirs.

  • source
  • [–] 13 points 2 months ago

    This could have all been avoided with a single, well-publicized torrent.

    💯 1
  • source
  • [–] 8 points 2 months ago (1 child)

    50 Tb is smaller than I expected

  • source
  • hideshow 1 child comment
  • [–] 2 points 2 months ago

    It's possible that 50 TB is just the portion that they could not recover from other backups. But even then, prior to the 80s and 90s, the archives would have been physical film, and any digitised versions of the footage could have possibly been just simple MP4 (or other format) video files, which at 1080p 24 FPS would be barely 1 MB/s.

  • source
  • parent
  • [–] 7 points 2 months ago (2 children)

    I really like PBS and what they've done for the world throughout my life but whoever is in charge of their archives should be drawn, quartered, razed to ash and then asked politely to leave for having absolutely no in house storage.

    I think PBS could probably handle the cost of one slightly above average old desktop PC (15 years old is FIINE) and 6 20TB hard drives FFS.

  • source
  • hideshow 2 child comments
  • [–] 2 points 2 months ago

    Archive-quality hard disk drives are like $50/TB

  • source
  • 15 comments in c/technology@lemmy.world view that post
    [–] 59 points 2 months ago (6 children)

    I keep saying, if this stuff is publically owned works made by the corporation for public broadcasting, put up some official torrents and let the public archive it for free. Mr. Rogers would surely approve of sharing for the common good.

  • source
  • hideshow 6 child comments
  • [–] 23 points 2 months ago

    Just because they are called "public" it doesn't mean they're owned by the public. It's a non profit created to produce media for public consumption, but they still have bills to pay, employees, etc.

    If they have content for streaming or DVD sales or (probably more likely) licensing to other PBS stations, they need to get income from it to maximize their ability to continue to produce the free broadcasts they were created for.

  • source
  • parent
  • [–] 10 points 2 months ago* (last edited 2 months ago) (1 child)

    None of it is solely funded by CPB. This is Nine PBS not PBS. 50TB is very low for broadcast footage so it’s likely a small handful of local shows, and a partial archive at that. For context, a 2000s consumer camcorder tape was 17GB per hour.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 2 months ago

    Woah that is some wild context. So wait does that mean it'd take 17GB to digitally store an hour's worth of that footage..."faithfully"?

    Not trying to paint you into some "gotcha!" corner or anything weird, ya just made me curious.

    I also just find weird and tortured units of measure fun, so "magnetic tape, ya know, the 17GB/hr kind" really tickles me.

  • source
  • parent
  • [–] 5 points 2 months ago (1 child)

    If their funding was 100% paid for by taxes, sure. But only a small fraction is actually paid for with public funds. They have to earn the rest to stay afloat.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 2 months ago

    There's that, and also part that's donations by viewers and sponsors. While they need funding to maintain operational costs I don't think they have the typical pressure of expecting a return on investment to deal with, so volunteer supported archiving, hosting via things like peertube or youtube without needing to show ads, there's room for some novel operations that other content producers don't have.

  • source
  • parent
  • [–] 26 points 2 months ago* (last edited 2 months ago) (2 children)

    And here's why public archival is so important.

    I would not be surprised if whatever person in charge of the archive, if there is/was one, has advocated for a 3-2-1 solution. But the hoops were too much to get through to have a fucking back up system beyond a relying on random storage company. That or it was someone who's been in the job sense they converted the tapes to digital and has just been coasting along uploading stuff for the past 20 years.

  • source
  • hideshow 2 child comments
  • [–] 19 points 2 months ago

    Damn, 50TB is not that much to depend on an external vendor, especially for archival purposes. Even I have ~40TB clanker disks shoved on my NAS at home.

  • source
  • [–] 8 points 2 months ago (3 children)
  • [–] 2 points 2 months ago (2 children)

    On prem is just a fire away from the same fate.

  • source
  • parent
  • hideshow 2 child comments
  • 10 comments in c/glam@lemmy.cafe view that post
    [–] 10 points 2 months ago (7 children)

    If it's important enough to pay a monthly fee for archival cloud hosting, it's important enough to maintain a local copy backup.

    If it's archival it doesn't even cost electricity, you only need the machine on when adding to the archive. Then just shit it back down and leave it alone.

  • source
  • hideshow 7 child comments
  • [–] 2 points 2 months ago

    You can say it to the C-Suite all you'd like, but Goodluck getting them to budget for anything sometimes. Spinning rust, lto tapes, etc and the software to handle them is generally considered out of budget, at least at most orgs I've worked with (despite the recommendation to practice 3-2-1). Also, you may want to further detail your advice, most storage mediums (save lto tapes) require ongoing power to prevent things like bitrot. It's quite common for hard drives for example to develop bad sectors over time, but algos running on the drives embedded firmware clean this up by repositioning bad sectors to a reserve space. Nvme/flash is even more vulnerable to this and behaves similarly.

  • source
  • parent
  • [–] [S] 3 points 2 months ago (5 children)

    important enough to maintain a local copy backup. tis why I tagged #3-2-1%20ignore you only need the machine on when adding to the archive. Then just shit it back down and leave it alone. Nope.
    Regular backup tests is part of backup hygiene.
    Otherwise you have duds you need to replace.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 3 points 2 months ago (1 child)

    To make your comment more readable, you need to edit your comment to add another line break between what you quoted and your response to it (i.e. a blank line before "Nope.").

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 2 months ago (2 children)

    I don't understand what you're trying to say, and I feel like that's because you don't know how comments work...

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 2 months ago

    They screwed up the formatting, but it's clear what they're saying. They quoted you saying you don't even need electricity for archives, and they're telling you that if you don't test your backups, you don't have backups.

  • source
  • parent
  • [–] 6 points 2 months ago (1 child)

    "3-2-1" is important when considering your backups. AT LEAST.

    3 copies of your data.

    2 separate locations.

    1 (or more) off-site.

    If you want to go a step further(a good idea), add in one more that's immutable(to prevent access to malware).

    🎯 1
  • source
  • hideshow 1 child comment
  • 6 comments in c/usa@lemmy.ml view that post
    [–] 6 points 2 months ago*

    They should have released them as torrents, us data hoarders would happily seed some of it! It's probably mostly on torrent trackers anyway.

  • source
  • [–] 6 points 2 months ago (3 children)

    Couldn't the cloud company just sell them the drive(s) their data was on? Since they're going out of business and all. I mean what else are they gonna do with that data?

  • source
  • hideshow 3 child comments
  • [–] 3 points 2 months ago (2 children)

    Afaik liquidation requires certain garantees of "highest price" like inventory auctions.

    Regardless, a deal or notification could and should have been made.

    Just goes to show the cloud is someone else's computer. A computer the someone else can do anything with, like declare bankrupcy and sell off.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 2 months ago (1 child)

    So someone else bought the drives at a higher price.

    So they could have been advised that the drives were being sold at auction and given the opportunity to download the data.

    I would say they should not be selling someone else's data at auction, but then again, that's how Google, Microsoft, Meta, and others operate.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 2 months ago

    Exactly. The right and responsible thing to do is give users access to their data: give notice, offer a competitor's similar service or offer to sell the drives to them.

    But reaponsibility is a high ask. Just start the bancrupcy procedure, sell everything off and be done with it is what most folks do.

    The possibilities are there, but most won't follow through.

  • source
  • parent
  • [–] 3 points 2 months ago

    50TB isn't even that much for a large company. This smells like pure incompetence.

  • source
  • 6 comments in c/hackernews@lemmy.bestiver.se view that post
    [–] 16 points 2 months ago (4 children)

    Throwing shade at PBS for not adhering to expensive but best practice 3-2-1 rule is Dumb. PBS doesn't have the money to do things right.

    I hope they're able to get their data back.

  • source
  • hideshow 4 child comments
  • [–] 4 points 2 months ago* (2 children)

    Nine PBS found an extra $30 million to refurb their headquarters this year. They could have afforded a NAS.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 2 months ago

    Just over here filling another square in my "We Are Completely Fucked" bingo card.

  • source
  • 3 comments in c/television@piefed.social view that post