But the cloud is safe secure and forever how could this happen
post
50 TB isn't even all that much for your own NAS.
Well, it is now.
This story is in regards to a single PBS station in Missouri and not the national organization, in case anyone missed that.
IMO if a cloud storage company is gonna shut down, they should legally be required to send you the storage device(s) your data is on.
3-2-1 people
So the company pulled some shady shit and now the new company that bought out the old one doesn't want to do anything. PBS needs to get a court order in to keep Iron Mountain from deleting that data while the court case is ongoing. They didn't follow the data backup guidelines but they did take the appropriate steps to retrieve their data.
And that folks, is why you follow the 3 2 1 rule
This is the 3–2–1 rule that PBS taught me as a kid:

https://m.youtube.com/watch?v=s2-LEBc2sO8&t=3s&pp=2AEDkAIB&ra=m
3- THREE CDRWs should be used to record the daily backups
2- TWO of them should be kept at home. Drive the discs home nightly, preferably in a station wagon because they have higher bandwidth
1- ONE CDRW is burning data in real time. Make sure mirroring is set up so the CDRW is mirroring your live hard drive
What do I do if all I have are PS1 memory cards?
Make a PS1 Beowulf cluster and turn it into a NAS.
If you run out of PS1s, consider furbies.
Instructions unclear. Made self-aware furby army.
The data is all still there. They were using a subcontractor (who went under) that stored the data at Iron Mountain. Since the subcontractor was the customer and not PBS, PBS needs a judge to declare that they own the data so they can retrieve it from Iron Mountain.
Still, 3,2,1, folks.
50TB that they didn't have their own backup of? I could realistically get drives to store that if I really wanted to.
I administer a Plex server with more storage space than that. It’s totally doable.
Well, before 2026 it was. Now prices are bonkers and PBS is being defunded.
Your friendly local data hoarder can easily fit that much data in a small server of theirs.
This could have all been avoided with a single, well-publicized torrent.
50 Tb is smaller than I expected
It's possible that 50 TB is just the portion that they could not recover from other backups. But even then, prior to the 80s and 90s, the archives would have been physical film, and any digitised versions of the footage could have possibly been just simple MP4 (or other format) video files, which at 1080p 24 FPS would be barely 1 MB/s.
I really like PBS and what they've done for the world throughout my life but whoever is in charge of their archives should be drawn, quartered, razed to ash and then asked politely to leave for having absolutely no in house storage.
I think PBS could probably handle the cost of one slightly above average old desktop PC (15 years old is FIINE) and 6 20TB hard drives FFS.
I keep saying, if this stuff is publically owned works made by the corporation for public broadcasting, put up some official torrents and let the public archive it for free. Mr. Rogers would surely approve of sharing for the common good.
Just because they are called "public" it doesn't mean they're owned by the public. It's a non profit created to produce media for public consumption, but they still have bills to pay, employees, etc.
If they have content for streaming or DVD sales or (probably more likely) licensing to other PBS stations, they need to get income from it to maximize their ability to continue to produce the free broadcasts they were created for.
None of it is solely funded by CPB. This is Nine PBS not PBS. 50TB is very low for broadcast footage so it’s likely a small handful of local shows, and a partial archive at that. For context, a 2000s consumer camcorder tape was 17GB per hour.
Woah that is some wild context. So wait does that mean it'd take 17GB to digitally store an hour's worth of that footage..."faithfully"?
Not trying to paint you into some "gotcha!" corner or anything weird, ya just made me curious.
I also just find weird and tortured units of measure fun, so "magnetic tape, ya know, the 17GB/hr kind" really tickles me.
If their funding was 100% paid for by taxes, sure. But only a small fraction is actually paid for with public funds. They have to earn the rest to stay afloat.
There's that, and also part that's donations by viewers and sponsors. While they need funding to maintain operational costs I don't think they have the typical pressure of expecting a return on investment to deal with, so volunteer supported archiving, hosting via things like peertube or youtube without needing to show ads, there's room for some novel operations that other content producers don't have.
Mr. Rogers would surely approve of sharing for the common good.
Correct :)
And here's why public archival is so important.
I would not be surprised if whatever person in charge of the archive, if there is/was one, has advocated for a 3-2-1 solution. But the hoops were too much to get through to have a fucking back up system beyond a relying on random storage company. That or it was someone who's been in the job sense they converted the tapes to digital and has just been coasting along uploading stuff for the past 20 years.
Damn, 50TB is not that much to depend on an external vendor, especially for archival purposes. Even I have ~40TB clanker disks shoved on my NAS at home.
On prem forever
On prem is just a fire away from the same fate.
If it's important enough to pay a monthly fee for archival cloud hosting, it's important enough to maintain a local copy backup.
If it's archival it doesn't even cost electricity, you only need the machine on when adding to the archive. Then just shit it back down and leave it alone.
You can say it to the C-Suite all you'd like, but Goodluck getting them to budget for anything sometimes. Spinning rust, lto tapes, etc and the software to handle them is generally considered out of budget, at least at most orgs I've worked with (despite the recommendation to practice 3-2-1). Also, you may want to further detail your advice, most storage mediums (save lto tapes) require ongoing power to prevent things like bitrot. It's quite common for hard drives for example to develop bad sectors over time, but algos running on the drives embedded firmware clean this up by repositioning bad sectors to a reserve space. Nvme/flash is even more vulnerable to this and behaves similarly.
important enough to maintain a local copy backup. tis why I tagged
#3-2-1%20ignoreyou only need the machine on when adding to the archive. Then just shit it back down and leave it alone. Nope.
Regular backup tests is part of backup hygiene.
Otherwise you have duds you need to replace.
To make your comment more readable, you need to edit your comment to add another line break between what you quoted and your response to it (i.e. a blank line before "Nope.").
I don't understand what you're trying to say, and I feel like that's because you don't know how comments work...
They screwed up the formatting, but it's clear what they're saying. They quoted you saying you don't even need electricity for archives, and they're telling you that if you don't test your backups, you don't have backups.
Do you comprehend actual backup/cyber hygiene?
"3-2-1" is important when considering your backups. AT LEAST.
3 copies of your data.
2 separate locations.
1 (or more) off-site.
If you want to go a step further(a good idea), add in one more that's immutable(to prevent access to malware).
They should have released them as torrents, us data hoarders would happily seed some of it! It's probably mostly on torrent trackers anyway.
Couldn't the cloud company just sell them the drive(s) their data was on? Since they're going out of business and all. I mean what else are they gonna do with that data?
Afaik liquidation requires certain garantees of "highest price" like inventory auctions.
Regardless, a deal or notification could and should have been made.
Just goes to show the cloud is someone else's computer. A computer the someone else can do anything with, like declare bankrupcy and sell off.
So someone else bought the drives at a higher price.
So they could have been advised that the drives were being sold at auction and given the opportunity to download the data.
I would say they should not be selling someone else's data at auction, but then again, that's how Google, Microsoft, Meta, and others operate.
Exactly. The right and responsible thing to do is give users access to their data: give notice, offer a competitor's similar service or offer to sell the drives to them.
But reaponsibility is a high ask. Just start the bancrupcy procedure, sell everything off and be done with it is what most folks do.
The possibilities are there, but most won't follow through.
50TB isn't even that much for a large company. This smells like pure incompetence.
Throwing shade at PBS for not adhering to expensive but best practice 3-2-1 rule is Dumb. PBS doesn't have the money to do things right.
I hope they're able to get their data back.
In most cases customers are relying on their cloud vendor to do all of this and follow best practices.
Without PBS’s contract we don’t know what was promised or stipulated
Nine PBS found an extra $30 million to refurb their headquarters this year. They could have afforded a NAS.
Just over here filling another square in my "We Are Completely Fucked" bingo card.

all 27 comments