For wikipedia you'll want to use Kiwix. A full backup of wikipedia is only like 100GB, and I think that includes pictures too.
post
The English Language Wikipedia probably wouldn't be hard, or Debian Stable.
All of Debian's packages might be a tad more expensive, though.
This might be a good place to start for Wikipedia;
https://meta.wikimedia.org/wiki/Data_dump_torrents#English_Wikipedia
It depends if you want the images or previous versions of wikipedia too. The current version is about 25Gb compressed, the dump with all versions is aparently multiple terabytes. They don't say how much media they have, but I'm guessing it's roughly "lots".
I stumbled across this sort of fascinating area of doomsday prepping a few weeks back.
A nice addition to that, don't just make it a USB, but a raspberry pi. So you'd have a reasonably low-powered computer you could easily take with you.
Not suggesting this one as it seems a bit expensive to me, but https://www.prepperdisk.com/products/prepper-disk-premium-over-512gb-of-survival-content?view=sl-8978CA41
Just built one of these myself. I went NVME M.2 instead of SD Card to avoid data corruption. I know SD Cards are fine if you don't write to them a lot but if you wanna update or add your own stuff, scares me. Plus NVME is just so much faster.
How would you access the info if electricity permanently goes out?
You find a generator, or solar panels, or wind mill, or water turbine, or a bicycle hooked up to a generator.
If electricity permanently goes out then we're in a scavenger situation and it is time to start taking apart things that are no longer necessary to build the things that are.
start taking apart things that are no longer necessary to build the things that are
Hey finally a good use for all of those cars, grab the alternators out for small generators (since bicycles are the ultimate apocalypse vehicle: simple, small, easy to maintain and don't require complex fuels)
You only need 20 watts of power. One of those dinky fold up solar panels would work. Add a USB power brick for cloudy days.
2W for a RPi Zero with data on a microSD
You're going to need a monitor as well.
I have a PaPiRus ePaper eInk e.g. https://media.digikey.com/pdf/Data%20Sheets/Pi%20Supply%20PDFs/PaPiRus_ePaper_Web.pdf and even though I don't know the watts for a refresh but I assume it's one of the lowest solution you can use.
PS: FWIW if you don't refresh the display can keep the information on for months, if not years.
Eink displays are pretty awesome for this sort of thing, I repuposed a kobo ereader as a household info display and it worked nicely. Those PaPiRus screens look easier to interface with, but a little small for reading wikipedia articles. They'd do in a pinch, but the eyestrain would have me looking for a bigger solution.
Pretty much what Sinthesis said; USB power brick and/or solar panels. Both at the ready and tested. Also got a big ass battery backup that will charge off solar panels.
at this point why not just use a phone running postmarketos?
You'd first have to buy a phone that can run postmarketos and these are much rarer than I wish they were. Is there even anything new that can run it? Pine64 stopped making phones and said they'll make a new one when they can make it RISC-V.
Fairphone maybe I guess. 4 is listed as a supported device, but someone has gotten it working on 6 too.
there's lots of devices it runs on iirc, something like the pixel 3a can be had for less than a new rpi3b+ where I live
Cause if ya wanna go overboard like I did, 1TB of NVME storage, can add with SD Card if necessary. 16GB RAM. Very little learning curve for my part as I use SBCs often. Plus almost every Docker container and program I want works on RPi without any hassle.
There's also more robust guides and community for RPi.
Just my thoughts.
Last I checked (3 years ago) postmarketOS drained the pinephone battery in record time :(
I have been archiving Linux builds for the last 20 years so I could effectively install Linux on almost any hardware since 1998-ish.
I have been archiving docker images to my locally hosted gitlab server for the past 3-5 years (not sure when I started tbh). I've got around 100gb of images ranging from core images like OS to full app images like Plex, ffmpeg, etc.
I also have been archiving foss projects into my gitlab and have been using pipelines to ensure they remain up-to-date.
the only thing I lack are packages from package managers like pip, bundler, npm, yum/dnf, apt. there's just so much to cache it's nigh impossible to get everything archived.
I have even set up my own local CDN for JS imports on HTML. I use rewrite rules in nginx to redirect them to my local sources.
my goal is to be as self-sustaining on local hosting as possible.
Everyone should have this mindset regarding their data. I always say to my friends and family, "If you like it, download it.". The internet is always changing and that piece of media that you like can be moved, deleted, or blocked at any time.
Neither are that bad honestly. I have jigdo scripts I run with every point release of Debian and have a copy of English Wikipedia on a Kiwix mirror I also host. Wikipedia is a tad over 100 GB. The source, arm64 and amd64 complete repos (DVD images) for Debian Trixie, including the network installer and a couple live boot images, are 353 GB.
Kiwix has copies of a LOT of stuff, including Wikipedia on their website. You can view their zim files with a desktop application or host your own web version. Their website is: https://kiwix.org/
If you want (or if Wikipedia is censored for you) you can also look at my mirror to see what a web hosted version looks like: https://kiwix.marcusadams.me/
Note: I use Anubis to help block scrapers. You should have no issues as a human other than you may see a little anime girl for a second on first load, but every once and a while Brave has a disagreement with her and a page won't load correctly. I've only seen it in Brave, and only rarely, but I've seen it once or twice so thought I'd mention it.
I also recommend downloading βFlashpoint archiveβ to have flash games and animations to stay entertained.
There is a 4gb version and a 2.3TB version.
There is a 4gb version and a 2.3TB version.
That's quite the range
Is that Flash exclusive or do they accept other games from that era?
@Maroon I thought torrent technology to be a godsend for package managers.Why none of them use it?I mean, damn.@AnimalsDream
What happens when they just cut the underwater cables? Torrent over carrier pigeon for a linux distro would take ages
Sneakernet to the rescue. Some of you are too young to know about walking around with boxes full of disks.
A wise man once said
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
It was trading CD-R's during my high school days.... good times. Napster was just starting to take off by the time we had a CD-R trading network set up, Napster just increased the amount of CD's that got passed around.
Pigeon latency is horrible, but the bandwidth is pretty great. You could probably load up an adult pigeon with at least 12TB of media.
https://en.wikipedia.org/wiki/IP_over_Avian_Carriers
Just gonna leave this here for whoever wants to read more on the methodology and potential risks.
Over a 30-mile (48 km) distance, a single pigeon may be able to carry tens of gigabytes of data in around an hour, which on an average bandwidth basis compared very favorably to early ADSL standards, even when accounting for lost drives.
Compared to what I use at home now, this sounds great
A good way to see what the future of places like the U.S are is to look at places like North Korea, where they do exactly this, move files around on flash media to avoid the state censors.
magnetlink to any linux package repo torrent if you don't mind?
i don't wanna scrape since it takes forever and burdens
FWIW :
fabien@debian2080ti:/media/fabien/slowdisk$ ls -lhS offline_prep/
total 341G
-rw-r--r-- 1 fabien fabien 103G Jul 6 2024 wikipedia_en_all_maxi_2024-01.zim
-rw-r--r-- 1 fabien fabien 81G Apr 22 2023 gutenberg_mul_all_2023-04.zim
-rw-r--r-- 1 fabien fabien 75G Jul 7 2024 stackoverflow.com_en_all_2023-11.zim
-rw-r--r-- 1 fabien fabien 74G Mar 10 2024 planet-240304.osm.pbf
-rw-r--r-- 1 fabien fabien 3.8G Oct 18 06:55 debian-13.1.0-amd64-DVD-1.iso
-rw-r--r-- 1 fabien fabien 2.6G May 7 2023 ifixit_en_all_2023-04.zim
-rw-r--r-- 1 fabien fabien 1.6G May 7 2023 developer.mozilla.org_en_all_2023-02.zim
-rw-r--r-- 1 fabien fabien 931M May 7 2023 diy.stackexchange.com_en_all_2023-03.zim
-rw-r--r-- 1 fabien fabien 808M Jun 5 2023 wikivoyage_en_all_maxi_2023-05.zim
-rw-r--r-- 1 fabien fabien 296M Apr 30 2023 raspberrypi.stackexchange.com_en_all_2022-11.zim
-rw-r--r-- 1 fabien fabien 131M May 7 2023 rapsberry_pi_docs_2023-01.zim
-rw-r--r-- 1 fabien fabien 100M May 7 2023 100r-off-the-grid_en_2022-06.zim
-rw-r--r-- 1 fabien fabien 61M May 7 2023 quantumcomputing.stackexchange.com_en_all_2022-11.zim
-rw-r--r-- 1 fabien fabien 45M May 7 2023 computergraphics.stackexchange.com_en_all_2022-11.zim
-rw-r--r-- 1 fabien fabien 37M May 7 2023 wordnet_en_all_2023-04.zim
-rw-r--r-- 1 fabien fabien 23M Jul 17 2023 kiwix-tools_linux-armv6-3.5.0-1.tar.gz
-rw-r--r-- 1 fabien fabien 16M Oct 6 21:32 be-stib-gtfs.zip
-rw-r--r-- 1 fabien fabien 3.8M Oct 6 21:32 be-sncb-gtfs.zip
-rw-r--r-- 1 fabien fabien 2.3M May 7 2023 termux_en_all_maxi_2022-12.zim
-rw-r--r-- 1 fabien fabien 1.9M May 7 2023 kiwix-firefox_3.8.0.xpi
but if you want the easier version just get Kiwix on whatever device in front of you right now (yes, even mobile phone assuming you have the space) then get whatever content you need.
If need a bit of help I recorded TechSovereignty at home, episode 11 - Offline Wikipedia, Kiwix and checksums with a friend just 3 weeks ago.
I also wrote randomly update https://fabien.benetou.fr/Content/Vademecum and coded https://git.benetou.fr/utopiah/offline-octopus but tbh KDE-Connect is much better now.
The point though is having such a repository takes minutes. If you don't have the space, buy a 512Go microSD for 50EUR then put that on, stuff it in a drawer then move on. If you want to every 3 months or whenever you feel like it, updated it.
TL;DR: takes longer to write such a meme than actually do it.
Watch out for flash data corruption. Lots of cheap flash (USB sticks, SD cards, SSDs) lose data after just a few years of offline storage. Something something quantum tunnel bullshit, iirc.
So either look for media that guarantee long cold storage retention (lots of businesses need to keep shit for 10 years for tax reasons), or occasionally plug it in and let do the housekeeping.
It's more that flash NAND uses a small electric charge to keep the NAND gates in the correct configuration. Over time, that charge dissipates. If you power the storage device every once in a while, you minimize these chances.
Here's a video explaining why it happens to Wii U's after being powered off for a while. https://youtu.be/JHME4zLs6Qs
User older flash tech can be useful here. You might not always need the highest density storage if you want to maintain files for a long time. Getting stuff built in a much larger process node makes for a much more stable form of storage.
Thanks but even though it's on a plugged HDD I don't even care for any of that data. What I mean is that none of that data is sensitive. It might be useful, potentially, but it's not unique. What I mean is that if somehow my .zim file for Wikipedia was corrupted I could download it again from https://library.kiwix.org/#lang=eng&category=wikipedia or elsewhere in ~30min (just checked).
What I'm trying to highlight here is more the process than the actual outcome.
TL;DR: yes, if one is actually serious about just getting and storing, they should verify periodically if the data is indeed fine. What I do want to highlight though is to first know how to do it at all. Anyway, you are right that for a proper solution on the long run one must understand how (cold) storage actually works. My heuristic is that it's like can food (which I don't use much), it might last a while, but not forever.
I thought the point of backing stuff up was to have things in case just downloading it again isn't a viable option?
It can be but not to me. To me the point is to test what's actually feasible and usable. It can be Wikipedia on my HDD but it could also be SO on a microSD or a RPi ... or it could be something totally different on another piece of hardware with another piece of storage. It will depend on the context.
So again, sure, having the data itself feels nice but in practice I never really needed it. If tomorrow my HDD would die I would shrug. If tomorrow Kiwix library wouldn't work anymore, I'd be disappointed but I could rely on .zim file elsewhere, e.g. on torrent trackers.
IMHO the point isn't files, the point is usable knowledge.
Edit : to be clear this isn't philosophy, you can see exactly what I mean and even HOW I do it (and even when) with the edits of my public wiki or my git repositories.
Whoa, what are all those things you have?
Commenting inline :
-rw-r--r-- 1 fabien fabien 103G Jul 6 2024 wikipedia_en_all_maxi_2024-01.zim
# encyclopedia Wikipedia English with images and more
-rw-r--r-- 1 fabien fabien 81G Apr 22 2023 gutenberg_mul_all_2023-04.zim
# Project Gutenberg, book collection in multiple languages
-rw-r--r-- 1 fabien fabien 75G Jul 7 2024 stackoverflow.com_en_all_2023-11.zim
# StackOverflow, programming questions and answers
-rw-r--r-- 1 fabien fabien 74G Mar 10 2024 planet-240304.osm.pbf
# OpenStreetMap low resolution for the whole World
-rw-r--r-- 1 fabien fabien 3.8G Oct 18 06:55 debian-13.1.0-amd64-DVD-1.iso
# Debian base ISO
-rw-r--r-- 1 fabien fabien 2.6G May 7 2023 ifixit_en_all_2023-04.zim
# iFixit colection of guides to fix appliances
-rw-r--r-- 1 fabien fabien 1.6G May 7 2023 developer.mozilla.org_en_all_2023-02.zim
# Web development documentation
-rw-r--r-- 1 fabien fabien 931M May 7 2023 diy.stackexchange.com_en_all_2023-03.zim
# Do It Yourself Q&A
-rw-r--r-- 1 fabien fabien 808M Jun 5 2023 wikivoyage_en_all_maxi_2023-05.zim
# WikiVoyage, the version of Wikipedia for traveling
-rw-r--r-- 1 fabien fabien 296M Apr 30 2023 raspberrypi.stackexchange.com_en_all_2022-11.zim
# Raspberry Pi Q&A
-rw-r--r-- 1 fabien fabien 131M May 7 2023 rapsberry_pi_docs_2023-01.zim
# Rasspberry Pi documentation
-rw-r--r-- 1 fabien fabien 100M May 7 2023 100r-off-the-grid_en_2022-06.zim
# Off the grid documents
-rw-r--r-- 1 fabien fabien 61M May 7 2023 quantumcomputing.stackexchange.com_en_all_2022-11.zim
# Quantum computer Q&A
-rw-r--r-- 1 fabien fabien 45M May 7 2023 computergraphics.stackexchange.com_en_all_2022-11.zim
# Computer graphics Q&A
-rw-r--r-- 1 fabien fabien 37M May 7 2023 wordnet_en_all_2023-04.zim
# Graph of words in English
-rw-r--r-- 1 fabien fabien 23M Jul 17 2023 kiwix-tools_linux-armv6-3.5.0-1.tar.gz
# Kiwix to read .zim files
-rw-r--r-- 1 fabien fabien 16M Oct 6 21:32 be-stib-gtfs.zip
# public transport database in Brussels, Belgium
-rw-r--r-- 1 fabien fabien 3.8M Oct 6 21:32 be-sncb-gtfs.zip
# train transport database in Belgium
-rw-r--r-- 1 fabien fabien 2.3M May 7 2023 termux_en_all_maxi_2022-12.zim
# Termux, Linux tooling on Android, documentation in English
-rw-r--r-- 1 fabien fabien 1.9M May 7 2023 kiwix-firefox_3.8.0.xpi
# Kiwix Web Extension for the Firefox browser
By the way, there's now a Wikipedia 2025 snapshot.
I am currently trying to fit that on my phone somehow. I wish I could just omit the index database at the end that can't be split it seems. I have to keep it, but when it's split up, it doesn't work anyway (search is broken that way) (https://github.com/openzim/zim-tools/issues/295).
My phone can only do FAT32 for SD cards...
For 2024 Wikipedia, that seems to be around 18GiB of wasted space.
Thanks, updating (~20min) accordingly.
FWIW I have a CMF Nothing 1 and I can put a 500Go microSD in it.
I've got Ulefone Armor 24. It can take a 1TB Micro SD, but only FAT32. Why a Linux-based OS can only do FAT32, despite supporting other FSs on internal storage goes beyond me.
Weird, assuming you have Android 13 it should be usable at least as exFAT and thus can be large enough
Yeah not gonna lie, i think i heard someone in a youtube video a while back talk about how the entirety of wikipedia takes up like 200 gigs or something like that, and it got me seriously considering to actually make that offline backup. Shit is scary when countries like the uk are basically blocking you from having easy access to knowledge.
Yeah, itβs surprisingly small when itβs compressed if you exclude things like images and media. Itβs just text, after all. But the high level of compression requires special software to actually read without uncompressing the entire archive. There are dedicated devices you can get, which pretty much only do that. Like there are literal Wikipedia readers, where you just give it an archive file and itβll allow you to search for and read articles.
if you remove topics you are not interessed it can shrink even more
Sure, but removing knowledge kind of goes against what creating a Wikipedia backup is about..
If my experience with mashing the random article button is any indicator, you could reduce the size by 30% just by removing articles on sports players. I doubt I'll need those
Well, i doubt i will ever need to know anything about a football player or a car
UKGOV haven't started on things like Wikipedia yet. They know kids use it for school and blinded by ideology though they are, even they can see there'd be an enormous backlash if they blocked it any time soon.
If that's going to happen at all, I doubt it would be before the next election. That's whether Labour get re-elected or the Tories make an unexpected comeback. You can tell how far Labour have fallen in the eyes of their party faithful when they've taken a Tory-drafted policy and made it their own.
Ironically, the up and coming third option fascist party, have said they're going to repeal the Online Safety Act. They have other fish to fry if they get in, and they'll want to keep their preferred demographic(s) happy while they do it.
I assume that eventually something like the OSA would come back to "protect the children". They love the current US President.
None of this is hopeful. Take this as more of a rant.
top 98 comments