685
Self-host Reddit – 2.38B posts, works offline, yours forever (github.com)
submitted 9 months ago by to c/selfhosted@lemmy.world
120 comments fedilink hide all child comments
 

Reddit's API is effectively dead for archival. Third-party apps are gone. Reddit has threatened to cut off access to the Pushshift dataset multiple times. But 3.28TB of Reddit history exists as a torrent right now, and I built a tool to turn it into something you can browse on your own hardware.

The key point: This doesn't touch Reddit's servers. Ever. Download the Pushshift dataset, run my tool locally, get a fully browsable archive. Works on an air-gapped machine. Works on a Raspberry Pi serving your LAN. Works on a USB drive you hand to someone.

What it does: Takes compressed data dumps from Reddit (.zst), Voat (SQL), and Ruqqus (.7z) and generates static HTML. No JavaScript, no external requests, no tracking. Open index.html and browse. Want search? Run the optional Docker stack with PostgreSQL – still entirely on your machine.

API & AI Integration: Full REST API with 30+ endpoints – posts, comments, users, subreddits, full-text search, aggregations. Also ships with an MCP server (29 tools) so you can query your archive directly from AI tools.

Self-hosting options:

  • USB drive / local folder (just open the HTML files)
  • Home server on your LAN
  • Tor hidden service (2 commands, no port forwarding needed)
  • VPS with HTTPS
  • GitHub Pages for small archives

Why this matters: Once you have the data, you own it. No API keys, no rate limits, no ToS changes can take it away.

Scale: Tens of millions of posts per instance. PostgreSQL backend keeps memory constant regardless of dataset size. For the full 2.38B post dataset, run multiple instances by topic.

How I built it: Python, PostgreSQL, Jinja2 templates, Docker. Used Claude Code throughout as an experiment in AI-assisted development. Learned that the workflow is "trust but verify" – it accelerates the boring parts but you still own the architecture.

Live demo: https://online-archives.github.io/redd-archiver-example/ GitHub: https://github.com/19-84/redd-archiver (Public Domain)

Pushshift torrent: https://academictorrents.com/details/1614740ac8c94505e4ecb9d88be8bed7b6afddd4

top 100 comments

sorted by: hot top controversial new old
[–] 54 points 9 months ago (4 children)

And only a 3.28 TB database? Oh, because it's compressed. Includes comments too, though.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 11 points 9 months ago* (2 children)

    Yeah, it should inflate to 15TB or more I think

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 16 points 9 months ago (1 child)

    If only I had the space and bandwidth. I would host a mirror via Lemmy and drag the traffic away.

    Actually, isn’t the a way to decentralize this that can be accessed from regular browsers on the internet? Live content here, archive everywhere.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • [–] 38 points 9 months ago (1 child)

    Wow, great idea. So much useful information and discussion that users have contributed. Looking forward to checking this out.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 56 points 9 months ago

    This seems especially handy for anyone who wants a snapshot of Reddit from pre-enshittification and AI era, where content was more authentic and less driven by bots and commercial manipulation of opinion. Just choose the cutoff date you want and stick with that dataset.

  • permalink
  • fedilink
  • source
    •
    [–] 120 points 9 months ago (19 children)

    Just so you're aware, it is very noticeable that you also used AI to help write this post and its use of language can throw a lot of people off.

    Not to detract from your project, which looks cool!

  • permalink
  • fedilink
  • source
  • hideshow 19 child comments
    •
  • [–] [S] 141 points 9 months ago (17 children)

    Yes I used AI, English is not my first language. Thank you for the kind words!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 17 child comments
    •
  • [+] -33 points 9 months ago (1 child)

    You're awesome. AI is fun and there's nothing wrong with using it especially how you did. Lemmy was hit hard with AI hate propaganda. China probably trying to stop it's growth and development in other countries or some stupid shit like that. But you're good. Fuck them

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 9 months ago

    Yup, if there was ever a decent use for AI, this is it. Lemmy can (and will) hate the shit out of it, but it took a little burden off the shoulders of someone doing us a great service.

  • permalink
  • fedilink
  • source
  • parent
  • [+] -22 points 9 months ago (15 children)
    [removed by mod]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 15 child comments
    •
  • [–] 40 points 9 months ago (11 children)

    Would love to see you learn an entire foreign language just so you are able to communicate with the world without being laughed at by people as hostile as yourself.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 11 child comments
    •
  • [–] 1 point 9 months ago

    I mean I can't see what the comment was and I'm assuming it must've been downright hateful, but that person almost certainly has learned a foreign language just to communicate with the world and in fact had to learn another foreign language in school because their name is Estonian for "gypsy" and learning two foreign languages (usually English and Russian, sometimes German or something else for the second foreign language) is required. Likely they speak 2.5 languages as is common here (my German is so bad I count it as half a language - native speakers speak too fast for me, but I can kinda get my point across if needed), but could be more.

    Just pointing out that even when trying to be accepting of others, subtle anglo-defaultism can show up in your comment, not that I necessarily agree with whatever the comment was.

  • permalink
  • fedilink
  • source
  • parent
  • [+] -7 points 9 months ago* (8 children)

    They said it wasn't their "first" lanugage. Which leads me to believe that they do speak English. If that's the case, then they indeed are kind of lazy. There have already been studies in the impact of AI when used for communication and the results are not positive.

    This isn't something I'd personally point out and criticize, just something I wouldn't do personally. Take the time to express your own ideas in your own words. The long term cost is higher than the short term gains.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 8 points 9 months ago (5 children)

    Hey I drove to the library, picked up all these things you needed, got dinner here ya go, free!

    You drove? man that's lazy...

    He used AI to clean up translation and save time after he spent a fuck ton of time curating and delivering us a helpful product. Calling him out as lazy is an awful take.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] [S] 6 points 9 months ago (1 child)

    there are the so called activists that complain alot then there are the activists that deliver projects and code... enough said

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] -3 points 9 months ago

    "Activists"? What are you even talking about?

    Regardless, I specifically said that what you did wasn't wrong or anything likw that. I simply think that it's going to do you more harm than good in the long run. You're free to do whatever you want though, obviously.

    Another piece of advice. When someone simply shares an opinion, don't get instantly butthurt over nothing. Otherwise this might as well be reddit.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 9 months ago (2 children)

    First, that's an awful analogy.

    Second, you're assuming (for some unknown reason) that they "cleaned up" the "translation" using ai. You have literally no idea exactly how they wrote the post. It's kinda weird to make up a random scenario but ok.

    Third, no, it's not an awful take. You can code something that requires a ton of effort but write awful documentation. One thing does not make the other impossible.

    Fourth, I already explained that there have already been studies that concluded that using AI to write stuff for you has a negative impact on your communication skills. This is not an opinion or me being ingrateful or whatever. I was just sharing information.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 9 months ago (1 child)

    I have A1 and A2 level in a couple of non-first languages, technically I can speak those, realistically I don't and will not be able to communicate something more complex than ‘here, take a look’

    So I don't agree with your absolutistic stance

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 4 points 9 months ago

    There's nothing "absolutistic" about my "stance". If you're rusty using a language, you won't get better if someone else does the homework for you. Make an effort, make mistakes, write in a way that sounds weird, who cares. But practice. If you only take the easy way out, that'll be your only option in the future.

    Although, like I already said, that's MY way of thinking about it. If you want to use ai to write your stuff, you do you. It doesn't negate the fact that, whle it's not "wrong", it's the lazy (or minimum effort) option. Don't know why it bothers you so much.

  • permalink
  • fedilink
  • source
  • parent
    • load more comments (-1 replies)
    • [–] 26 points 9 months ago (4 children)

    What is the timing of the dataset, up through which date in time?

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] [S] 45 points 9 months ago (1 child)

    2005-06 to 2024-12

    however the data from 2025-12 has been released already, it just needs to be split and reprocessed for 2025 by watchful1. once that happens then you can host archive up till end of 2025. i will probably add support for importing data from the arctic shift dumps instead so that archives can be updated monthly.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • [–] 20 points 9 months ago

    so kinda like kiwix but for reddit. That is so cool

  • permalink
  • fedilink
  • source
    •
    [–] 19 points 9 months ago (7 children)

    Fuck Reddit and Fuck Spez.

  • permalink
  • fedilink
  • source
  • hideshow 7 child comments
    •
  • [–] 6 points 9 months ago (6 children)

    You know what would be a good way to do t? Take all that content and throw it on a federated service like ours. Publicly visible. No bullshit. And no reason to visit Reddit to get that content. Take their traffic away.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [+] 3 points 9 months ago (5 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 9 months ago (4 children)

    What would they say? It’s information that’s freely available, no payment required, no accounts to simply read it, no copyrights, where’s the legal in hosting a duplicate of the content?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 9 months ago (1 child)

    It might fall under the same concept that recipes do - you can't copyright a recipe, but a collection of recipes (such as a book) is copyrightable.

    In any case, they have a lot more money to pay lawyers than you or I do, I'll bet, so even if you are right, that doesn't mean you'll have the money to actually win.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
    • [–] 53 points 9 months ago (16 children)

    It would be neat for someone to migrate this data set to a Lemmy instance

  • permalink
  • fedilink
  • source
  • hideshow 16 child comments
    •
  • [–] 26 points 9 months ago (13 children)

    It would be inviting a lawsuit for sure. I like the essence of the idea, but it's probably more trouble than it's worth for all but the most fanatic.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
  • [–] 9 points 9 months ago* (3 children)

    Is it though? That is (or was, and should be again) publicly accessible information that was created over the years by random internet users. I refuse the notion that an American company can "own it" just because they ran the servers. Sure they can hold copyright for their frontend and backend code, name and whatever. But posts and comments, no way.

    Of course it would be dumb for someone under US jurisdiction but we'll see how much an international DMCA claim is worth considering the current relations anyway.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 9 months ago (2 children)

    They don't own it, the individual posters own the content of their own posts, however, from the reddit terms of service:

    When Your Content is created with or submitted to the Services, you grant us a worldwide, royalty-free, perpetual, irrevocable, non-exclusive, transferable, and sublicensable license to use, copy, modify, adapt, prepare derivative works of, distribute, store, perform, and display Your Content and any name, username, voice, or likeness provided in connection with Your Content in all media formats and channels now known or later developed anywhere in the world. This license includes the right for us to make Your Content available for syndication, broadcast, distribution, or publication by other companies, organizations, or individuals who partner with Reddit.

    And with each of those rights granted, Reddit's lawyers can defend those rights. So no, they don't own it "just because they ran the servers" - they own specific rights to copy granted to them by each poster.

    (I don't like this arrangement, but ignorance of the terms of service isn't going to help someone who uploaded a full copy of the works they have extensive rights to) On this subject I think there needs to be an extensive overhaul to narrow what terms you can extend to the general public. The problem is I straight up don't trust anyone currently in power to make such a change to have our interests in mind.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 9 months ago (1 child)

    I'm not at all familiar with legalese, but wouldn't 'non-exclusive' in that statement mean that you, and others permitted by you, can redistribute the content as you see fit? Meaning that copying and redistributing reddit content doesn't necessarily violate reddit's terms of service but does violate the user's copyright?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 9 months ago

    Yeah so at worst you could get sued by some random reddit users that don't want their post history hosted on your site.

    Given how little traction artists and authors have had with suing AI companies for blatant copyright infringement, I kinda doubt it would go anywhere.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 7 points 9 months ago* (8 children)

    Might be easiest to set up an instance in a country that doesn't give a fuck about western IP law, then others can federate to it.

    So yeah, fanatic levels of effort.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 6 points 9 months ago (3 children)

    Brb, setting up a Lemmy server in Red Star OS

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 5 points 9 months ago* (2 children)

    (The machine with the only Steam account active in North Korea would like to already knows your location)

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 9 months ago (1 child)

    The chances are pretty high that is probably Kims computer, arent they?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 9 months ago

    I think we were all hoping that some loveable genius was quietly subverting their surveillance state and getting a view of the outside world via Team Fortress 2, but, yeah, if it's not North Korea's fattest man, it's probably a high ranking military crony.

    . . .Hey just musing here but that sounds like a kinda hilariously easy doxx. You don't think they'd keep state secrets on that same machine? . . . Surely. . .? Noooo. . . 🤔

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 9 months ago* (2 children)

    Post and comments are not Reddit's IP anyway :3

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 9 months ago*

    They might have set up the user agreement for it. Stackexchange did and their whole business model was about catching businesses where some worker copy/pasted code from a stackexchange answer and getting a settlement out of it.

    I agree with you in principle (hell, I'd even take it further and think only trademarks should be protected, other than maybe a short period for copyright and patent protection, like a few years), but the legal system might disagree.

    Edit: I'd also make trademarks non-transferrable and apply to individuals rather than corporations, so they can go back to representing quality rather than business decisions. Especially when some new entity that never had any relation to the original trademark user just throws some money at them or their estate to buy the trust associated with the trademark.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 17 points 9 months ago

    You should be very proud of this project!! Thank you for sharing.

  • permalink
  • fedilink
  • source
    •
    [–] 34 points 9 months ago (1 child)

    Reddit is hot stinky garbage but can be useful for stuff like technical support and home maintenance.

    Voat and Ruqqus are straight-up misinformation and fascist propaganda, and if you excise them from your data set, your data will dramatically improve.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] [S] 26 points 9 months ago (2 children)

    PLEASE SHARE ON REDDIT!!! I have never had a reddit account and they will NOT let me post about this!!

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
    load more comments (2 replies)
    [–] 13 points 9 months ago

    Thanks. This is great for mining data and urls.

  • permalink
  • fedilink
  • source
    •
    [–] 10 points 9 months ago (3 children)

    How does this compare to redarc? It seems to be similar.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 16 points 9 months ago (2 children)

    redarc uses reactjs to serve the web app, redd-archiver uses a hybrid architecture that combines static page generation with postgres search via flask. is more like a hybrid static site generator with web app capabilities through docker and flask. the static pages with sorted indexes can be viewed offline and served on hosts like github and codeberg pages.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • [–] 7 points 9 months ago* (3 children)

    How long it takes to download this 3TB torrent ?

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 12 points 9 months ago (2 children)

    week(s)

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • [–] 6 points 9 months ago (1 child)

    What's the size difference when you remove the porn stuff from the torrent?

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 5 points 9 months ago

    grabs external

  • permalink
  • fedilink
  • source
    •
    [–] 3 points 9 months ago

    Very cool! Do you know how your project may compare with arctic shift ? For those more interested in research with reddit data, is there benefit of one vs another?

  • permalink
  • fedilink
  • source
    •
    [–] 9 points 9 months ago (1 child)

    I think this is a good use case for AI and Impressed with it. wish the instructions were more clear how to set up though.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 1 point 9 months ago

    Does this decompress the files preemptively and leave them? Or is it only decompressing as a post/subreddit is accessed? Basically i am wondering what kind of storage footprint would be required to search through this

  • permalink
  • fedilink
  • source
    •
    [–] 1 point 9 months ago (3 children)

    Can anyone figure out what the minimum process is to just use the SSG function? I'm having a really hard time trying to understand the documentation.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 1 point 9 months ago (2 children)

    did you check the quickstart?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • [–] 1 point 9 months ago (5 children)

    Eww, Voat and Ruqqus.

  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] [S] 1 point 9 months ago (3 children)

    i will always take more data sources, including lemmy!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 9 months ago (2 children)

    ... for building your personal Grok?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • [–] -2 points 9 months ago (8 children)

    I would sooner download a tire fire.

  • permalink
  • fedilink
  • source
  • hideshow 8 child comments
    •
  • [–] [S] 44 points 9 months ago (3 children)

    thanks anyway for looking at my project 🙂

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 13 points 9 months ago (2 children)

    I use Reddit for reference through RedLib. I could see how having an on-premise repository would be helpful. How many subs were scrapped in this 3.28 TB backup? Reason for asking, I'd have little interest in say News or Politics, but there are some good subs that deal with Linux, networking, selfhosting, some old subs I used to help moderate like r/degoogle, r/deAmazon, etc.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 20 points 9 months ago (3 children)

    Say what you will about Reddit, but there is tons of information on that platform that's not available anywhere else.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [+] -10 points 9 months ago (2 children)

    :-/

    You can definitely mine a bit of gold out of that pile of turds. But you could also go to the library and receive a much higher ratio of signal to noise.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 9 months ago (1 child)

    This one specific bug in this one niche library has probably not been written about in a book, and even if it has I doubt that book is in my local library, and even if it is I doubt I can fucking find it

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 9 months ago

    obscure problems almost always have reddit comments as search results, and there's no forums or blogs with any of it anymore. be nice to have around solely for that... though I'm sure if shit like /pics or whatever else was removed it could get significantly smaller...

  • permalink
  • fedilink
  • source
  • parent
    • [–] 1 point 9 months ago

    I do not consent for this

  • permalink
  • fedilink
  • source
    •
    load more comments
    view more: next ›