Crossposted from https://lemmy.ml/post/50758732

Hi everyone,

I am the original author of Searx. I started Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control.

Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches that index entirely on your own infrastructure.

Hister can automatically index pages through its Firefox and Chrome extensions. It can also watch local directories, import browser history and bookmarks, index individual URLs, and crawl complete documentation sites.

The feature I find most useful is offline previews. Hister stores the readable content and HTML of indexed pages locally. You can open a result in a clean and sanitized preview beside the search results without visiting the original website again.

Some other features:

  1. Full text search across web pages, PDFs, docx files, Markdown, OrgMode and text files
  2. Phrase searches, field filters, date filters, wildcards, negation, aliases, labels, facets, and result priorities
  3. Optional semantic search using an embeddings endpoint you configure
  4. Persistent website crawls
  5. Imports from browser history, Linkwarden, Karakeep, Shaarli, Wallabag, and Linkding
  6. Web, terminal, command line, HTTP API, and MCP interfaces
  7. SQLite and PostgreSQL support, plus optional multiple user hosting

Hister cannot replace a global search engine (yet) for subjects you have never encountered because it only searches what you have indexed. My workflow is to search Hister first, then use its shortcut to fall back to traditional search when I need broader web results.

The project is free software under the AGPLv3+ license. It can be installed as a standalone binary or with Docker.

Project: https://github.com/asciimoo/hister

Website and documentation: https://hister.org/

Small read-only demo: https://demo.hister.org/

I'd appreciate feedback, questions, and suggestions as well as joining our growing community.

AI disclosure: AI assisted contributions are not strictly prohibited, but all contributions should be made by humans. More details: https://github.com/asciimoo/hister/blob/master/CONTRIBUTING.md#ai-policy

all 12 comments

sorted by: hot top controversial new old
[–] 3 points 3 months ago

Big fan of searx and use it on all of my devices. This will be a welcome addition.

  • source
  • [–] 2 points 3 months ago (3 children)

    I set hister up for a while, completely forget about it and checked it again after reading this Lemmy post. Well there are now 6.600 pages and 480 MB RAM usage.

    Looks like I need to integrate hister more into my web usage. But maybe I need some training. It isn't possible to setup hister as my default search engine in browser with automated fallback to normal Meta-search with whatever engine? And I use login credentials maybe a faster way like passkey possible in the future?

  • source
  • hideshow 3 child comments
  • [–] 1 point 3 months ago (2 children)

    It isn’t possible to setup hister as my default search engine in browser with automated fallback to normal Meta-search with whatever engine?

    It can be added as a default search engine and it provides a link and a hotkey to open the search query with an external search engine, however I have a different workflow: https://hister.org/posts/how-i-use-hister

    And I use login credentials maybe a faster way like passkey possible in the future?

    That's a good idea, thanks for the suggestion.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 months ago (1 child)

    Yay, bindsym, good to see i3 again, Bad that I switched to KDE, but there is an equivalent. Anyway it's a habit changer. And Lemmy is the perfect place for your app, because here are the last people that know how to change habits nowadays. I'll give it a chance again. Default search engine is a good starting point for me.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 3 months ago

    it’s a habit changer.

    It took some time to get used to it, but I find it more efficient, to spawn a Hister with a single hotkey and use the URL bar with a traditional search engine when I know that I'm looking for something new. But, of course this is subjective, everbody has their preferred workflows, I shared mine as an inspiration.

    Btw, you can start your query with !!, in this case Hister automatically redirects you to your external search engine with the rest of the search query.

  • source
  • parent
  • [–] 2 points 3 months ago
    [–] 2 points 3 months ago (1 child)

    Very cool! Is there a place to share index DB? Having a pre-fetched index on awesome lists and fmhy would be amazing!

  • source
  • hideshow 1 child comment
  • [–] 3 points 3 months ago

    I've been slowly migrating from buku to hister, but it's still early days. I do like þat hister indexes sites, which beats having to manually tag everyþing. So far it's looking pretty good, þough. Þanks for writing it.

  • source
  • [–] 1 point 3 months ago* (last edited 3 months ago) (2 children)

    Being able to navigate only through the content of my favorites websites, oh my, excellent news. I'm truly eager to try it.

    I have a newbie question tho : I would like to use Hister while still supporting the people who create website content. Say I'm online and don't wish to use the local preview, can I access to the website instead of loading my local content ? If I access the website via Hister instead of Searx for instance, would my request be added to their traffic numbers, or not ? (those lowering their referencing and their visibility).

    (Not like my individual usage would change anything anyway but still asking to get a graspe of what it implies).

    Thank you for the hard work !

  • source
  • hideshow 2 child comments
  • [–] 1 point 3 months ago (1 child)

    The results are standard search results, if you click on the title, it opens the original URL, just like in any other regular search engine. You can even close the preview pane if you are not interested in it.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 3 months ago* (last edited 3 months ago)

    I realize my question was more about direct links as a whole and the way they can impact website visibility (because no click registered by search engines), than about Hister specifically. Did some searching in the meantime.

    In any case thank you for your work and your answer ! Time to set up now. 😈

  • source
  • parent
  • 37 comments in c/selfhosted@lemmy.world view that post
    [–] 30 points 3 months ago (2 children)

    Any interested in making this a federated search?

    I've been toying with the idea of a web-ring style search engine where it uses the fediverse's post and sub system so that you can subscribe to "indexers" that are made by other users (or even mastodon users themselves) so that you can have highly targeted search engine systems. So, for example, you could make multiple search "scopes" and each of those scopes would follow specific other "sources" and search from them. Likewise, users could provide links and descriptions to add new entries to each scope which would, in essense, work like a fediverse post and then be distributed to all other interested parties.

    I haven't done a lick of implementation yet, but thought I'd share the idea here as I consider doing this more and more every day. The only thing I haven't figured out yet is images, because ideally that would be a special type of scope view.

  • source
  • hideshow 2 child comments
  • [–] [S] 37 points 3 months ago (1 child)

    Federated search is the direction I'd like to go when the core is mature enough.

    I'm still trying to figure out the best approach to make the federation secure (no accidental private/confidential data leak) and easy to use. Related conversations: https://github.com/asciimoo/hister/discussions/432 & https://github.com/asciimoo/hister/issues/387 . I'd appreciate help to figure out an optimal solution.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 12 points 3 months ago

    Already installed and loving this idea.

  • source
  • [–] [M] 22 points 3 months ago (1 child)

    @asciimoo@lemmy.ml please update the title with the appropriate tag (rule 7) and if AI was involved in development, add the disclosure per rule 8 (sample disclosures can be found here)

    Thanks!

  • source
  • hideshow 1 child comment
  • [–] 10 points 3 months ago* (2 children)

    This seems super cool, you can curate your own search index :O

    Being able to preview your search results offline is quite neat too!

    One question though, how will you solve the issue of "echo chambers", because from my understanding, it looks like it will only present results that you search for / have searched for, meaning for some people, you could be stuck with results that only support your view of the world.

  • source
  • hideshow 2 child comments
  • [–] 15 points 3 months ago (1 child)

    I mean it kinda looks like that's by design currently. It's an indexer for your corners of the web, not a search aggregator to find new ones.

  • source
  • parent
  • hideshow 1 child comment
  • [–] [S] 19 points 3 months ago

    Exactly, the echo chamber phenomenon is mostly problematic for "discovery type" searches while Hister is mainly for "recall type" search.

    Implementing federated search/index sharing could be a partial solution to this issue in the long run.

  • source
  • parent
  • [–] 9 points 3 months ago (1 child)

    Asciimoo always with the cool stuff!

    Think I'll try getting it setup on Kubernetes and using it for searching documentation when programming.

    Would you be interested in getting a Kubernetes example in the docs too then? If you have any requirements for it let me know.

  • source
  • hideshow 1 child comment
  • [–] 7 points 3 months ago

    I've been running this for a while since I first heard of it, it's already saved me a few times trying to recall a page for something.

    I treat it like a more advanced history and it's great!

  • source
  • [–] 6 points 3 months ago (4 children)

    Oh damn this could replace the bookmakers. I have Linkding with the Linkding Injector extension, but this would be next level.

    You can view the saved text too, nice. Would it be possible to attach an HTML file from like single file for sites that have heavy image content as part of the “view” button? That’d completely replace Linkding/Karakeep/Linkwarden use cases for me

    Understandable if not, that’s ancillary to the text search focus

  • source
  • hideshow 4 child comments
  • [–] [S] 11 points 3 months ago* (3 children)

    Oh damn this could replace the bookmakers.

    The inspiration for Hister was a bookmarking app, but I realized that I always forget to manually trigger the bookmarking and I miss so many great resources.

    Would it be possible to attach an HTML file from like single file

    Currently you can import SingleFile HTMLs using the hister import file command, but no further integrations are implemented yet. Although, rendering the exact SinglePage file as a preview can be added relatively quickly. It would be also nice to accept files directly from the SingleFile extension.

    Thanks for the good suggestion, I've added it to my TODO! =]

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 3 months ago (2 children)

    I have tons of webpages saved using the Firefox built-in page saver, which saves and html file and a corresponding folder for the other resources (images, javascript). Would be cool if these could be imported as well. Though maybe the resource folder can be ignored and the html file can already be imported?

  • source
  • parent
  • hideshow 2 child comments
  • [–] [S] 3 points 3 months ago (1 child)

    Though maybe the resource folder can be ignored and the html file can already be imported?

    It depends. Hister always requires a unique URL for each document. SingleFile snapshots include the original URL of the document as a meta HTML element. I'm not sure if the built-in page saver provides URL information.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 3 months ago* (last edited 3 months ago) (4 children)

    Can i use this in my self hosted environment? the docs talk mainly about running it in the terminal

    does it have a docker install? found it https://hister.org/docs/docker

    can i connect it to other services like paperless, or would i need to manually import files?

  • source
  • hideshow 4 child comments
  • [–] 2 points 3 months ago (3 children)

    I have tried it a bit today and I like it, but I have a few questions

    • Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file
    • After just playing with it for a day, the disk usage is 150 MB. I figure I should use postgres instead of sqlite. Do you have any examples on how to set that up, preferably with docker? I guess I can figure it out, but it might take me some time. I am not familiar with pgvector
  • source
  • parent
  • hideshow 3 child comments
  • [–] [S] 4 points 3 months ago (2 children)

    can i connect it to other services like paperless, or would i need to manually import files?

    Hister supports importing data from a few services, but paperless isn't supported yet. More details: https://hister.org/docs/import

    Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file

    Docker is perfectly fine. Every settings option from the config file can specified using environment variables. The syntax is HISTER__[SECTION]__[OPTION]=[VALUE].

    After just playing with it for a day, the disk usage is 150 MB

    Probably most of the disk space is occupied by the Hister binary which contains all the N-grams required to identify ~30 languages. The index should be much smaller.

    I figure I should use postgres instead of sqlite.

    SQLite is more than enough for personal use, but if you prefer to use postgres, just specify the standard DSN formatted connection data to the server.database config option: https://hister.org/docs/configuration#database-backends

    how to set that up, preferably with docker

    Use the HISTER__SERVER__DATABASE="host=localhost user=hister password=hister dbname=hister port=5432" environment variable.

    I am not familiar with pgvector

    Hister automatically creates the database model and handles the migrations if required.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 3 months ago (1 child)

    Thank you for your response 🙂 I will stick to sqlite

    How do i use environment variables to make a indexer? I couldn't get it to work

    i.e. the part with directories https://hister.org/posts/indexing-local-files-with-hister

    indexer:
      directories:
        - path: ~/Documents
          filetypes: ['md', 'txt']
        - path: ~/code/projects
          filetypes: ['txt']
          excludes: [, 'secret/*', '*key']
        - path: ~/notes
          patterns: ['*.org', 'doc_*', 'README']
    
  • source
  • parent
  • hideshow 1 child comment
  • [–] [S] 4 points 3 months ago

    How do i use environment variables to make a indexer? I couldn’t get it to work

    Repeatable config options are not supported from env vars. In this case you have to mount a config file to your container. But, if you want to track local files from docker, you have to mount those files/directories anyway. I suggest to use the standalone binary release if you'd like to track local files, it is way more convenient.

  • source
  • parent
  • [–] 4 points 3 months ago

    Awesomely cool and federated search would be a boon later! Hope you keep it going!

  • source
  • [–] 3 points 3 months ago (1 child)

    The name reminds me of this crazy nostradamus “documentary “ that used to play on HBO when I was a kid.

  • source
  • hideshow 1 child comment
  • [–] 3 points 3 months ago (2 children)

    Is there a future where Hister takes on the functions of SearXNG? I.E. new content is discovered through APIs but local or previous content is prioritized?

  • source
  • hideshow 2 child comments
  • [–] [S] 4 points 3 months ago (1 child)

    Currently it is not planned. Hister guarantees that non of your data/query/metadata leaves the service if you use it. As I see, this is a more valuable and unique feature than having an integrated metasearch. There are already great metasearch solutions and Hister provides an easy fallback to search providers, so in my opinion this direction would be more of a sacrifice than an improvement.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 3 months ago

    SearXNG user here. Thanks for your work!

    I'm looking forward to checking this out.

    Like others have mentioned, a federated version of this seems like it would be really, really powerful.

  • source
  • [–] 1 point 3 months ago

    Very likely a huge thing to ask, but have you considered adding federation to the indexing? I presume that generating and storing the index database would quickly put a severe storage bottleneck (maybe the idea here is that no single person explores the internet enough for their index database to become infeasibly large?). But, at least to me, it sounds like allowing different hister instances to share a common index (or index subsets) would greatly reduce this bottleneck. Just curious if you have any thoughts on this. Overall I love the idea and am looking forward to giving it a try soon, so well done!

  • source
  • [+] -10 points 3 months ago* (last edited 1 month ago) (7 children)
  • [–] [S] 12 points 3 months ago (6 children)

    The summary has numerous inaccuracies. Most importantly: it is pretty easy to delete content by topic or age. The hister delete command can accept a search query to remove only matched documents. The same is true on the web UI "actions -> remove all matching documents". You can quickly filter by age, simply query updated:>365d. Combine it with URLs, labels, domains or phrases.

  • source
  • parent
  • hideshow 6 child comments
  • [+] -1 points 3 months ago* (last edited 1 month ago) (6 children)
  • [–] [S] 5 points 3 months ago* (2 children)

    Is there a TTL / max database size per user setting?

    As I wrote, you can simply automate deletion by document age. Schedule a delete event on each day with the desired retention time defined as a filter expression. Database size limit isn't available yet.

    Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?

    No, sorry, I don't have time to correct a copy of a multiple screens long AI prompt.

    I would like to further recommend Hister but your documentation is somewhat confusing at first blush.

    Which parts are confusing?

  • source
  • parent
  • hideshow 2 child comments
  • [+] -8 points 3 months ago* (last edited 1 month ago) (2 children)
  • [–] [S] 7 points 3 months ago (1 child)

    However, dismissing it as “a multiple screens long AI prompt” does not only not answer the question, it comes off as abrasive.

    I think it is more abrasive to copy/paste a poorly formatted LLM output instead of taking the time and summarizing it to a few sentences just as you did in your previous post.

  • source
  • parent
  • hideshow 1 child comment
  • load more comments (-1 replies)
  • load more comments (-1 replies)
  • [–] 2 points 3 months ago* (last edited 3 months ago) (2 children)

    Additionally, is the other parenthetical information materially correct? If not, which points [1 thru to 7] are wrong?

    This is one of the problems with relying on AI... it can produce an overwhelming amount of content with errors and inaccuracies throughout. If you don't review and know the content yourself, you won't know what it got wrong.

    It's genuinely rude to lazily use AI to produce such a large babble of details and then ask someone else to review it for errors when you haven't reviewed it yourself. I know you didn't mean it that way but that's nonetheless the result. People are going to read your comment and be misled about this project all because they assume AI is accurate and you didn't review its results.

    Edit: Sheesh... I hadn't even got to your shitty comments that followed. You use AI to make a low effort but highly verbose post and then get mad that the repo author won't review it in detail for you when you can't be arsed with reading the docs yourself.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 months ago (1 child)

    This is one of the problems with relying on AI… it can produce an overwhelming amount of content with errors and inaccuracies throughout. If you don’t review and know the content yourself, you won’t know what it got wrong.

    Yep, and this induces anyone who wants to review the resulting slop to need to turn to AI as well to even deal with the amount (or dismiss as a whole).

    AI is a viral infection on software development.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 3 months ago

    Yep... couldn't agree more. I has its uses but too many people are using it and disconnecting their brain. Going to meetings where people have used it to determine requirements, do analysis or even transform data and then don't even review the results themselves before presenting and asking us to review them is rage-inducing. "What do you mean it has problems? Where? What's wrong??" And there's like 5 things I've spotted in 2 minutes and it's clear they've not even reviewed it themselves.

    This guy and his long-ass, poorly formatted AI-generated "documentation" no one asked for and then after numerous errors are pointed out.... "Will you review the rest?" The audacity!! 😄

  • source
  • parent
  • load more comments (-1 replies)
  • 14 comments in c/privacy@lemmy.ml view that post
    [–] 4 points 6 months ago

    Uhh, neat. Will definitely try it out.

  • source
  • [–] 3 points 6 months ago

    This is a great idea, I’m gonna try

  • source
  • [–] 3 points 6 months ago

    I have actually been looking for something like this, look forward to giving it a go!

  • source
  • [–] 2 points 6 months ago (1 child)

    Interesting, I didn't see it in the documentation so if you didn't document that already, you can have your local instance as search suggestion for Firefox on mobile and desktop. I use it for my own wiki, e.g. https://mastodon.pirateparty.be/@utopiah/116351732150481942

    Also how I would imagine it is default search there and if no hit then fallback to a default search engine, e.g. DDG.

  • source
  • hideshow 1 child comment
  • [–] [S] 5 points 6 months ago

    Also how I would imagine it is default search there and if no hit then fallback to a default search engine, e.g. DDG.

    This is exactly how I use it. Hister has even a hotkey to quickly jump to your preferred online search engine with the current search query if you cannot find what you are looking for.

  • source
  • parent
  • [+] 2 points 6 months ago (2 children)
  • [–] 2 points 6 months ago* (last edited 6 months ago)

    This looks really rad. I have been trying to build a leftist search engine using searxng and it has a lot of issue because of things getting blocked over tor due to the amount of background requests you have to be making to those sites. I have thought about building something like this to deal with that issue, but just had my hands full with other projects. Quite obviously (for those who like to come online to shit on really cool project people are building) the use case is people looking to build their own privately indexed search results without it having to get polluted by shit that comes up in other search engines. I feel like that was immediately apparent to me, because I have that use case and I immediately saw the utility here. For those who don't get, just because it doesn't fit the use case you have for internet related projects, doesn't mean it's not extremely useful to others. You literally don't have to come on here just to shit on what other people are doing simply because, "you don't get it." This project looks really cool and I am going to look into replacing my searxng instance with this soon. I am super stoked that somebody has done this work. It has needed to be done for a long time. So thank you!

  • source
  • [–] -1 points 6 months ago (4 children)

    I don't get it. It indexes pages which were already visited, right? So in order to find some website I need to first use another search engine. Afterwards, that website is in my browsing history and if I need it again, I don't need to search for it. So what's the use case for this project?

  • source
  • hideshow 4 child comments
  • [–] 4 points 6 months ago

    Your browsing history does not have full text search, so if you only remember the content of the page and not the title of it, you're SOL. Or if you browse across multiple devices, you have to check multiple places to hope to find it.

  • source
  • parent
  • [–] [S] 2 points 6 months ago (2 children)

    It indexes pages which were already visited, right?

    Yes, if you use the browser extension only, but Hister has an API and a crawler as well if you'd like to add content you have not visited yet. Also, Hister supports indexing local text files, not just websites.

    Afterwards, that website is in my browsing history and if I need it again, I don’t need to search for it

    • Unfortunately browser history does not include the page's content only the URL + title combo at best.
    • Browser's can't show an offline preview (Having offline previews is a huge privacy - and productivity - win in my opinion, it completely eliminates the need of creating external network requests)

    These are the biggest weaknesses of the browser history compared to Hister, but there are many more nuances where Hister can provide extra features and QoL improvements. I recommend checking the documentation & posts on the website if you are interested in the details.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 6 months ago* (1 child)

    So does that mean that the index starts off as empty? If so, is there a way to create a centralized (I know that's a bad word) starting repo such that the engine already knows some cool results? I have tabbed bookmarks for news that is not shitty, archives, video that isn't YouTube, privacy resources, etc. It would be cool if people could post indices focused on certain topics that they could add. Like indices for random stuff, like dog grooming, kayaking, or woodworking. It could be a hub like Docker Hub, but for cool results.

    Sorry. Ha ha. You know you have a good idea when people start asking for features. I haven't even started it yet. Maybe I can try self hosting on my desktop.

    This is exciting! I normally use Searxist on Android.

  • source
  • parent
  • hideshow 1 child comment
  • [–] [S] 4 points 6 months ago

    I'd absolutely love to do this! It's already on my future plans list: https://hister.org/support :

    Create infrastructure for importable, pre-indexed databases organized by topic, letting users quickly expand their local index with curated, relevant content.

    It could be a hub like Docker Hub, but for cool results.

    Exactly!

    Sorry. Ha ha. You know you have a good idea when people start asking for features. I haven’t even started it yet.

    <3 No need to apologize. I appreciate suggestions a lot (especially if those are well aligned with my ideas =] ).

  • source
  • parent
  • 3 comments in c/privacy@programming.dev view that post
    [–] 2 points 6 months ago (1 child)

    this sounds really cool!

    How did you come up with the name?

    iirc "Hister" is the name of an evil despot used in at least one of Nostradamus' "prophecies" and is often believed (by those who believe in these things) to be a reference to Hitler.

    its also an ancient name for the River Danube, which flows through much of WWII's battlefields

  • source
  • hideshow 1 child comment
  • [–] 1 point 6 months ago

    I thought it was obvious that “hister” was the verb-ificarion of the noun “history” since this thing is a service/tool which produces history.

    Who cares what crackpot Nostradamus said. Nosferatu was more metal anyway 🤘

  • source
  • parent
  • [–] 1 point 6 months ago

    Those are some impressive projects - some people are just built different huh

  • source