GitHub - asciimoo/hister: Your own search engine
github.com
Your own search engine. Contribute to asciimoo/hister development by creating an account on GitHub.

Hi everyone,

I am the original author of Searx. I started Hister with a similar motivation: reducing our dependence on external search engines while keeping searches and personal data under our control.

Searx is a metasearch engine that forwards queries to other search providers. Hister takes a different approach. It builds a private full text index from content you choose, then searches that index entirely on your own infrastructure.

Hister can automatically index pages through its Firefox and Chrome extensions. It can also watch local directories, import browser history and bookmarks, index individual URLs, and crawl complete documentation sites.

The feature I find most useful is offline previews. Hister stores the readable content and HTML of indexed pages locally. You can open a result in a clean and sanitized preview beside the search results without visiting the original website again.

Some other features:

  1. Full text search across web pages, PDFs, docx files, Markdown, OrgMode and text files
  2. Phrase searches, field filters, date filters, wildcards, negation, aliases, labels, facets, and result priorities
  3. Optional semantic search using an embeddings endpoint you configure
  4. Persistent website crawls
  5. Imports from browser history, Linkwarden, Karakeep, Shaarli, Wallabag, and Linkding
  6. Web, terminal, command line, HTTP API, and MCP interfaces
  7. SQLite and PostgreSQL support, plus optional multiple user hosting

Hister cannot replace a global search engine (yet) for subjects you have never encountered because it only searches what you have indexed. My workflow is to search Hister first, then use its shortcut to fall back to traditional search when I need broader web results.

The project is free software under the AGPLv3+ license. It can be installed as a standalone binary or with Docker.

Project: https://github.com/asciimoo/hister

Website and documentation: https://hister.org/

Small read-only demo: https://demo.hister.org/

I’d appreciate feedback, questions, and suggestions as well as joining our growing community.

AI disclosure: AI assisted contributions are not strictly prohibited, but all contributions should be made by humans. More details: https://github.com/asciimoo/hister/blob/master/CONTRIBUTING.md#ai-policy

  • conrad82@lemmy.worldEnglish
    3·
    1 day ago

    Can i use this in my self hosted environment? the docs talk mainly about running it in the terminal

    does it have a docker install? found it https://hister.org/docs/docker

    can i connect it to other services like paperless, or would i need to manually import files?

    • conrad82@lemmy.worldEnglish
      1·
      22 hours ago

      I have tried it a bit today and I like it, but I have a few questions

      • Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file
      • After just playing with it for a day, the disk usage is 150 MB. I figure I should use postgres instead of sqlite. Do you have any examples on how to set that up, preferably with docker? I guess I can figure it out, but it might take me some time. I am not familiar with pgvector
      • asciimoo@lemmy.mlOPEnglish
        3·
        20 hours ago

        can i connect it to other services like paperless, or would i need to manually import files?

        Hister supports importing data from a few services, but paperless isn’t supported yet. More details: https://hister.org/docs/import

        Is it better to run it in docker or e.g. a LXC in proxmox? If i want to index files, it seems it needs the config.yml file

        Docker is perfectly fine. Every settings option from the config file can specified using environment variables. The syntax is HISTER__[SECTION]__[OPTION]=[VALUE].

        After just playing with it for a day, the disk usage is 150 MB

        Probably most of the disk space is occupied by the Hister binary which contains all the N-grams required to identify ~30 languages. The index should be much smaller.

        I figure I should use postgres instead of sqlite.

        SQLite is more than enough for personal use, but if you prefer to use postgres, just specify the standard DSN formatted connection data to the server.database config option: https://hister.org/docs/configuration#database-backends

        how to set that up, preferably with docker

        Use the HISTER__SERVER__DATABASE="host=localhost user=hister password=hister dbname=hister port=5432" environment variable.

        I am not familiar with pgvector

        Hister automatically creates the database model and handles the migrations if required.

        • conrad82@lemmy.worldEnglish
          2·
          14 hours ago

          Thank you for your response 🙂 I will stick to sqlite

          How do i use environment variables to make a indexer? I couldn’t get it to work

          i.e. the part with directories https://hister.org/posts/indexing-local-files-with-hister

          indexer:
  directories:
    - path: ~/Documents
      filetypes: ['md', 'txt']
    - path: ~/code/projects
      filetypes: ['txt']
      excludes: [, 'secret/*', '*key']
    - path: ~/notes
      patterns: ['*.org', 'doc_*', 'README']
          • asciimoo@lemmy.mlOPEnglish
            3·
            11 hours ago

            How do i use environment variables to make a indexer? I couldn’t get it to work

            Repeatable config options are not supported from env vars. In this case you have to mount a config file to your container. But, if you want to track local files from docker, you have to mount those files/directories anyway. I suggest to use the standalone binary release if you’d like to track local files, it is way more convenient.