omnifeed

README

omnifeed gives an AI agent the full research loop — search → URLs → content — against self-hosted SearXNG + crawl4ai. A dedicated Reddit engine returns full comment trees as TOON (~40% fewer tokens than JSON, lossless) with no Reddit API key — and Hacker News, GitHub, and Discourse get the same treatment.

• web_search queries a SearXNG instance (Google/Bing/DDG, Reddit included) and returns ranked URLs with titles and snippets. A site argument scopes results to one hostname — use it instead of naming the site in the query text, which the engines read as a topic word.
• fetch_url renders any URL through crawl4ai as clean markdown — and dedicated engines return TOON instead: Reddit (threads and /r/{sub} listings — which honor the URL's own ?t= time window and ?limit= post count) through a real browser, plus Hacker News, GitHub issues / pull requests, Bluesky posts and profiles, and Discourse topics read straight from their public APIs.

Why omnifeed

	omnifeed	Cloud web MCPs / other Reddit MCPs
Works on Reddit	✅ your residential IP + real browser	❌ datacenter IPs → 403
Web search → crawl in one self-hosted service	✅ SearXNG + crawl4ai	❌ search-only or crawl-only
Full comment tree (/api/morechildren expansion)	✅ up to 40 rounds (~4k comments)	❌
Token-efficient output	✅ TOON, ~40% smaller than JSON	❌ verbose JSON or truncated bodies
Generic crawl fallback for non-Reddit URLs	✅ via crawl4ai	❌
Front-ends	MCP + Open WebUI + REST	MCP only (most)

Quick start

# Fetch the compose file + SearXNG settings, then start:
curl -fsSL https://raw.githubusercontent.com/kinorai/omnifeed/main/docker-compose.yml -o docker-compose.yml
curl -fsSL --create-dirs https://raw.githubusercontent.com/kinorai/omnifeed/main/searxng/settings.yml -o searxng/settings.yml
docker compose up

Starts omnifeed + SearXNG + crawl4ai — tokenless out of the box (the compose file sets OMNIFEED_DEV_NO_AUTH=true), so docker compose up just works. Point Open WebUI at http://localhost:8080 with WEB_LOADER_ENGINE=external. (SearXNG is mounted with searxng/settings.yml, which enables the json format web_search needs.) See Authentication below to require a bearer token.

On Apple Silicon you can skip Docker entirely and run the stack on Apple's native container runtime — see docs/apple-container.md.

As an MCP server

Works with any MCP client — Claude Code, Cursor, Codex, Gemini CLI, OpenCode, Windsurf, Pi, and more. Speaks both the current stateless MCP protocol and the older initialize-era revisions, so old and new clients share the same endpoint. Stateless-protocol requests get the spec's HTTP statuses (400 for header/version violations, 404 for unknown methods); initialize-era responses stay 200, and cross-origin browser requests are rejected unless allowlisted via OMNIFEED_ALLOWED_ORIGINS.

HTTP — recommended. docker compose up already runs the MCP server on :8081 (tokenless in dev mode), so the simplest setup is no extra container at all — point your client at the URL:

{ "mcpServers": { "omnifeed": { "url": "http://localhost:8081/mcp" } } }

Stdio — for clients that only speak stdio. A stdio server is spawned and owned by your client (it pipes JSON-RPC over the process's stdin/stdout), so it can't be a long-running compose service — but you can launch the mcp profile from this compose file, which keeps every setting (upstreams, network, image) in one place:

{
  "mcpServers": {
    "omnifeed": {
      "command": "docker",
      "args": ["compose", "-f", "/abs/path/to/docker-compose.yml", "run", "-T", "--rm", "mcp"]
    }
  }
}

run -T disables the TTY so JSON-RPC pipes cleanly; the container joins the stack's network and reuses crawl4ai/SearXNG. Bring the stack up first (docker compose up -d) so the upstreams are healthy.

Standalone stdio — without the compose stack

Spawn the container directly and tell it where crawl4ai/SearXNG are reachable (omnifeed exits at startup without OMNIFEED_CRAWL4AI_URL):

{
  "mcpServers": {
    "omnifeed": {
      "command": "docker",
      "args": [
        "run", "--rm", "-i",
        "-e", "OMNIFEED_CRAWL4AI_URL=http://host.docker.internal:11235/crawl",
        "-e", "OMNIFEED_SEARXNG_URL=http://host.docker.internal:8080",
        "kinorai/omnifeed:latest", "--mcp-stdio"
      ]
    }
  }
}

On Linux, add "--add-host=host.docker.internal:host-gateway" to the args so host.docker.internal resolves.

Tools: fetch_url (always) and web_search (only when OMNIFEED_SEARXNG_URL is set). The intended loop is web_search → pick URLs → fetch_url.

/crawl returns [{"page_content": "...", "metadata": {...}}] — already the shape of a LangChain / LlamaIndex Document, so wrapping it as a custom document loader takes only a few lines.

Authentication

The compose stack runs tokenless for local use (OMNIFEED_DEV_NO_AUTH=true). To require a bearer token instead, generate one — this is the value clients send as Authorization: Bearer <token>, so copy it:

openssl rand -hex 32        # ← your token; copy this

Then in docker-compose.yml set OMNIFEED_API_KEY to that value and remove OMNIFEED_DEV_NO_AUTH. Without a key (and without OMNIFEED_DEV_NO_AUTH=true) the proxy refuses to start, so it can't be left open by accident. Stdio MCP needs no token — it inherits the trust of the process that spawned it.

Configuration

Everything is configured with OMNIFEED_-prefixed environment variables. In practice you only ever set three — OMNIFEED_API_KEY, OMNIFEED_CRAWL4AI_URL, and (optionally) OMNIFEED_SEARXNG_URL. The rest have sane defaults.

The full reference lives in docs/configuration.md — every variable, content-size control (resumable max_chars / start_char truncation on fetch_url), infinite-scroll fetching, Reddit size knobs, and Prometheus metrics.

Running more than one replica? Set OMNIFEED_REDIS_URL and the rate limiters share their state through Redis, so the whole deployment obeys one limit instead of one limit per pod (N replicas otherwise send N times the configured rate, which is what upstream search engines notice). It is opt-in and fail-open: unset, every replica paces in its own memory exactly as before, and if Redis becomes unreachable the limiters fall straight back to that in-process pacing rather than failing a crawl.

Architecture

%%{init: {"theme":"base","themeVariables":{"background":"transparent","mainBkg":"#161b22","primaryColor":"#161b22","primaryTextColor":"#e6edf3","primaryBorderColor":"#FF4500","lineColor":"#8b949e","secondaryColor":"#161b22","tertiaryColor":"#161b22"},"flowchart":{"curve":"basis","htmlLabels":false}}}%%
flowchart TB
  crawl["POST /crawl"] e1@--> owt["Open WebUI<br/>transport"]
  search["POST /search"] e2@--> sat["SearchAPI<br/>transport"]
  mcpStdio["MCP stdio"] e3@--> mcp["MCP server"]
  mcpHTTP["MCP HTTP /mcp"] e4@--> mcp

  owt e5@--> reg["Engine Registry"]
  mcp -- crawl tools --> reg
  sat e6@--> searcher["Searcher<br/>(SearXNG)"]
  mcp -- search tool --> searcher

  reg e7@--> reddit["Reddit engine<br/>(TOON)"]
  reg e12@--> hn["Hacker News engine<br/>(TOON)"]
  reg e14@--> gh["GitHub engine<br/>(TOON)"]
  reg e16@--> disc["Discourse engine<br/>(TOON)"]
  reg e8@--> generic["Generic fallback<br/>(markdown)"]
  reddit e9@--> c4["crawl4ai upstream<br/>(headless browser)"]
  generic e10@--> c4
  hn e13@--> algolia["Algolia HN API<br/>(hn.algolia.com)"]
  gh e15@--> ghapi["GitHub REST API<br/>(api.github.com)"]
  disc e17@--> discapi["Discourse topic JSON<br/>(allowlisted forums)"]
  searcher e11@--> sx["SearXNG upstream<br/>(Google / Bing / DDG)"]

  classDef box fill:#161b22,stroke:#30363d,stroke-width:1px,color:#e6edf3;
  classDef accent fill:#0d1117,stroke:#FF4500,stroke-width:2px,color:#ffd9b3;
  classDef animate stroke:#FF4500,stroke-width:2px,stroke-dasharray:10 6,stroke-dashoffset:900,animation:dash 14s linear infinite;
  class crawl,search,mcpStdio,mcpHTTP,owt,sat box;
  class mcp,reg,searcher,reddit,hn,gh,disc,generic,c4,sx,algolia,ghapi,discapi accent;
  class e1,e2,e3,e4,e5,e6,e7,e8,e9,e10,e11,e12,e13,e14,e15,e16,e17 animate;

Reddit anti-bot handling

Reddit's edge 403-blocks non-browser HTTP clients, so the Reddit engine never calls Reddit directly: it drives a real headless browser to a www.reddit.com page, then fetches Reddit's JSON from inside that page — no auth, cookies, or API key. Sustained scraping can still raise your source IP's risk score, so slow down if fetches start returning the block page. Details and tuning: docs/configuration.md.

Extending it

New engines (Hacker News, Stack Overflow, …), searchers (Brave, Tavily, …), MCP tools, and transports each plug into one small port without touching the rest. See AGENTS.md → Adding things for the architecture and a step-by-step.

Development

git clone https://github.com/kinorai/omnifeed.git && cd omnifeed
make check        # vet + lint + test — hermetic, no upstreams or token needed
docker compose up # run the full stack locally (tokenless: ports 8080 / 8081 / 9090)

See CONTRIBUTING.md for the full workflow, SECURITY.md for vulnerability reporting, and AGENTS.md if you're a coding agent working in this repo.

Prometheus metrics are served on :9090/metrics — the full metric table is in docs/configuration.md.

Contributing

Star it if it's useful — it helps other AI builders find omnifeed.

New engines, searchers, MCP tools, and transports are all welcome — start with AGENTS.md and CONTRIBUTING.md.

License

MIT © kinorai