README
¶
DOWNLINK
A feed aggregator and content-analysis platform for security news. DOWNLINK collects RSS/Atom feeds, scrapes full article content, runs each article through an LLM for rubric-based scoring and summarization, and assembles ranked digests that it can publish to GitHub Pages and Discord.
feeds → scrape → LLM analysis & scoring → digest → publish (GitHub Pages / Discord)
Features
- Multi-provider LLMs — Anthropic Claude, OpenAI, vLLM, Ollama, and llama.cpp (any OpenAI-compatible endpoint). Configure several and pick one per run.
- Rubric scoring — articles are scored on six dimensions (specificity, severity,
breadth, novelty, actionability, credibility) rather than a single opaque number.
See
pkg/scoring. - Flexible scraping — plain RSS, dynamic scraping (Lightpanda), and full-browser scraping (Solimen) with per-feed CSS selectors, triggers, blacklists, and headers.
- Digest generation — categorized, importance-ranked digests with deduplication.
- Publishing — push static digests to GitHub Pages and notify a Discord webhook.
- Interfaces — gRPC API (default
:50051), an Atom feed export (:65261), and thedlkcommand-line client. - Storage — embedded SQLite via GORM; no external database required.
Architecture
| Component | Path | Description |
|---|---|---|
| Server | cmd/server |
gRPC services + feed manager + Atom feed export. The core daemon. |
CLI (dlk) |
cmd/dlk |
gRPC client for articles, feeds, analysis, digests, config, queue. |
| Shared packages | pkg/ |
Scoring, LLM gateway/providers, codex auth, models, protobufs. |
The server exposes gRPC services for articles, analysis, feeds, digests, queue,
config, and auth. The Atom feed server publishes analyzed articles at
http://localhost:65261.
Install
With Go 1.25+:
go install github.com/ma111e/downlink/cmd/dlk@latest # CLI
go install github.com/ma111e/downlink/cmd/server@latest # server
Or build from source:
make all # builds server + cli
make server # build just the server (embeds the git commit)
make cli # build just dlk
Or with Docker:
docker build -t downlink .
docker run --rm -p 50051:50051 -p 65261:65261 \
-v "$PWD/config.json:/app/config.json" \
-v "$PWD/feeds.yml:/app/feeds.yml" \
downlink
A docker-compose.yml is provided that also wires up the optional Solimen
full-browser scraper.
Configuration
DOWNLINK reads configuration from three sources. Copy the bundled examples and fill in your values:
cp config.example.json config.json # LLM providers, analysis, notifications
cp feeds.example.yml feeds.yml # feed sources and per-feed scraping rules
cp .env.example .env # runtime/env overrides
config.json— LLM provider definitions (name, type, model, base URL, API key), the active analysis provider and persona/writing style, and notification settings (Discord webhook, GitHub Pages repo/token/branch). See config.example.json.feeds.yml— the list of feeds with per-feed scraping strategy, CSS selectors, triggers, blacklists, and custom HTTP headers. See feeds.example.yml..env/ environment — every server flag has aDOWNLINK_*env var equivalent (loaded via Viper). See .env.example. Precedence: CLI flag → env/.env→config.json→ default.
Secrets:
config.json,.env, andfeeds.ymlare gitignored. Never commit real API keys, tokens, or webhook URLs.
Running
Start the server:
./server --refresh # fetch all feeds on startup
./server --port 50051 --log-level info
Use the CLI against a running server:
dlk feed list
dlk article list
dlk analysis run
dlk digest generate
dlk --help
Check versions:
./server --version
dlk --version
Publishing digests
DOWNLINK can publish generated digests to GitHub Pages and announce them on Discord.
See docs/github-pages.md for the full setup, including
--init-gh-pages and the required token scopes.
Deployment
A sample systemd unit is provided at etc/downlink.service
for running the server under /opt/downlink.
License
MIT © 2026 ma111e
Contributions are not being accepted at this time.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
dlk
command
|
|
|
server
command
|
|
|
server/notification
Package notification re-exports the GitHub Pages publisher from the internal notification package so that packages outside cmd/server (such as the CLI) can use it without violating Go's internal-package import restrictions.
|
Package notification re-exports the GitHub Pages publisher from the internal notification package so that packages outside cmd/server (such as the CLI) can use it without violating Go's internal-package import restrictions. |
|
pkg
|
|
|
llmgateway
Package llmgateway is the single chokepoint through which every LLM call in the process must pass.
|
Package llmgateway is the single chokepoint through which every LLM call in the process must pass. |
|
llmprovider
Package llmprovider exposes a single Provider interface that every LLM backend in the project must satisfy.
|
Package llmprovider exposes a single Provider interface that every LLM backend in the project must satisfy. |
|
llmutil
Package llmutil holds small helpers shared between LLM-calling code paths.
|
Package llmutil holds small helpers shared between LLM-calling code paths. |
|
openaicompat
Package openaicompat provides a lightweight client for any OpenAI-compatible chat completions API (llama.cpp, vLLM, LM Studio, Ollama /v1, etc.).
|
Package openaicompat provides a lightweight client for any OpenAI-compatible chat completions API (llama.cpp, vLLM, LM Studio, Ollama /v1, etc.). |
|
scoring
Package scoring owns the article importance model: the rubric dimensions an LLM rates, the deterministic aggregation of those dimensions into a 0-100 score, and the tier thresholds used to bucket articles in digests.
|
Package scoring owns the article importance model: the rubric dimensions an LLM rates, the deterministic aggregation of those dimensions into a 0-100 score, and the tier thresholds used to bucket articles in digests. |
|
trace
Package trace is an opt-in, content-level debug tracer.
|
Package trace is an opt-in, content-level debug tracer. |