feedmill

README

feedmill

English | 中文 | 日本語

feedmill：an open-source, zero-dependency CLI that normalizes RSS, Atom, and JSON Feed into deduplicated JSONL with canonical UTC timestamps — feed-format hell handled once, so your pipeline never parses a broken date again.

git clone https://github.com/JaydenCJ/feedmill && cd feedmill
go build -o feedmill ./cmd/feedmill    # single static binary, stdlib only

Pre-release: v0.1.0 is not tagged on a package registry yet; build from source as above (any Go ≥1.22).

Why feedmill?

The RSS revival met data pipelines, and everyone scraping feeds now re-solves the same three problems: dates published as Thur, 05 Feb 2026 10:00:00 EST (misspelled weekday, named zone that Go and many libraries silently parse as +00:00), the same article appearing under three GUIDs and two utm_-suffixed URLs, and "UTF-8" documents full of raw windows-1252 bytes and double-escaped titles. Parser libraries like feedparser and gofeed give you a struct per feed and stop there — date repair is partial, URLs stay dirty, deduplication and output are your problem. Reader apps solve it internally but keep the clean data inside their database. feedmill is the missing pipeline stage: files or stdin in, one JSON object per item out, every timestamp RFC 3339 UTC, every URL canonical, every duplicate collapsed under an evidence-prefixed key — and every repair it made reported to stderr, so you can audit what changed.

	feedmill	feedparser (Python)	gofeed (Go)	reader apps (Miniflux etc.)
Pipeline-ready JSONL in one command	✅	❌ library	❌ library	❌ app + database
RSS 0.9x/1.0/2.0 + Atom + JSON Feed	✅	✅	✅	✅
Broken dates repaired to canonical RFC 3339 UTC	✅	partial, keeps struct time	partial, keeps raw strings	internal only
Cross-feed dedupe with canonical URLs	✅	❌	❌	per-feed only
Reports every repair it applied	✅	❌	❌	❌
cp1252-in-UTF-8 and double-escape recovery	✅	partial	❌	varies
Offline, no server, no database	✅	✅	✅	❌
Runtime dependencies	0	1 PyPI package	6+ Go modules	full application

_{Dependency counts checked 2026-07-13: feedmill imports the Go standard library only; feedparser pulls sgmllib3k; gofeed's go.mod lists goxpp, goquery, and friends.}

Features

• Broken dates handled once — misspelled weekdays, named zones resolved against a real offset table (EST means -05:00, not Go's silent +00:00), GMT+0200, missing seconds, two-digit years, bare dates, Unix epochs; everything lands as fixed-width RFC 3339 UTC, so string order is time order.
• Encoding chaos survived — UTF-16 BOMs, latin-1 declared or lied about, control bytes, HTML entities inside XML, —-style cp1252 references, and double-escaped titles repaired with a guard that never over-decodes AT&T.
• Honest deduplication — keys are prefixed with their evidence (guid: > url: > hash:); URL-shaped GUIDs are canonicalized so http→https migrations don't duplicate history, and --key url merges syndicated copies across feeds.
• Tracking-free canonical URLs — utm_* and a curated parameter list stripped (tunable), query keys sorted, fragments and default ports dropped, relative links resolved against the feed.
• Repairs are reported, not hidden — every fix goes to stderr with the original string and the result; inspect summarizes a feed's data quality before you wire it in, and --strict turns lossy repairs into exit code 1.
• Pipeline-native output — one JSON object per line, fixed field order, HTML not \u-escaped, chronological sort, --since/--until windows; jq-friendly but grep-sufficient.
• Zero dependencies, fully offline — Go standard library only; reads files and stdin, writes stdout. Fetching is your scheduler's job. No telemetry, no network, ever.

Quickstart

go build -o feedmill ./cmd/feedmill
./feedmill inspect examples/messy-rss2.xml

Real captured output:

examples/messy-rss2.xml
  format:     rss2
  feed:       Grist & Grain — https://grist.example.test
  items:      4 (3 dated, 2026-02-05 → 2026-02-11)
  ids:        4 guid, 0 url-keyed, 0 hashed
  repairs:    3 timestamps repaired, 1 dropped
  warnings:   5
    - item 1: published "Thur, 05 Feb 2026 10:00:00 EST" repaired (nonstandard weekday "Thur" dropped; named zone "EST" resolved to -0500) -> 2026-02-05T15:00:00Z
    - item 2: repaired double-escaped title
    - item 2: published "Mon, 09 Feb 2026 08:30 +0000" repaired (nonstandard layout 2 Jan 2006 15:04 -0700) -> 2026-02-09T08:30:00Z
    - item 3: published "2026-02-11" repaired (assumed UTC (no zone given)) -> 2026-02-11T00:00:00Z
    - item 4: published dropped: unparseable timestamp "sometime last week"

Normalize it (./feedmill normalize --quiet examples/messy-rss2.xml | head -2, real output):

{"id":"guid:grist-0001","feed":{"title":"Grist & Grain","url":"https://grist.example.test","format":"rss2"},"title":"Stone mills, revisited","url":"https://grist.example.test/posts/stone-mills","published":"2026-02-05T15:00:00Z","authors":["Miller Jones"],"tags":["milling"],"summary":"Why the old ways still grind finest.","content_html":"<p>Why the old ways still <b>grind</b> finest. A longer body follows.</p>","language":"en-us"}
{"id":"guid:grist-0002","feed":{"title":"Grist & Grain","url":"https://grist.example.test","format":"rss2"},"title":"It’s harvest week","url":"https://grist.example.test/posts/harvest-week","published":"2026-02-09T08:30:00Z","summary":"Double-escaped title, relative link, missing seconds — a classic.","language":"en-us"}

Merge three feeds of different formats, collapsing the syndicated duplicate:

./feedmill normalize --key url --quiet --stats examples/messy-rss2.xml examples/blog.atom examples/micro.json > all.jsonl
# stderr: feedmill: feeds=3 parse_errors=0 items=7 duplicates_removed=1 filtered_out=0 date_repairs=4 dates_dropped=1 hash_keys=0 bad_urls=0

CLI reference

feedmill [normalize|inspect|version] [flags] [file ...] — normalize is the default; no file (or -) reads stdin. Exit codes: 0 ok, 1 strict violation, 2 usage error, 3 runtime error.

Flag	Default	Effect
--since / --until	—	keep items published inside the UTC day window (YYYY-MM-DD)
--key	auto	dedupe key: auto (guid > url > hash), url, or guid
--no-dedupe	off	keep duplicate items
--sort	published	published (chronological) or none (input order)
--require-date	off	drop items without a parseable published timestamp
--max-items	0	emit at most N items after sorting (0 = unlimited)
--keep-param / --strip-param	—	tune the tracking-parameter list (repeatable)
--stats	off	print a one-line run summary to stderr
--quiet	off	suppress per-item repair warnings
--strict	off	exit 1 on parse failures, dropped dates, or hash keys
--max-warnings (inspect)	5	warnings shown per feed (0 = all)

The output schema — field meanings, key prefixes, URL and date rules — is documented in docs/record-format.md.

Verification

This repository ships no CI; every claim above is verified by local runs:

go test ./...            # 90 deterministic tests, offline, < 5 s
bash scripts/smoke.sh    # end-to-end CLI check, prints SMOKE OK

Architecture

flowchart LR
    IN[(feeds / stdin)] --> ENC[charset: repair to UTF-8]
    ENC --> DET[detect: sniff format]
    DET --> P{parse}
    P --> RSS[rss 0.9x / 1.0 / 2.0]
    P --> AT[atom 0.3 / 1.0]
    P --> JF[json feed 1.x]
    RSS --> N[normalize]
    AT --> N
    JF --> N
    N --> DA[dates: RFC 3339 UTC]
    N --> UR[urlnorm: canonical URLs]
    N --> HT[htmltext: entities and tags]
    DA --> K[record: derive keys]
    UR --> K
    HT --> K
    K --> DD[dedupe + sort]
    DD --> OUT[JSONL on stdout]

Roadmap

• v0.1.0 — RSS/RDF/Atom/JSON-Feed parsing, timestamp repair to RFC 3339 UTC, encoding recovery, canonical URLs, prefixed dedupe keys, JSONL output with windows/sorting/strict gate, inspect, 90 tests + smoke script
• --dedupe-state FILE for incremental runs (remember keys across polls)
• OPML input to normalize a whole subscription list in one call
• --format ndjson-envelope with per-run metadata records
• Extension-field passthrough (podcast namespaces, media RSS) behind --keep-ext
• Fuzz corpus grown from real-world feed pathologies

See the open issues for the full list.

Contributing

Issues, discussions and pull requests are welcome — see CONTRIBUTING.md for the local workflow (format, vet, tests, SMOKE OK). Good entry points are labelled good first issue, and design questions live in Discussions.

License

MIT