usenet

README

usenet plugin

A self-contained Usenet indexer. It crawls a curated set of newsgroups, stages article overviews until a release's parts are all present, assembles complete sets into gzipped NZB files, and serves search / group-listing / NZB download through capabilities the host's pages consume. It also health-checks the catalogue over time (are the articles still on the server?) and filters machine-generated junk at ingest.

Users see it two ways. Operators get one admin page at /admin/p/usenet (loon SlotAdminPage), tabbed: the provider fleet (with per-row connection test), indexing knobs, newsgroup curation, a live Crawlers dashboard (a slim provider strip with live dial state, newsgroup coverage bars, live crawl progress with the group-by-group bar, index stats, recently built releases, an aggregate backfill ETA, worker panels), a Jobs tab (one pane per pipeline job: status, next run, Run-now, a live log tail, plus the Builder and NZB-health panels), and Filters — the operator blacklist plus per-rule hit counters. End users get whatever the host builds on the published index capability — search results, group browse, and a Newznab/Torznab /api + /rss endpoint.

The plugin runs on one host or several at once. Multiple providers crawl in parallel and complete each other's releases through shared staging; multiple worker hosts divide the newsgroups between them without double-crawling — and a worker that loses a lease mid-pass (deploy takeover, expiry) cancels that pass immediately rather than overlapping the new owner's writes. It can own its own minimal catalogue (standalone / demo) or, in host-sink mode, hand every assembled release to a rich host's NZB domain — the seam by which a production site adopts this plugin in place of an in-tree crawler.

Surface

Routes:

• admin (Core.Router.Admin("usenet"), RequireRole(Admin) applied by the host): GET /admin/plugin/usenet/status.json — machine-readable crawl status (both passes, fleet, workers, staging/build depth, recent errors) for monitors and scripts. Carries provider hostnames and error text, so it is admin-gated, not public. Note for monitors: since 2026-07 active_groups counts active newsgroups (it used to count per-(backbone, group) state rows, so the value dropped on multi-backbone installs at that deploy), the total_nzbs/staged_articles figures are planner estimates, not exact counts, and the never-assigned pending_releases/ready_releases fields (constant 0 for their whole life) were removed. Accurate from ANY process: the worker publishes its in-memory pass trackers + error ring to the shared settings table every few seconds (worker_telemetry), so a split web/worker deployment serves live numbers too — the Crawlers tab polls this endpoint to tick in place.
• admin views (registered via Core.RegisterView, mounted by the host):
  • SlotAdminPage slug usenet — the single tabbed admin page (/admin/p/usenet): the provider fleet (per-row save/test/remove), indexing knobs, newsgroups, the Crawlers dashboard, the Jobs tab and the Filters blacklist. Actions: knobs, fetch-groups, group, provider, provider-del, provider-test, provider-probe (backbone fingerprint: compares article numbering against the reference provider via STAT), group-tune, group-move, group-del, groups-purge, crawl, backfill, run-crawl, run-backfill, run-build, run-tagfill, run-prune, run-health, reset-backfill, junk-move, junk-order, junk-toggle (the junk-rule ORDER editor: rules listed in evaluation order with lifetime hit counts, share and drift, so a high-volume rule sitting late is visible — see PIPELINE.md §6), filter-add, filter-toggle, filter-del, filter-reset, poster-watch-add, poster-watch-del. Each action redirects back to its own tab.
  • SlotJobsWidget anchor Usenet — a richer card for the Usenet job group on the host's /admin/jobs.
• public / api: the plugin publishes read capabilities rather than mounting its own public routes — the host mounts /api + /rss (Newznab/Torznab) and the search/browse/download pages, delegating to UsenetIndexName. Auth on those is the host's to enforce (the demo leaves the API open; a real host checks an apikey against its user store).

Process kinds (Metadata.Processes): web, worker, api.

• web / all: registers the index + newznab + admin capabilities and the admin views.
• worker / all: registers and runs the six jobs (below); no view system. The worker ALSO registers the admin capability — a host's worker-side stats-cache job reads Stats() through it for its public stats page.
• api: registers the index + newznab capabilities — no jobs, no admin surface.

See PIPELINE.md for the article flow end to end — buffers, cadences, measured timings, and where the cost actually sits. Read it before changing the crawl or build path.

Data

Owns the usenet Postgres schema (loon scopes search_path to it; unqualified names in migrations resolve there). Migrations 001–028, embedded via //go:embed migrations/*.sql, run under loon's plugin-migration path on boot. Every statement is IF NOT EXISTS / idempotent.

Principal tables:

• servers — configured NNTP providers (host, port, TLS, creds, role, priority, connections, backbone).
• newsgroups — the curated group list + per-group tuning (active, crawl-depth override, throttle, priority tier, manual order).
• newsgroup_state — crawl state keyed (backbone, group): watermarks, server bounds, backfill-done. Article numbers are per-backbone, so state cannot be shared across backbones (see below).
• newsgroup_ranges — fetched-article-number coverage per (backbone, group), for gap-filling backfill.
• articles — the staging buffer (pg mode); drained by the builder, swept by prune. Redis mode keeps this transient set in Redis instead.
• nzbs — assembled releases (internal sink mode only; host mode writes the host's table and never touches this one).
• junk_rules — the junk-filter rule set, seeded from seed/junk_rules.tsv.
• blacklist_regexes — the operator's editorial blacklist.
• filter_hits — two populations under one table, read separately. Rule counters (kind = junk/blacklist) are one row per configured rule: bounded, lifetime totals, never pruned, read whole for the Filters tab's rule card. Instrument counters (ungrouped, merge_suspect, parse_dropped) are one row per distinct observation — unbounded: the grouping watch's first day produced 2,260 stems against 26 rules. Those are paged, filtered by kind, and pruned by last_seen_at on the diagnostic horizon. Reading them together made the page's cost grow with the diagnostics and buried the rules; an operator reasonably read the merged list as "we now have 100k rules". Migration 029 indexes both access paths.
• poster_watch / poster_hits — the same accounting asked the other way round: filter_hits answers "which rule drops the most", which is right when tuning rules and useless when an operator says "this poster puts out a hundred releases a day and I have four of them". A watched pattern is matched case-insensitively against each article's From, and every outcome for it is tallied per (poster, stage, reason) — including the SUCCESSES, because "nothing at all for this poster" and "all of it junked at ingest" look identical from the outside and have opposite fixes. Migration 023.
• build_outcomes — per-day, per-reason counts of what the build pass did with every candidate set (built, incomplete, duplicate, junk, blacklist, blocked_ext, empty, and the four error reasons), plus one sample subject each. Where filter_hits attributes a drop to a rule, this accounts for every candidate — including the two outcomes the pass never used to report, incomplete and duplicate, which are usually the largest. Bucketed by day rather than all-time because the question is almost always "what changed".
• staging_census — one row per build pass covering the stretch between "the articles are staged" and "a release exists", which was previously unobservable end to end. Three different mechanisms can destroy a completed release in that gap — Redis evicting keys under maxmemory, the staging TTL, and a ready queue deeper than the per-pass draw — and all three remove work silently, so a release could stage, complete, queue itself and vanish without touching build_outcomes, the job log, the error log or any counter. The columns are cheap (one INFO plus two O(1) reads) and the signal is in the DELTA between rows: evicted_keys climbing means Redis is destroying staged work; ready_depth far above sampled pass after pass means arrivals outpace the drain; fossil_dropped counts releases that completed and then expired before the builder drew them. Rendered as the Staging health card on the Jobs tab, and pruned to 14 days. Migration 024.
• leases, crawler_workers — multi-host coordination.

Reads public.newsgroups / public.blacklist_regexes once, at host adoption only (see Lifecycle), and never again.

Dependencies

Core services consumed: Storage (SchemaDB("usenet")), Scheduler (the six jobs + their loops), Router (the admin status route), Logger, Errors (error sink), and Config (plugins.usenet.*). Redis is consumed optionally — only when staging: redis is configured; its absence in that mode is a boot refusal, not a silent fallback.

Store: self-contained — builds a *PGStore over SchemaDB("usenet") at Provision. No SetDeps.

Config keys (plugins.usenet.*, all optional — the wizard/knobs own the live values; these are defaults):

• server.* — seed a provider on first boot if the table is empty.
• staging — pg (durable, default) | redis (prod's pipeline, needs Redis). Redis is the tuned path (inline completeness, hopeless-set eviction, O(1) ready queue, staging_ttl_hours knob). The pg backend is CORRECT and its hot write path is batched (stageArticles inserts through a chunked unnest INSERT); prefer redis when crawling a full feed, pg when durability matters more than throughput. One known pg cost: the SQL candidate pre-filter is looser than the in-memory completeness check (it cannot see per-file segment totals), so a multi-file set the SQL admits but the check refuses is re-drawn — articles re-loaded — every build pass until it completes or ages out on the prune horizon.
• sink — internal (own nzbs table, default) | host (hand releases to the host's ReleaseSink; requires the host to register the sink + health capabilities). A production host pins this to host.
• Numeric knobs (crawl cadence, batch size, connection count, retention/crawl depth, backfill budget, lease TTL, assignment term, health cadence, staging pressure thresholds, …) — all overridable live from the admin knobs form.

Metadata.Requires: none. The catalog capability is consumed optionally (nil-degrades to Newznab category Other).

Hooks & Callbacks

Host hooks SET (web/handlers/plugin_hooks.go): (none) — this is a self-contained plugin; it exposes itself through the capability registry and RegisterView, not host func-var hooks.

Extensions PUBLISHED (Core.Register):

• pluginapi.UsenetIndexName (usenet.index) — the read surface: search, group listing, NZB fetch, Newznab query. The host's public pages + API consume this.
• pluginapi.UsenetAdminName (usenet.admin) — the admin/service surface (manual job triggers, wizard operations, crawl-progress stats). Registered in web/all AND worker (see process kinds above).
• pluginapi.UsenetNewznabName (usenet.newznab) — the whole Newznab/Torznab XML contract (caps / search / rss / get); the host mounts /api + /rss and delegates the parsed request here.
• pluginapi.UsenetActivityName (usenet.activity) — counts-only crawler liveness (current/last pass articles, staged, batches, wire bytes) sourced from the published worker telemetry. Sanitized for non-admin surfaces — no group names, hostnames, or error text — so a host can drive a public stats-page live widget from it. Registered in every process.
• pluginapi.RegisterStats — contributes indexer totals to the host's stats snapshot.

Extensions CONSUMED (Core.Lookup):

• pluginapi.CatalogName — optional; categorises releases for Newznab. Absent → category Other.
• pluginapi.UsenetReleaseSinkName / UsenetHealthStoreName — the host's NZB domain, in sink: host mode only. The plugin publishes the contracts (ReleaseSink, ReleaseHealthStore in pluginapi) and consumes the host's implementations: assembled releases go out through the sink, health candidates/verdicts flow through the health store. Host mode without them is a loud refusal. ReleaseHealthStore also carries ClearHealthRecheckRequest, for the one check outcome that writes neither a verdict nor a touch — see "How the health sweep gives up" below.
• pluginapi.UsenetCatalogStatsName — optional, sink: host only: catalog totals, the health breakdown, and the per-group release census, for the dashboard's Index Stats and NZB Health cards and the Newsgroups table's NZBs column (in host mode the releases and verdicts live in the host's domain, invisible to the plugin's tables). A host should serve it from CACHED numbers; absent, those cards degrade to empty and the per-group column hides. CatalogStats.PerGroup is separately optional — nil means "this host does not answer", which hides the column, where an empty map means "answered: every group has zero" and would declare the fleet dead. A group absent from the map has produced nothing, which is the point: a fossil group is crawled every pass like any other, so a fresh last_crawl says nothing, and the release count is the only thing that separates "quiet" from "dead".
• pluginapi.UsenetJunkSweepName — optional: the host's stored-catalogue junk-sweep attribution counters, shown as a third card on the Filters tab (ingest hits say what was dropped; the sweep says what got past ingest and had to be tagged afterwards). Absent, the card is hidden.

Events: emits pluginapi.EventIngested after a build pass creates releases, so a host subscriber (e.g. a cache invalidator) can react. Best-effort — no host event bus means no-op.

Lifecycle

Provision (all processes): builds the *PGStore, reads config + applies defaults, validates the sink mode (an unknown value fails boot rather than silently splitting the catalogue), constructs the staging backend behind the stagingStore seam, and registers the per-process capabilities (see process kinds). The admin status.json route registers on web/all only. The optional catalog plugin is looked up in Start (provision order isn't guaranteed), in every process.

Start (worker/all only): seeds a provider from config if the table is empty; runs host adoption (adopt.go) once in host-sink mode — carrying a legacy in-tree crawler's newsgroups, watermarks (into per-backbone state, deriving backfill-done from the legacy schema's convention), per-group tuning, and blacklist so a production flip resumes rather than restarts; starts the heartbeat (multi-host presence); seeds junk rules + the curated newsgroup pack; and launches the six job loops.

The six jobs (registered on the worker, all carrying the "Usenet" prefix so they never collide with a host's own job names in the shared registry): Usenet Crawler (forward-fetch recent overviews; keeps going without waiting for the interval while the servers still hold a backlog, unless crawl_no_catchup is set), Usenet Backfill (walk history downward, filling coverage gaps), Usenet Builder (assemble complete staged sets, filter, store through the sink), Usenet Tag Fill, Usenet Prune (stale-staging + junk sweep; NZB retention is opt-in, default keep-forever), Usenet Health Check (STAT segments, record healthy/broken/dead).

Stop: no-op — the jobs derive from the root context and unwind on SIGTERM; leases expire (or are taken over by a replacement worker once the heartbeat goes stale).

Architecture notes

Two facts shape most of the design:

• Article numbers are per-backbone; message-ids are global. Watermarks, coverage ranges, and backfill state are keyed by backbone because two backbones number the same articles differently — merging them would silently skip real content. Staging dedup, by contrast, is keyed by message-id, so two providers on different backbones complete each other's releases. Content identity (content_hash) is sha256(sorted message-ids)[:16] — the same scheme the host uses, so dedup is shared across the plugin and the host at adoption.
• Assembly is delivery-agnostic below "the bytes arrived." Everything the builder does after a set is complete — junk/blacklist filtering, title + category classification, NZB XML — is identical whether the release lands in the plugin's table or the host's. Only the final store step differs, which is exactly the ReleaseSink seam.

Coordination across hosts is two layers: leases (a row with an expiry per (backbone, group) or per job) guarantee no two workers crawl the same thing; term-based assignment (heartbeat presence + hash partitioning within a fixed time term) divides the groups so workers don't merely race for the leases. A worker that joins mid-term waits for the next boundary. Exclusivity is heartbeat-backed: a lease whose owner has written no heartbeat for two minutes is claimable IMMEDIATELY, regardless of expiry — a deploy renames the worker (container hostname), so without takeover every deploy idled the crawler until the TTL lapsed. The catch-up loop retries an all-blocked pass every 45s to ride out the takeover window.

Subjects are RFC 2047 decoded at ingest, before anything reads them. A subject is a raw header, and a poster writing outside ASCII sends =?UTF-8?q?...?=; undecoded, the title is unparseable — a base64 encoded-word can even swallow the yEnc segment counter — and reads as punctuation soup to the junk engine, which had dropped 41.6 million articles that way. Charsets Go has no table for are passed through unconverted (mojibake in the right release beats an absent release), except the stateful ISO-2022/UTF-7 family, where doing so spills ESC bytes into the title and measurably makes things worse; those keep the raw header.

high_special_chars ("the title is mostly spam-grade punctuation") judges letters and digits with Unicode, not ASCII, and takes its ratio over runes rather than bytes. Deciding it the ASCII way counted every CJK ideograph, kana and Hangul syllable as punctuation, so a native-script title scored ~100% special and was dropped at ingest — on an anime indexer that excluded exactly the Japanese-titled releases the catalogue most wants, while the rule's counter climbed as if it were working. The host-side mirror in pkg/services/junk_title.go carries the same fix (the SQL sweep clause counts a fixed ASCII set and was never affected).

Three further corrections came out of running the engine over 20,000 titles already in the production catalogue (TestJunkCorpusAudit, which skips unless USENET_AUDIT_CORPUS is set): a trailing {Tags:L0;A=ja,en;S=en,ar;} metadata block is stripped before the ratio (it is ;=, almost end to end, so a long language list read as soup); only ASCII punctuation counts, because the obfuscation bots work in ASCII while 【】（）｜ are ordinary structure in their own script; and a RUN of the same mark counts once, since Keijo!!!!!!!! and New Game!! are title styles rather than garble. Together those took the false-positive rate from 96 titles to 27 — 23 of the remainder are the deliberate under_1mib/under_5mib size bands catching subtitle packs, manga and audio rather than a rule defect. Run that audit before changing a junk rule; none of these were visible by inspection.

Each staged set records the article-number span it covers (art_lo/art_hi in its meta, folded per batch by a Lua min/max script so out-of-order and descending-backfill arrival still record true bounds). Article numbers ascend with posting time and one upload happens in a single run, so a real release is near-contiguous even with other posters interleaved. A set spanning a million or more article numbers — roughly half a day of posting on a busy group — is therefore not one release but several unrelated posts that collided on the same base subject, and it can NEVER complete: it is waiting on files belonging to somebody else's upload. The forming-releases card shows the span and flags those as collision, which is a different fact from "incomplete" and calls for a different fix (tighten the base derivation, not wait longer).

The threshold is absolute rather than a ratio of articles held. Scaling it with Have — the first attempt — flags every set early in its arrival, when it holds few articles but already covers a real range, which is precisely the window an operator is watching. A test with a genuine four-article release caught that.

The forward crawl yields when staging is full, at crawl_pressure_high_pct (default 95, deliberately above the backfill's 85: new articles matter more than history, so the crawl stops only when storing would actively destroy what is already there). Under redis with an allkeys-* policy at its ceiling every write evicts something to make room, and the coldest keys are precisely the forming sets waiting between crawl visits — production evicted 97.4 million keys that way, roughly 640 staged releases a minute, while the crawler kept feeding it. Skipping the round leaves the watermark where it is, so nothing is skipped: those articles are re-read next pass. Storing them into a full backend is what loses them. Set to 0 to disable, which is right on a backend that cannot destroy what it holds (pg staging, or redis under noeviction, where a full server refuses the write instead).

The ready queue is swept before each draw. candidateGroups takes a random sample of build_drain_per_pass entries, and nothing else ever removed the dead ones — so a queue growing faster than the draw accumulates fossils without bound and dilutes itself. Production reached 7,403,408 entries against a 500-entry draw, 407 of every 500 already dead: a completed release had roughly a 1-in- 15,000 chance of being picked in a pass, and its articles expire after two hours. That is not a queue, it is a lottery nobody wins. reapReadyQueue SSCANs a bounded slice per build ROUND (ready_reap_per_pass, default 50,000 — the key name predates the round/pass split), pipelines EXISTS against each entry's grp: metadata, and SRems the dead — the same liveness definition candidateGroups already used, applied by something that runs often enough to matter. Bounded, with a cursor that persists across rounds, so a multi-million- entry queue is worked down instead of stalling the round clearing it.

Staged sets that can never complete are evicted by the walk-past sweep (same round cadence): a set still short of its claimed totals, idle past walk_past_grace_min (default 15), whose whole recorded article span (art_lo/art_hi) lies inside the group's fetched coverage (newsgroup_ranges) has been offered every article it could ever receive — absence is final, and every hour it waits for the staging TTL is an hour of memory held against the backfill's pressure gate. walk_past_sweep_per_round (default 2,000) bounds each round's examination with persistent per-group cursors; groups covered on more than one backbone are skipped (article numbers are per-backbone, so a mixed span cannot be judged); walk_past_no_evict disables the sweep. Evictions surface as walk_past in the telemetry and as their own census column — distinct from hopeless shedding, because the two remove different populations for different reasons.

Dead sets still holding most of their articles are salvaged, not destroyed (walk_past_no_salvage to disable): the sweep hands them back (up to 25 per round), and each is scored with the SAME rule the health job applies to stored releases — healthVerdict over a data/par2 split of its gaps. Gaps covered by surviving par2 build into a release stored marked broken through the health backend (both sink modes, no contract change — a downloader's par2 repair completes it); par2-only gaps build as a NORMAL release (all data is present; only the completeness check was holding it); gaps beyond repair evict. Junk, blocked-extension and blacklist gates run first — salvage never resurrects what the build path would drop. When a later re-walk completes a salvaged release for real, the broken NZB's segment set is a strict subset of the new one, which is exactly what nzb-heal purges.

Note that stagingInfo issues two single-section INFO calls, not one multi-section call: INFO memory stats requires Redis 7.0, and against 6.x it returns nothing usable so every memory and eviction field silently reads zero — reporting a server at its ceiling as unbounded and never-evicting. The integration test catches this only when pointed at a 6.x server, which is noted there.

Tier decides ORDER, not entitlement — and on a site where the critical groups are caught up, that gap is a starvation bug rather than a nuance. Critical and normal plan in an instant, the whole remaining pass budget falls to whichever LOW group still has a backlog, and the articles it stages fill the staging memory whose pressure gate pauses the BACKFILL — the only job that can serve the critical group's history. hold_low_until_backfilled closes it by REMOVING the low tier from the pass (not merely ranking it last) while any critical group on that backbone still has history to pull. Asked per backbone, because article numbers and therefore backfill progress are per backbone. It fails OPEN: if the check errors, the crawl proceeds rather than converting a transient query failure into an outage for every low group. Off by default, since it deliberately starves a tier — right when the critical group is far behind, wrong on a site that is caught up everywhere. Both the hold and an all-held pass log what they did; a tier that silently stops crawling is indistinguishable from a broken crawler.

The junk engine is a full, data-driven port of the production filter: 24 rules (regex + named heuristics for shapes regexes can't express) in the production evaluation order, shipped in seed/junk_rules.tsv, seeded to junk_rules, compiled once into an atomic in-memory matcher, and reloadable without a restart. Size-banded rules only fire once a release's total size is known (the build path); the per-article ingest path runs the unsized subset.

Files

• plugin.go — registration, Metadata, Provision/Start/Stop, job + view + route wiring.
• config.go — the Config struct, defaults, live-override knob mapping.
• store.go / store_iface.go — the Store interface + PGStore construction.
• providers.go / provider_store.go / provider_state.go — the provider fleet, per-backbone crawl state.
• backbones.go — hostname → backbone identity mapping.
• crawl.go — the forward crawl pass (plan → fetch → stage → advance watermarks).
• backfill.go — the downward history walk, gap-filling.
• ranges.go — fetched-range coverage + the gap complement.
• nntp.go / pool.go (loon) — NNTP conn handling; the shared connection pool.
• staging.go / redis_staging.go — the stagingStore seam (pg | redis).
• assemble.go — completeness detection, NZB XML build, classification, the store handoff.
• staging_census.go — per-build-pass staging health series (migration 024).
• subject_mime.go — RFC 2047 encoded-word decoding at ingest.
• subject.go / tags.go — subject parsing; title-derived quality tags + category helpers.
• junk.go / junk_store.go / seed/junk_rules.tsv — the junk-filter engine + its data.
• blacklist.go / blacklist_store.go — the operator blacklist + filter-hit counters.
• poster_watch.go / poster_watch_store.go — per-poster outcome attribution.
• health.go / health_store.go — NZB health checking + its backend seam (internal | host).
• lease.go / assign.go — multi-host leases + term assignment.
• adopt.go — one-time host-state adoption for a production flip.
• dashboard.go / telemetry.go / views.go — status JSON, live counters, admin views.
• newznab.go — the Newznab/Torznab caps + feed XML.
• newsgroup_seed.go / seed.go / seed/*.tsv — the curated group pack + shipped data.

Testing

Unit-tested (no DB): the junk engine (a 47-vector parity suite differentially verified against the production filter, plus rule-order/attribution and the size-band contract), encoded-word decoding (the production subject that was being junked, a base64 subject hiding its segment counter, the stateful-charset refusal, and malformed input falling back to the raw header), the poster watch (substring matching keyed by PATTERN rather than raw From so one poster's history does not scatter across header variants, nil/empty safety on the per-article path, and accumulate/drain with a stable first sample), the blacklist matcher (per-field matching, fail-closed on unknown field, one-bad-pattern isolation), coverage-cell math, backfill gap complement, term-assignment partitioning (every group to exactly one worker, stable across churn), content-hash identity (pinned cross-repo to the host's scheme), assembly classification order, the sink/health backend contracts, and the Newznab/telemetry formatting.

Needs integration (live Postgres, -tags=integration, USENET_TEST_DSN): the coordination SQL (lease exclusivity, heartbeat, the atomic claim guard), the per-backbone stats/coverage queries, blacklist + filter-hit persistence, and host adoption — the last one asserts against the real legacy schema, which is how it caught two column-shape divergences the in-memory tests could not. Run the full suite (unit + integration, -race) against a throwaway Postgres; the coordination and adoption guarantees are the pieces that fail silently and expensively if wrong, so they are only trustworthy against a real database.

Also needs integration (live Redis, -tags=integration, USENET_TEST_REDIS): the nzb:ready LIST→SET migration — resumption across calls, discarding entries whose grp: metadata has expired while keeping live ones, two concurrent converters losing nothing, and the WRONGTYPE self-heal driven through stageArticles. Real server, not a fake: the cases turn on Redis' own semantics (script atomicity, WRONGTYPE, LTrim dropping an emptied key, SUnionStore), which a double would only restate as the assumptions under test. A stale pre-migration list of 7.3M entries stalled prod's crawl→stage→build pipeline for 23 hours, and the first fix for it introduced a converter race that destroyed 16k of 40k entries in a differential run — both are now pinned here, so these paths are worth a real server:

docker run -d --name usenet-test-redis -p 6399:6379 redis:7-alpine
USENET_TEST_REDIS=127.0.0.1:6399 go test -tags=integration -race ./usenet/

How the health sweep gives up

The sweep borrows the fleet's primary pool, so it must be a considerate neighbour to the crawler. It distinguishes three ways a release can fail to produce a verdict, and they are not interchangeable — collapsing two of them is what left the checker doing no work at all for weeks while logging a plausible "connection pool busy or failing":

what happened	outcome	effect
the pool had nothing to lend (ErrPoolBusy / ErrPoolEmpty)	healthSkipTransient	ends the pass at once — the crawler is using those connections and waiting for them is its loss
we held a connection and the provider failed the read	healthSkipTransport	abandons that release, tries the next; the pass ends only after health_transport_yield of them in a row
the server answered, but too much of the answer was unusable	healthSkipRow	abandons that release AND stamps it, like an unreadable blob — the server answered, so it is exactly as inconclusive next pass, and unstamped it re-led every batch

health_stat_timeout_sec (default 10) bounds ONE STAT, and is the other half of the same problem. The sweep borrows the crawler's pool and inherited its op_timeout_sec — 60s, sized for a 3000-article OVER. A socket the provider had quietly closed therefore cost a full minute to discover, up to three times per release, so a measured pass spent 19 minutes to check one release of fifty. A STAT is a single short line; the whole value of learning the connection is dead is learning it cheaply. The deadline is set per STAT inside the lease, and the pool clears whatever deadline it finds when the lease ends, so it cannot follow the connection to the crawler's next use.

The middle row is the fix. It used to be the first: one release timing out ended the whole pass, and against a provider that times out routinely the FIRST release tripped it on every single pass. health_transport_yield (default 5) is the run length that means the pool itself is sick — providers kill idle NNTP sessions, so after a quiet gap every socket is a corpse and grinding on costs an op-timeout each. Any release that reaches an answer resets the run, because that proves the connections work.

Transport and transient stay unstamped — their doubt is minted by the pool, not the server, so the release is retried promptly rather than waiting out the recheck window. Row-doubt is STAMPED like an unreadable blob, verdict untouched: the server answered, so the row answers identically next pass, and candidates sort unstamped rows first — once health_batch_size deterministic- inconclusive rows accumulated, every hourly sweep re-STATted the same fifty releases while the rest of the catalogue silently went unchecked. A skip also clears a user's recheck request, and only for healthSkipRow: a person pressed a button, the answer is "we genuinely cannot tell", and leaving the request set means re-STATting that release on every pass forever while their page says "queued". healthSkipTransport and healthSkipTransient leave the request alone — their check never really happened, so the request has not been serviced.

Documentation

Overview

Package usenet is the delivery-axis plugin: a basic Usenet indexer that crawls the last few days of a set of newsgroups, assembles complete article sets into downloadable NZB files, and serves search / group-list / download through a capability the host's pages consume. It owns the "usenet" Postgres schema and groups its jobs — Crawler, Backfill, Builder, Tag Fill, Prune, Health — under one "Usenet" family.

Staging (the transient article-assembly buffer) is pluggable behind the stagingStore seam: durable Postgres by default (never-lost, the base site's mode), or prod's Redis pipeline lifted verbatim via staging: redis (fast, best-effort) when the host has Redis. See README.md.

Index

• Constants
• Variables
• func DecodeSpotBase64(s string) ([]byte, error)
• func JoinSpotXML(values []string) string
• func ParseSpotKey(header string) (*rsa.PublicKey, error)
• func SpotSignatureBytes(sig string) ([]byte, error)
• func SpotTrust(err error) (string, bool)
• func VerifySpot(pub *rsa.PublicKey, messageID string, signature []byte) error
• type AssemblerStore
• type BackfillStore
• type BlacklistStore
• type BuilderInfo
• type Config
• type Episode
  • func ParseEpisode(title string) Episode
  • func (e Episode) Found() bool
• type ErrorReport
• type GroupStore
• type HealthStore
• type JobReport
• type JunkStore
• type LeaseStore
• type MaintenanceStore
• type PGStore
  • func NewPGStore(db *core.SchemaDB) *PGStore
• type PassReport
• type PendingRelease
  • func (p PendingRelease) Pct() int
• type Plugin
  • func (p *Plugin) Metadata() core.Metadata
  • func (p *Plugin) Provision(c *core.Core) error
  • func (p *Plugin) Start(ctx context.Context) error
  • func (p *Plugin) Stop(ctx context.Context) error
• type ProviderReport
• type ReleaseIndexed
• type ReleaseReader
• type ServerConfig
• type ServerStore
• type SettingStore
• type SinkMode
• type SpotHeader
  • func ParseSpotFrom(from string) (*SpotHeader, error)
  • func (h *SpotHeader) FullSubCats() []string
• type SpotKey
• type SpotXML
  • func ParseSpotXML(values []string) (*SpotXML, error)
  • func (s *SpotXML) CategoryValue() string
  • func (s *SpotXML) NZBSegment() string
  • func (s *SpotXML) NZBSegments() []string
• type StagingMode
• type StatusReport
• type Store
• type Tags
  • func (t Tags) Empty() bool
• type Tier
  • func (t Tier) Label() string
• type WorkerReport
• type WorkerStore

Constants

const (
	// SpotTrustVerified: signature checked against a key worth checking.
	SpotTrustVerified = "verified"
	// SpotTrustWeakKey: the signature is arithmetically valid and proves
	// nothing, because the key is small enough to forge cheaply.
	SpotTrustWeakKey = "weak-key"
	// SpotTrustUnsigned: no key or no signature to check at all.
	SpotTrustUnsigned = "unsigned"
)

Trust labels for a spot, stored on the release as nzbs.origin_trust.

const EventReleaseIndexed = "usenet.release.indexed"

What the crawler announces.

The first SYSTEM event on the site, and the reason core has a kind at all. Nobody did this: a crawler assembled a release out of articles that were already on a news server. There is no member to credit, `UserID` stays zero, and core refuses to let a system event be countable — so no achievement can ever be scored on "releases indexed", which is correct. Rewarding a member for something a machine did is the failure the kind field exists to make impossible rather than merely unwise.

Who wants it: caches (a release page that should stop saying "not found"), stats, and anything showing recent activity. All of which previously had to poll or be told by the host.

const MinSpotKeyBits = 1024

MinSpotKeyBits is the smallest modulus this package will call verified.

This is not theoretical. Of the 12 spots sampled from free.pt, SIX carried 384-bit keys — a size that factors on a laptop in minutes, so anyone can mint signatures for those posters at will. Verifying such a signature and reporting success would be the exact failure this file exists to prevent: a verifier that says yes without proving anything.

1024 is the protocol's own norm rather than a modern recommendation. It is the line between "weak but costly" and "trivially forged", which is the line that matters here. Raising it further would reject most of the live feed.

const SpotGroup = "free.pt"

SpotGroup is the Spotnet index group. A fixed property of the protocol, not operator config: comments live in free.usenet and the NZB and image payloads in alt.binaries.ftd, and a client that reads a different group is not reading Spotnet.

const SpotNZBGroup = "alt.binaries.ftd"

SpotNZBGroup carries the NZB and image payloads a spot points at. A fixed property of the protocol; fetched by message-id, never crawled.

Variables

var (
	// ErrNotASpot is returned for a From header that is not in spot form. It
	// is not a failure worth logging per article: free.pt carries ordinary
	// posts too, and a listing pass will meet plenty of them.
	ErrNotASpot = errors.New("spotnet: From header is not a spot")
	// ErrSpotMalformed means it LOOKED like a spot and was not parseable —
	// which is worth noticing, because it means either a format change or a
	// bug here.
	ErrSpotMalformed = errors.New("spotnet: malformed spot header")
)

var (
	// ErrSpotBadSignature means the bytes were checked and did not match.
	ErrSpotBadSignature = errors.New("spotnet: signature does not match the spot's key")
	// ErrSpotBadKey means the carried key could not be parsed at all.
	ErrSpotBadKey = errors.New("spotnet: unparseable public key")
	// ErrSpotWeakKey means the key is too small for a signature over it to mean
	// anything. Distinct from a bad signature because the spot is not forged —
	// it is merely unprovable, and a caller may knowingly accept it at lower
	// trust.
	ErrSpotWeakKey = errors.New("spotnet: key is too small to be worth verifying")
)

var (
	// ErrNoSpotXML means the article carried no X-Xml header at all.
	ErrNoSpotXML = errors.New("spotnet: article has no X-Xml header")
	// ErrSpotXMLTruncated means the joined document did not parse. Given the
	// pieces arrive as separate headers, the overwhelmingly likely cause is a
	// missing or reordered piece rather than a malformed spot.
	ErrSpotXMLTruncated = errors.New("spotnet: X-Xml document did not parse (missing a piece?)")
)

var AllTiers = []Tier{TierCritical, TierNormal, TierLow}

AllTiers is the admin UI's option list, in priority order.

Functions

func DecodeSpotBase64

func DecodeSpotBase64(s string) ([]byte, error)

DecodeSpotBase64 decodes Spotnet's escaped base64.

It is NOT the URL-safe alphabet, which is the trap: '+' and '/' — the only two non-alphanumeric characters in the standard alphabet — are escaped as the TWO-CHARACTER sequences "-p" and "-s". Treating '-' as a single-character substitution (the URL-safe assumption) leaves a stray 'p' or 's' in the stream, shifting every subsequent byte. That produced signatures one and two bytes LONGER than the modulus, which is arithmetically impossible and read as "these spots are forged" rather than "we decoded them wrongly": 11 of 12 live spots failed that way before the escaping was understood.

Standard base64 never emits '-', so applying this to an unescaped value is a no-op. That is why the same decoder serves both the key (which arrives unescaped, '/' and all) and the signature (which does not) — one function, no per-field guessing about which encoding a value uses.

func JoinSpotXML

func JoinSpotXML(values []string) string

JoinSpotXML concatenates the repeated X-Xml header values in the order they were received.

No separator, no trimming of the pieces: the split is at an arbitrary byte offset, so a boundary can fall in the middle of a tag name or an attribute value, and trimming whitespace at the seam would corrupt a document that happened to split on a space inside a description.

func ParseSpotKey

func ParseSpotKey(header string) (*rsa.PublicKey, error)

ParseSpotKey turns the X-User-Key header into an rsa.PublicKey.

func SpotSignatureBytes

func SpotSignatureBytes(sig string) ([]byte, error)

SpotSignatureBytes decodes the X-User-Signature header.

func SpotTrust

func SpotTrust(err error) (string, bool)

SpotTrust turns a VerifySpot result into the label stored with the release.

This exists so the import path has ONE place that decides what a verification outcome means, rather than each call site inventing its own mapping — the difference between "unprovable" and "forged" is the whole value of the check, and it is exactly the distinction an ad-hoc `if err != nil` at the call site would flatten.

A false second return means DO NOT IMPORT. Note that a weak key is importable: refusing it would drop half the live feed, and the honest treatment is to carry it with a label saying the signature proved nothing.

func VerifySpot

func VerifySpot(pub *rsa.PublicKey, messageID string, signature []byte) error

VerifySpot checks a spot's signature against the key it carries.

messageID may arrive with or without its angle brackets; what gets signed is always the bracketed form, because that is what Spotweb signs and a spot verified against the wrong shape would silently fail for every poster.

A nil return means the signature is genuine AND the key is large enough for that to be evidence. ErrSpotWeakKey means the maths was not attempted because the answer would not have meant anything.

Types

type AssemblerStore

type AssemblerStore interface {
	// contains filtered or unexported methods
}

AssemblerStore is the staging area the NZB assembler reads + drains.

type BackfillStore

type BackfillStore interface {
	// contains filtered or unexported methods
}

BackfillStore drives the backward crawl + its builder view.

type BlacklistStore

type BlacklistStore interface {
	// contains filtered or unexported methods
}

BlacklistStore is the operator blacklist + the per-rule filter-hit counters (blacklist_store.go).

type BuilderInfo

type BuilderInfo struct {
	StagedArticles int
	Releases       int
	Ready          int
	Pending        []PendingRelease
}

BuilderInfo is the NZB Builder's view of staging: how many articles are staged, how many distinct releases they form, how many are ready to assemble, and the largest still-incomplete releases (with unit progress) — so an admin can see WHY nothing is building (usually huge multi-file releases only partly crawled).

type Config

type Config struct {
	// Enabled, ABSENT, is true — the opposite default from the tracker, and
	// deliberately so: a tracker answers announces the moment it is reachable
	// and must be asked for, while an indexer that vanished because an
	// operator upgraded and never added a key would be a catalogue going
	// quietly stale. A pointer so absence is distinguishable from an explicit
	// false, which is a torrent-flavour host saying it means it: nothing
	// crawls, no pages mount, no jobs register.
	Enabled *bool `json:"enabled"`

	Server ServerConfig `json:"server"`
	// RetentionDays is CRAWL DEPTH: how far back to fetch and backfill. It does
	// NOT delete anything.
	RetentionDays int `json:"retention_days"` // default 6431 (~17.6y, prod parity)

	// NZBRetentionDays deletes assembled releases older than N days. 0 = keep
	// forever, which is the default and what prod does. Deleting a catalogue is
	// not something a default should ever do quietly.
	NZBRetentionDays int `json:"nzb_retention_days"` // default 0 = never delete

	CrawlIntervalMin   int `json:"crawl_interval_min"`   // crawl cadence (default 15)
	TagFillIntervalMin int `json:"tagfill_interval_min"` // tag-fill + recategorize cadence (default 360)
	PruneIntervalMin   int `json:"prune_interval_min"`   // prune cadence (default 1440)
	BuildDrainPerPass  int `json:"build_drain_per_pass"` // completed sets assembled per build pass (default 500)
	Batch              int `json:"batch"`                // article-number span per OVER request (default 3000)
	MaxGroups          int `json:"max_groups"`           // cap active groups crawled per run (default 20; 0 = all, no cap)
	CrawlMaxBatches    int `json:"crawl_max_batches"`    // forward-pass batch budget (default 20000) — the catch-up loop rolls the remainder into the next round
	// CrawlHeadroom is how many articles below the server's reported high water
	// mark the forward crawl stops, leaving the newest articles for the next
	// pass. 0 disables it.
	//
	// Articles do not appear atomically. An article number can exist while its
	// overview line is still being written or still propagating between peers,
	// so a batch that runs right up to the high water mark comes back short —
	// and crawl.go then records the whole requested range as fetched coverage
	// anyway. Walk-past eviction reasons FROM that coverage, treating "covered
	// and still short" as proof the missing articles are never coming, so a
	// frontier fetched too eagerly produces false dead verdicts and salvaged
	// BROKEN releases out of content that was merely still arriving.
	//
	// Nothing is lost by waiting: the next pass picks the articles up, and the
	// catch-up loop means "the next pass" is usually seconds away. NNTmux
	// leaves a comparable window for the same reason.
	CrawlHeadroom       int `json:"crawl_headroom"`         // articles left below the high water mark (default 2 batches)
	MaxArticlesPerGroup int `json:"max_articles_per_group"` // cap the first-pass volume so a busy group can't pull millions (default 20000)

	// Connections is the NNTP pool size — how many articles can be fetched in
	// parallel. Providers cap concurrent connections per account; the pool keeps
	// whatever it can open, so overshooting is safe but pointless.
	Connections int `json:"connections"` // default 10

	// KeepaliveMin is how often idle pool connections are probed, in minutes.
	// 0 disables keepalive.
	//
	// Providers reap idle connections, and a crawl pass leaves most of the pool
	// untouched between runs — so without probing, the steady state is a pool
	// full of connections the server already closed, discovered only when the
	// next pass leases one. Not a hardcoded constant because the right value is
	// the provider's idle timeout, which differs per provider and is rarely
	// documented.
	KeepaliveMin int `json:"keepalive_min"` // default 2

	SkipBackfill   bool `json:"skip_backfill"`    // "new articles only" — disable the backfill job
	CrawlNoCatchup bool `json:"crawl_no_catchup"` // disable the catch-up loop (default off = catch-up ON)
	// BackfillNoCatchup disables the backfill's catch-up loop. Same inverted
	// sense as the crawl one: the zero value keeps catching up, because a job
	// with hundreds of millions of articles outstanding should not sleep.
	BackfillNoCatchup bool `json:"backfill_no_catchup"`
	// BuildNoCatchup disables the builder's catch-up loop. Same inverted sense:
	// a builder holding the backfill's release valve should not nap.
	BuildNoCatchup bool `json:"build_no_catchup"`
	// BackfillDrainWaitSec is how long the backfill will wait for the builder to
	// make room before ending its pass. It waits rather than returning so the
	// two jobs run together instead of taking turns — the builder is the only
	// thing that can relieve the pressure the backfill is blocked on.
	BackfillDrainWaitSec int `json:"backfill_drain_wait_sec"`
	// BackfillPressureCeilingPct is the hard stop that applies even when there is
	// nothing for the builder to drain. Above the normal high-water mark because
	// in that state pausing achieves nothing — but still short of full, because
	// at maxmemory Redis EVICTS rather than refusing the write, and the sets it
	// evicts are the ones still assembling.
	BackfillPressureCeilingPct int `json:"backfill_pressure_ceiling_pct"`
	// HoldLowUntilBackfilled stops LOW-tier groups being crawled forward
	// while any CRITICAL group still has history to backfill. See
	// holdLowTier in provider_state.go for why ordering alone is not enough.
	HoldLowUntilBackfilled bool `json:"hold_low_until_backfilled"`
	// WalkPastNoEvict disables the walk-past sweep (inverted so the zero value
	// sweeps): a set whose whole article span has been fetched and is still
	// incomplete can never complete, and every hour it waits for the TTL is an
	// hour of staging memory held against the pressure gate.
	WalkPastNoEvict bool `json:"walk_past_no_evict"`
	// WalkPastGraceMin is how long a set must go without a new article before
	// the walk-past sweep may judge it (default 15) — covers retried batches
	// and staging latency at the walk edge.
	WalkPastGraceMin int `json:"walk_past_grace_min"`
	// WalkPastSweepPerRound bounds how many staged sets the walk-past sweep
	// examines per build round (default 2000). The cursor persists, so the
	// sweep RATE is this budget times the round frequency.
	WalkPastSweepPerRound int `json:"walk_past_sweep_per_round"`
	// WalkPastNoSalvage disables broken-release salvage (inverted so the zero
	// value salvages): walk-past-dead sets holding most of their articles are
	// then evicted like the rest instead of being assembled and stored marked
	// broken (repairable gaps) or normal (par2-only gaps).
	WalkPastNoSalvage bool `json:"walk_past_no_salvage"`
	// ReadyReapPerPass bounds the dead-entry sweep of nzb:ready per build
	// ROUND (the name predates the round/pass split; the stored key stays for
	// compatibility). Default 50000: a full circuit of a multi-million-entry
	// queue takes several rounds, which is the point — the sweep must not cost
	// more than the round it is clearing the way for. Per round matters: the
	// cursor persists, so the sweep RATE is this budget times the call
	// frequency, and a catch-up pass has no round cap.
	ReadyReapPerPass      int `json:"ready_reap_per_pass"`
	BackfillBatchesPerRun int `json:"backfill_batches_per_run"` // cap backward batches per backfill pass, across all groups (default 25)
	BackfillIntervalMin   int `json:"backfill_interval_min"`    // backfill cadence (default 5)
	// DiagKeepDays is the rolling window for the observe-only diagnostic
	// series: staging_census, subject_corpus, set_resolutions.
	//
	// A knob because their volume is driven by the CRAWLER's behaviour, not by
	// ours: set_resolutions took 1,070 rows/minute while the walk-past sweep
	// cleared a backlog (settling to ~120), and the next reset will burst
	// again. At 195 bytes a row that is the difference between half a gigabyte
	// and several, and the fix must not require a deploy. Default 14 days.
	DiagKeepDays int `json:"diag_keep_days"`

	// Staging backend (README.md). Boot config, not a live knob:
	// switching backends at runtime would strand staged data.
	Staging StagingMode `json:"staging"` // pg (durable, default) | redis (fast, best-effort)

	// Sink is where assembled releases go: SinkInternal (the plugin's own minimal
	// nzbs table — standalone installs, the demo) or SinkHost (the host registers
	// the ReleaseSink capability and owns the NZB domain — how prod adopts the
	// crawler). Boot config: switching sinks live would split the catalogue.
	Sink              SinkMode `json:"sink"`
	StagingMaxRows    int      `json:"staging_max_rows"`     // pg back-pressure denominator: staged rows / this (default 2_000_000)
	StagingPruneHours int      `json:"staging_prune_hours"`  // pg stale-staging horizon in hours (default 6)
	StagingTTLHours   int      `json:"staging_ttl_hours"`    // redis staged-key TTL in hours (default 2) — must exceed the gap between passes that stage parts of one release
	EvictStaleSecs    int      `json:"evict_staleness_secs"` // redis inline hopeless-eviction staleness window in seconds (default 300) — must exceed routine staging-pressure pauses or resumed sets are judged abandoned

	// Splitting groups between crawlers (assign.go). Membership is fixed for a
	// TERM, so a crawler that joins mid-term waits for the next boundary rather
	// than changing everyone's share underneath a pass in flight.
	AssignTermMin  int `json:"assign_term_min"`  // default 15
	WorkerStaleSec int `json:"worker_stale_sec"` // presence timeout, default 90

	// Cross-host coordination (lease.go). How long a claimed lease survives
	// without renewal — long enough that a slow pass never loses its own claim,
	// short enough that a killed worker's work is picked up promptly.
	LeaseTTLMin int `json:"lease_ttl_min"` // default 15

	// NZB health checking (health.go). Segments are STATted on idle connections
	// only, so these bound how much bookkeeping runs, not how fast it must.
	HealthIntervalMin int `json:"health_interval_min"`  // sweep cadence (default 60)
	HealthBatchSize   int `json:"health_batch_size"`    // releases per sweep (default 50)
	HealthRecheckDays int `json:"health_recheck_days"`  // re-check a release this often (default 30)
	HealthMinAgeHours int `json:"health_min_age_hours"` // propagation guard: skip releases newer than this (default 24)
	HealthStatChunk   int `json:"health_stat_chunk"`    // segments STATted per connection lease (default 200)
	// HealthStatTimeoutSec bounds ONE STAT, as opposed to OpTimeoutSec which
	// bounds a whole command exchange and is sized for a 3000-article OVER.
	//
	// The sweep borrows the crawler's pool and inherited its 60s, so a socket
	// the provider had already closed cost a full minute to discover — three
	// times per release before the release was abandoned. A measured pass
	// spent 19 minutes to check ONE release. A STAT is a single short line:
	// if it has not answered in seconds the connection is dead, and the whole
	// value of finding that out is finding it out cheaply.
	HealthStatTimeoutSec int `json:"health_stat_timeout_sec"` // per-STAT deadline (default 10)
	// HealthTransportYield: how many releases in a row may fail on TRANSPORT
	// (the provider timed out mid-STAT) before the pass gives up. Not the same
	// as the pool being busy, which still yields on the first refusal so the
	// crawler keeps priority. This exists because the yield used to be decided
	// per release and end the whole pass: against a provider that times out
	// routinely the first release tripped it every time, and the sweep checked
	// nothing for weeks while logging a plausible "pool busy or failing".
	HealthTransportYield int `json:"health_transport_yield"` // consecutive transport-failed releases before yielding (default 5)

	// NFO extraction (nfo.go). The first feature built on article bodies --
	// the crawler indexes from OVERVIEW lines and has never read one.
	//
	// NFOEnabled defaults FALSE. Every other job here is bookkeeping against
	// data already paid for; this one spends provider bytes, and a block
	// account's bytes are finite and metered. An operator should choose to
	// spend them rather than discover the choice was made for them by an
	// upgrade.
	NFOEnabled bool `json:"nfo_enabled"` // read .nfo articles at all (default false)
	// Spotnet. The index pass is cheap by design -- one XOVER round trip per
	// SpotBatchSize articles -- so its budget is expressed in BATCHES, and a
	// full history sweep is a few thousand of them rather than millions.
	SpotIntervalMin int `json:"spot_interval_min"` // pass cadence (default 15)
	SpotBatchSize   int `json:"spot_batch_size"`   // articles per XOVER (default 1000)
	SpotMaxBatches  int `json:"spot_max_batches"`  // XOVER round trips per pass, forward + backfill (default 200)
	// The fetch pass is the expensive half: TWO article reads per spot (the
	// document, then the NZB). Its batch is therefore in SPOTS, not batches,
	// and is two orders of magnitude smaller than the index pass's budget.
	SpotFetchIntervalMin int `json:"spot_fetch_interval_min"` // pass cadence (default 10)
	SpotFetchBatch       int `json:"spot_fetch_batch"`        // spots per pass (default 200)

	NFOIntervalMin int `json:"nfo_interval_min"` // pass cadence (default 60)
	NFOBatchSize   int `json:"nfo_batch_size"`   // releases per pass (default 100)
	// NFOBudgetMB caps the bytes ONE PASS may read. The genuinely new control
	// this feature needs: providers meter bytes, so unlike connection pressure
	// -- which the pool already expresses and TryDo already yields to -- there
	// is nothing in the existing machinery that notices bytes being consumed.
	// Checked BEFORE each fetch, since a ceiling that one whole article can
	// exceed is not a ceiling.
	NFOBudgetMB int `json:"nfo_budget_mb"` // per-pass byte ceiling (default 64)
	// The junk-recovery probe (junk_probe.go). Off by default for the same
	// reason NFO is -- it spends metered bytes -- and additionally because it
	// answers a question rather than serving a feature: is the crawler
	// discarding real releases on the strength of a scrambled subject? The
	// batch is expressed in ARTICLES rather than MB because the wire cost of
	// one probe is a whole segment however few bytes we keep.
	// The ROT18 title repair (rot18_repair.go). Off by default because it
	// REWRITES catalogue titles: the decode is safe on rows a literal marker
	// matches, but "safe" is a property of the marker list, and an operator
	// should switch that on deliberately rather than find a thousand titles
	// changed after an upgrade. It spends no provider bytes.
	Rot18RepairEnabled     bool `json:"rot18_repair_enabled"`      // repair ROT18 titles (default false)
	Rot18RepairIntervalMin int  `json:"rot18_repair_interval_min"` // pass cadence (default 60)
	// Rot18RepairMaxMin bounds ONE pass. The walk is the whole catalogue the
	// first time (~1M rows) and nothing after that, so the budget exists to
	// keep the first pass from holding the job lease for an unbounded stretch,
	// not to ration work.
	Rot18RepairMaxMin int `json:"rot18_repair_max_min"` // minutes one pass may run (default 10)

	JunkProbeEnabled     bool `json:"junk_probe_enabled"`      // read dropped-junk bodies at all (default false)
	JunkProbeIntervalMin int  `json:"junk_probe_interval_min"` // pass cadence (default 360)
	JunkProbeBatchSize   int  `json:"junk_probe_batch_size"`   // drops per pass (default 50)
	// NFOMaxRetries bounds how many TRANSPORT failures one release may cost
	// before it is written off. A 430 is permanent and written off at once;
	// a timeout says nothing about the article, so it is counted instead --
	// but uncounted it would be retried forever, and a few unreachable
	// articles at the head of the queue consume every pass. Newznab bounds
	// the same thing by decrementing nfostatus toward a floor. 0 disables the
	// ceiling and restores retry-forever.
	NFOMaxRetries int `json:"nfo_max_retries"` // transport failures before write-off (default 3)

	// Proof-image extraction (image.go), the second body-fetch feature. Same
	// default-off reasoning as NFO — it spends metered provider bytes — and a
	// bigger per-item cost: a proof JPG spans several whole articles where an
	// NFO is one small one.
	ImageEnabled     bool `json:"image_enabled"`      // fetch proof images at all (default false)
	ImageIntervalMin int  `json:"image_interval_min"` // pass cadence (default 60)
	ImageBatchSize   int  `json:"image_batch_size"`   // releases per pass (default 25)
	ImageBudgetMB    int  `json:"image_budget_mb"`    // per-pass byte ceiling (default 128)
	ImageMaxRetries  int  `json:"image_max_retries"`  // transport failures before write-off (default 3)

	// NNTP transport bounds. Per-provider behavior lives on the servers table;
	// these are the plugin-wide dial/operation limits every pool is built with.
	// DialTimeoutSec bounds one connect+greeting attempt. OpTimeoutSec bounds
	// one whole command exchange — one GROUP+OVER round — and interacts with
	// `batch`: a bigger batch on a slow provider legitimately takes longer, and
	// an OpTimeout below the honest fetch time turns every batch into a
	// discarded connection and a reconnect storm.
	DialTimeoutSec int `json:"dial_timeout_sec"` // default 30
	OpTimeoutSec   int `json:"op_timeout_sec"`   // default 60
	// ProviderDownCooldownMin is how long a provider stays benched after
	// failing. Long enough to stop re-dialling a dead server every pass, short
	// enough that recovery is noticed the same hour.
	ProviderDownCooldownMin int `json:"provider_down_cooldown_min"` // default 10

	// Backfill back-pressure thresholds (percent of staging pressure). Backfill
	// pauses at high, resumes below low; the forward crawl is never paused.
	BackfillPressureHighPct int `json:"backfill_pressure_high_pct"` // default 85
	// CrawlPressureHighPct stops the FORWARD crawl staging when the staging
	// backend is this full. Higher than the backfill gate on purpose: new
	// articles matter more than history, so the forward crawl yields only when
	// storing would actively destroy what is already there.
	CrawlPressureHighPct   int `json:"crawl_pressure_high_pct"`   // default 95
	BackfillPressureLowPct int `json:"backfill_pressure_low_pct"` // default 70
}

Config is the plugins.usenet section of config.yml. The server here seeds the servers table on first boot if it's empty; after that the wizard owns it. The numeric knobs are DEFAULTS — rows in the plugin's settings table (edited on the host's /admin/settings page) override them at job run time via withOverrides.

type Episode

type Episode struct {
	// Series is the show's name as it appeared, cleaned of separators:
	// "The.Blacklist" → "The Blacklist".
	Series string
	// SeriesKey is Series folded for grouping and lookup — lowercase, no
	// punctuation, no spaces. It is what "the same show" means, because
	// "Marvels.Agents.of.S.H.I.E.L.D." and "Marvel's Agents of SHIELD" are one
	// show and no operator should have to reconcile them by hand.
	SeriesKey string
	Season    int
	// Episode is 0 for a whole-season pack (S03, S03.COMPLETE), which is a
	// real thing to index and a different thing from episode zero.
	Episode int
	// Pack marks that whole-season release, so a page can group it with the
	// season rather than losing it among the episodes.
	Pack bool
}

Episode is what a title says about where a release sits in a series.

func ParseEpisode

func ParseEpisode(title string) Episode

ParseEpisode reads a title. Zero value when it says nothing usable.

func (Episode) Found

func (e Episode) Found() bool

Found reports whether the title said anything usable.

type ErrorReport

type ErrorReport struct {
	At  time.Time `json:"at"`
	Op  string    `json:"op"`
	Msg string    `json:"message"`
}

type GroupStore

type GroupStore interface {
	// contains filtered or unexported methods
}

GroupStore manages the newsgroup catalog.

type HealthStore

type HealthStore interface {
	// contains filtered or unexported methods
}

HealthStore is the NZB health surface (health.go).

type JobReport

type JobReport struct {
	Name     string `json:"name"`
	Status   string `json:"status"`
	Activity string `json:"activity"`
	Next     string `json:"next_run"`
	Running  bool   `json:"running"`
	// DutyPct is the trailing-hour busy percentage — "runs on schedule" and
	// "actually works" are different claims, and only this one tells them
	// apart from outside.
	DutyPct float64 `json:"duty_pct"`
	// Logs is the recent job-log tail (jobLogTail lines) — what the Jobs
	// tab's per-job panes poll for live logging.
	Logs []string `json:"logs,omitempty"`
}

JobReport is one scheduler job's live state (mirrors crawlerJobVM).

type JunkStore

type JunkStore interface {
	// contains filtered or unexported methods
}

JunkStore is the tunable junk-rule set (seeded from the embedded TSV, loaded into memory — see junk.go).

type LeaseStore

type LeaseStore interface {
	// contains filtered or unexported methods
}

LeaseStore is cross-host coordination (lease.go): who crawls which backbone, and which worker owns the cluster-wide jobs.

type MaintenanceStore

type MaintenanceStore interface {
	// contains filtered or unexported methods
}

MaintenanceStore is the nzbs cleanup / retagging surface (off-peak jobs). The staging-side cleanup (prune) moved to stagingStore (staging.go) so it swaps with the backend.

type PGStore

type PGStore struct {
	// contains filtered or unexported fields
}

PGStore is the Postgres implementation of Store. Every method runs through the SchemaDB's WithTx, which scopes search_path to "usenet" so unqualified table names resolve into the plugin's own schema.

func NewPGStore

func NewPGStore(db *core.SchemaDB) *PGStore

NewPGStore builds the Postgres-backed store over a plugin-scoped SchemaDB.

type PassReport

type PassReport struct {
	Running bool `json:"running"`
	Groups  int  `json:"groups"`
	// GroupsDone / BatchesTotal / Reading are the legacy dashboard's live
	// progress trio: "Group N / M — <what it is reading>" plus the bar's
	// denominator (batches/batches_total).
	GroupsDone   int `json:"groups_done"`
	Batches      int `json:"batches"`
	BatchesTotal int `json:"batches_total"`
	// PassBatches/PassBatchesTotal accumulate across the whole pass, and are
	// what a consumer should read for "how much did this pass do" — batches/
	// batches_total are ROUND-scoped for the live bar, and a completed pass
	// always ends on an empty round, so they read ~0 the moment it finishes.
	// Additive fields: the round pair keeps its name and meaning.
	PassBatches      int     `json:"pass_batches"`
	PassBatchesTotal int     `json:"pass_batches_total"`
	Reading          string  `json:"reading"`
	Failed           int     `json:"failed_batches"`
	Articles         int     `json:"articles"`
	Staged           int     `json:"staged"`
	WireBytes        int64   `json:"wire_bytes"`
	DurationSec      float64 `json:"duration_seconds"`
	ArticlesSec      float64 `json:"articles_per_second"`
}

PassReport is one job's current or last pass.

type PendingRelease

type PendingRelease struct {
	Base     string
	Have     int
	Need     int
	Segments int
	Multi    bool
}

PendingRelease is one incomplete staged release. Units are files for multi-file releases, else segments.

func (PendingRelease) Pct

func (p PendingRelease) Pct() int

Pct is the unit-completion percentage (0-100).

type Plugin

type Plugin struct {
	// contains filtered or unexported fields
}

func (*Plugin) Metadata

func (p *Plugin) Metadata() core.Metadata

func (*Plugin) Provision

func (p *Plugin) Provision(c *core.Core) error

func (*Plugin) Start

func (p *Plugin) Start(ctx context.Context) error

func (*Plugin) Stop

func (p *Plugin) Stop(ctx context.Context) error

type ProviderReport

type ProviderReport struct {
	ID       int    `json:"id"`
	Name     string `json:"name"`
	Host     string `json:"host"`
	Backbone string `json:"backbone"`
	Role     string `json:"role"`
	Enabled  bool   `json:"enabled"`
	// Live dial state, merged from the worker-published fleet stats so the
	// dashboard's provider strip can tick without a page reload. Dialled is
	// false when the fleet has no entry for the provider yet.
	Dialled bool `json:"dialled"`
	Down    bool `json:"down"`
	Open    int  `json:"open"`
	Target  int  `json:"target"`
	Busy    int  `json:"busy"`
	// Fetch volume since worker start, per account. Deltas between polls are
	// the per-provider rate — the number that catches a degraded account on a
	// shared backbone.
	Articles      int   `json:"articles"`
	Staged        int   `json:"staged"`
	WireBytes     int64 `json:"wire_bytes"`
	FailedBatches int   `json:"failed_batches"`
	// Resets counts pool rebuilds. It is the signal that separates "the
	// provider is slow" from "the pool is thrashing": a climbing Resets with
	// steady Open/Target means connections are being torn down and re-dialled
	// under the crawl, which is what surfaces to operators as
	// "nntp: no usable connection in pool".
	Resets int64 `json:"resets"`
}

type ReleaseIndexed

type ReleaseIndexed struct {
	Title string
	Group string
	// Size in bytes of the assembled release.
	Size int64
}

ReleaseIndexed is the Data payload of EventReleaseIndexed.

type ReleaseReader

type ReleaseReader interface {
	// contains filtered or unexported methods
}

ReleaseReader is the read side: search, browse, feed, detail, raw NZB, stats.

type ServerConfig

type ServerConfig struct {
	Host     string `json:"host"`
	Port     int    `json:"port"`
	TLS      bool   `json:"tls"`
	Username string `json:"username"`
	Password string `json:"password"`
}

type ServerStore

type ServerStore interface {
	// contains filtered or unexported methods
}

ServerStore holds the single NNTP server row.

type SettingStore

type SettingStore interface {
	// contains filtered or unexported methods
}

SettingStore is the plugin's key/value settings.

type SinkMode

type SinkMode string

SinkMode selects where assembled releases are stored. It drives the catalogue-splitting branch in resolveSink/resolveHealthBackend, so it is a closed type rather than a raw string: a mistyped literal would silently fall through to internal mode and split the catalogue across two tables.

const (
	SinkInternal SinkMode = "internal" // the plugin's own nzbs table (default)
	SinkHost     SinkMode = "host"     // the host's NZB domain, via the ReleaseSink capability
)

type SpotHeader

type SpotHeader struct {
	Poster    string // display name, before the angle bracket
	PublicKey string // travels WITH the spot — there is no key directory
	Signature string // the last dotted field

	Category int      // 1 video, 2 audio, 3 game (observed; canonical table is Spotweb's)
	KeyID    int      // matches <Key> in the XML document
	SubCats  []string // "a02", "b00", … letter-prefixed, three characters each

	SizeBytes int64 // checked against titles during the spike and consistent
	PostedAt  int64 // unix seconds
	Locale    string

	// Unknown1 and Unknown2 are the two fields nobody has identified: the
	// value after the size and the value after the timestamp. Carried rather
	// than dropped, because a parser that silently discards fields it does not
	// understand is how a format change becomes invisible.
	Unknown1 string
	Unknown2 string
}

SpotHeader is what XOVER alone yields: enough to list a spot without a second round trip. Spotnet clients build their whole listing from this, roughly one round trip per thousand spots, which is why they feel fast.

func ParseSpotFrom

func ParseSpotFrom(from string) (*SpotHeader, error)

ParseSpotFrom reads the From header of a spot.

Paaldanser <KEY@27a02b00c08d13z00.3365188124.20.1786812549.1.NL.SIG>
            │    ││  └ subcats ┘  └ size ──┘ └┘ └ posted ─┘ │ └ locale
            │    │└ key id                   ?              ?
            │    └ category
            └ public key                             signature ┘

The address local part is the public key; everything after the @ is a dotted tuple whose FIRST element packs three values with no separator: one digit of category, one digit of key id, then subcategories in three-character groups.

func (*SpotHeader) FullSubCats

func (h *SpotHeader) FullSubCats() []string

FullSubCats renders the subcategories the way the XML document does, with the category prefixed onto each: category 2 + "a02" -> "02a02".

The header and the XML disagree in FORM but not in content, and the XML's form is the one Spotweb's category table is keyed on.

type SpotKey

type SpotKey struct {
	XMLName  xml.Name `xml:"RSAKeyValue"`
	Modulus  string   `xml:"Modulus"`
	Exponent string   `xml:"Exponent"`
}

SpotKey is the RSA key a spot carries in X-User-Key.

<RSAKeyValue><Modulus>…base64…</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>

The .NET RSAKeyValue form, because Spotweb and the original client are .NET — big-endian base64 for both components rather than any PEM encoding.

type SpotXML

type SpotXML struct {
	XMLName xml.Name `xml:"Spotnet"`
	Posting struct {
		Key         int    `xml:"Key"`
		Created     int64  `xml:"Created"`
		Poster      string `xml:"Poster"`
		Title       string `xml:"Title"`
		Description string `xml:"Description"`
		Size        int64  `xml:"Size"`

		Image struct {
			Width   int    `xml:"Width,attr"`
			Height  int    `xml:"Height,attr"`
			Segment string `xml:"Segment"`
		} `xml:"Image"`

		Category struct {
			// Value is the leading text node ("02"); Subs are the nested
			// <Sub> elements ("02a02", …). Mixed content, so the category
			// itself is chardata rather than an element of its own.
			Value string   `xml:",chardata"`
			Subs  []string `xml:"Sub"`
		} `xml:"Category"`

		// NZB.Segment is a bare message-id in alt.binaries.ftd. It is the
		// whole point of the spot: fetch it and a finished NZB comes back,
		// which is why importing skips the crawler's expensive half entirely.
		NZB struct {
			Segment []string `xml:"Segment"`
		} `xml:"NZB"`
	} `xml:"Posting"`
}

SpotXML is the document a spot carries. Field names follow the wire.

func ParseSpotXML

func ParseSpotXML(values []string) (*SpotXML, error)

ParseSpotXML joins the pieces and parses the result.

Takes the SLICE rather than a joined string on purpose, so a caller cannot accidentally pass the first header and get a plausible-looking answer. The count is reported in the error precisely because "we only had one piece" is the failure that otherwise looks like success.

func (*SpotXML) CategoryValue

func (s *SpotXML) CategoryValue() string

CategoryValue is the numeric category as written in the document ("02"), with the surrounding whitespace of the mixed content removed.

chardata on a mixed-content element collects the text around the children too, so the raw value carries the indentation between <Sub> elements.

func (*SpotXML) NZBSegment

func (s *SpotXML) NZBSegment() string

NZBSegment is the first segment, kept for the stored column and the "does this spot carry an NZB at all" check. Never use it to FETCH.

func (*SpotXML) NZBSegments

func (s *SpotXML) NZBSegments() []string

NZBSegments are the message-ids of the articles holding this spot's NZB, in posting order. Empty when the spot points at none.

PLURAL, and that is the whole point. A spot's NZB is one DEFLATE stream cut across as many articles as it takes, so a big release announces several segments and only their concatenation inflates to a document. Reading the first one alone yields a stream that decodes part-way and then stops — which is not a decode failure, it is a shorter NZB that parses far enough to look real. An 89GB release shipped with roughly a tenth of its segments that way, and the only outward sign was the file list failing to load.

The same trap already had a warning on it twelve lines up: ParseSpotXML takes the slice of X-Xml pieces rather than a string precisely "so a caller cannot accidentally pass the first header and get a plausible-looking answer". The NZB pointer needed the same treatment and did not have it.

type StagingMode

type StagingMode string

StagingMode selects the transient article-assembly backend.

const (
	StagingPG    StagingMode = "pg"    // durable Postgres (default)
	StagingRedis StagingMode = "redis" // prod's Redis pipeline (fast, best-effort)
)

type StatusReport

type StatusReport struct {
	GeneratedAt time.Time `json:"generated_at"`

	Crawl    PassReport `json:"crawl"`
	Backfill PassReport `json:"backfill"`

	Providers []ProviderReport `json:"providers"`
	Workers   []WorkerReport   `json:"workers"`

	// Groups counts ACTIVE NEWSGROUPS. Before 2026-07 it was the row count of
	// the per-(backbone, group) state join, which double-counted every group a
	// second backbone carried — on multi-backbone installs the value dropped
	// at that deploy; re-baseline any external monitor thresholds on it.
	Groups         int   `json:"active_groups"`
	StagedArticles int   `json:"staged_articles"`
	TotalNZBs      int   `json:"total_nzbs"`
	BackfillLeft   int64 `json:"backfill_remaining"`
	// BackfillETASeconds is 0 when there is nothing left or no measured rate.
	// A zero here means "unknown", never "done" — check backfill_remaining.
	BackfillETASeconds int64 `json:"backfill_eta_seconds"`

	// Jobs is the scheduler's view of the plugin's own jobs — status, last
	// activity line, and next scheduled run. On a split deployment these come
	// from the worker's published telemetry, so the web poll shows the truth.
	Jobs []JobReport `json:"jobs"`
	// ReadyGroups is redis staging's assembly queue depth (LLEN — O(1)).
	// Always 0 in pg mode: the equivalent there is a COUNT scan, which this
	// endpoint is forbidden from running per poll.
	ReadyGroups int64 `json:"ready_groups"`
	// Evicted counts hopeless sets shed by redis staging since worker start.
	Evicted int64 `json:"evicted"`
	// PendingCount is the size of the last incomplete-sets sample.
	PendingCount int `json:"pending_count"`

	// WorkerLastSeen is when the worker last published telemetry;
	// WorkerStale means that heartbeat has lapsed and every "running"
	// claim above is history, not state — the dead-worker case that used
	// to render as a crawl whose duration climbed forever.
	WorkerLastSeen time.Time `json:"worker_last_seen"`
	WorkerStale    bool      `json:"worker_stale"`
	// CrawlStalledPasses: consecutive crawl passes with zero forward
	// progress against a large backlog. Non-zero deserves a look; the
	// third also lands in the error log.
	CrawlStalledPasses int `json:"crawl_stalled_passes"`

	RecentErrors []ErrorReport `json:"recent_errors"`
}

StatusReport is the machine-readable crawler status. Exposed as JSON so a run can be watched without scraping the admin HTML — useful for a first live run, for an external monitor, and for the operator's own scripts.

Field names are stable; treat this as an API, not a view model.

type Store

type Store interface {
	ReleaseReader
	GroupStore
	ServerStore
	SettingStore
	BackfillStore
	BlacklistStore
	AssemblerStore
	MaintenanceStore
	JunkStore
	HealthStore
	LeaseStore
	WorkerStore
}

Store is usenet's persistence contract. It's segmented into concern-based interfaces (interface-segregation) so a consumer can depend on only the slice it uses — internalHealth (health.go) takes just HealthStore, and the read tier could one day bind ReleaseReader to a replica. The plugin field holds the union; PGStore is the Postgres impl.

The methods are package-private on purpose: this is an internal contract, so only an in-package impl (PGStore) or test double can satisfy it.

type Tags

type Tags struct {
	Resolution string // 2160p / 1080p / 720p / 480p
	Source     string // BluRay / WEB-DL / WEBRip / HDTV / DVD / Remux
	Codec      string // x265 / x264 / AV1 / XviD
	Audio      string // FLAC / AAC / DTS / AC3 / TrueHD / Opus
	Language   string // English / Japanese / Multi / Dual Audio / …
}

Tags is the quality metadata parsed from a release title.

func (Tags) Empty

func (t Tags) Empty() bool

Empty reports whether nothing was parsed.

type Tier

type Tier string

Tier is a group's crawl priority. Closed set, and the crawler branches on it, so it is a type rather than a bare string: a mistyped literal in a comparison takes the wrong branch silently, and the symptom (one group quietly crawled last) is exactly the bug the tier exists to fix.

The schema carries the same constraint (migration 019), so a bad value cannot reach here from the database either.

const (
	// TierCritical is crawled before everything else, every pass. For the one
	// or two groups the content is actually posted to.
	TierCritical Tier = "critical"
	// TierNormal is the default.
	TierNormal Tier = "normal"
	// TierLow is only crawled with whatever capacity is left after the others.
	TierLow Tier = "low"
)

func (Tier) Label

func (t Tier) Label() string

Label is the human name for the tier, used by the settings template.

type WorkerReport

type WorkerReport struct {
	ID     string `json:"id"`
	Groups int    `json:"groups_held"`
}

type WorkerStore

type WorkerStore interface {
	// contains filtered or unexported methods
}

WorkerStore is crawler presence, used to split groups between hosts (assign.go).