firehose

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 14, 2026 License: MIT Imports: 9 Imported by: 0

README

GitHub go.mod Go version Test ci Coverage

firehose

firehose is an RSS/Atom feed aggregator and static HTML generator that renders your chosen feeds as a single reverse-chronological river of news. Inbound feed content is sanitized for safety, typography is normalized, and the layout and styling are kept deliberately simple to favor reading on small and larger devices alike.

firehose runs as a batch job — fetch, render, write, exit — typically from a systemd timer or cron. Nothing stays resident and nothing listens on a port. Deploy locally or remote using the web infrastructure you already have.

Why the name? Think drinking from a fire hose.

Install

firehose is deliberately not dependent on cgo and thus is portable across operating systems and architectures. The releases page contains pre-built static binaries for amd64/arm64 architecture macOS and Linux distributions.

Or, build it yourself:

go install github.com/mwyvr/firehose/cmd/firehose@latest

Quick start

firehose init > config.toml     # every option, annotated, at its default
$EDITOR config.toml             # add feeds and sections; set output_dir, cache_db
firehose -config config.toml check
firehose -config config.toml    # generate

Usage:

firehose               generate: fetch, cache, render, write (default)
firehose -force        generate, ignoring failure-backoff gates
firehose check         validate config + dry-run templates (no network)
firehose test URL      diagnose one feed, verbose (-ua, -H "K: V" to bisect)
firehose export        feed list as OPML to stdout (-section NAME)
firehose init          annotated default config to stdout
firehose version       print the version

Try it locally:

make dev                    # builds, fetches sample feeds into ./build/dev, renders
open build/dev/index.html   # build/dev/firehose.html = health page

Deployment

firehose generates static HTML to serve up via the local or remote web server infrastructure you already have in place. See examples and more in docs/deploy.md.

Changelog

  • v0.2.0, 2026-07-13: Cross-feed dedupe with "also via" attribution; per-feed display_window; health page items column (shown / cached) and honest last-success (yielded items, not merely parsed).
  • v0.1.0, 2026-07-12: Eat my own dog food initial beta release.

Future

Not a lot planned. Possibly:

  • OPML import (firehose export already writes OPML)
  • Maybe user-definable templates and CSS overrides.

License

MIT — see LICENSE.

Documentation

Overview

Package firehose defines the domain types and service interfaces for firehose, the application. firehose, the application, is an RSS aggregator that produces static HTML in a river-of-news format.

Index

Constants

View Source 
const (
	ECONFLICT = "conflict"  // action cannot proceed against current state
	EINTERNAL = "internal"  // unexpected internal error
	EINVALID  = "invalid"   // validation failed / malformed input
	ENOTFOUND = "not_found" // entity not found (e.g. HTTP 404 feed)
	EPANIC    = "panic"     // recovered per-feed panic
	EPARSE    = "parse"     // fetched OK but content would not parse (bad XML)
	ETIMEOUT  = "timeout"   // fetch timed out
)

Application error codes.

Variables

View Source 
var Version = "dev"

Version is the build version, injected at build time via -ldflags "-X github.com/mwyvr/firehose.Version=..." (see Makefile). Plain `go build` produces "dev". Shown by -h, `firehose version`, and the page footer.

Functions

func CategoriesIntersect added in v0.2.1 

func CategoriesIntersect(cats, want []string) bool

CategoriesIntersect reports whether cats intersects want, case-insensitively; "*" in want matches everything.

func ContainsCategory added in v0.2.1 

func ContainsCategory(cats []string, want string) bool

ContainsCategory reports whether cats contains want, case-insensitively.

func DefaultConfigTOML 

func DefaultConfigTOML() string

DefaultConfigTOML returns a complete, annotated configuration with every option shown at its default value, Options with no default are shown commented.

func ErrorCode 

func ErrorCode(err error) string

ErrorCode unwraps an application error code from err, or returns EINTERNAL if none is present, or "" if err is nil.

Types

type BodyScope 

type BodyScope string

BodyScope is a rendering scope for item bodies, resolved three-tier (settings -> output -> feed). 

const (
	BodyTitle         BodyScope = "title"          // headline only
	BodyExcerpt       BodyScope = "excerpt"        // excerpt + title-link
	BodyFull          BodyScope = "full"           // full feed body, always expanded
	BodyExcerptExpand BodyScope = "excerpt+expand" // excerpt + collapsed full body (<details>)
)

func ResolveBody 

func ResolveBody(settings, output, feed BodyScope) BodyScope

ResolveBody applies three-tier resolution: feed wins over output wins over settings. Empty string means "inherit".

type Config 

type Config struct {
	Settings Settings     `toml:"settings"`
	Fetch    FetchConfig  `toml:"fetch"`
	Fonts    FontConfig   `toml:"fonts"`
	Outputs  []OutputConf `toml:"output"`
	Feeds    []FeedConf   `toml:"feed"`

	// Location is the resolved display *time.Location, threaded through parsing
	// (ParseInLocation) and rendering. A TZ slip reorders the river, not just
	// misdisplays it
	Location *time.Location `toml:"-"`

	// Warnings collects non-fatal problems found while loading (e.g. unknown
	// keys, likely typos).
	Warnings []string `toml:"-"`
}

Config is the decoded TOML configuration.

func DefaultConfig 

func DefaultConfig() *Config

DefaultFetchConfig returns the built-in fetch settings, used also for tools that do not depend on a live config file, such as `firehose test...`

func LoadConfig 

func LoadConfig(path string) (*Config, error)

LoadConfig reads and decodes the TOML config at path, applies defaults, resolves the display Location, and validates.

func (*Config) FeedConfByURL 

func (c *Config) FeedConfByURL() map[string]FeedConf

FeedConfByURL maps config feed blocks by URL

func (*Config) FontsCSSURL 

func (c *Config) FontsCSSURL() string

FontsCSSURL resolves the remote font stylesheet at runtime

func (*Config) MaxDisplayWindow added in v0.2.0 

func (c *Config) MaxDisplayWindow() time.Duration

MaxDisplayWindow is the widest window any feed uses — the single Since bound for the item query; per-feed narrowing happens at render.

func (*Config) ToFeeds 

func (c *Config) ToFeeds() []*Feed

ToFeeds converts config feeds into domain Feed values

func (*Config) ToOutputs 

func (c *Config) ToOutputs() []*Output

ToOutputs converts config outputs into domain Output values, applying the InNav/Health flags. The synthetic health output (firehose.html) is appended: generated every run, excluded from nav.

func (*Config) Validate 

func (c *Config) Validate() error

Validate checks the fully-defaulted config for internal consistency. Returns the first problem as an EINVALID error. This is the core of `firehose check`.

func (*Config) WindowFor added in v0.2.0 

func (c *Config) WindowFor(fc FeedConf) time.Duration

WindowFor resolves a feed's display window: the per-feed override when set, else the global setting. Slow civic feeds get long windows without bloating every section.

type Duration 

type Duration time.Duration

Duration is a TOML-decodable time.Duration accepting Go duration strings ("336h", "20s"). Keeps the config surface human-friendly.

func (Duration) D 

func (d Duration) D() time.Duration

D returns the value as a time.Duration.

func (*Duration) UnmarshalText 

func (d *Duration) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler for TOML decoding.

type Error 

type Error struct {
	Code    string
	Message string
}

Error is a domain error carrying a machine-readable Code and a human-readable Message. Mirrors the WTF error pattern.

func Errorf 

func Errorf(code, format string, args ...any) *Error

Errorf constructs an *Error with a formatted message.

func (*Error) Error 

func (e *Error) Error() string

Error implements the error interface.

type ExcerptImage 

type ExcerptImage string

ExcerptImage controls whether the lead image survives into an excerpt. 

const (
	ExcerptImageLead ExcerptImage = "lead" // keep first image, lazy, capped height
	ExcerptImageNone ExcerptImage = "none" // drop images from the excerpt
)

func ResolveExcerptImage 

func ResolveExcerptImage(settings, output, feed ExcerptImage) ExcerptImage

ResolveExcerptImage applies the same three-tier resolution for the lead-image, if any

type Feed 

type Feed struct {
	ID  int64  // cache primary key
	URL string // canonical URL; updated in place when a 301 is followed

	// Title is the feed's self-reported title. A config Title override wins
	// when the feed's own title is garbage ("Untitled Feed", a bare domain).
	Title string

	// Categories tag the feed into sections. A feed may belong to several; an
	// item inherits its feed's categories. Empty means it appears only in the
	// ALL river.
	Categories []string

	// Config-derived per-feed overrides. Nil/zero means "inherit from output
	// then settings" per the three-tier resolution.
	Body           string   // "", title, excerpt, full, excerpt+expand
	ExcerptImage   string   // "", lead, none
	Exclude        []string // simple keyword filters (config, not hook)
	Include        []string // simple keyword filters (config, not hook)
	StripSelectors []string // per-feed cruft removal, applied post-parse

	// Per-feed fetch overrides for CDN-hostile endpoints. UserAgent replaces
	// the global one; Headers are set verbatim on the request. Identifying
	// honestly is the default; impersonation is the operator's per-feed call.
	UserAgent string
	Headers   map[string]string

	// Conditional-GET validators, persisted between runs.
	ETag         string
	LastModified string

	// Health / backoff state. LastStatus is an application error code
	// (ENOTFOUND, EPARSE, ETIMEOUT, ...) reported in firehose.html.
	FailCount    int
	LastStatus   string
	LastSuccess  time.Time // last fetch that produced items
	LastFetched  time.Time
	NextEarliest time.Time // backoff; do not fetch before this
}

Feed is a subscribed source. Fetch-state fields (ETag, LastModified, FailCount, backoff) live here because the cache tracks per-feed health and conditional-GET validators.

ID is a private cache-internal join key and must never travel outside the database or a single run: identity that travels is the URL. (This invariant is what makes rowid reuse safe; see the schema.)

type FeedConf 

type FeedConf struct {
	URL            string       `toml:"url"`
	Title          string       `toml:"title"` // override garbage self-reported titles
	Categories     []string     `toml:"categories"`
	Body           BodyScope    `toml:"body"`
	ExcerptImage   ExcerptImage `toml:"excerpt_image"`
	Exclude        []string     `toml:"exclude"`
	Include        []string     `toml:"include"`
	StripSelectors []string     `toml:"strip_selectors"`
	DisplayWindow  Duration     `toml:"display_window"` // per-feed override; zero inherits settings

	// Per-feed fetch overrides (CDN-hostile endpoints).
	UserAgent string            `toml:"user_agent"`
	Headers   map[string]string `toml:"headers"`
}

FeedConf is a configured feed and its per-feed overrides.

type FeedFilter 

type FeedFilter struct {
	ID       *int64
	URL      *string
	Category *string

	// DueOnly, when true, returns only feeds whose NextEarliest has passed
	// (i.e. not currently in backoff).
	DueOnly bool

	Offset int
	Limit  int
}

FeedFilter narrows FindFeeds. Zero-value fields are ignored.

type FeedService 

type FeedService interface {
	// FindFeeds returns feeds matching filter, and the total count.
	FindFeeds(ctx context.Context, filter FeedFilter) ([]*Feed, int, error)

	// FindFeedByURL returns the feed with the given URL, or an ENOTFOUND error.
	FindFeedByURL(ctx context.Context, url string) (*Feed, error)

	// SyncFeeds reconciles the configured feed set into the cache: inserts new
	// feeds, updates categories/config on existing ones, and removes feeds no
	// longer in config (cascading their items).
	SyncFeeds(ctx context.Context, feeds []*Feed) error

	// UpdateFeed applies fetch/health state to a feed.
	UpdateFeed(ctx context.Context, id int64, upd FeedUpdate) (*Feed, error)
}

FeedService is the cache's feed store. Implemented by the sqlite and mock packages

type FeedUpdate 

type FeedUpdate struct {
	URL          *string // set when a 301 redirect is persisted
	Title        *string
	ETag         *string
	LastModified *string
	FailCount    *int
	LastStatus   *string
	LastSuccess  *time.Time
	LastFetched  *time.Time
	NextEarliest *time.Time
}

FeedUpdate carries mutable fetch/health state written back after a fetch. Pointer fields left nil are unchanged.

type FetchConfig 

type FetchConfig struct {
	Concurrency   int      `toml:"concurrency"`
	PerHostSerial bool     `toml:"per_host_serial"`
	Timeout       Duration `toml:"timeout"`
	UserAgent     string   `toml:"user_agent"`

	// AcceptLanguage is sent on every request when non-empty. Browsers always
	// send it; its absence is a bot tell for CDN filtering. We are honest
	// except when we cannot be, but this is a personal use tool after all.
	AcceptLanguage string `toml:"accept_language"`
}

FetchConfig holds politeness and concurrency controls.

func DefaultFetchConfig 

func DefaultFetchConfig() FetchConfig

type FontConfig 

type FontConfig struct {
	ContentFamily string `toml:"content_family"`
	ChromeFamily  string `toml:"chrome_family"`

	// CSSURL is a remote font stylesheet imported by style.css. Defaulted to
	// Google Fonts for the default families when no self-hosted sources are
	// configured. If you change the families, change this too (or self-host).
	CSSURL string `toml:"css_url"`

	// Self-hosted overrides: when set, @font-face rules are emitted and no
	// remote stylesheet is defaulted.
	ContentSrc string `toml:"content_src"`
	ChromeSrc  string `toml:"chrome_src"`
}

FontConfig holds the content/chrome family split and where the font files come from: a remote stylesheet (default: Google Fonts for the default families) or self-hosted woff2 sources, which take precedence when set.

type Item 

type Item struct {
	ID     kid.ID // first-class value type; id.Time() etc.
	FeedID int64
	GUID   string // feed-provided; dedupe key with FeedID

	Title  string
	URL    string // source link; MAY be empty (see linkless-title rule)
	Author string

	Published time.Time // sort key for the river (TZ-correct, ParseInLocation)

	// BodyHTML is the sanitized item body. When FullContent is true this is the
	// whole item (content:encoded); otherwise it is a teaser (description).
	BodyHTML string

	// SummaryHTML is the sanitized feed-provided summary when it is distinct
	// from the body (feed shipped both content:encoded and description).
	// Excerpt source-of-truth prefers this over truncating BodyHTML.
	SummaryHTML string

	// LeadImage is the first image URL for the item — the first inline <img>
	// in the sanitized body, else the first image enclosure/media thumbnail.
	// Used when excerpt_image = "lead" restores an image to a text excerpt.
	LeadImage string

	// FullContent gates excerpt+expand: the expand affordance is only offered
	// when the feed actually shipped a full body. Inferred at fetch time.
	FullContent bool

	// WordCount of the sanitized text, for the reading-time hint.
	WordCount int

	FetchedAt time.Time

	// Categories are inherited from the item's feed at render time (not stored
	// on the row). Populated when rendering the ALL river's section label.
	Categories []string
}

Item is a single feed entry, already sanitized (and, where the feed declared a language, highlighted). BodyHTML is safe to render directly. 

func (it *Item) HasLink() bool

HasLink reports whether the item has a usable source URL. When false, the template renders the title as plain text rather than a dead anchor.

func (*Item) ReadingTime 

func (it *Item) ReadingTime() time.Duration

ReadingTime returns an estimated read duration from WordCount (~220 wpm), rounded up to whole minutes with a floor of one.

type ItemFilter 

type ItemFilter struct {
	// Categories selects items in any of these categories. Empty selects all
	// (the ALL river).
	Categories []string

	// Since bounds the display window (e.g. now - 14d). Zero means unbounded.
	Since time.Time

	Limit  int
	Offset int
}

type ItemService 

type ItemService interface {
	// FindItems returns items matching filter, newest-first (Published desc,
	// then ID as a stable tiebreaker for deterministic output), and the total
	// count.
	FindItems(ctx context.Context, filter ItemFilter) ([]*Item, int, error)

	// UpsertItems inserts new items and updates changed ones, keyed by
	// (FeedID, GUID). Existing IDs are preserved so kid.ID stays stable.
	UpsertItems(ctx context.Context, items []*Item) error

	// ItemStats returns (FeedID, Published) for every cached item — enough
	// for the health page to compute per-feed shown/cached counts without
	// loading bodies.
	ItemStats(ctx context.Context) ([]ItemStat, error)

	// PurgeExpired deletes items older than the cache retention cutoff. Returns
	// the number removed. (Distinct from the display window: retention keeps
	// GUID history longer to prevent re-published old items reappearing.)
	PurgeExpired(ctx context.Context, olderThan time.Time) (int, error)
}

ItemService is the cache's item store. Implemented by the sqlite package and mock for tests.

type ItemStat added in v0.2.0 

type ItemStat struct {
	FeedID    int64
	Published time.Time
}

ItemFilter narrows which items render into a given output. A section IS a filter — this is the WTF XxxFilter idiom, and it is the whole section mechanism. The ALL river is an ItemFilter with empty Categories. ItemStat is the health page's view of one cached item.

type OPML 

type OPML struct {
	XMLName xml.Name `xml:"opml"`
	Version string   `xml:"version,attr"`
	Head    OPMLHead `xml:"head"`
	Body    OPMLBody `xml:"body"`
}

OPML is the domain representation of the feed list for interchange (export today; import is planned). Sections map to nested outline groups; a feed in multiple sections is duplicated under each — lossless, and truest to how firehose treats categories.

func BuildOPML 

func BuildOPML(cfg *Config, section string) *OPML

BuildOPML assembles the OPML document for a config. Section groups are emitted in output order (ALL and health outputs are not groups); feeds matching no section land at the top level. A non-empty section name limits the document to that one group.

type OPMLBody 

type OPMLBody struct {
	Outlines []Outline `xml:"outline"`
}

OPMLBody holds the top-level outlines.

type OPMLHead 

type OPMLHead struct {
	Title string `xml:"title"`
}

OPMLHead carries document metadata.

type Outline 

type Outline struct {
	Text     string    `xml:"text,attr"`
	Title    string    `xml:"title,attr,omitempty"`
	Type     string    `xml:"type,attr,omitempty"`
	XMLURL   string    `xml:"xmlUrl,attr,omitempty"`
	Children []Outline `xml:"outline,omitempty"`
}

Outline is one OPML node: a section group (Children set) or a feed (Type "rss", XMLURL set).

type Output 

type Output struct {
	Name  string // stable identifier ("gov", "all")
	File  string // output filename ("gov.html", "index.html")
	Title string // page <h1> ("Government & EM")

	// Categories selects which items appear. ["*"] (or empty) is the ALL river.
	Categories []string

	// Per-output overrides (win over settings, lose to per-feed).
	Body         BodyScope
	ExcerptImage ExcerptImage
	ReadingTime  *bool

	// InNav controls whether this output appears in the cross-page nav strip.
	// The firehose.html health page sets this false — generated every run, but
	// unlinked to the "public" nav. There is no security in this scheme.
	InNav bool

	// Health marks the special firehose.html output, rendered from feed
	// health state rather than the item river.
	Health bool
}

Output is one rendered river page (a section, or the ALL river, or the unlinked health page). Sections are not a special HTML type — an Output is the same template rendered against a different ItemFilter. The only section-aware element on a page is the nav strip, which is data-driven from the set of Outputs.

func (*Output) Filter 

func (o *Output) Filter(since time.Time) ItemFilter

Filter builds the ItemFilter for this output given the display-window cutoff (items published before since are excluded).

func (*Output) IsAll 

func (o *Output) IsAll() bool

IsAll reports whether this output is the ALL river (empty or "*" categories).

type OutputConf 

type OutputConf struct {
	Name         string       `toml:"name"`
	File         string       `toml:"file"`
	Title        string       `toml:"title"`
	Categories   []string     `toml:"categories"`
	Body         BodyScope    `toml:"body"`
	ExcerptImage ExcerptImage `toml:"excerpt_image"`
	ReadingTime  *bool        `toml:"reading_time"`
}

OutputConf is a configured section/river.

type Settings 

type Settings struct {
	OutputDir          string       `toml:"output_dir"`
	CacheDB            string       `toml:"cache_db"`
	DisplayWindow      Duration     `toml:"display_window"`  // what renders (e.g. 14d)
	CacheRetention     Duration     `toml:"cache_retention"` // GUID history (longer)
	Timezone           string       `toml:"timezone"`        // IANA name; resolved into Location
	Body               BodyScope    `toml:"body"`
	ExcerptWords       int          `toml:"excerpt_words"`
	ExcerptImage       ExcerptImage `toml:"excerpt_image"`
	Typography         bool         `toml:"typography"`
	ReadingTime        bool         `toml:"reading_time"`
	Highlight          bool         `toml:"highlight"`       // declared-language-only; never guess
	Dedupe             bool         `toml:"dedupe"`          // collapse the same story arriving via multiple feeds
	Theme              string       `toml:"theme"`           // auto | light | dark (page default; toggle overrides)
	HighlightTheme     string       `toml:"highlight_theme"` // chroma style, light mode
	HighlightThemeDark string       `toml:"highlight_theme_dark"`

	// Integration with the Author's CMS; NoteTemplate, when set, renders a per-item "note" link with {title} and
	// {url} substituted (query-escaped) No backend: it is just a URL and could be used to support other CMS.
	NoteTemplate string `toml:"note_template"`
}

Settings holds global defaults and paths.

Source Files

View all Source files

Directories

Path Synopsis
cmd
firehose command
Command firehose fetches subscribed feeds and writes static river-of-news HTML.
 Command firehose fetches subscribed feeds and writes static river-of-news HTML.
Package feed fetches and converts syndication feeds.
 Package feed fetches and converts syndication feeds.
Package htmlx ensures HTML5-correct self-closing elements as x/net/html.Render emits XML-style self-closing elements ("<br/>"),
 Package htmlx ensures HTML5-correct self-closing elements as x/net/html.Render emits XML-style self-closing elements ("<br/>"),
Package mock provides hand-written mocks of the root service interfaces for testing, in the WTF style: a struct of function fields, each method delegating to its field.
 Package mock provides hand-written mocks of the root service interfaces for testing, in the WTF style: a struct of function fields, each method delegating to its field.
Package render turns cached items into the static site.
 Package render turns cached items into the static site.
Package sqlite implements the firehose cache using the pure-Go (no cgo) SQLite driver, modernc.org/sqlite, for portability reasons across various Linux architectures.
 Package sqlite implements the firehose cache using the pure-Go (no cgo) SQLite driver, modernc.org/sqlite, for portability reasons across various Linux architectures.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.