feedparser

README

simplest-feed-parser

A format-agnostic feed parser for Go that supports RSS (0.90, 0.91, 0.92, 1.0, 2.0), Atom, and JSON Feed formats. Built on Go's standard library, with a single official golang.org/x dependency for character set decoding.

Features

• Multi-format support: Automatically detects and parses RSS, Atom, and JSON Feed formats
• Minimal dependencies: Go's standard library plus golang.org/x/net/html/charset
• Real-world tolerance: Handles legacy character encodings and HTML character entities
• Honest recovery: RSS documents, and JSON Feed items, that break their own specification are parsed rather than rejected, and every deviation is reported on Feed.Warnings
• Unified API: Single canonical model for all feed formats
• Format preservation: Format-specific data preserved in dedicated branches
• Bounded: A feed is read whole into memory, and how much may be read is capped — by default 10 MiB, see Memory and Limits
• Cancellable: A context cancels the fetch, the read and the parse
• Polite polling: Sends Accept and conditional-request headers, and reports a 304 as an answer rather than a failure — see Fetch
• Type-safe errors: Structured error types, each carrying the cause it wraps

Supported Formats

• RSS 0.90 - Netscape's original RSS format
• RSS 0.91 - UserLand RSS 0.91
• RSS 0.92 - UserLand RSS 0.92
• RSS 1.0 - RDF Site Summary
• RSS 2.0 - Really Simple Syndication
• Atom - Atom Syndication Format 1.0 (RFC 4287); Atom 0.3 is not supported
• JSON Feed - JSON Feed v1.0 and v1.1

Installation

go get gitlab.com/sunderee/simplest-feed-parser

Quick Start

Parse from URL

package main

import (
    "context"
    "fmt"
    "log"
    
    "gitlab.com/sunderee/simplest-feed-parser"
)

func main() {
    ctx := context.Background()
    
    feed, err := feedparser.ParseFromURL(ctx, "https://example.com/feed.xml")
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Printf("Feed: %s\n", feed.Title)
    fmt.Printf("Items: %d\n", len(feed.Items))
    
    for _, item := range feed.Items {
        fmt.Printf("- %s: %s\n", item.Title, item.Link)
    }
}

Parse from Reader

package main

import (
    "context"
    "fmt"
    "os"
    
    "gitlab.com/sunderee/simplest-feed-parser"
)

func main() {
    file, err := os.Open("feed.xml")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()
    
    ctx := context.Background()
    feed, err := feedparser.ParseFromSource(ctx, file)
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Printf("Feed: %s\n", feed.Title)
}

With Options

package main

import (
    "context"
    "net/http"
    "time"
    
    "gitlab.com/sunderee/simplest-feed-parser"
)

func main() {
    ctx := context.Background()

    feed, err := feedparser.ParseFromURL(
        ctx,
        "https://example.com/feed.xml",
        feedparser.WithUserAgent("MyApp/1.0"),
        feedparser.WithTimeout(15*time.Second),
        feedparser.WithMaxSize(2<<20), // refuse anything over 2 MiB
    )
    if err != nil {
        log.Fatal(err)
    }

    // Use feed...
}

Supply your own client when you need one — a shared transport, a proxy, a redirect policy. It brings its own timeout, and WithTimeout is then ignored rather than silently overriding it:

client := &http.Client{Timeout: 10 * time.Second}

feed, err := feedparser.ParseFromURL(
    ctx,
    "https://example.com/feed.xml",
    feedparser.WithHTTPClient(client),
)

API Reference

Functions

ParseFromURL(ctx context.Context, url string, opts ...Option) (*model.Feed, error)

Fetches a feed from the given URL and parses it. Returns a canonical *model.Feed or an error.

Parameters:

• ctx: Context for cancellation and timeout
• url: URL of the feed to fetch
• opts: Optional configuration (see Options below)

Returns:

• *model.Feed: Parsed feed in canonical format
• error: Error if parsing or network request fails

A 304 answer to a conditional request has no feed to return, so it surfaces as an error matching ErrNotModified. Use Fetch to poll a feed properly.

ParseFromSource(ctx context.Context, r io.Reader, opts ...Option) (*model.Feed, error)

Parses a feed from the provided reader. Returns a canonical *model.Feed or an error.

Parameters:

• ctx: Context for cancellation and timeout
• r: Reader containing feed data
• opts: Optional configuration (see Options below)

Returns:

• *model.Feed: Parsed feed in canonical format
• error: Error if parsing fails

Fetch(ctx context.Context, url string, opts ...Option) (*Response, error)

Retrieves a feed over HTTP and parses it, reporting what the transport said alongside the feed. It is ParseFromURL with room for the answers that do not fit in a model.Feed.

type Response struct {
    Feed         *model.Feed // nil when NotModified
    URL          string      // where the feed was finally read from
    StatusCode   int
    ETag         string      // hand back with WithETag
    LastModified time.Time   // hand back with WithLastModified
    NotModified  bool        // the server said 304: your copy is current
}

URL is the URL after any redirects, which is what a document's relative links resolve against — not necessarily the URL you asked for.

Polling a feed without re-downloading it:

resp, err := feedparser.Fetch(ctx, url,
    feedparser.WithETag(saved.ETag),
    feedparser.WithLastModified(saved.LastModified))
if err != nil {
    return err
}
if resp.NotModified {
    return nil // what we already have is current
}
save(resp.Feed, resp.ETag, resp.LastModified)

Options

WithHTTPClient(client *http.Client) Option

Sets a custom HTTP client for network requests. When provided, this client is used instead of the default client, and WithTimeout has no effect — a caller supplying a client owns its timeout policy.

WithUserAgent(ua string) Option

Sets a custom User-Agent header for HTTP requests. Defaults to "simplest-feed-parser/1.0".

WithTimeout(timeout time.Duration) Option

Sets the timeout for HTTP requests. Defaults to 30 seconds. Ignored when WithHTTPClient is supplied.

WithMaxSize(maxSize int64) Option

Bounds how many bytes are read from the input, for both entry points. Defaults to 10 MiB. Input past the bound is not read, and the parse fails with an error matching ErrTooLarge rather than returning a feed built from a truncated document. Zero or less removes the bound — see Memory and Limits before doing that.

WithAccept(accept string) Option

Replaces the Accept header sent with a fetch. The default lists the feed media types ahead of the generic ones and ends in */*, so that a server which would otherwise answer 406 still serves the feed. An empty value sends no Accept header.

WithETag(etag string) Option

Sends If-None-Match carrying the entity tag a previous fetch returned.

WithLastModified(t time.Time) Option

Sends If-Modified-Since carrying the time a previous fetch returned. The zero time sends no header.

WithContentType(contentType string) Option

Declares the media type the input arrived with, for input whose type is known from somewhere other than the content. Fetch sets this from the response for you.

It is a fallback, not an override. The format is read from the content first, because feeds are routinely served as text/plain, text/html and application/octet-stream, and trusting the server over the document would break feeds that parse today. The declared type is consulted only when the content names no format at all, and even then only if parsing under it recovers actual entries — the RSS parsers do not fail on content that is not RSS, so without that condition every mislabelled website would become an empty feed. A parse resting on the declaration is reported on Feed.Warnings.

Error Types

The library provides structured error types for different failure modes. Each carries the cause it wraps, so errors.As says where a failure happened and errors.Is says what went wrong about the same error value.

DetectionError

Indicates that the feed format could not be determined.

var detErr *feedparser.DetectionError
if errors.As(err, &detErr) {
    fmt.Printf("Could not detect format: %s\n", detErr.Reason)
}

ParseError

Indicates that the feed content is malformed or invalid. Format names the format the content was parsed as, and is empty when the failure happened before a format was settled on.

var parseErr *feedparser.ParseError
if errors.As(err, &parseErr) {
    fmt.Printf("Parse error in %s: %s\n", parseErr.Format, parseErr.Reason)
}

NetworkError

Indicates an HTTP or network failure. StatusCode is the status the server answered with, or zero if the request failed before a response was read.

var netErr *feedparser.NetworkError
if errors.As(err, &netErr) {
    fmt.Printf("Network error for %s: %s\n", netErr.URL, netErr.Reason)
}

Error Sentinels

These classify the conditions a caller is likely to branch on. They are never returned bare — each is carried as the cause of one of the typed errors above.

Sentinel	Means
ErrUnknownFormat	The content names no format, and no declared media type recovered one
ErrMalformedXML	An XML parser rejected the document; the *xml.SyntaxError and its line number stay reachable with errors.As
ErrMissingRequired	An element the format requires a reader to reject the document for — Atom and JSON Feed only, see Validation and Strictness
ErrTooLarge	The input exceeded WithMaxSize
ErrNotModified	A 304 answer to a conditional request; an expected outcome of polling, not a failure

feed, err := feedparser.ParseFromURL(ctx, url)
switch {
case errors.Is(err, feedparser.ErrTooLarge):
    // over the size limit
case errors.Is(err, feedparser.ErrUnknownFormat):
    // not a feed we know
case errors.Is(err, context.Canceled):
    // the caller gave up; also reachable through a NetworkError
}

Cancellation is the exception to the "errors are typed" rule. A context cancelled or expired during ParseFromSource surfaces as context.Canceled or context.DeadlineExceeded itself: a caller withdrawing its own request is not a defect in the feed, and reporting it as a ParseError would say that it was. A fetch cancelled mid-request still surfaces as a NetworkError, which unwraps to the context error.

Feed Model

The library uses a canonical feed model that unifies all supported formats while preserving format-specific data:

type Feed struct {
    ID          string
    Title       string
    Link        string
    Description string
    Language    string
    Updated     *time.Time
    Authors     []Person
    Items       []Entry
    Image       *Image
    Warnings    []string      // deviations the parse recovered from
    
    // Format-specific branches
    RSS      *RSSData      // RSS-specific data
    Atom     *AtomData     // Atom-specific data
    JSONFeed *JSONData     // JSON Feed-specific data
}

Common Fields

All feeds expose these common fields:

• ID: Unique identifier for the feed
• Title: Feed title
• Link: Feed URL
• Description: Feed description/subtitle
• Language: Feed language code
• Updated: Last update timestamp
• Authors: List of feed authors
• Items: List of feed entries/items
• Image: Feed image/logo
• Warnings: What the document did that its specification does not allow, in document order (see Validation and Strictness)

Format-Specific Branches

Format-specific data is preserved in dedicated branches:

• RSS: Contains RSS-specific fields (version, generator, TTL, cloud, the feed's own SelfURL from <atom:link rel="self">, the Dublin Core and syndication modules, etc.)
• Atom: Contains Atom-specific fields (generator, icon, logo, subtitle, rights, contributors, etc.)
• JSONFeed: Contains JSON Feed-specific fields (version, feedURL, nextURL, icon, favicon, hubs, and the document's _-prefixed Extensions)

Only the branch corresponding to the parsed format will be populated; others will be nil.

JSONFeed.Extensions and Entry.JSONFeed.Extensions carry JSON Feed's whole extension mechanism: every member whose name begins with an underscore, keyed by that name and holding the value as decoded JSON (map[string]any, []any, string, json.Number, bool or nil). JSON Feed's core vocabulary is deliberately small, so this is where the podcast and microblog namespaces put everything the core has no field for. Both are nil when the document declared none.

if podcast, ok := feed.JSONFeed.Extensions["_podcast"].(map[string]any); ok {
    fmt.Println(podcast["subtitle"])
}

Entry Model

Each feed item/entry is represented as:

type Entry struct {
    ID          string
    Title       string
    Link        string
    Description string
    Content     string
    Language    string
    Published   *time.Time
    Updated     *time.Time
    Rights      string
    Authors     []Person
    Enclosures  []Enclosure
    Categories  []Category
    Source      *Source
    CommentsURL string
    
    // Format-specific branches
    RSS      *RSSEntryData
    Atom     *AtomEntryData
    JSONFeed *JSONEntryData
}

Language is populated where the format carries a per-entry language: xml:lang in Atom, and language on a JSON Feed 1.1 item. It falls back to nothing rather than to the feed's language when an entry explicitly declares none.

Authors is []Person, and Person.Avatar is populated where the format carries a picture of the person — JSON Feed's author.avatar, and nothing in RSS or Atom. A JSON Feed item that names no author of its own takes the feed's, as JSON Feed 1.1 directs.

Enclosures is []Enclosure, whose Title and DurationInSeconds are populated where the format states them. A JSON Feed attachment states both; an RSS <enclosure> is a url, a length and a type and nothing else, so they are empty and nil there. DurationInSeconds is what a podcast client needs from an episode:

for _, enc := range entry.Enclosures {
    if enc.DurationInSeconds != nil {
        fmt.Printf("%s runs %ds\n", enc.Title, *enc.DurationInSeconds)
    }
}

Content is the entry's full body and Description its excerpt; they are different values, and neither is filled in from the other. For RSS 2.0 and 0.92 the body comes from <content:encoded> (the content module), which is how most publishers ship it.

RSS.GUID carries what <guid> declared. Entry.ID holds its value, but only IsPermaLink says whether that value is a URL to the item or an opaque identifier. An item declaring a permalink guid and no <link> takes its address from the guid, provided the guid really is an absolute http(s) URL — publishers routinely leave isPermaLink at its default of true on tag: URIs and hashes that were never addresses.

RSS.DublinCore carries an RSS 1.0 item's Dublin Core metadata. RSS 1.0 defines an item as title, link and description, so everything else a document says about one — author, date, subject, rights, publisher — it says through that module. creator, date, subject and rights are also mapped onto Authors, Published, Categories and Rights; the other eleven live here only. At channel level, dc:language, dc:date and dc:creator populate Feed.Language, Feed.Updated and Feed.Authors, which RSS 1.0 has no elements of its own for.

Atom.Content describes what atom:content declared, including the case Content cannot express. RFC 4287 §4.1.3.2 lets an entry hold its content at another IRI, and such an entry is required to be empty, so Content is "" and Atom.Content.Src is where the content actually is:

if c := entry.Atom.Content; c != nil && c.Src != "" {
    fmt.Printf("content of type %s lives at %s\n", c.Type, c.Src)
}

Validation and Strictness

Strictness is decided per format, because the formats are not comparable. Atom is validated against its specification and rejected when it breaks it. JSON Feed is validated at the level of the document, and its items are parsed permissively. RSS is parsed permissively throughout. Every deviation that is recovered rather than rejected is reported on Feed.Warnings.

Atom is a standards-track specification (RFC 4287) whose required elements are normative and whose feeds are produced by software written against it; a reader that accepts an entry with no id is not reading Atom. RSS's rules come from a 1999 DTD and from prose that publishers have ignored for two decades, and the elements they miss are ones no consumer needs. Rejecting a feed for a missing channel <link> rejects feeds people actually read.

JSON Feed falls between the two, and the line runs between the document and its items. A feed that does not say which version it is, what it is called, or what its items are is not something a reader can act on, so it is rejected. What one item says is that item's business: no item-level defect ever rejects the feed, and none ever costs a sibling item. Where the specification directs readers to discard an item — which it does for an item with no id, since an item that cannot be identified cannot be recognised in tomorrow's fetch — that item is discarded and the discard is reported. Everything else is kept and reported. An item carrying only a url and a title breaks the rule that one of content_html and content_text must be present, and it is also what every link blog and microblog publishes; discarding it would lose the url, title and attachments it does carry.

Nothing is ever silently repaired. Recovery that leaves no trace is indistinguishable from a conformant document, so each recovered deviation appends one line to Feed.Warnings, naming the element the way the document spells it — channel/item[2]/guid for an XML format, items[2]/authors for JSON:

feed, err := feedparser.ParseFromSource(ctx, r)
if err != nil {
    log.Fatal(err)
}
for _, w := range feed.Warnings {
    log.Printf("feed deviation: %s", w) // channel/skipHours/hour: 99 is outside 0-23
}

Warnings is empty for a conformant document.

What is rejected

• Input that is not a well-formed document of a recognisable format: malformed XML or JSON, an unknown root element, an encoding the decoder does not know.
• Atom: id, title and updated on the feed and on every entry (§4.1.1, §4.1.2), the mandated <div> wrapper on type="xhtml" constructs, and two rel="alternate" links with the same type and hreflang. An unparseable timestamp or a non-numeric atom:link/@length is an error quoting the offending value, not a silently nil timestamp. A timestamp that is readable but states no UTC offset — which RFC 3339, and so §3.3, requires — is read as UTC and reported on Feed.Warnings, because the feed still names every entry it publishes.
• JSON Feed: version (and a version the library does not implement), title, and items — the three things the document has to state about itself. A stream carrying anything after the top-level object is rejected too: two concatenated feeds are not one feed, and returning the first as though it were the whole of what arrived hides the second entirely.

Atom entries are never silently dropped: a caller cannot distinguish a feed that published nine entries from one that published ten and had one discarded on the way through. Neither are JSON Feed items — the specification calls for discarding some of them, so every discard is reported.

What is recovered and reported (RSS)

Deviation	Behaviour
Channel missing title, link or description	Parsed; warning
RSS 0.91 missing its required language or image	Parsed; warning
Incomplete <image> (no url, title or link)	Kept as-is; warning
RSS 2.0 item with neither title nor description	Item kept — it may carry a guid, link and enclosure; warning
RSS 1.0 missing rdf:about, item title/link, or all items	Parsed; warning
RSS 0.90 item missing title/link, and its 1–15 item rule	Parsed; warning
RSS 0.91's DTD limits: 15 items, 100-character title, 500-character description	All kept whole; warning
<skipHours><hour> outside 0–23, <skipDays><day> that is not a day	Value kept; warning
<cloud>/@protocol outside xml-rpc, soap, http-post	Value kept; warning
Negative or non-numeric <ttl>, more than one <enclosure> per item	Value kept; warning
Image dimensions above RSS 0.91's 144×400 maxima	Clamped; warning
Unparseable pubDate, lastBuildDate or dc:date	Timestamp left nil; warning quoting the value
A date whose timezone abbreviation is unknown or ambiguous (IST, BST)	Timestamp left nil; warning quoting the value and why
A date stating no timezone at all	Read as UTC; warning, since the offset is this library's guess
An <rdf:li> naming no item, or an item named by no <rdf:li>	Items kept, sequence order applied; warning
An rss@version this library does not know	Parsed as RSS 2.0; warning

Leniency also covers things no specification rules on: unknown elements, legacy encodings, HTML entities, and unescaped inline markup are all accommodated (see below).

What is recovered and reported (JSON Feed)

Deviation	Behaviour
Item with neither content_html nor content_text	Item kept — it may carry a url, title and attachments; warning
Item with no id, or an empty one	Item discarded, as the specification directs; warning
Item whose id is an object, an array, a boolean or null	Item discarded; warning
An entry of items that is not an object	Discarded; warning
authors that is not an array, or holds no author object	The singular author is used instead; warning
author that is not an object	Ignored; warning
expired that is not a boolean	Left nil rather than read as false; warning
Unparseable date_published or date_modified	Timestamp left nil; warning quoting the value
A date stating no UTC offset, which RFC 3339 requires	Read as UTC; warning

A numeric id is converted to its string form, which the specification asks for explicitly; that is a conversion, not a deviation, and it warns about nothing.

Dates and Timezones

Every format carries dates, and a timestamp read wrongly is worse than one left empty: it is a falsehood presented with the same confidence as a fact. Two rules follow from that.

A timezone abbreviation resolves from a fixed table, never from the host. Go's time.Parse resolves an abbreviation against the machine's own zone database, and for one the machine does not know it invents a zone with that name and an offset of zero, returning no error. A feed timestamped EST therefore parsed to 14:55 UTC in New York and 09:55 UTC everywhere else — five hours apart, silently, from the same bytes. Abbreviations are now resolved from RFC 822 §5.1's list (UT, GMT, EST/EDT, CST/CDT, MST/MDT, PST/PDT, Z) plus the common European, Asian and Pacific ones, so the instant depends on the input alone.

An abbreviation naming more than one offset — IST is +05:30, +01:00 or +02:00; BST is +01:00 or +06:00 — is refused, and the warning says why. Picking one would record a coin toss as a fact.

A date that states no offset is read as UTC and reported. RFC 822, RFC 3339 and W3CDTF all require the offset once a time of day is given, so a value without one is a deviation, and the instant produced is wrong by however far the publisher sits from UTC. The wall clock and the day are still worth keeping, so the value is read — and the guess appears on Feed.Warnings:

item[0]/pubDate: "Mon, 05 Jun 2000 09:55:00": no timezone; read as UTC

W3CDTF's bare date is the one exception: <dc:date>2000-01-01</dc:date> is conformant RSS 1.0, so reading it as midnight UTC warns about nothing.

Beyond that, RFC 822 parsing accepts what feeds actually contain: an optional and possibly wrong day-of-week, one- or two-digit days, two- or four-digit years, optional seconds, full month and weekday names, surrounding and folded whitespace, a parenthesised zone comment (-0700 (PDT)), an abbreviation carrying an offset (GMT+02:00), and — since publishers do it — an ISO 8601 timestamp where RFC 822 was specified. ISO 8601 parsing accepts RFC 3339 with or without the colon in the offset, with or without seconds, and with fractional seconds. It does not accept ISO 8601's basic format, ordinal dates or week dates, none of which appear in feeds. The fallback runs one way only: an RSS date may be ISO 8601, but an Atom or JSON Feed date may not be RFC 822.

Format Notes

• Image dimensions are defaulted only where a specification defines them. RSS 0.91 declares 88×31 as its DTD's attribute defaults and caps dimensions at 144×400; RSS 0.92 and 2.0 inherit the defaults and drop the caps. RSS 0.90 and RSS 1.0 have no image dimensions at all, and none are invented for them — a zero Width means the document stated none.
• An rss@version this library does not know is parsed as RSS 2.0, whose element vocabulary is a superset of 0.91's and 0.92's, and the version is preserved verbatim on RSS.Version.
• <content:encoded> is read for RSS 2.0 and 0.92. The element's text is the markup, whether it arrives CDATA-wrapped or escaped, and it is kept intact.

Atom Conformance Notes

• Atom 1.0 (RFC 4287) only. Atom 0.3 (http://purl.org/atom/ns#) is not supported and is reported as an unknown format. It was superseded in 2005 and never standardised.
• Every entry needs id, title and updated (§4.1.2), as does the feed itself (§4.1.1).
• An element must not carry two rel="alternate" links with the same type and hreflang (§4.1.1), because nothing then says which of them is the entry. An absent rel counts as alternate (§4.2.7.2), as does the IANA IRI form (http://www.iana.org/assignments/relation/alternate).
• type="xhtml" constructs must carry the mandated wrapper <div> (§3.1.1.3). The <div> itself is not part of the content and is removed.
• XHTML atom:content keeps its markup; an XHTML text construct — title, subtitle, summary, rights — is flattened to a plain string, since that is what a text construct is defined to be however it was marked up.
• An entry with no atom:author takes its atom:source's, then the feed's (§4.2.1).

JSON Feed Conformance Notes

• JSON Feed 1.0 and 1.1 only. The version member decides, and a version string the library does not implement is reported as an unknown format.
• version may sit anywhere in the object. JSON object members are unordered, so the specification can only recommend that it comes first. The whole top-level object is scanned, at any size, and a version inside an item is not the feed's.
• authors and author fall back to one another in both directions. 1.1 deprecated the singular form and told publishers to keep emitting it for 1.0 readers, so documents carry both; an authors array that holds nothing usable does not suppress the author beside it.
• An item that names no author takes the feed's. JSON Feed 1.1 states this for authors, and the same reading is applied to 1.0's author: a one-author blog states its authorship once.
• An extension name is any member name beginning with _, which is the only rule the specification states. _1st is a name it allows.
• Relative URLs are not resolved. home_page_url, feed_url and an item's url are returned exactly as the document wrote them; the specification suggests resolving them against feed_url, and this library does not yet do so for any format.

Character Encodings and Entities

Feeds in the wild are frequently not well-formed UTF-8 XML, so the shared XML decoder makes two allowances:

• Legacy character encodings. A feed declaring any encoding in its XML declaration is transcoded to UTF-8 before parsing. Long-lived RSS feeds still commonly declare ISO-8859-1 or windows-1252; CJK encodings such as Shift_JIS and GB2312 are supported too. This is what golang.org/x/net/html/charset provides. An encoding the decoder does not recognise is reported as a DetectionError or ParseError rather than silently mangled.
• HTML character entities. XML predefines only <, >, &, " and ', but feeds routinely use HTML entities such as  , — and ’. These are resolved. A genuinely unknown entity is still an error.

The decoder is deliberately left in strict mode, so a document with mismatched or unclosed tags is rejected rather than silently reinterpreted.

Text elements containing unescaped inline markup are flattened to their text content — for example <description>Hello <b>bold</b> world</description> yields Hello bold world. Where the markup is block-level, the boundary becomes a single space, so <p>one</p><p>two</p> yields one two rather than fusing the two words together. Markup that is the value — HTML and XHTML atom:content — is kept intact rather than flattened.

Architecture

The library follows a layered architecture:

1. Intake: Reads the input whole, under the bound set by WithMaxSize, and wraps it so that the context can interrupt everything downstream — see Memory and Limits
2. Detection Layer: Identifies feed format using minimal inspection, falling back to a declared media type only where that recovers a feed
3. Format Parsers: Independent parsers for each format using streaming decoders
4. Adapter Layer: Maps format-specific structures to canonical model
5. Builder Pattern: Enforces invariants and validates the canonical model
6. Post-Processing: Optional transformations on the canonical model

This design ensures:

• Losslessness: All semantically meaningful data is preserved
• Extensibility: New formats can be added without modifying existing code
• Predictability: Deterministic parsing with explicit state management
• Minimal Dependencies: Go's standard library plus golang.org/x/net/html/charset

Memory and Limits

This library is not streaming. The parsers use streaming decoders, but the library around them does not: ParseFromSource reads the whole input into memory before parsing begins. It has to, because the format is detected from the content and each parser then reads that content from the start. Peak memory is therefore a multiple of the feed's size — the buffered document, plus the parsed model built from it.

Earlier versions of this README described the library as "streaming-friendly, designed for efficient memory usage". That was not true of the architecture and is not claimed here.

What is true is that the cost is now bounded. Reads stop at WithMaxSize, 10 MiB by default, and input past the bound fails with ErrTooLarge:

feed, err := feedparser.ParseFromURL(ctx, url, feedparser.WithMaxSize(2<<20))
if errors.Is(err, feedparser.ErrTooLarge) {
    // the feed is bigger than we are willing to hold
}

The bound matters most on ParseFromURL, where the input is a remote body whose size is decided by whoever serves it. Without one, a response that never ends is read until the process dies. Fetch also refuses a body whose declared Content-Length is over the bound before downloading any of it, though a server may declare nothing or lie — the read bound is what actually holds.

Passing WithMaxSize(0) removes the bound. Do that only for input you produced yourself.

Known limit: there is no separate cap on XML nesting depth or element count. A deeply nested document costs memory in proportion to its depth, and the size bound is what constrains it — a 10 MiB input cannot nest more than a few million levels deep. Go's encoding/xml is iterative, so this is not a stack overflow. If you parse untrusted feeds with an unbounded WithMaxSize, you have removed the only thing bounding this.

Cancellation

ctx cancels the fetch, the read and the parse. The context is checked on entry, on each read of the input, and on each buffer the parsers pull while decoding — a cancelled parse of a 4 MiB feed stops after roughly 12 KiB rather than at the end of the document.

This costs one 4 KiB buffer and about seven allocations per parse, constant in the size of the feed, and 1–6% of parse time.

Testing

Run the test suite:

make tests

The full non-fuzz CI gate (format, lint, race tests, coverage floors, tidy, govulncheck):

make verify

Or directly with Go:

go test -race -cover ./...

Requirements

• A current stable Go toolchain. The language version is recorded in go.mod (go); CI uses the matching golang: image.

License

Open-sourced under GNU AGPLv3.

Documentation

Overview

Package feedparser parses syndication feeds into one canonical model.

Seven formats are supported — RSS 0.90, 0.91, 0.92, 1.0 and 2.0, Atom 1.0, and JSON Feed 1.0/1.1 — and the format is detected from the input rather than declared by the caller. Both entry points return a model.Feed; the format-specific data that has no cross-format equivalent is preserved on the Feed.RSS, Feed.Atom and Feed.JSONFeed branches.

Errors are typed: DetectionError when the format cannot be determined, ParseError when the content is malformed or fails validation, and NetworkError for HTTP and transport failures. Match them with errors.As. Each carries its cause, so errors.Is reaches the sentinels that classify what went wrong — ErrUnknownFormat, ErrMalformedXML, ErrMissingRequired and ErrTooLarge — and reaches context.Canceled for a cancelled fetch.

Index

• Constants
• Variables
• func ParseFromSource(ctx context.Context, r io.Reader, opts ...Option) (*model.Feed, error)
• func ParseFromURL(ctx context.Context, url string, opts ...Option) (*model.Feed, error)
• type DetectionError
  • func (e *DetectionError) Error() string
  • func (e *DetectionError) Unwrap() error
• type NetworkError
  • func (e *NetworkError) Error() string
  • func (e *NetworkError) Unwrap() error
• type Option
  • func WithAccept(accept string) Option
  • func WithContentType(contentType string) Option
  • func WithETag(etag string) Option
  • func WithHTTPClient(client *http.Client) Option
  • func WithLastModified(t time.Time) Option
  • func WithMaxSize(maxSize int64) Option
  • func WithTimeout(timeout time.Duration) Option
  • func WithUserAgent(ua string) Option
• type Options
• type ParseError
  • func (e *ParseError) Error() string
  • func (e *ParseError) Unwrap() error
• type Response
  • func Fetch(ctx context.Context, url string, opts ...Option) (*Response, error)

Constants

const DefaultAccept = "application/atom+xml, application/rss+xml, application/feed+json, " +
	"application/xml;q=0.9, application/json;q=0.9, text/xml;q=0.8, */*;q=0.5"

DefaultAccept is the Accept header sent when WithAccept is not supplied.

The feed types come first and unweighted; the generic types that a feed is also legitimately served as follow, ranked below them. The trailing */* is what keeps a server that would otherwise answer 406 serving feeds — the header is there to break a tie between representations, not to refuse any.

const DefaultMaxSize int64 = 10 << 20

DefaultMaxSize is the largest input read when WithMaxSize is not supplied.

Ten mebibytes is far above any feed a publisher intends to serve and far below what it costs to hold one. The bound exists because the size of a feed is decided by whoever serves it: without one, a response that never ends is read until the process dies.

const DefaultTimeout = 30 * time.Second

DefaultTimeout is the request timeout applied when neither WithTimeout nor WithHTTPClient is supplied.

const DefaultUserAgent = "simplest-feed-parser/1.0"

DefaultUserAgent is the User-Agent sent when WithUserAgent is not supplied.

Variables

var (
	// ErrUnknownFormat classifies content whose format could not be determined
	// — neither by inspecting it nor from any declared media type.
	ErrUnknownFormat = errors.New("unknown feed format")

	// ErrMalformedXML classifies a document one of the XML parsers rejected as
	// syntactically invalid. The underlying [xml.SyntaxError], which carries
	// the line number, remains reachable with errors.As.
	ErrMalformedXML = errors.New("malformed XML structure")

	// ErrMissingRequired classifies a document that omits an element its own
	// specification requires a reader to reject it for. Only Atom and JSON Feed
	// are validated this way; see the README's "Validation and Strictness".
	ErrMissingRequired = errors.New("missing required element")

	// ErrTooLarge classifies input that exceeded the configured maximum size.
	// See [WithMaxSize].
	ErrTooLarge = errors.New("feed exceeds the maximum size")

	// ErrNotModified classifies a 304 answer to a conditional request. It is an
	// expected outcome of polling with [WithETag] or [WithLastModified], not a
	// failure — [Fetch] reports it as [Response.NotModified] and no error at
	// all. [ParseFromURL], having no feed to return, reports it as this.
	ErrNotModified = errors.New("not modified")
)

Sentinel errors classifying the conditions a caller is likely to branch on.

They are never returned bare. Each is carried as the cause of one of the typed errors above, so both questions a caller asks are answerable about the same error value: errors.As names where the failure happened, and errors.Is names what went wrong.

feed, err := feedparser.ParseFromURL(ctx, url)
switch {
case errors.Is(err, feedparser.ErrTooLarge):      // over the size limit
case errors.Is(err, feedparser.ErrUnknownFormat): // not a feed we know
}

Functions

func ParseFromSource

func ParseFromSource(ctx context.Context, r io.Reader, opts ...Option) (*model.Feed, error)

ParseFromSource parses a feed from the provided reader.

The whole input is read into memory before parsing begins, because the format is detected from the content and every parser then reads that content from the start. How much may be read is bounded by WithMaxSize; input past the bound fails the parse with ErrTooLarge rather than being buffered.

ctx cancels the read and the parse, not merely the fetch that preceded them. A cancelled or expired context surfaces as context.Canceled or context.DeadlineExceeded itself rather than as one of this package's typed errors: the caller withdrawing its own request is not a defect in the feed, and reporting it as a ParseError would say that it was.

func ParseFromURL

func ParseFromURL(ctx context.Context, url string, opts ...Option) (*model.Feed, error)

ParseFromURL fetches a feed from the given URL and parses it.

It is Fetch for callers who need only the feed. Because it has nowhere to report one, a 304 answer to a conditional request surfaces as an error matching ErrNotModified; use Fetch to poll a feed properly.

Types

type DetectionError

type DetectionError struct {
	Reason string

	// Err is the cause, kept so that [errors.Is] and [errors.As] can reach it.
	// Reason renders that cause for a human; Err is what a program branches on.
	Err error
}

DetectionError indicates that the feed format could not be determined.

func (*DetectionError) Error

func (e *DetectionError) Error() string

func (*DetectionError) Unwrap

func (e *DetectionError) Unwrap() error

Unwrap returns the cause, or nil when the error carries none.

type NetworkError

type NetworkError struct {
	URL    string
	Reason string

	// StatusCode is the HTTP status the server answered with, or zero when the
	// request failed before a response was read.
	StatusCode int

	// Err is the cause, kept so that [errors.Is] and [errors.As] can reach it.
	// A request cancelled by its context unwraps to [context.Canceled], and one
	// that ran out of time to [context.DeadlineExceeded].
	Err error
}

NetworkError indicates an HTTP or network failure.

func (*NetworkError) Error

func (e *NetworkError) Error() string

func (*NetworkError) Unwrap

func (e *NetworkError) Unwrap() error

Unwrap returns the cause, or nil when the error carries none.

type Option

type Option func(*Options)

Option is a function that modifies Options.

func WithAccept

func WithAccept(accept string) Option

WithAccept sets the Accept header sent with a fetch, replacing DefaultAccept. An empty value sends no Accept header at all.

Reach for it when a server answers the default badly — some reject a request whose Accept they do not recognise rather than ignoring it.

func WithContentType

func WithContentType(contentType string) Option

WithContentType declares the media type the input arrived with, for input whose type is known from somewhere other than the content — a Content-Type header, a file extension, a database column.

It is a fallback, not an override. The format is read from the content first, because a server's idea of what it is serving is wrong often enough that trusting it would break feeds that parse today: feeds are routinely served as text/plain, text/html, or application/octet-stream. The declared type is consulted only when the content names no format at all, and when a parse rests on it that is recorded on Feed.Warnings.

Fetch sets this from the response for you.

func WithETag

func WithETag(etag string) Option

WithETag sends an If-None-Match header carrying the entity tag a previous fetch returned as Response.ETag.

If the feed has not changed, the server answers 304 and sends no body: Fetch reports that as Response.NotModified, and ParseFromURL, having no feed to return, as an error matching ErrNotModified.

func WithHTTPClient

func WithHTTPClient(client *http.Client) Option

WithHTTPClient sets a custom HTTP client for network requests.

func WithLastModified

func WithLastModified(t time.Time) Option

WithLastModified sends an If-Modified-Since header carrying the time a previous fetch returned as Response.LastModified. The zero time sends no header. See WithETag for what a server answers.

func WithMaxSize

func WithMaxSize(maxSize int64) Option

WithMaxSize bounds how many bytes are read from the input, for both ParseFromURL and ParseFromSource. Input beyond the bound is not read, and the parse fails with an error matching ErrTooLarge rather than a feed built from a truncated document.

A size of zero or less removes the bound. Do that only for input you produced yourself — a file on disk, a buffer you filled. It is not a setting to reach for because a publisher's feed grew: raise the bound to a size you are willing to hold in memory instead.

func WithTimeout

func WithTimeout(timeout time.Duration) Option

WithTimeout sets the timeout for HTTP requests. It has no effect when WithHTTPClient is also supplied, because that client carries its own.

func WithUserAgent

func WithUserAgent(ua string) Option

WithUserAgent sets a custom User-Agent header for HTTP requests.

type Options

type Options struct {
	// HTTPClient, when non-nil, is used verbatim and Timeout is ignored —
	// a caller supplying their own client owns its timeout policy.
	HTTPClient *http.Client
	UserAgent  string
	Timeout    time.Duration

	// MaxSize bounds how many bytes are read from the input. Zero or less
	// means unbounded. See [WithMaxSize].
	MaxSize int64

	// Accept is the Accept header sent with a fetch. Empty sends none.
	Accept string

	// ETag and LastModified are the validators for a conditional request. Both
	// are empty or zero unless the caller supplied what a previous fetch
	// returned. See [WithETag] and [WithLastModified].
	ETag         string
	LastModified time.Time

	// ContentType is the media type the input was declared as, consulted only
	// when the content itself names no format. See [WithContentType].
	ContentType string
}

Options holds configuration for feed parsing operations.

type ParseError

type ParseError struct {
	// Format names the feed format the content was parsed as. It is empty when
	// the failure happened before a format was settled on.
	Format string
	Reason string

	// Err is the cause, kept so that [errors.Is] and [errors.As] can reach it.
	Err error
}

ParseError indicates that the feed content is malformed or invalid.

func (*ParseError) Error

func (e *ParseError) Error() string

func (*ParseError) Unwrap

func (e *ParseError) Unwrap() error

Unwrap returns the cause, or nil when the error carries none.

type Response

type Response struct {
	// Feed is the parsed feed, or nil when NotModified is set.
	Feed *model.Feed

	// URL is the URL the feed was finally read from, which differs from the
	// one requested when the request was redirected. Relative links in the
	// document resolve against this, not against the URL asked for.
	URL string

	// StatusCode is the HTTP status of the response.
	StatusCode int

	// ETag is the entity tag the server gave this version of the feed, empty
	// if it gave none. Hand it back with [WithETag] on the next poll.
	ETag string

	// LastModified is when the server said the feed last changed, zero if it
	// said nothing or said something unparseable. Hand it back with
	// [WithLastModified] on the next poll.
	LastModified time.Time

	// NotModified reports that the server answered 304 to a conditional
	// request: the feed has not changed since the validators sent with it.
	// This is the successful outcome of a poll, not a failure — Feed is nil
	// and the error is nil, and the caller keeps what it already had.
	NotModified bool
}

Response is what one fetch returned: the feed, and the transport facts a caller needs that a feed has nowhere to record.

The validators are the reason this type exists. A poller that re-downloads every feed on every pass wastes its own bandwidth and the publisher's, and the way not to is to send back what the server last said — see Fetch.

func Fetch

func Fetch(ctx context.Context, url string, opts ...Option) (*Response, error)

Fetch retrieves a feed over HTTP and parses it, reporting what the transport said alongside the feed.

It is ParseFromURL with room for the answers that do not fit in a model.Feed: the validators for a conditional request, the URL after redirects, and a 304 that means the caller's copy is current rather than that anything went wrong.

resp, err := feedparser.Fetch(ctx, url,
    feedparser.WithETag(prev.ETag),
    feedparser.WithLastModified(prev.LastModified))
if err != nil {
    return err
}
if resp.NotModified {
    return nil // what we have is current
}
save(resp.Feed, resp.ETag, resp.LastModified)