sitemap

Documentation

Overview

Package sitemap parses the Sitemaps 0.9 protocol (sitemaps.org) into the discoveries the frontier feeds on. It reads a <urlset> as a list of page URLs and a <sitemapindex> as a list of child sitemap files to fetch next.

The parser is deliberately narrow. It keeps <loc> and <lastmod> and drops changefreq and priority: those two are self-declared, unverifiable, and almost always wrong, so meguri does not let them steer the crawl. lastmod survives only as a weak freshness prior, parsed leniently so one bad date never sinks a whole file.

The reader is hardened for the open web. It detects gzip by magic bytes and decompresses transparently, caps the decompressed body at MaxDecompressed to blunt a gzip bomb, streams the XML so a 50 MB file never materializes at once, and stops after MaxEntries.

Index

• Constants
• type Entry
• type Result
  • func Parse(r io.Reader) (*Result, error)
  • func (res *Result) ChildURLs(base string, pol *dedup.CanonPolicy) []string
  • func (res *Result) Discoveries(base string, g meguri.HostGrouping, pol *dedup.CanonPolicy, now uint32) []meguri.Discovery

Constants

const MaxDecompressed = 50 << 20 // 50 MB, the protocol's uncompressed limit

MaxDecompressed caps the decompressed body to defend against a gzip bomb.

const MaxEntries = 50_000

MaxEntries caps the number of URLs parsed from one sitemap file.

Types

type Entry

type Entry struct {
	Loc        string // the <loc> URL, the only required field
	LastMod    uint32 // <lastmod> as epoch-hours, 0 when absent or unparseable
	HasLastMod bool   // true when a parseable <lastmod> was present
}

Entry is one <url> from a urlset. meguri consumes Loc as a discovery and LastMod as a weak freshness prior. changefreq and priority are deliberately NOT parsed: they are self-declared, unverifiable, and almost always wrong.

type Result

type Result struct {
	Entries  []Entry  // from <urlset><url><loc>
	Children []string // from <sitemapindex><sitemap><loc>
}

Result is one parsed sitemap. Exactly one of Entries or Children is populated in practice: a <urlset> yields Entries, a <sitemapindex> yields Children (the URLs of child sitemap files to fetch next).

func Parse

func Parse(r io.Reader) (*Result, error)

Parse reads a sitemap from r. It transparently decompresses gzip, caps the decompressed bytes at MaxDecompressed and the entries at MaxEntries, ignores changefreq and priority, and parses <lastmod> in W3C datetime form. It returns a Result with either Entries (urlset) or Children (sitemapindex).

func (*Result) ChildURLs

func (res *Result) ChildURLs(base string, pol *dedup.CanonPolicy) []string

ChildURLs canonicalizes a sitemapindex's child sitemap locations against base, the URLs ami fetches next to walk an index down to its urlsets. It drops a child whose location is not a fetchable URL. It carries no discovery: a child is a sitemap to fetch, not a page to crawl.

func (*Result) Discoveries

func (res *Result) Discoveries(base string, g meguri.HostGrouping, pol *dedup.CanonPolicy, now uint32) []meguri.Discovery

Discoveries turns a parsed urlset into the frontier discoveries it feeds, the intake half of the sitemap path: the parser reads <loc> and <lastmod>, and this maps each surviving entry onto a meguri.Discovery the frontier's Discover folds in. It is the freshness/prioritization consumer the audit names, kept on the meguri side because canonicalization and the URLKey must be trusted before a URL enters the frontier (invariant 10); ami fetches the sitemap bytes and hands them to Parse, this turns the result into routable discoveries.

Each entry's Loc is canonicalized against base (the sitemap's own URL, so a relative <loc> resolves) and keyed; an entry whose Loc is not a frontier URL is dropped rather than admitted with a bad key. The discovery is stamped with meguri.SourceSitemap so the record records how it entered.

lastmod rides in as the weak freshness prior, the only sitemap-declared signal meguri trusts: a parseable <lastmod> becomes the discovery's ObservedAt, which the frontier stamps as the record's FirstSeen, so a never-crawled URL enters with a prior on its age (an old lastmod reads as a long-lived, stable page; a recent one as fresh) that the change-rate estimator starts from. When lastmod is absent the discovery is observed now. changefreq and priority are never consulted, the same refusal the parser makes.