crawl

Documentation

Overview

Package crawl is the single-host crawl engine behind kumo. It owns the frontier, the polite fetcher, robots and sitemap handling, URL normalization and scoping, and the worker loop that turns a host into a stream of structured pages. It depends on the extract package for HTML-to-structured conversion and on nothing in the kumo package, so the engine stays reusable on its own.

Index

• func Normalize(raw string) (string, bool)
• func Sitemap(ctx context.Context, opt Options) ([]string, error)
• type Crawler
  • func New(opt Options) *Crawler
  • func (c *Crawler) Run(ctx context.Context, emit func(*Page) error) (Stats, error)
• type Options
• type Page
  • func One(ctx context.Context, opt Options, target string) (*Page, error)
• type Scope
  • func (s Scope) DedupKey(normalized string) string
  • func (s Scope) InScope(raw string) bool
• type Stats

Functions

func Normalize

func Normalize(raw string) (string, bool)

Normalize canonicalizes a URL for fetching and storage: it lowercases the scheme and host, drops the fragment, drops a default port, strips tracking parameters, and sorts the remaining query so two equivalent URLs share one string. It returns ok=false for anything that is not an absolute http(s) URL.

func Sitemap

func Sitemap(ctx context.Context, opt Options) ([]string, error)

Sitemap discovers the in-scope URL frontier for a host from robots.txt and its sitemaps, fetching no page content. It backs the "kumo sitemap" command.

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler runs one crawl. Build it with New and drive it with Run.

func New

func New(opt Options) *Crawler

New builds a Crawler from options, applying defaults.

func (*Crawler) Run

func (c *Crawler) Run(ctx context.Context, emit func(*Page) error) (Stats, error)

Run crawls the host, calling emit for each page. emit is serialized, so it need not be safe for concurrent use. It returns when the frontier drains, the page cap is reached, or the context is cancelled.

type Options

type Options struct {
	Host          string
	Seeds         []string // extra seed URLs beyond the host root
	Workers       int
	MaxPages      int    // 0 = unlimited
	MaxDepth      int    // 0 = unlimited
	Traversal     string // "bfs" (default) or "dfs"
	IncludeSubs   bool
	ScopePrefix   string
	Exclude       []string
	IncludeSearch bool
	RespectRobots bool
	FollowSitemap bool
	ProbeRawMD    bool
	UserAgent     string
	Rate          time.Duration
	Retries       int
	HTTP          *http.Client
	DryRun        bool

	// Known returns the stored validators for a URL so a re-crawl can issue a
	// conditional GET. It may be nil, in which case every fetch is unconditional.
	Known func(url string) (etag, lastModified string)
}

Options configures one crawl. Only Host is required; the zero value of every other field is a sensible default applied by New.

type Page

type Page struct {
	URL          string
	Status       int
	Depth        int
	FetchedAt    time.Time
	ContentType  string
	ETag         string
	LastModified string
	NotModified  bool
	Extract      *extract.Result
	RawMarkdown  string
	Error        string
}

Page is one crawled page as the engine sees it: the fetch outcome plus the extracted structured content. The kumo package maps this onto its own URI addressed record.

func One

func One(ctx context.Context, opt Options, target string) (*Page, error)

One fetches and structures a single URL outside a crawl, the pointwise case behind "kumo page". It applies the same fetch and extraction path.

type Scope

type Scope struct {
	Host           string   // the bare host being crawled (no scheme, no port)
	IncludeSubs    bool     // allow *.Host as well as Host
	Prefix         string   // restrict to URLs whose path starts with this
	Exclude        []string // skip URLs whose path contains any of these
	IncludeSearch  bool     // allow site-search routes
	CollapseLocale bool     // treat localized copies of a page as one
}

Scope decides which URLs a crawl is allowed to enqueue. It is built once from the crawl options and consulted for every discovered link.

func (Scope) DedupKey

func (s Scope) DedupKey(normalized string) string

DedupKey returns the key the frontier uses to decide whether a URL has been seen. It is the normalized URL, with the leading locale segment removed when CollapseLocale is set, so a site that serves the same page under /en/, /fr/, and /ja/ is crawled once.

func (Scope) InScope

func (s Scope) InScope(raw string) bool

InScope reports whether a normalized URL is eligible to be crawled.

type Stats

type Stats struct {
	Fetched int
	Pages   int
	RawMD   int
	Errors  int
	Skipped int
	Elapsed time.Duration
}

Stats summarizes a finished crawl.