crawl

Documentation

Index

• func Canonicalize(rawURL string) (string, error)
• func DefaultExcludes() []*regexp.Regexp
• func MaybeGunzip(r io.ReadCloser, hdr http.Header, url string) (io.Reader, func() error, error)
• func ParseSitemap(r io.Reader, sourceURL string, onEntry func(SitemapEntry) error) error
• func SameHost(origin, candidate string) bool
• func SitemapURLs(host string) []string
• type Config
• type Crawler
  • func (c *Crawler) Run(ctx context.Context) (int, error)
• type SitemapEntry

Functions

func Canonicalize

func Canonicalize(rawURL string) (string, error)

Canonicalize returns the canonical form of rawURL:

• scheme + host lowercased
• default ports stripped (:80, :443)
• fragment removed
• query params sorted alphabetically; tracking params and utm_* stripped
• dot segments resolved
• single trailing slash removed from non-root paths

func DefaultExcludes

func DefaultExcludes() []*regexp.Regexp

DefaultExcludes are the patterns described in the schema.

func MaybeGunzip

func MaybeGunzip(r io.ReadCloser, hdr http.Header, url string) (io.Reader, func() error, error)

MaybeGunzip wraps r in a gzip.Reader if header says so or URL ends .gz.

func ParseSitemap

func ParseSitemap(r io.Reader, sourceURL string, onEntry func(SitemapEntry) error) error

ParseSitemap streams an XML sitemap or sitemap-index, yielding entries via callback. depth caps recursive sitemap-index processing. The reader is closed by the caller.

func SameHost

func SameHost(origin, candidate string) bool

SameHost reports whether candidate is on the same host as origin.

func SitemapURLs

func SitemapURLs(host string) []string

SitemapURLs returns the candidate sitemap URL paths to probe when robots.txt doesn't declare one. step 2.

Types

type Config

type Config struct {
	Site           string
	Seeds          []string
	MaxPages       int
	MaxDepth       int
	Concurrency    int
	Rate           float64
	IgnoreRobots   bool
	RespectCanon   bool
	UserAgent      string
	Include        []*regexp.Regexp
	Exclude        []*regexp.Regexp
	HTTPTimeout    time.Duration
	ThinThreshold  int
	MaxRetries     int
	RetryBaseDelay time.Duration
	RetryMaxDelay  time.Duration
}

Config bundles the crawl-relevant settings derived from CLI flags.

type Crawler

type Crawler struct {
	Cfg     Config
	Client  *httpx.Client
	DB      *store.DB
	Logger  *slog.Logger
	Robots  *robotstxt.RobotsData
	RunID   string
	BaseURL *url.URL
	// contains filtered or unexported fields
}

Crawler runs the BFS + sitemap + robots discovery and persists pages.

func (*Crawler) Run

func (c *Crawler) Run(ctx context.Context) (int, error)

Run executes the crawl until --max-pages is reached or the queue is exhausted. Returns the count of pages fetched + error.

type SitemapEntry

type SitemapEntry struct {
	Loc          string
	LastModified string
	Source       string // sitemap URL it came from
	IsIndex      bool   // when true, Loc points to a nested sitemap
}

SitemapEntry is one URL discovered via a sitemap.