config

Documentation

Overview

Package config holds the tunables for one ami run. Defaults come from a prior 50K-worker recrawl engine and are sized for a 10 Gbps box; lower them for a laptop or a polite crawl.

Index

• type Config
  • func Default() Config
• type Mode

Types

type Config

type Config struct {
	// Output.
	OutDir string
	RunID  string

	// Concurrency.
	//
	// Workers is the size of the goroutine pool and the ceiling on in-flight
	// requests. The actual number of concurrent requests floats between
	// MinInflight and Workers under an adaptive controller that tracks latency,
	// so a thin uplink is never asked to open more connections than it sustains
	// (which would collapse into timeouts and false-skip live hosts), while a
	// fat pipe is driven up to Workers.
	Workers         int
	MinInflight     int
	StartInflight   int
	DNSWorkers      int
	TransportShards int

	// Timeouts.
	Timeout      time.Duration
	ProbeTimeout time.Duration
	DNSTimeout   time.Duration

	// HeaderTimeout caps how long to wait for a server's response headers after
	// the request has been written, separate from Timeout which bounds the whole
	// exchange including the body read. It is the lever for the dominant dead-host
	// cost on a raw crawl: a host that accepts the TCP connection then goes silent
	// pays only HeaderTimeout instead of the full request deadline, while a
	// slow-but-live host that has started streaming a body is unaffected because
	// its headers already arrived. Set it above the slowest first-byte latency you
	// are willing to wait for (a distant government or university server can take
	// two to three seconds) and below Timeout. Zero disables it, falling back to
	// Timeout for the whole exchange.
	HeaderTimeout time.Duration

	// TimeoutFloor is the lower clamp on the adaptive request timeout. The
	// adaptive timeout serves roughly P95 latency times two, and never drops
	// below this floor, so a fast run does not abort a host that is merely slow to
	// connect or answer. Zero selects the engine default.
	TimeoutFloor time.Duration

	// Politeness.
	MaxConnsPerHost     int
	MaxConnsPerIP       int
	DomainFailThreshold int
	MaxRetries          int
	MaxRedirects        int
	PerHostDelay        time.Duration

	// Behaviour.
	Mode           Mode
	UserAgent      string
	StoreUnchanged bool
	MaxBodyBytes   int64

	// Markdown converts each HTML response to Markdown and stores it in the
	// markdown column of the capture (parquet format only). It is CPU work done
	// on the worker pool, so on a network-bound crawl it is close to free; on a
	// fast local source it lowers throughput. Off by default.
	Markdown bool

	// Output.
	//
	// Format selects how captured exchanges are stored:
	//   parquet  bodies and reconstructed headers go into rotated, zstd-compressed
	//            Parquet files (captures-NNNNN.parquet). Columnar zstd compresses
	//            thousands of similar pages together, so this is several times
	//            smaller on disk than per-record gzip WARC, and the crawl is
	//            network-bound so the extra compression is effectively free.
	//   warc     classic WARC/1.1 files (one gzip member per record) plus a
	//            metadata-only Parquet index pointing into them, for archival
	//            fidelity and interop with the web-archiving ecosystem.
	Format string

	// Output sizing.
	WARCTargetSize int64
	// CaptureTargetSize rotates the Parquet capture files once this many
	// uncompressed payload bytes have accumulated, so a long run produces a series
	// of bounded, independently readable files instead of one giant file finalized
	// only at the end. Completed files can be offloaded and deleted mid-run.
	CaptureTargetSize int64
	IndexBatchRows    int

	// Sharded distribution (process only partition Shard of ShardCount).
	Shard      int
	ShardCount int

	// Reorder spreads the seed across hosts before it reaches the workers, so
	// throughput does not depend on the order the caller listed the URLs in. A
	// raw Common Crawl shard arrives host-clustered (many consecutive URLs share
	// a host); fed in that order the worker pool stalls on the per-host
	// concurrency cap. With Reorder on, the engine buffers a window of seeds and
	// emits them round-robin across hosts, keeping a wide host set in flight
	// whatever the input order. On by default.
	Reorder bool
	// ReorderWindow is how many seeds to buffer for the round-robin spread. A
	// larger window holds more distinct hosts at once at the cost of memory.
	// Zero selects an automatic size derived from Workers.
	ReorderWindow int
}

Config is the full set of knobs for a crawl.

func Default

func Default() Config

Default returns the standard configuration.

type Mode

type Mode string

Mode selects the request header profile.

const (
	// ModeFast sends minimal headers for the highest throughput.
	ModeFast Mode = "fast"
	// ModePolite sends a full browser-like header set, less likely to be
	// fingerprinted and blocked by bot-detection WAFs.
	ModePolite Mode = "polite"
)