fetch

Documentation

Overview

Package fetch gets the bytes of a page. It has two backends: a plain HTTP client (the fast default) and a headless-browser renderer that reuses kage's Chrome pool. The auto policy fetches statically first and escalates to the browser only when the static DOM looks empty or JavaScript-gated.

Index

• type Fetcher
  • func New(opts Options) *Fetcher
  • func (f *Fetcher) Close() error
  • func (f *Fetcher) Fetch(ctx context.Context, url string) (*Response, error)
• type Mode
• type Options
• type Response

Types

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher fetches pages with a static client, a renderer, or static-first auto escalation. It is safe for concurrent use; the browser pool it owns launches Chrome once and is shared across every Fetch.

func New

func New(opts Options) *Fetcher

New builds a Fetcher. Chrome is not launched until a render is actually needed.

func (*Fetcher) Close

func (f *Fetcher) Close() error

Close shuts down the browser pool if one was started.

func (*Fetcher) Fetch

func (f *Fetcher) Fetch(ctx context.Context, url string) (*Response, error)

Fetch returns the bytes of url under the configured mode.

type Mode

type Mode string

Mode mirrors yomi's render mode without importing the yomi package, keeping fetch a leaf.

const (
	Auto Mode = "auto"
	On   Mode = "on"
	Off  Mode = "off"
)

type Options

type Options struct {
	Mode       Mode
	UserAgent  string
	Timeout    time.Duration
	Scroll     bool
	ChromeBin  string
	ControlURL string
	// Logf, when set, receives a line each time auto escalates to the browser.
	Logf func(format string, args ...any)
}

Options configure a Fetcher.

type Response

type Response struct {
	URL      string // final URL after redirects
	Body     []byte // decoded to UTF-8
	MIME     string // sniffed or declared content type
	Rendered bool   // produced by the headless browser
}

Response is the outcome of fetching one URL.