fetch

Documentation

Overview

Package fetch is a plain HTTP client that identifies itself honestly.

Requests go out through Go's standard net/http transport and TLS stack, with forage's own User-Agent unless the caller sets another. Nothing here imitates a browser. HTTP/2 is used when the server offers it, gzip, deflate, Brotli and zstd bodies are decoded, cookies are kept in a jar, and each request may name its own proxy (http, https or socks5).

Fetcher is the interface the rest of forage depends on. The browser package implements it with headless Chrome, so a caller can swap one for the other without changing anything else.

Index

• Constants
• Variables
• func AcceptLanguage(lang string) string
• type Client
  • func New(opts Options) *Client
  • func (c *Client) CloseIdleConnections()
  • func (c *Client) Fetch(ctx context.Context, req Request) (*Response, error)
• type Fetcher
• type Header
• type Options
• type Profile
  • func (p Profile) UserAgent() string
  • func (p Profile) WithUserAgent(userAgent string) Profile
• type Request
• type Response

Constants

const DefaultUserAgent = "forage/0.2 (+https://github.com/pigfox/forage)"

DefaultUserAgent is the User-Agent forage sends unless the caller sets another. It names forage and links to its source, so a site operator who sees it in a log can tell what made the request.

Variables

var DefaultProfile = Profile{
	Name: "forage",
	Headers: []Header{
		{"User-Agent", DefaultUserAgent},
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
		{"Accept-Encoding", "gzip, deflate, br, zstd"},
	},
}

DefaultProfile is the profile used when none is given: forage's own User-Agent, plus the Accept and Accept-Encoding the client can handle.

var ErrBodyTooLarge = errors.New("fetch: response body exceeds the size limit")

ErrBodyTooLarge is returned when a response body, before or after decoding, exceeds Options.MaxBodyBytes.

var ErrUnsupportedProxy = errors.New("fetch: proxy scheme must be http, https, socks5 or socks5h")

ErrUnsupportedProxy is returned for a proxy whose scheme is not http, https, socks5 or socks5h.

var ErrUnsupportedScheme = errors.New("fetch: URL scheme must be http or https")

ErrUnsupportedScheme is returned for URLs that are not http or https.

Functions

func AcceptLanguage

func AcceptLanguage(lang string) string

AcceptLanguage turns a language tag into an Accept-Language value: "de-DE" becomes "de-DE,de;q=0.9". A value that already lists several languages is returned unchanged.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is a Fetcher over Go's standard HTTP stack. It is safe for concurrent use.

func New

func New(opts Options) *Client

New returns a Client with the given options.

func (*Client) CloseIdleConnections

func (c *Client) CloseIdleConnections()

CloseIdleConnections closes every idle connection the Client holds.

func (*Client) Fetch

func (c *Client) Fetch(ctx context.Context, req Request) (*Response, error)

Fetch implements Fetcher.

type Fetcher

type Fetcher interface {
	Fetch(ctx context.Context, req Request) (*Response, error)
}

Fetcher fetches one URL. Both *fetch.Client and *browser.Browser implement it.

type Header

type Header struct {
	Name  string
	Value string
}

Header is one request header.

type Options

type Options struct {
	// Profile is the default profile. The zero value means DefaultProfile.
	Profile Profile
	// Proxy is the default proxy, nil for a direct connection.
	Proxy *url.URL
	// Jar is the default cookie jar. Nil gives the Client a jar of its own.
	Jar http.CookieJar

	Timeout             time.Duration // whole request, default 30s
	DialTimeout         time.Duration // default 10s
	TLSHandshakeTimeout time.Duration // default 10s
	IdleConnTimeout     time.Duration // default 90s
	MaxBodyBytes        int64         // default 10 MiB, applied before and after decoding
	MaxRedirects        int           // default 10; negative disables redirects

	// RootCAs replaces the system roots, for servers with private
	// certificates.
	RootCAs *x509.CertPool
	// InsecureSkipVerify disables certificate verification. Use it only
	// against test servers.
	InsecureSkipVerify bool
}

Options configures a Client. The zero value is usable.

type Profile

type Profile struct {
	Name    string
	Headers []Header
}

Profile is a named set of request headers sent with every request that uses it. It does not imitate any browser: the default profile identifies forage, and a caller who wants a different User-Agent (their own crawler name and contact address, say) builds one with WithUserAgent.

func (Profile) UserAgent

func (p Profile) UserAgent() string

UserAgent returns the profile's User-Agent header, or "" if it has none.

func (Profile) WithUserAgent

func (p Profile) WithUserAgent(userAgent string) Profile

WithUserAgent returns a copy of p whose User-Agent is userAgent. The original profile is not changed.

type Request

type Request struct {
	URL    string
	Method string // default GET
	Header http.Header
	Body   []byte

	// Profile is the header set sent with the request. Nil uses the
	// fetcher's default profile.
	Profile *Profile
	// Proxy routes this request through an http, https or socks5 proxy.
	// Nil uses the fetcher's default, which is a direct connection unless
	// set.
	Proxy *url.URL
	// Jar holds cookies for this request. Nil uses the fetcher's own jar.
	Jar http.CookieJar
	// Language, such as "de-DE", sets Accept-Language.
	Language string
	// Timezone, such as "Europe/Berlin", is applied by fetchers that run a
	// browser. The HTTP client has no use for it.
	Timezone string
}

Request describes one fetch. Only URL is required; every other field falls back to the fetcher's defaults.

type Response

type Response struct {
	URL    string // the final URL, after redirects
	Status int
	Proto  string
	Header http.Header
	Body   []byte
}

Response is a fetched page. Body is already decoded.