httpx

Documentation

Overview

Package httpx provides a shared HTTP client and a retry helper used by every scraper. Keeping this in one place means retry/backoff, timeout, and connection-pool tuning live in exactly one file.

Index

• Constants
• Variables
• func BrowserHeaders(ua string) map[string]string
• func DecodeJSON(r io.Reader, v any) error
• func DecodeJSONN(r io.Reader, v any, maxBytes int64) error
• func Do(ctx context.Context, client *http.Client, r Request) (*http.Response, error)
• func DoWithStatus(ctx context.Context, client *http.Client, r Request) (*http.Response, error)
• func NewBrowserTLSClient(timeout time.Duration) *http.Client
• func NewClient(timeout time.Duration) *http.Client
• func NewLegacyTLSClient(timeout time.Duration) *http.Client
• func NewRetryClient(timeout time.Duration) *http.Client
• func ReadBody(body io.ReadCloser) ([]byte, error)
• func ReadBodyN(body io.ReadCloser, maxBytes int64) ([]byte, error)
• func RequestID(ctx context.Context) uint64
• func ResolveUA(ua string) string
• func SetDefaultUA(ua string)
• func WithRequestID(ctx context.Context) context.Context
• type Request
• type StatusError
  • func (e *StatusError) Error() string
  • func (e *StatusError) FailureKind() scraper.FailureKind

Constants

const MaxPageBytes = 10 * 1024 * 1024

MaxPageBytes caps ReadBody response reads to prevent an oversized or malicious response from exhausting memory.

Variables

var (
	UserAgentFirefox = builtinFirefox
	UserAgentChrome  = builtinChrome
)

UserAgentFirefox and UserAgentChrome are the active UA strings used by scrapers. Call SetDefaultUA at startup to override both with a config value.

Functions

func BrowserHeaders

func BrowserHeaders(ua string) map[string]string

BrowserHeaders returns headers that mimic a real browser navigation request, including Sec-Fetch-* headers that WAFs like Wordfence check. Pass a UA string, or empty to use the active default.

func DecodeJSON

func DecodeJSON(r io.Reader, v any) error

DecodeJSON JSON-decodes from r into v, reading at most MaxPageBytes.

func DecodeJSONN

func DecodeJSONN(r io.Reader, v any, maxBytes int64) error

DecodeJSONN JSON-decodes from r into v, reading at most maxBytes.

Reads one byte past the limit so an oversized body is reported as such rather than surfacing as a confusing "unexpected EOF" from the decoder. A body that decodes successfully within the limit is fine even if bytes remain unread, so the size is only reported when decoding actually failed.

func Do

func Do(ctx context.Context, client *http.Client, r Request) (*http.Response, error)

Do performs the request with exponential backoff: it retries network errors, 429, and 5xx up to MaxAttempts times, sleeping (attempt * 2s) between tries. Non-retryable 4xx responses fail fast with a *StatusError — the caller does not have to guard against decoding an error page as a successful body.

func DoWithStatus

func DoWithStatus(ctx context.Context, client *http.Client, r Request) (*http.Response, error)

DoWithStatus is like Do but passes any HTTP status (including 4xx and 5xx) through to the caller without classifying — useful for endpoints that legitimately return non-2xx with a meaningful body (e.g. SexMex's CMS returns HTTP 500 + valid HTML on model pages). Network errors are still retried with the same backoff as Do, but 429/5xx are NOT retried — the caller asked for the raw response and presumably wants to act on it. Default for `Do` (4xx fail-fast, 429/5xx retried) is the safer choice for everything else.

func NewBrowserTLSClient

func NewBrowserTLSClient(timeout time.Duration) *http.Client

NewBrowserTLSClient returns a client that presents a browser's TLS fingerprint instead of Go's.

Reach for it only after a scraper has been shown to fail without it — the signature is a WAF answering an identical 403 to every request including the site's own homepage, while a browser on the same machine loads it. It is the mirror image of NewLegacyTLSClient: that one widens what Go will accept from an old server, this one changes what Go presents to a picky one. Both verify certificates.

The connection pool is separate from NewClient's, so using it for one site does not change how every other site is reached.

func NewClient

func NewClient(timeout time.Duration) *http.Client

NewClient returns an http.Client backed by the shared transport.

func NewLegacyTLSClient

func NewLegacyTLSClient(timeout time.Duration) *http.Client

NewLegacyTLSClient returns a client for hosts that cannot complete a handshake with Go's default cipher list. Certificates are still verified — this widens the accepted key exchange, it does not disable checks. The negotiated suites lack forward secrecy, so use it only where a scraper has been shown to fail otherwise, and only for reading public metadata.

func NewRetryClient

func NewRetryClient(timeout time.Duration) *http.Client

NewRetryClient returns an http.Client that retries the way Do does — network errors, 429 and 5xx, with the same jittered backoff — for libraries that take an *http.Client and drive the request themselves.

timeout bounds each attempt rather than the sequence, so a retried request gets the full budget every time. Non-retryable responses, 4xx included, come back untouched with their body intact.

func ReadBody

func ReadBody(body io.ReadCloser) ([]byte, error)

ReadBody reads an HTTP response body up to MaxPageBytes. Use this instead of io.ReadAll(resp.Body) in scrapers to bound memory usage.

func ReadBodyN

func ReadBodyN(body io.ReadCloser, maxBytes int64) ([]byte, error)

ReadBodyN reads an HTTP response body up to maxBytes.

func RequestID

func RequestID(ctx context.Context) uint64

RequestID returns the request ID from the context, or 0 if none is set.

func ResolveUA

func ResolveUA(ua string) string

ResolveUA maps a shorthand to a full UA string. "firefox" and "chrome" return the built-in strings; anything else is returned as-is.

func SetDefaultUA

func SetDefaultUA(ua string)

SetDefaultUA overrides both exported UA variables. Accepts "firefox", "chrome", or a full custom string. Empty is a no-op (keeps built-ins). Call once at startup before any scrapers run.

func WithRequestID

func WithRequestID(ctx context.Context) context.Context

WithRequestID attaches an auto-incrementing request ID to the context. The ID appears in level-2 debug logs for correlation.

Types

type Request

type Request struct {
	Method      string
	URL         string
	Body        []byte
	Headers     map[string]string
	MaxAttempts int
	// Note: there is deliberately no per-request body limit here. Do returns
	// the response without reading the body, so it could not enforce one;
	// bounding the read is the caller's job via ReadBody/ReadBodyN or
	// DecodeJSON/DecodeJSONN.
	BackoffSleep func(ctx context.Context, d time.Duration) error // nil uses default (real sleep with ctx)
}

Request describes a single HTTP call. Method defaults to GET (or POST if Body is non-nil). MaxAttempts defaults to 3.

type StatusError

type StatusError struct {
	StatusCode int
}

StatusError is returned when the server replies with a non-2xx status that we do not retry. The body has already been closed.

func (*StatusError) Error

func (e *StatusError) Error() string

func (*StatusError) FailureKind

func (e *StatusError) FailureKind() scraper.FailureKind

FailureKind reports that the page did not arrive, whatever the status was.

Deliberately including 404 and 410. It is tempting to call those "absent" and let them off the traversal-completeness hook, but a status code does not say whether the missing resource mattered: a 404 on an optional sub-listing the scraper was merely probing costs nothing, while a 404 on the detail page of a scene the listing already returned means a known scene went uncollected — and under `--full` the authoritative Save would then delete it. Only the call site knows which it is, so httpx always reports missing data and a scraper that knows better opts out explicitly with scraper.AbsentError.