forage

README

forage

A reusable scraping engine for Go. Forage fetches pages politely, says who it is, and tells you what it got. Parsing a particular site is up to the application that imports it.

• Honest HTTP. Requests go out through Go's standard net/http stack with the User-Agent forage/0.2 (+https://github.com/pigfox/forage), or one you set. Forage does not imitate a browser: no borrowed TLS fingerprint, no browser User-Agent, no rotating headers. HTTP/2 is used when the server offers it. Bodies compressed with gzip, deflate, Brotli or zstd are decoded. Each request can name its own proxy (http, https or socks5).
• Headless Chrome for JavaScript pages. Pages that need JavaScript are rendered with chromedp. The browser implements the same Fetcher interface as the HTTP client, so callers never need to know which path ran. Chrome sends the same User-Agent as the HTTP path and does not hide that it is automated.
• Blocks are final. When a page is detected as a block or a challenge, forage does not retry it, does not switch identity and does not hand it to the browser. The client stops sending requests to that host for the rest of its life and tells you: Get returns the page with a *forage.BlockedError (errors.Is(err, forage.ErrHostBlocked)), later calls for that host fail at once without sending anything, and BlockedHosts lists every stopped host.
• Identities with health scores. Each identity in a pool is a proxy (or a direct connection) and a cookie jar. All identities send the same headers, and they are not a way around blocks.
• Pacing. Each host gets its own token bucket. Retries use jittered exponential backoff and honor Retry-After. Network errors and 5xx responses are retried with the same identity; blocks, captchas and 404s never are. Under the default detectors a 429 counts as a block. robots.txt is checked by default.
• Detection. Every page is classified as ok, blocked, captcha or empty. Generic detectors are built in, and you add your own.
• Extraction that fails loudly. Each field lists selectors, best first, and later ones are fallbacks. When no selector matches a required field, you get a typed error that names the field, not a record full of empty strings.
• A job queue. It runs in memory, or in Postgres with FOR UPDATE SKIP LOCKED, attempt counts and dead-lettering.
• Metrics hooks. Requests, blocks, captchas and parse failures are counted per host and per identity.

Go only. Forage's own code reads no environment variables and no files. Proxies, credentials and all other settings are passed in code. In particular, the HTTP client ignores HTTP_PROXY and similar variables.

Install

go get github.com/pigfox/forage@v0.2.0

Requires Go 1.27. Chrome or Chromium is needed only if you use the browser package.

Minimal example

client, err := forage.New(forage.Config{})
if err != nil {
	log.Fatal(err)
}
res, err := client.Get(context.Background(), "https://example.com/")
if errors.Is(err, forage.ErrHostBlocked) {
	log.Fatal(err) // the site said no; this client will not ask it again
}
if err != nil {
	log.Fatal(err) // no response at all, or robots.txt said no
}
fmt.Println(res.Status, res.Outcome, res.Identity.ID, len(res.Body))

product := extract.Set{Fields: []extract.Field{
	{Name: "title", Required: true, Selectors: []extract.Selector{extract.Text("h1")}},
	{Name: "link", Selectors: []extract.Selector{extract.Attr("a", "href")}},
}}
rec, err := product.ExtractHTML(bytes.NewReader(res.Body))
if errors.Is(err, extract.ErrMissingField) {
	log.Fatal(err) // the markup changed: fail loudly
}
fmt.Println(rec["title"], rec["link"])

With the zero Config you get one direct identity sending forage's own User-Agent, one request per second per host, three attempts for network and server errors, the robots.txt check and the generic detectors. Set Config.UserAgent to name your own crawler and a contact address, or build a profile with fetch.DefaultProfile.WithUserAgent. Get returns a Result whenever an attempt got a response, so check Outcome and Status. It returns an error when no attempt got a response, when robots.txt forbids the URL, when the context ends, or when the host has blocked the client; the blocked page itself comes back alongside the first such error.

example_test.go has a fuller setup: your own User-Agent, an identity pool, a slower pace for one host, and Chrome for JavaScript pages. Both examples are compiled by go test.

Packages

Package	What it does
forage	Client: identity, robots.txt, pacing, fetch, detect, retry, stop on block, browser for JavaScript, metrics
fetch	Plain net/http client with an honest default User-Agent, HTTP/2, cookie jars, proxies, decoding
browser	Headless Chrome behind the same fetch.Fetcher interface
session	Identity pool (a proxy and a cookie jar each) with health scores
pace	Per-host token buckets, backoff, retry policy, robots.txt
detect	Detector interface and the generic ok / blocked / captcha / empty detectors
extract	goquery selector sets with ordered fallbacks and required-field errors
queue	Queue interface with in-memory and Postgres implementations
metrics	Hook interface, plus no-op and in-memory implementations

The subpackages can be used on their own. The root package does not import browser, so a program that never renders JavaScript does not link chromedp.

Site-specific detection

myDetector := detect.Chain{
	detect.Func(func(p detect.Page) detect.Outcome {
		if bytes.Contains(p.Body, []byte(`id="geo-blocked"`)) {
			return detect.Blocked
		}
		return detect.OK
	}),
	detect.Default(),
}
client, _ := forage.New(forage.Config{Detector: myDetector})

Postgres queue

pool, _ := pgxpool.New(ctx, dsn)
q, _ := queue.NewPostgres(pool, queue.PostgresOptions{})
_ = q.Migrate(ctx) // creates forage_jobs if it does not exist

id, _ := q.Enqueue(ctx, queue.Job{URL: "https://example.com/item/1"})
job, err := q.Dequeue(ctx) // queue.ErrEmpty when nothing is ready
// ... fetch job.URL ...
dead, err := q.Fail(ctx, job, err, time.Minute) // or q.Complete(ctx, job)

A worker that stalls past its lease loses the job to another worker. Its late Complete or Fail then returns queue.ErrLeaseLost, so it cannot overwrite the other worker's result.

What it does not do

• It does not solve captchas or get around blocks. It detects them, stops sending requests to that host and tells you.
• It does not disguise itself. There are no browser TLS fingerprints, no browser User-Agents, no header rotation and no fingerprint randomisation.

Tests

./scripts/test.sh

This runs go vet and then every test with -race, showing coverage per package. If Docker is available, it starts a throwaway postgres:16-alpine for the queue tests and removes it on exit. The browser tests run when Chrome or Chromium is installed. CI runs go test -short, which skips both groups and runs the rest. With the full suite, every package is at 100% statement coverage.

Responsible use

You are responsible for how you use this library. Follow each site's terms of service and its robots.txt, keep your request rate low, and respect the law where you and the site operate. The robots.txt check is on by default, the default rate is one request per second per host, every request names forage (or the User-Agent you set), and a block stops the client from contacting that host again. Turning the robots.txt check off or raising the rate is your decision, and so is the responsibility that comes with it.

License

MIT. See LICENSE.

Documentation

Overview

Package forage is a scraping engine: a Client that fetches a URL politely, identifies itself honestly, and reports what it got.

Get picks an identity from the pool, checks robots.txt, waits its turn for the host, fetches over plain HTTP with forage's own User-Agent, classifies the page, and retries with backoff after a network error or a server error. A block or challenge is never retried: it stops the Client from sending any further request to that host and is reported to the caller as a *BlockedError. When a browser is configured it is used for pages that need JavaScript.

The parts live in subpackages and can be used on their own: fetch, browser, session, pace, detect, extract, queue and metrics. Site-specific parsing belongs in the application, not here.

Example

The README's minimal example. It compiles as part of the test suite and does not run, because it reaches a real site.

package main

import (
	"bytes"
	"context"
	"errors"
	"fmt"
	"log"

	"github.com/pigfox/forage"
	"github.com/pigfox/forage/extract"
)

func main() {
	client, err := forage.New(forage.Config{})
	if err != nil {
		log.Fatal(err)
	}
	res, err := client.Get(context.Background(), "https://example.com/")
	if errors.Is(err, forage.ErrHostBlocked) {
		log.Fatal(err) // the site said no; this client will not ask it again
	}
	if err != nil {
		log.Fatal(err) // no response at all, or robots.txt said no
	}
	fmt.Println(res.Status, res.Outcome, res.Identity.ID, len(res.Body))

	product := extract.Set{Fields: []extract.Field{
		{Name: "title", Required: true, Selectors: []extract.Selector{extract.Text("h1")}},
		{Name: "link", Selectors: []extract.Selector{extract.Attr("a", "href")}},
	}}
	rec, err := product.ExtractHTML(bytes.NewReader(res.Body))
	if errors.Is(err, extract.ErrMissingField) {
		log.Fatal(err) // the markup changed: fail loudly
	}
	fmt.Println(rec["title"], rec["link"])
}

Example (Configured)

A fuller setup: your own User-Agent, several identities, a slower pace for one host, and headless Chrome for pages that need JavaScript. A host that blocks or challenges the client is not asked again.

package main

import (
	"context"
	"errors"
	"fmt"
	"log"

	"github.com/pigfox/forage"
	"github.com/pigfox/forage/browser"
	"github.com/pigfox/forage/pace"
	"github.com/pigfox/forage/session"
)

func main() {
	if err := configured(context.Background()); err != nil {
		log.Fatal(err)
	}
}

func configured(ctx context.Context) error {
	pool, err := session.NewPool(session.Config{
		Size:     3,
		Language: "en-GB",
		Timezone: "Europe/London",
	})
	if err != nil {
		return err
	}
	limiter := pace.NewLimiter(1, 1)
	limiter.SetHost("slow.example", 0.2, 1)

	chrome, err := browser.New(browser.Options{})
	if err != nil {
		return err
	}
	defer chrome.Close()

	client, err := forage.New(forage.Config{
		UserAgent: "examplebot/1.0 (+mailto:ops@example.com)",
		Pool:      pool,
		Limiter:   limiter,
		Browser:   chrome,
		Retry:     pace.Policy{MaxAttempts: 4},
	})
	if err != nil {
		return err
	}
	res, err := client.Get(ctx, "https://example.com/app")
	var blocked *forage.BlockedError
	if errors.As(err, &blocked) {

		fmt.Println("stopped at", blocked.Host, blocked.Outcome)
		return nil
	}
	if err != nil {
		return err
	}
	fmt.Println(res.Via, res.Outcome)
	return nil
}

Index

• Variables
• type BlockedError
  • func (e *BlockedError) Error() string
  • func (e *BlockedError) Is(target error) bool
• type Client
  • func New(cfg Config) (*Client, error)
  • func (c *Client) BlockedHosts() []BlockedError
  • func (c *Client) Extract(res *Result, set extract.Set) (extract.Record, error)
  • func (c *Client) ExtractAll(res *Result, set extract.Set, itemCSS string) ([]extract.Record, error)
  • func (c *Client) Get(ctx context.Context, rawURL string) (*Result, error)
• type Config
• type DisallowedError
  • func (e *DisallowedError) Error() string
  • func (e *DisallowedError) Is(target error) bool
• type Result
• type Via

Examples

• Package
• Package (Configured)

Variables

var ErrDisallowed = errors.New("forage: disallowed by robots.txt")

ErrDisallowed is returned, wrapped in a *DisallowedError, when robots.txt forbids the URL.

var ErrHostBlocked = errors.New("forage: host blocked this client")

ErrHostBlocked is returned, wrapped in a *BlockedError, when a host has blocked or challenged this Client.

Types

type BlockedError

type BlockedError struct {
	Host    string         // lowercase host, with the port if the URL had one
	URL     string         // the page that was blocked
	Status  int            // its status code
	Outcome detect.Outcome // detect.Blocked or detect.Captcha
	At      time.Time      // when the block was seen, in UTC
}

BlockedError reports the block or challenge that stopped a Client from sending further requests to a host.

func (*BlockedError) Error

func (e *BlockedError) Error() string

func (*BlockedError) Is

func (e *BlockedError) Is(target error) bool

Is makes errors.Is(err, ErrHostBlocked) true.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client fetches pages. It is safe for concurrent use.

func New

func New(cfg Config) (*Client, error)

New returns a Client, filling in defaults for anything cfg leaves out.

func (*Client) BlockedHosts

func (c *Client) BlockedHosts() []BlockedError

BlockedHosts returns every host this Client has stopped sending requests to, sorted by host.

func (*Client) Extract

func (c *Client) Extract(res *Result, set extract.Set) (extract.Record, error)

Extract runs a selector set over a Result's body and counts a parse failure against the host and identity when a required field is missing.

func (*Client) ExtractAll

func (c *Client) ExtractAll(res *Result, set extract.Set, itemCSS string) ([]extract.Record, error)

ExtractAll is Extract for a list of items, such as search results.

func (*Client) Get

func (c *Client) Get(ctx context.Context, rawURL string) (*Result, error)

Get fetches rawURL. It returns a Result whenever some attempt got a response — check Outcome and Status. It returns an error when no attempt got a response at all, when robots.txt forbids the URL, when ctx ends, or when the host blocked this Client.

A block or challenge is final. The first time a host answers with one, Get returns that page together with a *BlockedError, and from then on every Get to the same host returns a *BlockedError at once without sending anything. The request is never retried, not through another identity and not through the browser. The stop lasts as long as the Client does; BlockedHosts lists the hosts it covers.

type Config

type Config struct {
	// HTTP is the primary fetcher. Default fetch.New(fetch.Options{}).
	HTTP fetch.Fetcher
	// Browser, when set, renders pages that need JavaScript. A
	// *browser.Browser is the usual choice. Nil means HTTP only. A page
	// that was blocked or challenged is never sent to the Browser.
	Browser fetch.Fetcher
	// NeedsJS decides whether a page needs the browser. Default
	// detect.NeedsJS.
	NeedsJS func(detect.Page) bool

	// UserAgent, when set, replaces the User-Agent on every request.
	// Otherwise the identity's profile decides, and that is
	// fetch.DefaultUserAgent unless the pool was given another profile.
	// Name your crawler and give a contact address, such as
	// "examplebot/1.0 (+mailto:ops@example.com)".
	UserAgent string

	// Pool supplies identities. Default: one direct identity with
	// fetch.DefaultProfile.
	Pool *session.Pool
	// Limiter spaces requests per host. Default one request per second
	// per host.
	Limiter *pace.Limiter
	// Retry decides retries and their backoff. Default three attempts.
	Retry pace.Policy
	// Detector classifies pages. Default detect.Default().
	Detector detect.Detector
	// Metrics receives counts. Default metrics.Nop.
	Metrics metrics.Hook

	// IgnoreRobots turns the robots.txt check off. It is on by default.
	IgnoreRobots bool
	// RobotsAgent is the product token matched against robots.txt groups.
	// Default "forage", which falls back to the "*" group.
	RobotsAgent string
	// RobotsTTL is how long a robots.txt is cached. Default 24h.
	RobotsTTL time.Duration
}

Config wires a Client. Every field is optional.

type DisallowedError

type DisallowedError struct {
	URL string
}

DisallowedError names the URL robots.txt forbids.

func (*DisallowedError) Error

func (e *DisallowedError) Error() string

func (*DisallowedError) Is

func (e *DisallowedError) Is(target error) bool

Is makes errors.Is(err, ErrDisallowed) true.

type Result

type Result struct {
	URL      string // final URL, after redirects
	Body     []byte
	Status   int
	Header   http.Header
	Identity *session.Identity
	Outcome  detect.Outcome
	Via      Via
	Attempts int
}

Result is one fetched page.

type Via

type Via string

Via names the path that produced a Result.

const (
	ViaHTTP    Via = "http"
	ViaBrowser Via = "browser"
)

The fetch paths.