forage

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2026 License: MIT Imports: 14 Imported by: 0

README

forage

A reusable scraping engine for Go. Forage fetches pages the way a careful browser user would, and tells you what it got. Parsing a particular site is up to the application that imports it.

  • Browser-grade HTTP. The TLS handshake is done with uTLS, so the ClientHello matches Chrome, Firefox or Safari, and each request carries the headers that browser sends. HTTP/2 is used when the server offers it. Bodies compressed with gzip, deflate, Brotli or zstd are decoded. Each request can name its own proxy (http or socks5).
  • Headless Chrome fallback. Pages that need JavaScript are rendered with chromedp. The browser implements the same Fetcher interface as the HTTP client, so callers never need to know which path ran.
  • Identities with health scores. Each identity in a pool is a proxy, a browser profile, a cookie jar, a language and a timezone. Each block puts that identity on a cooldown that doubles every time. After N blocks in a row it is retired. With no proxies, the pool uses direct connections.
  • Pacing. Each host gets its own token bucket. Retries use jittered exponential backoff and honor Retry-After. 429s, 5xx responses, network errors and blocks are retried; a 404 never is. robots.txt is checked by default.
  • Detection. Every page is classified as ok, blocked, captcha or empty. Generic detectors are built in, and you add your own.
  • Extraction that fails loudly. Each field lists selectors, best first, and later ones are fallbacks. When no selector matches a required field, you get a typed error that names the field, not a record full of empty strings.
  • A job queue. It runs in memory, or in Postgres with FOR UPDATE SKIP LOCKED, attempt counts and dead-lettering.
  • Metrics hooks. Requests, blocks, captchas and parse failures are counted per host and per identity.

Go only. Forage's own code reads no environment variables and no files. Proxies, credentials and all other settings are passed in code. In particular, the HTTP client ignores HTTP_PROXY and similar variables.

Install

go get github.com/pigfox/forage@v0.1.0

Requires Go 1.27. Chrome or Chromium is needed only if you use the browser package.

Minimal example

client, err := forage.New(forage.Config{})
if err != nil {
	log.Fatal(err)
}
res, err := client.Get(context.Background(), "https://example.com/")
if err != nil {
	log.Fatal(err) // no response at all, or robots.txt said no
}
fmt.Println(res.Status, res.Outcome, res.Identity.ID, len(res.Body))

product := extract.Set{Fields: []extract.Field{
	{Name: "title", Required: true, Selectors: []extract.Selector{extract.Text("h1")}},
	{Name: "link", Selectors: []extract.Selector{extract.Attr("a", "href")}},
}}
rec, err := product.ExtractHTML(bytes.NewReader(res.Body))
if errors.Is(err, extract.ErrMissingField) {
	log.Fatal(err) // the markup changed: fail loudly
}
fmt.Println(rec["title"], rec["link"])

With the zero Config you get one direct Chrome-profile identity, one request per second per host, three attempts, the robots.txt check and the generic detectors. Get returns a Result whenever any attempt got a response, including a blocked one, so check Outcome and Status. It returns an error only when no attempt got a response, when robots.txt forbids the URL, or when the context ends.

example_test.go has a fuller setup: an identity pool, a slower pace for one host, and Chrome used both for JavaScript pages and as a second try after a block. Both examples are compiled by go test.

Packages

Package What it does
forage Client: identity, robots.txt, pacing, fetch, detect, retry, browser fallback, metrics
fetch HTTP client with uTLS fingerprints, browser header profiles, HTTP/2, cookie jars, proxies, decoding
browser Headless Chrome behind the same fetch.Fetcher interface
session Identity pool with health scores, cooldowns and retirement
pace Per-host token buckets, backoff, retry policy, robots.txt
detect Detector interface and the generic ok / blocked / captcha / empty detectors
extract goquery selector sets with ordered fallbacks and required-field errors
queue Queue interface with in-memory and Postgres implementations
metrics Hook interface, plus no-op and in-memory implementations

The subpackages can be used on their own. The root package does not import browser, so a program that never renders JavaScript does not link chromedp.

Site-specific detection
myDetector := detect.Chain{
	detect.Func(func(p detect.Page) detect.Outcome {
		if bytes.Contains(p.Body, []byte(`id="geo-blocked"`)) {
			return detect.Blocked
		}
		return detect.OK
	}),
	detect.Default(),
}
client, _ := forage.New(forage.Config{Detector: myDetector})
Postgres queue
pool, _ := pgxpool.New(ctx, dsn)
q, _ := queue.NewPostgres(pool, queue.PostgresOptions{})
_ = q.Migrate(ctx) // creates forage_jobs if it does not exist

id, _ := q.Enqueue(ctx, queue.Job{URL: "https://example.com/item/1"})
job, err := q.Dequeue(ctx) // queue.ErrEmpty when nothing is ready
// ... fetch job.URL ...
dead, err := q.Fail(ctx, job, err, time.Minute) // or q.Complete(ctx, job)

A worker that stalls past its lease loses the job to another worker. Its late Complete or Fail then returns queue.ErrLeaseLost, so it cannot overwrite the other worker's result.

What it does not do

  • It does not solve captchas. It detects them, backs off and changes identity.
  • The header profiles match each browser's header values, but Go's HTTP stack sends headers in its own order. The HTTP/2 SETTINGS frame is Go's, not the browser's. Sites that fingerprint either will still tell the difference.
  • The built-in profiles are snapshots of one browser version each. Build your own fetch.Profile when you need another.

Tests

./scripts/test.sh

This runs go vet and then every test with -race, showing coverage per package. If Docker is available, it starts a throwaway postgres:16-alpine for the queue tests and removes it on exit. The browser tests run when Chrome or Chromium is installed. CI runs go test -short, which skips both groups and runs the rest. With the full suite, every package is at 100% statement coverage.

Responsible use

You are responsible for how you use this library. Follow each site's terms of service and its robots.txt, keep your request rate low, and respect the law where you and the site operate. The robots.txt check is on by default and the default rate is one request per second per host. Turning either off is your decision, and so is the responsibility that comes with it.

License

MIT. See LICENSE.

Documentation

Overview

Package forage is a scraping engine: a Client that fetches a URL the way a careful person with a browser would, and reports what it got.

Get picks an identity from the pool, checks robots.txt, waits its turn for the host, fetches over HTTP with a real browser's TLS fingerprint, classifies the page, and retries with backoff when the page is blocked, rate-limited or broken. When a browser is configured it is used for pages that need JavaScript, and optionally for pages the HTTP path could not get past.

The parts live in subpackages and can be used on their own: fetch, browser, session, pace, detect, extract, queue and metrics. Site-specific parsing belongs in the application, not here.

Example

The README's minimal example. It compiles as part of the test suite and does not run, because it reaches a real site. 

package main

import (
	"bytes"
	"context"
	"errors"
	"fmt"
	"log"

	"github.com/pigfox/forage"
	"github.com/pigfox/forage/extract"
)

func main() {
	client, err := forage.New(forage.Config{})
	if err != nil {
		log.Fatal(err)
	}
	res, err := client.Get(context.Background(), "https://example.com/")
	if err != nil {
		log.Fatal(err) // no response at all, or robots.txt said no
	}
	fmt.Println(res.Status, res.Outcome, res.Identity.ID, len(res.Body))

	product := extract.Set{Fields: []extract.Field{
		{Name: "title", Required: true, Selectors: []extract.Selector{extract.Text("h1")}},
		{Name: "link", Selectors: []extract.Selector{extract.Attr("a", "href")}},
	}}
	rec, err := product.ExtractHTML(bytes.NewReader(res.Body))
	if errors.Is(err, extract.ErrMissingField) {
		log.Fatal(err) // the markup changed: fail loudly
	}
	fmt.Println(rec["title"], rec["link"])
}
Example (Configured)

A fuller setup: several identities, a slower pace for one host, and headless Chrome for pages that need JavaScript. 

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/pigfox/forage"
	"github.com/pigfox/forage/browser"
	"github.com/pigfox/forage/pace"
	"github.com/pigfox/forage/session"
)

func main() {
	if err := configured(context.Background()); err != nil {
		log.Fatal(err)
	}
}

func configured(ctx context.Context) error {
	pool, err := session.NewPool(session.Config{
		Size:      3,
		Languages: []string{"en-US", "en-GB"},
		Timezones: []string{"America/New_York", "Europe/London"},
	})
	if err != nil {
		return err
	}
	limiter := pace.NewLimiter(1, 1)
	limiter.SetHost("slow.example", 0.2, 1)

	chrome, err := browser.New(browser.Options{})
	if err != nil {
		return err
	}
	defer chrome.Close()

	client, err := forage.New(forage.Config{
		Pool:           pool,
		Limiter:        limiter,
		Browser:        chrome,
		BrowserOnBlock: true,
		Retry:          pace.Policy{MaxAttempts: 4},
	})
	if err != nil {
		return err
	}
	res, err := client.Get(ctx, "https://example.com/app")
	if err != nil {
		return err
	}
	fmt.Println(res.Via, res.Outcome)
	return nil
}

Index

Examples

Constants

This section is empty.

Variables

View Source 
var ErrDisallowed = errors.New("forage: disallowed by robots.txt")

ErrDisallowed is returned, wrapped in a *DisallowedError, when robots.txt forbids the URL.

Functions

This section is empty.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client fetches pages. It is safe for concurrent use.

func New 

func New(cfg Config) (*Client, error)

New returns a Client, filling in defaults for anything cfg leaves out.

func (*Client) Extract 

func (c *Client) Extract(res *Result, set extract.Set) (extract.Record, error)

Extract runs a selector set over a Result's body and counts a parse failure against the host and identity when a required field is missing.

func (*Client) ExtractAll 

func (c *Client) ExtractAll(res *Result, set extract.Set, itemCSS string) ([]extract.Record, error)

ExtractAll is Extract for a list of items, such as search results.

func (*Client) Get 

func (c *Client) Get(ctx context.Context, rawURL string) (*Result, error)

Get fetches rawURL. It returns a Result whenever some attempt got a response, even a blocked one — check Outcome and Status. It returns an error only when no attempt got a response at all, when robots.txt forbids the URL, or when ctx ends.

type Config 

type Config struct {
	// HTTP is the primary fetcher. Default fetch.New(fetch.Options{}).
	HTTP fetch.Fetcher
	// Browser, when set, renders pages that need JavaScript. A
	// *browser.Browser is the usual choice. Nil means HTTP only.
	Browser fetch.Fetcher
	// BrowserOnBlock also sends a page to the Browser when the HTTP path
	// was blocked, challenged or got an empty page.
	BrowserOnBlock bool
	// NeedsJS decides whether a page needs the browser. Default
	// detect.NeedsJS.
	NeedsJS func(detect.Page) bool

	// Pool supplies identities. Default: one direct Chrome identity.
	Pool *session.Pool
	// Limiter spaces requests per host. Default one request per second
	// per host.
	Limiter *pace.Limiter
	// Retry decides retries and their backoff. Default three attempts.
	Retry pace.Policy
	// Detector classifies pages. Default detect.Default().
	Detector detect.Detector
	// Metrics receives counts. Default metrics.Nop.
	Metrics metrics.Hook

	// IgnoreRobots turns the robots.txt check off. It is on by default.
	IgnoreRobots bool
	// RobotsAgent is the product token matched against robots.txt groups.
	// Default "forage", which falls back to the "*" group.
	RobotsAgent string
	// RobotsTTL is how long a robots.txt is cached. Default 24h.
	RobotsTTL time.Duration
}

Config wires a Client. Every field is optional.

type DisallowedError 

type DisallowedError struct {
	URL string
}

DisallowedError names the URL robots.txt forbids.

func (*DisallowedError) Error 

func (e *DisallowedError) Error() string

func (*DisallowedError) Is 

func (e *DisallowedError) Is(target error) bool

Is makes errors.Is(err, ErrDisallowed) true.

type Result 

type Result struct {
	URL      string // final URL, after redirects
	Body     []byte
	Status   int
	Header   http.Header
	Identity *session.Identity
	Outcome  detect.Outcome
	Via      Via
	Attempts int
}

Result is one fetched page.

type Via 

type Via string

Via names the path that produced a Result. 

const (
	ViaHTTP    Via = "http"
	ViaBrowser Via = "browser"
)

The fetch paths.

Source Files

View all Source files

Directories

Path Synopsis
Package browser fetches pages with headless Chrome, for sites that build their content with JavaScript.
 Package browser fetches pages with headless Chrome, for sites that build their content with JavaScript.
Package detect classifies a fetched page as usable, blocked, a captcha or challenge, or suspiciously empty.
 Package detect classifies a fetched page as usable, blocked, a captcha or challenge, or suspiciously empty.
Package extract pulls named fields out of HTML with CSS selectors.
 Package extract pulls named fields out of HTML with CSS selectors.
Package fetch is an HTTP client that presents a real browser's TLS fingerprint and headers.
 Package fetch is an HTTP client that presents a real browser's TLS fingerprint and headers.
Package metrics counts what a scraper does, per host and per identity.
 Package metrics counts what a scraper does, per host and per identity.
Package pace decides when the next request may go: a token bucket per host, jittered exponential backoff between attempts, a retry policy, and a robots.txt check.
 Package pace decides when the next request may go: a token bucket per host, jittered exponential backoff between attempts, a retry policy, and a robots.txt check.
Package queue holds scrape jobs: a URL and an optional payload, leased to one worker at a time, retried on failure, and moved to a dead-letter state after too many attempts.
 Package queue holds scrape jobs: a URL and an optional payload, leased to one worker at a time, retried on failure, and moved to a dead-letter state after too many attempts.
Package session keeps a pool of identities — a proxy, a browser profile, a cookie jar, a language and a timezone each — and tracks how well each one is doing.
 Package session keeps a pool of identities — a proxy, a browser profile, a cookie jar, a language and a timezone each — and tracks how well each one is doing.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.