Help us shape the future of Go! Take the 2026 Go Developer Survey

scrapeland

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 30, 2026 License: MIT Imports: 12 Imported by: 0

README

scrapeland-go

Go client for the scrape.land web data extraction API. Standard library only — no dependencies.

go get github.com/scrape-land/scrapeland-go

Two ways to use it

1. As an *http.Client — every request exits a fresh IP

This is usually what you want. Nothing else in your code changes: hand it to any library that takes an *http.Client.

package main

import (
	"fmt"
	"io"

	"github.com/scrape-land/scrapeland-go"
)

func main() {
	c, err := scrapeland.New("pb_live_YOURKEY")
	if err != nil {
		panic(err)
	}

	res, err := c.HTTPClient(nil).Get("https://api.ipify.org")
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	body, _ := io.ReadAll(res.Body)
	fmt.Println("exit IP:", string(body)) // different on every call
}

Route through a specific country, or pin a sticky session so a sequence of requests shares one exit IP:

us := c.HTTPClient(&scrapeland.Options{Country: "us"})
job := c.HTTPClient(&scrapeland.Options{Session: "checkout-42"})
2. As a data API — get structured data back
ctx := context.Background()

// Clean Markdown, ready for an LLM: navigation, headers, footers and sidebars
// stripped, tables preserved, relative links rewritten to absolute.
page, err := c.Fetch(ctx, "https://example.com", &scrapeland.Options{Format: "markdown"})
fmt.Println(page.Markdown)

// CSS or XPath selectors.
out, err := c.Extract(ctx, "https://example.com/product/1", map[string]any{
	"title": "h1",
	"price": ".price",
	"specs": map[string]any{"css": "li.spec", "all": true},
}, nil)
fmt.Println(out.Data["title"])

// Or describe what you want and let the model find it. AI extraction (Prompt,
// Schema, ExtractType) needs a Scale plan or above; selector Fields work on every
// plan. Check c.Capabilities(ctx) if you need to branch.
out, err = c.Extract(ctx, "https://example.com/product/1", nil, &scrapeland.Options{
	Prompt: "the product name, price in USD, and whether it is in stock",
})

// Asking a QUESTION rather than listing fields? Structured: scrapeland.Bool(false)
// answers in prose via out.Answer, leaving out.Data nil — so you never have to guess
// the key names a model invented. Schema and ExtractType pin a JSON shape and so
// cannot be combined with it (the API returns 400).
out, err = c.Extract(ctx, "https://scrape.land/", nil, &scrapeland.Options{
	Prompt:     "what's this business about?",
	Structured: scrapeland.Bool(false),
})
fmt.Println(out.Answer)

// JavaScript pages: Render drives a real browser. It bills 5 request-units
// instead of 1 (10 with a screenshot, 1 with BlockResources) and downloads
// everything the page asks for — measured at 1–300x the bytes (median ~17x,
// image-heavy shops worst). Set
// BlockResources unless you need a screenshot or the images: fonts, images and
// media are skipped and the DOM your selectors read is identical.
out, err = c.Extract(ctx, "https://example.com/product/1",
	map[string]any{"title": "h1"},
	&scrapeland.Options{Render: true, WaitFor: "h1", BlockResources: true})

// Many URLs in one call.
batch, err := c.Batch(ctx, []string{"https://a.com", "https://b.com"}, nil)

// Web search.
hits, err := c.Search(ctx, "golang web scraping", &scrapeland.Options{Count: 10})

// Rank a page's links by relevance — pick what to fetch next instead of crawling all.
r, err := c.Rank(ctx, "https://news.example/", "articles about interest rates",
	&scrapeland.Options{TopK: 10})
for _, link := range r.Links {
	fmt.Println(link.Score, link.URL, "-", link.Reason)
}
// AI transparency: on an AI failure Rank returns NO error — links come back in
// document order with r.RankingError set, so check it when the order matters.
if r.RankingError != "" {
	log.Println("ranking degraded:", r.RankingError)
}

// BatchIter: any number of URLs, auto-chunked into <=20-URL calls, in order.
err = c.BatchIter(ctx, allURLs, &scrapeland.Options{Fields: map[string]any{"title": "h1"}},
	func(res scrapeland.Result) bool {
		if res.Error != "" {
			log.Println("failed", res.URL, res.Error)
		} else {
			save(res.URL, res.Data)
		}
		return true // return false to stop early
	})

Configuration

New reads the environment when arguments are omitted:

Setting Option Environment Default
API key first arg SCRAPELAND_API_KEY required
Gateway WithGateway SCRAPELAND_GATEWAY http://gateway.scrape.land:8080
Data API WithAPIBase SCRAPELAND_API_BASE https://scrape.land

Blocked port? The gateway also answers on gateway.scrape.land:443, for corporate, university and hotel networks that filter high ports. Identical behaviour and billing — only the port differs, and the scheme stays http:// with your key in the username:

c, err := scrapeland.New("pb_live_YOURKEY",
	scrapeland.WithGateway("http://gateway.scrape.land:443"))

Client-wide defaults, overridable per request:

c, err := scrapeland.New("pb_live_YOURKEY",
	scrapeland.WithCountry("de"),
	scrapeland.WithMaxLatency(3000),
	scrapeland.WithRetries(3),
)

Errors

Non-2xx responses come back as *APIError, carrying the status and the server's message — check for a 402 to detect an exhausted quota:

var apiErr *scrapeland.APIError
if errors.As(err, &apiErr) {
	switch {
	case apiErr.IsPaymentRequired(): // out of credit / quota
	case apiErr.IsRateLimited():     time.Sleep(apiErr.RetryAfter) // honors Retry-After
	case apiErr.IsUnauthorized():    // bad or revoked key
	case apiErr.IsServer():          // transient 5xx
	}
	// or check apiErr.Status directly; apiErr.Body has the server's message.
}

*APIError carries Status, Body, RetryAfter, and IsBadRequest / IsUnauthorized / IsPaymentRequired / IsNotFound / IsRateLimited / IsServer helpers.

429 and 5xx are retried automatically (2 extra attempts by default, with exponential backoff). Each retry rotates to a different exit IP, which is often enough on its own. Other 4xx responses are returned immediately — they will not improve on a retry.

Every call that touches the network takes a context.Context and honours cancellation, including between retries.

A 413 means the response was larger than your plan's cap; the message names the size, the limit and the plan. It is refused whole — never truncated — and never billed. A 503 means we are momentarily at capacity (usually every browser slot is busy on a Render); it carries Retry-After and is not billed either.

Plan limits

c.Account(ctx) returns your own numbers; these are the shapes to code against.

Free Starter Growth Scale Business+ Pay as you go
Rate limit 5/s 50/s 100/s 200/s 1,000/s none
Max response 2 MB 2 MB 5 MB 10 MB 25 MB 5 MB
AI extraction — — — yes yes —

Past your included requests, overage draws on prepaid credit and floors at zero — at zero you get a 402, never an invoice after the fact.

License

MIT

Documentation

Overview

Package scrapeland is a client for the scrape.land web data extraction API.

There are two ways in, and most Go programs want the first:

  • Client.HTTPClient returns a *http.Client wired to the rotating gateway. Every request it makes exits from a fresh IP. Nothing else in your code has to change — hand it to any library that accepts an *http.Client.
  • Client.Fetch / Extract / Batch / Search call the data API directly and return parsed results, for when you want structured data rather than a transport.

Control parameters (country, sticky session, protocol, latency ceiling) are encoded into the proxy username, so they work through the plain transport too: 

pb_live_KEY-country-us-session-job42-protocol-socks5-maxlatency-3000

Zero dependencies: standard library only.

Index

Constants

View Source 
const (
	// DefaultGateway is the forward proxy. Port 8080 is the long-standing
	// endpoint; 443 also serves the proxy for networks that filter 8080.
	DefaultGateway = "http://gateway.scrape.land:8080"
	// DefaultAPIBase is the data API — a different host from the proxy.
	DefaultAPIBase = "https://scrape.land"
)
View Source 
const BatchMaxURLs = 20

BatchMaxURLs is the server's per-call cap for Batch; BatchIter chunks by it.

Variables

View Source 
var ErrNoAPIKey = errors.New("scrapeland: api key required (pass it or set $SCRAPELAND_API_KEY)")

ErrNoAPIKey is returned by New when no key was passed and none is in the environment.

Functions

func Bool added in v1.0.0 

func Bool(b bool) *bool

Bool returns a pointer to b, for the optional tri-state Options fields (today: Structured, where nil and false mean different things). Saves every caller writing the same two-line temporary.

Types

type APIError 

type APIError struct {
	Status     int
	Endpoint   string
	Body       string
	RetryAfter time.Duration
}

APIError is a non-2xx response from the data API. Status is the HTTP code; Body is the (truncated) response body, which usually carries a JSON message; RetryAfter is the server's Retry-After, when it sent one. Branch on the kind with the Is* helpers or check Status directly: 

var apiErr *scrapeland.APIError
if errors.As(err, &apiErr) && apiErr.IsRateLimited() { time.Sleep(apiErr.RetryAfter) }

func (*APIError) Error 

func (e *APIError) Error() string

func (*APIError) IsBadRequest added in v1.0.0 

func (e *APIError) IsBadRequest() bool

IsBadRequest reports a 400 — a malformed or contradictory request (a retry won't help).

func (*APIError) IsNotFound added in v1.0.0 

func (e *APIError) IsNotFound() bool

IsNotFound reports a 404 — no such resource (e.g. an unknown job id).

func (*APIError) IsPaymentRequired added in v1.0.0 

func (e *APIError) IsPaymentRequired() bool

IsPaymentRequired reports a 402 — out of quota / prepaid credit, or key budget spent.

func (*APIError) IsRateLimited added in v1.0.0 

func (e *APIError) IsRateLimited() bool

IsRateLimited reports a 429 — too many requests; honor RetryAfter.

func (*APIError) IsServer added in v1.0.0 

func (e *APIError) IsServer() bool

IsServer reports a 5xx — a transient server-side failure, usually worth retrying.

func (*APIError) IsUnauthorized added in v1.0.0 

func (e *APIError) IsUnauthorized() bool

IsUnauthorized reports a 401 — the API key is missing, invalid, or revoked.

type BatchResult 

type BatchResult struct {
	Results []Result `json:"results"`
}

BatchResult is the response from Batch: one entry per requested URL, in order.

type Client 

type Client struct {
	APIKey  string
	Gateway string
	APIBase string

	// Defaults applied to every request; per-request Options override them.
	Country      string
	Session      string
	Protocol     string
	MaxLatencyMS int

	// Retries counts EXTRA attempts after the first (default 2). Backoff is the
	// base delay, doubled per attempt.
	Retries int
	Backoff time.Duration

	// HTTP is used for data API calls. It is NOT proxied — the data API is a
	// normal HTTPS endpoint. Use HTTPClient for proxied traffic.
	HTTP *http.Client
}

Client talks to the scrape.land gateway and data API. The zero value is not usable — construct it with New.

func New 

func New(apiKey string, opts ...Option) (*Client, error)

New builds a Client. apiKey falls back to $SCRAPELAND_API_KEY. Gateway and API base fall back to $SCRAPELAND_GATEWAY / $SCRAPELAND_API_BASE, then the public endpoints.

func (*Client) Account 

func (c *Client) Account(ctx context.Context) (map[string]any, error)

Account returns the calling key's plan, quota and usage.

func (*Client) Batch 

func (c *Client) Batch(ctx context.Context, urls []string, o *Options) (*BatchResult, error)

Batch fetches or extracts many URLs in one call.

func (*Client) BatchIter added in v1.0.0 

func (c *Client) BatchIter(ctx context.Context, urls []string, o *Options, yield func(Result) bool) error

BatchIter fetches or extracts any number of URLs, auto-chunking into BatchMaxURLs-sized Batch calls and invoking yield for each Result in input order as chunks complete — the streaming counterpart to Batch, so you can feed a slice of any size without holding every response in memory or tripping the 20-URL cap. yield returns false to stop early (the range-over-func convention, so this composes with `for r := range ...` on Go 1.23+). o applies to every URL, exactly like Batch. A failed chunk stops iteration and returns its error (the same *APIError as Batch). 

err := c.BatchIter(ctx, urls, &scrapeland.Options{Fields: f}, func(r scrapeland.Result) bool {
	save(r); return true
})

func (*Client) Capabilities 

func (c *Client) Capabilities(ctx context.Context) (map[string]any, error)

Capabilities reports what this deployment supports. No API key required.

func (*Client) Extract 

func (c *Client) Extract(ctx context.Context, target string, fields map[string]any, o *Options) (*Result, error)

Extract pulls structured data from a page. fields maps a name to a CSS selector, an XPath string (leading "/"), or an object with css/xpath/attr/all. Pass nil fields and set Options.Prompt to use AI extraction instead.

func (*Client) Fetch 

func (c *Client) Fetch(ctx context.Context, target string, o *Options) (*Result, error)

Fetch retrieves one page through the rotating proxy.

func (*Client) GetJob 

func (c *Client) GetJob(ctx context.Context, id string) (*Job, error)

GetJob returns one job by id.

func (*Client) HTTPClient 

func (c *Client) HTTPClient(o *Options) *http.Client

HTTPClient returns an *http.Client whose every request exits from a fresh IP. Pass nil for o to use the client defaults. 

c, _ := scrapeland.New(key)
res, err := c.HTTPClient(nil).Get("https://api.ipify.org")

Each call builds a fresh Transport, because the control parameters live in the proxy credentials — sharing one across different Options would silently apply the first caller's country and session to everyone else.

func (*Client) Jobs 

func (c *Client) Jobs(ctx context.Context, limit int) ([]Job, error)

Jobs lists recent jobs. limit <= 0 uses the server default.

func (*Client) ProxyURL 

func (c *Client) ProxyURL(o *Options) (*url.URL, error)

ProxyURL returns the proxy URL for use with http.Transport.Proxy.

func (*Client) ProxyUsername 

func (c *Client) ProxyUsername(o *Options) string

ProxyUsername builds the proxy username: the API key plus control parameters. Per-request values in o override the client defaults.

func (*Client) Rank added in v1.0.0 

func (c *Client) Rank(ctx context.Context, target, query string, o *Options) (*RankResult, error)

Rank scores a page's links by how relevant each is to query, so you can pick which sub-pages to fetch next instead of crawling everything. Set o.TopK to cap the list and o.Model ("fast"/"smart") to choose the backend. It is stateless: it ranks the links a page already has and never fetches them.

AI transparency: if the ranking step fails, Rank still returns a result (no error) with the links in document order and RankResult.RankingError set — so check that field when the order matters. Requires the AI backend to be configured. Billed as one fetch plus the AI surcharge; a failed ranking bills only the fetch.

func (*Client) Search 

func (c *Client) Search(ctx context.Context, query string, o *Options) (*SearchResult, error)

Search runs a web search and returns organic results.

func (*Client) SubmitJob 

func (c *Client) SubmitJob(ctx context.Context, jobType string, params map[string]any) (*Job, error)

SubmitJob queues an async job and returns its record.

type Job 

type Job struct {
	ID     string          `json:"id"`
	Type   string          `json:"type"`
	Status string          `json:"status"`
	Result json.RawMessage `json:"result,omitempty"`
	Error  string          `json:"error,omitempty"`
}

Job is an async job record.

type Option 

type Option func(*Client)

Option configures a Client.

func WithAPIBase 

func WithAPIBase(base string) Option

WithAPIBase overrides the data API base URL.

func WithCountry 

func WithCountry(cc string) Option

WithCountry sets a default exit country (ISO code, e.g. "us").

func WithGateway 

func WithGateway(gw string) Option

WithGateway overrides the proxy endpoint.

func WithHTTPClient 

func WithHTTPClient(h *http.Client) Option

WithHTTPClient supplies the client used for data API calls (timeouts, tracing).

func WithMaxLatency 

func WithMaxLatency(ms int) Option

WithMaxLatency only uses upstreams faster than the given milliseconds.

func WithProtocol 

func WithProtocol(p string) Option

WithProtocol restricts the upstream protocol ("http", "https", "socks5").

func WithRetries 

func WithRetries(n int) Option

WithRetries sets how many EXTRA attempts follow a failure.

func WithSession 

func WithSession(id string) Option

WithSession sets a default sticky-session id (same exit IP across requests).

type Options 

type Options struct {
	Format         string            `json:"format,omitempty"` // "html" (default), "text", "markdown", "raw"
	Country        string            `json:"country,omitempty"`
	Session        string            `json:"session,omitempty"`
	Protocol       string            `json:"protocol,omitempty"`
	MaxLatency     int               `json:"max_latency,omitempty"`
	Render         bool              `json:"render,omitempty"`
	Device         string            `json:"device,omitempty"`
	BlockResources bool              `json:"block_resources,omitempty"`
	Fingerprint    bool              `json:"fingerprint,omitempty"`
	WaitFor        string            `json:"wait_for,omitempty"`
	Headers        bool              `json:"headers,omitempty"`
	Screenshot     bool              `json:"screenshot,omitempty"`
	FullPage       bool              `json:"full_page,omitempty"`
	SendHeaders    map[string]string `json:"send_headers,omitempty"`
	Cookies        string            `json:"cookies,omitempty"`
	Method         string            `json:"method,omitempty"`
	Body           string            `json:"body,omitempty"`
	Actions        []any             `json:"actions,omitempty"`
	Metadata       bool              `json:"metadata,omitempty"`
	Links          bool              `json:"links,omitempty"`

	// Extraction. Fields holds CSS/XPath selectors; Prompt drives AI extraction.
	Fields      map[string]any `json:"fields,omitempty"`
	Prompt      string         `json:"prompt,omitempty"`
	Schema      any            `json:"schema,omitempty"`
	Model       string         `json:"model,omitempty"`
	ExtractType string         `json:"extract_type,omitempty"`

	// TopK caps the list returned by Rank (rank only; ignored elsewhere).
	TopK int `json:"top_k,omitempty"`

	// Structured selects what a Prompt returns. Nil (the default) and true give a JSON
	// object in Result.Data; false gives a plain-language reply in Result.Answer and
	// leaves Data nil. A pointer so "unset" and "explicitly false" differ — use
	// scrapeland.Bool(false) to ask for prose.
	//
	// Schema and ExtractType both pin a JSON shape and so imply structured output;
	// combining either with Structured=Bool(false) is rejected with a 400.
	Structured *bool `json:"structured,omitempty"`

	// Search only.
	Count int `json:"count,omitempty"`
}

Options are the per-request knobs shared by Fetch, Extract, Batch and Search. Every field is optional; the zero value means "use the client default, or the server default". It doubles as the request payload, which is why the json tags matter as much as the field names.

type RankResult added in v1.0.0 

type RankResult struct {
	URL          string       `json:"url"`
	Status       int          `json:"status"`
	Query        string       `json:"query"`
	Links        []RankedLink `json:"links"`
	RankingError string       `json:"ranking_error,omitempty"`
}

RankResult is the response from Rank. If the AI step failed, Links come back in document order (not ranked) and RankingError explains why — Rank returns no error in that case, so always check RankingError when the order matters. 

type RankedLink struct {
	URL    string  `json:"url"`
	Text   string  `json:"text"`
	Score  float64 `json:"score"`
	Reason string  `json:"reason"`
}

RankedLink is one link scored by Rank: Score is 0-1, Reason is a short rationale.

type Result 

type Result struct {
	URL         string            `json:"url"`
	Status      int               `json:"status"`
	HTML        string            `json:"html,omitempty"`
	Text        string            `json:"text,omitempty"`
	Markdown    string            `json:"markdown,omitempty"`
	Base64      string            `json:"base64,omitempty"`
	ContentType string            `json:"content_type,omitempty"`
	Screenshot  string            `json:"screenshot,omitempty"`
	Data        map[string]any    `json:"data,omitempty"`
	Answer      string            `json:"answer,omitempty"`
	Metadata    map[string]any    `json:"metadata,omitempty"`
	Links       []string          `json:"links,omitempty"`
	Headers     map[string]string `json:"headers,omitempty"`
	Error       string            `json:"error,omitempty"`
}

Result is one fetched page. Which body field is populated depends on Format.

type SearchResult 

type SearchResult struct {
	Query   string `json:"query"`
	Results []struct {
		Title   string `json:"title"`
		URL     string `json:"url"`
		Snippet string `json:"snippet"`
	} `json:"results"`
}

SearchResult is the response from Search.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.