scrapeland

README

scrapeland-go

Go client for the scrape.land web data extraction API. Standard library only — no dependencies.

go get github.com/scrape-land/scrapeland-go

Two ways to use it

1. As an *http.Client — every request exits a fresh IP

This is usually what you want. Nothing else in your code changes: hand it to any library that takes an *http.Client.

package main

import (
	"fmt"
	"io"

	"github.com/scrape-land/scrapeland-go"
)

func main() {
	c, err := scrapeland.New("pb_live_YOURKEY")
	if err != nil {
		panic(err)
	}

	res, err := c.HTTPClient(nil).Get("https://api.ipify.org")
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	body, _ := io.ReadAll(res.Body)
	fmt.Println("exit IP:", string(body)) // different on every call
}

Route through a specific country, or pin a sticky session so a sequence of requests shares one exit IP:

us := c.HTTPClient(&scrapeland.Options{Country: "us"})
job := c.HTTPClient(&scrapeland.Options{Session: "checkout-42"})

2. As a data API — get structured data back

ctx := context.Background()

// Clean Markdown, ready for an LLM: navigation, headers, footers and sidebars
// stripped, tables preserved, relative links rewritten to absolute.
page, err := c.Fetch(ctx, "https://example.com", &scrapeland.Options{Format: "markdown"})
fmt.Println(page.Markdown)

// CSS or XPath selectors.
out, err := c.Extract(ctx, "https://example.com/product/1", map[string]any{
	"title": "h1",
	"price": ".price",
	"specs": map[string]any{"css": "li.spec", "all": true},
}, nil)
fmt.Println(out.Data["title"])

// Or describe what you want and let the model find it.
out, err = c.Extract(ctx, "https://example.com/product/1", nil, &scrapeland.Options{
	Prompt: "the product name, price in USD, and whether it is in stock",
})

// Many URLs in one call.
batch, err := c.Batch(ctx, []string{"https://a.com", "https://b.com"}, nil)

// Web search.
hits, err := c.Search(ctx, "golang web scraping", &scrapeland.Options{Count: 10})

Configuration

New reads the environment when arguments are omitted:

Setting	Option	Environment	Default
API key	first arg	SCRAPELAND_API_KEY	required
Gateway	WithGateway	SCRAPELAND_GATEWAY	http://gateway.scrape.land:8080
Data API	WithAPIBase	SCRAPELAND_API_BASE	https://scrape.land

Client-wide defaults, overridable per request:

c, err := scrapeland.New("pb_live_YOURKEY",
	scrapeland.WithCountry("de"),
	scrapeland.WithMaxLatency(3000),
	scrapeland.WithRetries(3),
)

Errors

Non-2xx responses come back as *APIError, carrying the status and the server's message — check for a 402 to detect an exhausted quota:

var apiErr *scrapeland.APIError
if errors.As(err, &apiErr) && apiErr.Status == http.StatusPaymentRequired {
	// out of credit
}

429 and 5xx are retried automatically (2 extra attempts by default, with exponential backoff). Each retry rotates to a different exit IP, which is often enough on its own. Other 4xx responses are returned immediately — they will not improve on a retry.

Every call that touches the network takes a context.Context and honours cancellation, including between retries.

License

MIT

Documentation

Overview

Package scrapeland is a client for the scrape.land web data extraction API.

There are two ways in, and most Go programs want the first:

• Client.HTTPClient returns a *http.Client wired to the rotating gateway. Every request it makes exits from a fresh IP. Nothing else in your code has to change — hand it to any library that accepts an *http.Client.
• Client.Fetch / Extract / Batch / Search call the data API directly and return parsed results, for when you want structured data rather than a transport.

Control parameters (country, sticky session, protocol, latency ceiling) are encoded into the proxy username, so they work through the plain transport too:

pb_live_KEY-country-us-session-job42-protocol-socks5-maxlatency-3000

Zero dependencies: standard library only.

Index

• Constants
• Variables
• type APIError
  • func (e *APIError) Error() string
• type BatchResult
• type Client
  • func New(apiKey string, opts ...Option) (*Client, error)
  • func (c *Client) Account(ctx context.Context) (map[string]any, error)
  • func (c *Client) Batch(ctx context.Context, urls []string, o *Options) (*BatchResult, error)
  • func (c *Client) Capabilities(ctx context.Context) (map[string]any, error)
  • func (c *Client) Extract(ctx context.Context, target string, fields map[string]any, o *Options) (*Result, error)
  • func (c *Client) Fetch(ctx context.Context, target string, o *Options) (*Result, error)
  • func (c *Client) GetJob(ctx context.Context, id string) (*Job, error)
  • func (c *Client) HTTPClient(o *Options) *http.Client
  • func (c *Client) Jobs(ctx context.Context, limit int) ([]Job, error)
  • func (c *Client) ProxyURL(o *Options) (*url.URL, error)
  • func (c *Client) ProxyUsername(o *Options) string
  • func (c *Client) Search(ctx context.Context, query string, o *Options) (*SearchResult, error)
  • func (c *Client) SubmitJob(ctx context.Context, jobType string, params map[string]any) (*Job, error)
• type Job
• type Option
  • func WithAPIBase(base string) Option
  • func WithCountry(cc string) Option
  • func WithGateway(gw string) Option
  • func WithHTTPClient(h *http.Client) Option
  • func WithMaxLatency(ms int) Option
  • func WithProtocol(p string) Option
  • func WithRetries(n int) Option
  • func WithSession(id string) Option
• type Options
• type Result
• type SearchResult

Constants

const (
	// DefaultGateway is the forward proxy. Port 8080 is the long-standing
	// endpoint; 443 also serves the proxy for networks that filter 8080.
	DefaultGateway = "http://gateway.scrape.land:8080"
	// DefaultAPIBase is the data API — a different host from the proxy.
	DefaultAPIBase = "https://scrape.land"
)

Variables

var ErrNoAPIKey = errors.New("scrapeland: api key required (pass it or set $SCRAPELAND_API_KEY)")

ErrNoAPIKey is returned by New when no key was passed and none is in the environment.

Types

type APIError

type APIError struct {
	Status   int
	Endpoint string
	Body     string
}

APIError is a non-2xx response from the data API. Status is the HTTP code; Body is the (truncated) response body, which usually carries a JSON message.

func (*APIError) Error

func (e *APIError) Error() string

type BatchResult

type BatchResult struct {
	Results []Result `json:"results"`
}

BatchResult is the response from Batch: one entry per requested URL, in order.

type Client

type Client struct {
	APIKey  string
	Gateway string
	APIBase string

	// Defaults applied to every request; per-request Options override them.
	Country      string
	Session      string
	Protocol     string
	MaxLatencyMS int

	// Retries counts EXTRA attempts after the first (default 2). Backoff is the
	// base delay, doubled per attempt.
	Retries int
	Backoff time.Duration

	// HTTP is used for data API calls. It is NOT proxied — the data API is a
	// normal HTTPS endpoint. Use HTTPClient for proxied traffic.
	HTTP *http.Client
}

Client talks to the scrape.land gateway and data API. The zero value is not usable — construct it with New.

func New

func New(apiKey string, opts ...Option) (*Client, error)

New builds a Client. apiKey falls back to $SCRAPELAND_API_KEY. Gateway and API base fall back to $SCRAPELAND_GATEWAY / $SCRAPELAND_API_BASE, then the public endpoints.

func (*Client) Account

func (c *Client) Account(ctx context.Context) (map[string]any, error)

Account returns the calling key's plan, quota and usage.

func (*Client) Batch

func (c *Client) Batch(ctx context.Context, urls []string, o *Options) (*BatchResult, error)

Batch fetches or extracts many URLs in one call.

func (*Client) Capabilities

func (c *Client) Capabilities(ctx context.Context) (map[string]any, error)

Capabilities reports what this deployment supports. No API key required.

func (*Client) Extract

func (c *Client) Extract(ctx context.Context, target string, fields map[string]any, o *Options) (*Result, error)

Extract pulls structured data from a page. fields maps a name to a CSS selector, an XPath string (leading "/"), or an object with css/xpath/attr/all. Pass nil fields and set Options.Prompt to use AI extraction instead.

func (*Client) Fetch

func (c *Client) Fetch(ctx context.Context, target string, o *Options) (*Result, error)

Fetch retrieves one page through the rotating proxy.

func (*Client) GetJob

func (c *Client) GetJob(ctx context.Context, id string) (*Job, error)

GetJob returns one job by id.

func (*Client) HTTPClient

func (c *Client) HTTPClient(o *Options) *http.Client

HTTPClient returns an *http.Client whose every request exits from a fresh IP. Pass nil for o to use the client defaults.

c, _ := scrapeland.New(key)
res, err := c.HTTPClient(nil).Get("https://api.ipify.org")

Each call builds a fresh Transport, because the control parameters live in the proxy credentials — sharing one across different Options would silently apply the first caller's country and session to everyone else.

func (*Client) Jobs

func (c *Client) Jobs(ctx context.Context, limit int) ([]Job, error)

Jobs lists recent jobs. limit <= 0 uses the server default.

func (*Client) ProxyURL

func (c *Client) ProxyURL(o *Options) (*url.URL, error)

ProxyURL returns the proxy URL for use with http.Transport.Proxy.

func (*Client) ProxyUsername

func (c *Client) ProxyUsername(o *Options) string

ProxyUsername builds the proxy username: the API key plus control parameters. Per-request values in o override the client defaults.

func (*Client) Search

func (c *Client) Search(ctx context.Context, query string, o *Options) (*SearchResult, error)

Search runs a web search and returns organic results.

func (*Client) SubmitJob

func (c *Client) SubmitJob(ctx context.Context, jobType string, params map[string]any) (*Job, error)

SubmitJob queues an async job and returns its record.

type Job

type Job struct {
	ID     string          `json:"id"`
	Type   string          `json:"type"`
	Status string          `json:"status"`
	Result json.RawMessage `json:"result,omitempty"`
	Error  string          `json:"error,omitempty"`
}

Job is an async job record.

type Option

type Option func(*Client)

Option configures a Client.

func WithAPIBase

func WithAPIBase(base string) Option

WithAPIBase overrides the data API base URL.

func WithCountry

func WithCountry(cc string) Option

WithCountry sets a default exit country (ISO code, e.g. "us").

func WithGateway

func WithGateway(gw string) Option

WithGateway overrides the proxy endpoint.

func WithHTTPClient

func WithHTTPClient(h *http.Client) Option

WithHTTPClient supplies the client used for data API calls (timeouts, tracing).

func WithMaxLatency

func WithMaxLatency(ms int) Option

WithMaxLatency only uses upstreams faster than the given milliseconds.

func WithProtocol

func WithProtocol(p string) Option

WithProtocol restricts the upstream protocol ("http", "https", "socks5").

func WithRetries

func WithRetries(n int) Option

WithRetries sets how many EXTRA attempts follow a failure.

func WithSession

func WithSession(id string) Option

WithSession sets a default sticky-session id (same exit IP across requests).

type Options

type Options struct {
	Format         string            `json:"format,omitempty"` // "html" (default), "text", "markdown", "raw"
	Country        string            `json:"country,omitempty"`
	Session        string            `json:"session,omitempty"`
	Protocol       string            `json:"protocol,omitempty"`
	MaxLatency     int               `json:"max_latency,omitempty"`
	Render         bool              `json:"render,omitempty"`
	Device         string            `json:"device,omitempty"`
	BlockResources bool              `json:"block_resources,omitempty"`
	Fingerprint    bool              `json:"fingerprint,omitempty"`
	WaitFor        string            `json:"wait_for,omitempty"`
	Headers        bool              `json:"headers,omitempty"`
	Screenshot     bool              `json:"screenshot,omitempty"`
	FullPage       bool              `json:"full_page,omitempty"`
	SendHeaders    map[string]string `json:"send_headers,omitempty"`
	Cookies        string            `json:"cookies,omitempty"`
	Method         string            `json:"method,omitempty"`
	Body           string            `json:"body,omitempty"`
	Actions        []any             `json:"actions,omitempty"`
	Metadata       bool              `json:"metadata,omitempty"`
	Links          bool              `json:"links,omitempty"`

	// Extraction. Fields holds CSS/XPath selectors; Prompt drives AI extraction.
	Fields      map[string]any `json:"fields,omitempty"`
	Prompt      string         `json:"prompt,omitempty"`
	Schema      any            `json:"schema,omitempty"`
	Model       string         `json:"model,omitempty"`
	ExtractType string         `json:"extract_type,omitempty"`

	// Search only.
	Count int `json:"count,omitempty"`
}

Options are the per-request knobs shared by Fetch, Extract, Batch and Search. Every field is optional; the zero value means "use the client default, or the server default". It doubles as the request payload, which is why the json tags matter as much as the field names.

type Result

type Result struct {
	URL         string            `json:"url"`
	Status      int               `json:"status"`
	HTML        string            `json:"html,omitempty"`
	Text        string            `json:"text,omitempty"`
	Markdown    string            `json:"markdown,omitempty"`
	Base64      string            `json:"base64,omitempty"`
	ContentType string            `json:"content_type,omitempty"`
	Screenshot  string            `json:"screenshot,omitempty"`
	Data        map[string]any    `json:"data,omitempty"`
	Metadata    map[string]any    `json:"metadata,omitempty"`
	Links       []string          `json:"links,omitempty"`
	Headers     map[string]string `json:"headers,omitempty"`
	Error       string            `json:"error,omitempty"`
}

Result is one fetched page. Which body field is populated depends on Format.

type SearchResult

type SearchResult struct {
	Query   string `json:"query"`
	Results []struct {
		Title   string `json:"title"`
		URL     string `json:"url"`
		Snippet string `json:"snippet"`
	} `json:"results"`
}

SearchResult is the response from Search.