Help us shape the future of Go! Take the 2026 Go Developer Survey

sdo

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2026 License: Apache-2.0 Imports: 38 Imported by: 0

README

sdo — Go SDK for Scrape.do

The official Go SDK for the Scrape.do web-scraping API. Token-only auth, transparent compression, a typed result, and a generic plugin core — no telemetry, no UI.

go get github.com/scrape-do/sdo-go

Quickstart

package main

import (
	"context"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	// Token precedence: New("TOKEN") > SDO_TOKEN env var.
	client, err := sdo.New("YOUR_TOKEN")
	if err != nil {
		panic(err)
	}

	res, err := client.Scrape(context.Background(), &sdo.ScrapeConfig{
		URL:     "https://example.com",
		Render:  true,
		GeoCode: "us",
	})
	if err != nil {
		panic(err)
	}
	cost, _ := res.Cost()
	fmt.Println(res.StatusCode, cost)
	text := res.Text()
	fmt.Println(text[:min(200, len(text))]) // the first 200 bytes, or fewer
}

Go 1.26 or newer is required.

Method, body and headers

Scrape(ctx, cfg) is a GET. Do sends any method through the gateway:

res, err := client.Do(ctx, &sdo.ScrapeConfig{URL: "https://example.com/api"}, &sdo.Request{
	Method:  "PUT",                       // default GET, or POST when a body is given
	JSON:    map[string]any{"name": "x"}, // or Data: []byte("a=1&b=2")
	Headers: map[string]string{"Accept": "application/json"},
})

A JSON body is sent as compact JSON with sorted keys and Content-Type: application/json (unless you set one). GET/HEAD with a body, or JSON and Data together, are a *sdo.ConfigError.

Header names are case-insensitive, and your value replaces the SDK's own (User-Agent, Accept-Encoding). A name must be an RFC 9110 token and a value printable ASCII (space and tab allowed); Host, Content-Length, Transfer-Encoding and Connection cannot be set. Anything else is a *sdo.ConfigError before the request is sent.

Retries and timeouts

A retryable status (429, 502, 503, 504, 520–524) is retried with exponential backoff (or Retry-After, capped at 30 s); POST, PUT, PATCH and DELETE only on 429, so a request with side effects is never repeated after the target saw it. A failure before anything was sent (DNS, connect, TLS) is retried for GET, HEAD and OPTIONS. A failure after the request was sent — a read timeout, a dropped connection, a body that cannot be decompressed — is never retried: the gateway may still be running (and billing) it. A 510 is the gateway's record of a request whose HTTP client cancelled it before it finished; it is not an upstream failure and is not retried.

The client timeout covers one whole attempt. Its default, 70 s, is the gateway's own default timeout (60 s) plus 10 s, so the gateway's answer always arrives first. When a ScrapeConfig sets the gateway Timeout (or a plugin call passes a timeout param), that request's attempts wait at least the gateway timeout plus 10 s, so the client never gives up on a request the gateway was allowed to take longer for. A timeout error names the deadline that expired.

Errors

Successful scrapes — including target 404/410 pages, any status the gateway billed and 3xx responses — return a *Result (check res.StatusCode; res.Header holds every header, repeated ones included). Only gateway-origin failures return an error.

Scrape.do-Request-Cost is the authoritative billing figure. A response with status 400 or above that carries a positive cost is the target's own answer (its 400, 401, 403, 422, ...): a *Result, never retried. Without one, the status keeps its gateway meaning: the gateway's own 400 (a parameter it rejected, such as render=mert) is KindValidation, its 401 (a refused token, no credits) KindAuth, and so on. APIError.Billed follows the cost header when it is present (0 is not billed) and the documented status table otherwise; APIError.Cost() and Result.Cost() return the header:

var apiErr *sdo.APIError
if errors.As(err, &apiErr) {
	switch apiErr.Kind {
	case sdo.KindConcurrency: // 429, retryable
	case sdo.KindAuth:        // 401/403 — a bad token or no credits left
	}
}

With TransparentResponse: true, a response the target produced itself (marked by Scrape.do-Initial-Status-Code) is always a *Result, whatever its status, and is never retried. Invalid options are a *sdo.ConfigError before any request: unknown enum values (matched case-insensitively), out-of-range millisecond values (Timeout 5000–120000, RetryTimeout 5000–55000, CustomWait 0–35000), render-only fields (Height and Width included) without rendering, and a target URL that is not absolute http(s) (its user:password@ is masked in the message).

The documented parameter dependencies are checked too, each one the gateway rejects (or documents as invalid), so a mistake costs nothing:

Rule Why
ShowFrames, ShowWebsocketRequests and every screenshot field need ReturnJSON (and rendering) their output is part of the JSON response
at most one of Screenshot, FullScreenshot, ParticularScreenshot the gateway takes one
ParticularScreenshot cannot be combined with PlayWithBrowser use a ScreenShot action instead
RegionalGeoCode needs SuperProxy and excludes GeoCode the gateway rejects both
ZipCode needs SuperProxy and GeoCode without super it is silently ignored
SessionID is an integer 0–1000000 (decimal digits) the documented range
RetryTimeout cannot be combined with rendering the gateway rejects it
at most one of CustomHeaders, ExtraHeaders, ForwardHeaders the gateway rejects any two

ReturnJSON on its own is allowed: without rendering the gateway wraps the plain response in the same JSON envelope.

Configuration

Option Environment Default
New(token) SDO_TOKEN — (required)
WithBaseURL SDO_BASE_URL https://api.scrape.do
WithAsyncBaseURL SDO_ASYNC_BASE_URL https://q.scrape.do
WithTimeout SDO_TIMEOUT (seconds) 70 s, total per attempt

An option wins over the environment; an empty value means "not given". A token is trimmed of surrounding whitespace (a secret file's trailing newline is fine) and must otherwise be visible ASCII. SDO_TIMEOUT is a decimal number of seconds, at least 0.001 (30, 1.5). Base URLs must be https — the token travels with every request — except for localhost, 127.0.0.1 and ::1; sdo.WithAllowInsecureHTTP() lifts that for other hosts. A base URL may carry a path prefix (https://gw.example/pfx), but no query, fragment or credentials. Invalid values are a *sdo.ConfigError from New. Redirects are never followed, even with your own WithHTTPClient.

Error messages, retained error bodies and wrapped causes have the token and any credential-shaped value masked, and printing a Client never shows the token. errors.As through an SDK error hands back a *url.Error as a redacted copy and never returns a cause whose text carries a credential. client.Redact(text) applies the same masking to your own output.

Compression

Every request sends Accept-Encoding: gzip, deflate, br and the SDK decompresses the response transparently (res.Bytes() is decompressed): gzip/x-gzip, deflate and br, stacked codings included; any other coding is returned as it is. Decompression is capped (32 MiB by default, WithMaxResponseBytes). Use sdo.WithPreferBrotli() to advertise br alone.

Text, JSON and HTML

res.Text() decodes the body as a browser would (WHATWG Encoding Standard: BOM, charset, <meta>, then UTF-8). res.JSON(&v) parses it as UTF-8 and returns a *sdo.TransportError for anything that is not JSON. For CSS selection, the html subpackage returns a goquery document (a separate package, so programs that do not parse HTML do not link goquery):

import sdohtml "github.com/scrape-do/sdo-go/html"

doc, err := sdohtml.Document(res)
title := doc.Find("h1").First().Text()

Plugins

// Google plugins take Google's own locale params: gl (country), hl (language).
data, err := client.Plugin(ctx, "google/shopping", map[string]any{"q": "laptop", "gl": "us"})

// PluginResult returns the same response with its status and headers.
res, err := client.PluginResult(ctx, "google/search", map[string]any{"q": "laptop", "gl": "us"})
cost, _ := res.Cost()
var serp map[string]any
err = res.JSON(&serp)

// The Amazon raw-HTML plugin lives at /plugin/amazon/; the route "amazon"
// (or "amazon/") calls it.
html, err := client.Plugin(ctx, "amazon", map[string]any{
	"url": "https://www.amazon.com/dp/B0C7BKZ883", "geocode": "us", "zipcode": "10001",
})

Params are sent sorted; a list repeats its key, an object is sent as compact JSON. An empty key, the key token (the client supplies it), a key that is not an ASCII letter followed by letters, digits, _, . or -, or a list inside a list is a *sdo.ConfigError. ScrapeConfig.ExtraParams keys follow the same shape rule.

Usage statistics

u, err := client.Usage(ctx) // GET https://api.scrape.do/info; no credits
fmt.Println(u.IsActive, u.RemainingMonthlyRequest, u.RemainingConcurrentRequest)

Usage returns IsActive, ConcurrentRequest, MaxMonthlyRequest, RemainingConcurrentRequest and RemainingMonthlyRequest. The endpoint allows 10 requests per minute per IP and answers 429 above that; the SDK does not retry its error statuses (a retry within seconds would only spend more of the same minute). An unknown token is answered with a 502 "authentication refused".

Async jobs

Asynchronous scraping is the REST job API on https://q.scrape.do, reached via client.Jobs(). Every request authenticates with the X-Token header (never a token= query) and returns typed values. The callback option was removed — poll a job's status and pull each task's result instead.

jobs := client.Jobs()

// Create a job for one or more targets. The options map uses the job API's own
// PascalCase body keys (e.g. Super, GeoCode, Render, Output) verbatim.
job, err := jobs.Create(ctx, []string{"https://example.com"},
	map[string]any{"Super": true, "GeoCode": "us"})

status, err := jobs.Status(ctx, job.JobID)          // job + per-task state
task, err := jobs.Result(ctx, job.JobID, "TASK_ID") // one task's delivered content
body, err := task.ContentBytes()                     // base64-decoded for binary targets
all, err := jobs.List(ctx)                           // every visible job
err = jobs.Cancel(ctx, job.JobID)                    // cancel a job
acct, err := jobs.Me(ctx)                            // concurrency + credits snapshot

Job and task ids are validated (^[A-Za-z0-9_-]{1,128}$) before they reach the URL. Job calls share the client's transport — timeout, size cap, retry policy, redaction. Any non-2xx job response, a redirect included, is a *sdo.APIError whose message comes from the API's {"Error": ...} body (job API errors are never billed). Responses are decoded leniently: a missing or mistyped field reads as its zero value instead of failing the call, and a 2xx body that is not a JSON object is a *sdo.TransportError. Cancel ignores the response body.

CI/CD workflows

Workflow Runs on What it does
ci.yml every push to main and every pull request go vet and the race-enabled test suite on Go 1.26 and 1.27 (Linux), plus macOS and Windows on 1.27, including the cross-language conformance suite and its integrity check; a quality job for gofmt and the 99% statement-coverage floor; govulncheck; the public-API snapshot (api-surface.txt) must be up to date; CodeQL code scanning. Everything runs on GitHub-hosted runners; CodeQL uploads its results from the org's self-hosted runners, and a pull request from a fork keeps its SARIF as a build artifact instead.
security.yml push to main, pull requests, and weekly secret scanning over the full git history (gitleaks), OSV-Scanner over go.mod/go.sum, and an SBOM uploaded as a build artifact. OpenSSF Scorecard runs on pushes to main and weekly, with results in code scanning.
tag.yml a version tag only A Go module is published by its tag (the module proxy fetches it), so there is no publish job. This workflow checks that the tag is vX.Y.Z or vX.Y.Z-rc.N, is on main, and matches version.go. A -rc.N tag is a Go pre-release, so @latest never selects it.
Dependabot weekly update pull requests for Go modules and the GitHub Actions, taken only after a 7-day cooldown; each one runs the checks above.

Every action is pinned to a full commit SHA and every job declares least-privilege permissions:. Publishing never happens outside the scrape-do organisation, never from a commit that is not on main, and never without a maintainer's approval.

Responsible use

You are responsible for complying with the terms of service, robots.txt, and applicable laws of any site you scrape. Third-party names are trademarks of their respective owners; Scrape.do is not affiliated with them.

License

Apache-2.0.

Documentation

Overview

Package sdo is the official Go SDK for the Scrape.do web-scraping API.

Example (Quickstart)

Example_quickstart is the README quickstart, run against a local stand-in for the gateway that returns a page shorter than 200 bytes. 

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	gateway := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Scrape.do-Request-Cost", "5")
		fmt.Fprint(w, "tiny page")
	}))
	defer gateway.Close()

	client, err := sdo.New("YOUR_TOKEN", sdo.WithBaseURL(gateway.URL))
	if err != nil {
		panic(err)
	}
	res, err := client.Scrape(context.Background(), &sdo.ScrapeConfig{
		URL:     "https://example.com",
		Render:  true,
		GeoCode: "us",
	})
	if err != nil {
		panic(err)
	}
	cost, _ := res.Cost()
	fmt.Println(res.StatusCode, cost)
	text := res.Text()
	fmt.Println(text[:min(200, len(text))]) // the first 200 bytes, or fewer
}

Output:
200 5
tiny page

Index

Examples

Constants

View Source 
const Version = "0.1.0"

Version is the single source of truth for the installed SDK version. Its only use on the wire is the User-Agent (sdo-sdk-go/<Version>).

Variables

This section is empty.

Functions

func Bool 

func Bool(b bool) *bool

Bool returns a pointer to b, for setting tri-state *bool fields such as ScrapeConfig.BlockResources from a literal (Go cannot take the address of one): BlockResources: sdo.Bool(false).

func RedactSecrets 

func RedactSecrets(text string) string

RedactSecrets masks credential-shaped values in text: token=, api_key= and access_token= values (in a URL or in prose), the same names as JSON string fields, and an X-Token header value.

It works by shape, so it also covers URLs the SDK never built (echoed by a proxy, or embedded in a *url.Error). Client.Redact additionally masks the client's own token wherever it appears.

Types

type APIError 

type APIError struct {
	// Message is the gateway's own explanation when its error body carries
	// one, otherwise a default for the status. Credentials are redacted.
	Message    string
	StatusCode int
	Kind       ErrorKind
	// Billed reports whether the gateway charged for the response: the
	// Scrape.do-Request-Cost header when it is readable (positive is billed,
	// 0 is not), else the documented table (the gateway's own 400, 401, 429,
	// 502 and 510 are not billed). Never for the async job API.
	Billed bool
	// IsRetryable reports whether the status is transient. The client retries
	// such statuses itself; POST, PUT, PATCH and DELETE only on 429.
	IsRetryable bool
	// Body is the start of the response body as text (up to 64 KiB is read),
	// redacted, truncated to 2048 characters.
	Body string
	// contains filtered or unexported fields
}

APIError is the single, normative gateway-origin error. Match it with errors.As(err, &apiErr) then inspect Kind. A successful scrape of a page that itself returned 404/410 is NOT an APIError — it is a *Result, and so is any response the gateway billed (a positive Scrape.do-Request-Cost: the target's 400, 401, 403, 422, ...) and any response the target itself produced under TransparentResponse. A 400 or 401 the gateway produced (a parameter it rejected, a refused token) carries no cost and is an APIError.

func (*APIError) Cost 

func (e *APIError) Cost() (int, bool)

Cost returns the Scrape.do-Request-Cost header and whether it was present as a single non-negative integer, as Result.Cost reads it (never set for the async job API). An error's cost is 0: a positive cost makes the response a *Result.

func (*APIError) Error 

func (e *APIError) Error() string

func (*APIError) IsAuth 

func (e *APIError) IsAuth() bool

Convenience predicates.

func (*APIError) IsConcurrency 

func (e *APIError) IsConcurrency() bool

func (*APIError) IsGateway 

func (e *APIError) IsGateway() bool

func (*APIError) IsValidation 

func (e *APIError) IsValidation() bool

func (*APIError) RemainingCredits 

func (e *APIError) RemainingCredits() (int, bool)

RemainingCredits returns the Scrape.do-Remaining-Credits header of a 429 and whether it was present.

func (*APIError) RetryAfter 

func (e *APIError) RetryAfter() (time.Duration, bool)

RetryAfter returns the Retry-After header as a duration (whole seconds) and whether it was present as a plain non-negative integer.

type Account 

type Account struct {
	TotalConcurrency int `json:"TotalConcurrency"`
	FreeConcurrency  int `json:"FreeConcurrency"`
	ActiveJobs       int `json:"ActiveJobs"`
	// AvailableCredits reads "AvailableCredits", falling back to the API's
	// misspelled "AvaliableCredits".
	AvailableCredits int `json:"AvailableCredits"`
}

Account is the caller's concurrency and credit snapshot.

func (*Account) UnmarshalJSON 

func (a *Account) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes an account snapshot leniently.

type AsyncJobs 

type AsyncJobs struct {
	// contains filtered or unexported fields
}

AsyncJobs wraps the q.scrape.do REST job surface. Every request authenticates via the X-Token header (never a token= query) and goes through the owning client's transport: its timeout, size cap, retry policy and redaction. Any non-2xx response, a redirect included, is an *APIError; a success body that is not valid JSON is a *TransportError.

func (*AsyncJobs) Cancel 

func (j *AsyncJobs) Cancel(ctx context.Context, jobID string) error

Cancel cancels a job by id. Any 2xx is success; the body is ignored.

func (*AsyncJobs) Create 

func (j *AsyncJobs) Create(ctx context.Context, targets []string, options map[string]any) (*Job, error)

Create submits a job for one or more target URLs. options is merged into the JSON body using the API's own PascalCase job-body keys (e.g. "Super", "GeoCode", "Render", "Output"); it is not ScrapeConfig query serialization. The body is canonical JSON. targets must be non-empty, and options may not set "Targets".

func (AsyncJobs) GoString 

func (j AsyncJobs) GoString() string

GoString is String, for %#v.

func (*AsyncJobs) List 

func (j *AsyncJobs) List(ctx context.Context) ([]JobStatus, error)

List returns every job visible to the caller.

func (*AsyncJobs) Me 

func (j *AsyncJobs) Me(ctx context.Context) (*Account, error)

Me returns the caller's concurrency and credit snapshot.

func (*AsyncJobs) Result 

func (j *AsyncJobs) Result(ctx context.Context, jobID, taskID string) (*TaskResult, error)

Result returns the delivered content of a task within a job.

func (*AsyncJobs) Status 

func (j *AsyncJobs) Status(ctx context.Context, jobID string) (*JobStatus, error)

Status returns the current state of a job and its tasks.

func (AsyncJobs) String 

func (j AsyncJobs) String() string

String describes the job client without its token.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the Scrape.do client. It is safe for concurrent use.

func New 

func New(token string, opts ...Option) (*Client, error)

New constructs a Client.

Token precedence: the explicit argument, then SDO_TOKEN. Either is trimmed of ASCII whitespace ("" then means not given) and must otherwise be visible ASCII. The environment also supplies SDO_BASE_URL, SDO_ASYNC_BASE_URL and SDO_TIMEOUT (a decimal number of seconds, at least 0.001) when the matching option is not given. Invalid settings, from options or the environment, return a *ConfigError.

Example

ExampleNew shows constructing a client. The token is read from the explicit argument, falling back to the SDO_TOKEN environment variable. 

package main

import (
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New("your-token")
	if err != nil {
		fmt.Println("config error:", err)
		return
	}
	_ = client
}
Example (Options)

ExampleNew_options configures timeouts, retries, and brotli negotiation. 

package main

import (
	"fmt"
	"time"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New(
		"your-token",
		sdo.WithTimeout(30*time.Second),
		sdo.WithRetries(3),
		sdo.WithBackoff(250*time.Millisecond),
		sdo.WithPreferBrotli(),
	)
	if err != nil {
		fmt.Println("config error:", err)
		return
	}
	_ = client
}

func (*Client) Do 

func (c *Client) Do(ctx context.Context, config *ScrapeConfig, req *Request) (*Result, error)

Do scrapes config.URL with the method, body and headers in req (nil is a plain GET).

Retries follow the method: GET, HEAD and OPTIONS retry a retryable status and a failure before anything was sent (DNS, connect, TLS); POST, PUT, PATCH and DELETE, which can have side effects on the target, retry only a 429 (the gateway refused before contacting the target). A failure after the request was sent (a read timeout, a dropped connection, a body that cannot be read or decoded) is never retried: the gateway may still run and bill it.

Example

ExampleClient_Do sends a POST with a JSON body through the gateway. The body is canonical JSON; a Request with no Method and a body is a POST. 

package main

import (
	"context"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New("your-token")
	if err != nil {
		return
	}
	res, err := client.Do(context.Background(),
		&sdo.ScrapeConfig{URL: "https://httpbin.org/post", CustomHeaders: true},
		&sdo.Request{
			JSON:    map[string]any{"query": "laptop", "page": 1},
			Headers: map[string]string{"Accept": "application/json"},
		})
	if err != nil {
		return
	}
	fmt.Println(res.StatusCode, res.Header.Values("Set-Cookie"))
}

func (Client) GoString 

func (c Client) GoString() string

GoString is String, for %#v.

func (*Client) Jobs 

func (c *Client) Jobs() *AsyncJobs

Jobs returns the async (q.scrape.do) job API. It shares the client's transport: the same timeout, size cap, retry policy and redaction.

Example

ExampleClient_Jobs submits an async job via the REST job API, polls it and reads a task's content. The options map uses the job API's own PascalCase body keys (e.g. Super, GeoCode). 

package main

import (
	"context"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New("your-token")
	if err != nil {
		return
	}
	ctx := context.Background()
	job, err := client.Jobs().Create(ctx, []string{"https://example.com"},
		map[string]any{"Super": true, "GeoCode": "us"})
	if err != nil {
		return
	}
	status, err := client.Jobs().Status(ctx, job.JobID)
	if err != nil || len(status.Tasks) == 0 {
		return
	}
	task, err := client.Jobs().Result(ctx, job.JobID, status.Tasks[0].TaskID)
	if err != nil {
		return
	}
	// Binary targets arrive base64-encoded; ContentBytes decodes them.
	body, err := task.ContentBytes()
	if err != nil {
		return
	}
	fmt.Println(task.Status, task.StatusCode, len(body))
}

func (*Client) Plugin 

func (c *Client) Plugin(ctx context.Context, route string, params map[string]any) (any, error)

Plugin calls a documented plugin endpoint generically. Returns parsed JSON (as any) or raw bytes for non-JSON plugins. PluginResult returns the same response with its status and headers (cost included).

route is the path under /plugin/, e.g. "google/shopping"; "amazon" (or "amazon/") is the Amazon raw-HTML plugin at /plugin/amazon/.

Example

ExampleClient_Plugin calls a structured-data plugin endpoint. JSON responses come back as a decoded any; non-JSON plugins return raw []byte. 

package main

import (
	"context"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New("your-token")
	if err != nil {
		return
	}
	out, err := client.Plugin(context.Background(), "google/shopping", map[string]any{
		"q":       "laptop",
		"geoCode": "us",
	})
	if err != nil {
		return
	}
	fmt.Printf("%T\n", out)
}

func (*Client) PluginResult 

func (c *Client) PluginResult(ctx context.Context, route string, params map[string]any) (*Result, error)

PluginResult calls a plugin like Plugin but returns the delivered response as a *Result: its status, its headers (Cost, RemainingCredits) and its body (Result.JSON parses it). Result.URL is "".

func (*Client) Redact 

func (c *Client) Redact(text string) string

Redact masks this client's token (literally and percent-encoded) and every credential shape RedactSecrets knows in text. Use it on anything derived from a response before it leaves your process (logs, an LLM's context).

func (*Client) Scrape 

func (c *Client) Scrape(ctx context.Context, config *ScrapeConfig) (*Result, error)

Scrape fetches a URL with a GET and returns a *Result. It is Do(ctx, config, nil).

Example

ExampleClient_Scrape performs a basic scrape and reads the decoded body. 

package main

import (
	"context"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New("your-token")
	if err != nil {
		return
	}
	res, err := client.Scrape(context.Background(), &sdo.ScrapeConfig{
		URL: "https://example.com",
	})
	if err != nil {
		return
	}
	fmt.Println(res.StatusCode)
	fmt.Println(res.Text())
}
Example (Render)

ExampleClient_Scrape_render enables headless rendering with a geo target and a wait condition — the render-only fields require Render (or RenderMode) set. 

package main

import (
	"context"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New("your-token")
	if err != nil {
		return
	}
	res, err := client.Scrape(context.Background(), &sdo.ScrapeConfig{
		URL:          "https://example.com",
		Render:       true,
		GeoCode:      "us",
		WaitUntil:    "networkidle2",
		WaitSelector: "#app",
		Output:       "markdown",
	})
	if err != nil {
		return
	}
	fmt.Println(res.Text())
}

func (Client) String 

func (c Client) String() string

String describes the client without its token, so logging a Client or a copy of one (with %v, %+v, %s or %q) cannot leak the credential.

func (*Client) Usage 

func (c *Client) Usage(ctx context.Context) (*Usage, error)

Usage returns the account's usage statistics (GET <base URL>/info, the token as a query parameter). It costs no credits.

The endpoint allows 10 requests per minute per IP address and answers 429 above that, so a retryable status is not retried here (a retry within seconds would only spend more of the same minute); a failure before the request was sent still is. A token the gateway does not know is answered with a 502 "authentication refused".

type ConfigError 

type ConfigError struct{ Message string }

ConfigError is returned for client-side misconfiguration (no HTTP call made), e.g. a missing token or mutually exclusive options.

func (*ConfigError) Error 

func (e *ConfigError) Error() string

type ErrorKind 

type ErrorKind int

ErrorKind classifies a gateway-origin API error. It maps to the shared cross-language taxonomy (Auth/Validation/Concurrency/Gateway).

There is no quota kind: the gateway has no 402 tier and returns 401 for an exhausted credit balance too, so that case is KindAuth and is told apart by the message rather than the status code. 

const (
	KindOther ErrorKind = iota
	KindAuth
	KindValidation
	KindConcurrency
	KindGateway
)

func (ErrorKind) String 

func (k ErrorKind) String() string

type Job 

type Job struct {
	JobID   string   `json:"JobID"`
	Message string   `json:"Message"`
	TaskIDs []string `json:"TaskIDs"`
}

Job is the acknowledgement returned by Create.

func (*Job) UnmarshalJSON 

func (j *Job) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes a job acknowledgement leniently.

type JobStatus 

type JobStatus struct {
	JobID     string   `json:"JobID"`
	TaskIDs   []string `json:"TaskIDs"`
	Status    string   `json:"Status"`
	StartTime string   `json:"StartTime"`
	EndTime   string   `json:"EndTime"`
	Canceled  bool     `json:"Canceled"`
	Tasks     []Task   `json:"Tasks"`
}

JobStatus is the full state of a job and its tasks.

func (*JobStatus) UnmarshalJSON 

func (s *JobStatus) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes a job status leniently.

type Option 

type Option func(*clientConfig)

Option configures a Client.

func WithAcceptEncoding 

func WithAcceptEncoding(s string) Option

WithAcceptEncoding overrides the advertised Accept-Encoding (default "gzip, deflate, br"). The value must be printable ASCII.

func WithAllowInsecureHTTP 

func WithAllowInsecureHTTP() Option

WithAllowInsecureHTTP permits an http:// base URL on any host. Without it, plain http is accepted only for localhost, 127.0.0.1 and ::1, because the token travels with every request.

func WithAsyncBaseURL 

func WithAsyncBaseURL(u string) Option

WithAsyncBaseURL sets the async job API base (default https://q.scrape.do, or SDO_ASYNC_BASE_URL). The same https rule as WithBaseURL applies.

func WithBackoff 

func WithBackoff(d time.Duration) Option

WithBackoff sets the base of the exponential backoff between retries (default 500ms). A Retry-After header takes precedence; waits are capped at 30s.

func WithBaseURL 

func WithBaseURL(u string) Option

WithBaseURL sets the sync API base (default https://api.scrape.do, or the SDO_BASE_URL environment variable). It must be https, unless the host is localhost, 127.0.0.1 or ::1, or WithAllowInsecureHTTP is given. "" means not given.

func WithHTTPClient 

func WithHTTPClient(h *http.Client) Option

WithHTTPClient supplies the http.Client to send through (for proxies, mTLS or tests). The SDK uses a copy that never follows redirects, so a 3xx cannot carry the token elsewhere. Its Timeout is kept unless WithTimeout is given; when it is 0 the SDK's timeout applies.

func WithMaxResponseBytes 

func WithMaxResponseBytes(n int64) Option

WithMaxResponseBytes sets the ceiling for a decompressed response body (default 32 MiB). Exceeding it returns *ResponseTooLargeError instead of inflating the whole payload.

func WithPreferBrotli 

func WithPreferBrotli() Option

WithPreferBrotli advertises "br" alone so the gateway actually returns brotli.

func WithRetries 

func WithRetries(n int) Option

WithRetries sets how many times a transient failure is retried (default 2). A negative count is treated as 0: the request is still sent once.

func WithTimeout 

func WithTimeout(d time.Duration) Option

WithTimeout sets the total deadline of one HTTP attempt: connect, headers and the whole body. The default is 70s (the gateway's 60s default timeout plus 10s), or SDO_TIMEOUT seconds. 0 disables it. When a ScrapeConfig sets the gateway Timeout (or a plugin call the "timeout" param), that request's attempts wait at least the gateway timeout plus 10s.

func WithUserAgent 

func WithUserAgent(ua string) Option

WithUserAgent replaces the User-Agent, for tools built on this SDK (the sdo CLI sends "sdo-cli/<version>"). The value must match ^sdo-(sdk-(python|js|go)|cli)/<semver>$; anything else makes New return a *ConfigError. The default is "sdo-sdk-go/<Version>".

Example

ExampleWithUserAgent is for tools built on the SDK: the value must be a Scrape.do client identifier, or New returns a *ConfigError. 

package main

import (
	"errors"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	_, err := sdo.New("your-token", sdo.WithUserAgent("sdo-cli/1.0.0"))
	fmt.Println(err)
	_, err = sdo.New("your-token", sdo.WithUserAgent("my-bot/1.0"))
	var cfgErr *sdo.ConfigError
	fmt.Println(errors.As(err, &cfgErr))
}

Output:
<nil>
true

type Request 

type Request struct {
	// Method is case-insensitive and one of GET, POST, PUT, PATCH, DELETE,
	// HEAD or OPTIONS. Empty means GET, or POST when a body is given.
	Method string
	// JSON, when non-nil, is the body, in the SDK's canonical JSON encoding
	// (sorted keys, no HTML escaping), sent with Content-Type:
	// application/json unless Headers sets one.
	JSON any
	// Data, when non-nil, is the body, sent verbatim with no Content-Type.
	// JSON and Data are mutually exclusive.
	Data []byte
	// Headers are sent with the request and win over ScrapeConfig.Headers.
	// Names are case-insensitive; a value replaces the SDK's own header of
	// the same name (User-Agent, Accept-Encoding). See ScrapeConfig.Headers for
	// what is accepted.
	Headers map[string]string
}

Request is the HTTP side of a scrape: the method, body and headers the gateway sends to the target. A nil *Request is a plain GET.

type ResponseTooLargeError 

type ResponseTooLargeError struct {
	Limit int64
}

ResponseTooLargeError is returned when a response exceeds the client's size ceiling.

Transparent decompression is an attack surface: the scraped target controls the body, so a small compressed payload can inflate without bound. Reading and inflating are both capped and abort as soon as the ceiling is crossed, before the full buffer is materialized.

func (*ResponseTooLargeError) Error 

func (e *ResponseTooLargeError) Error() string

type Result 

type Result struct {
	StatusCode int
	// Header holds every response header, repeated ones included
	// (Header.Values), as the gateway sent them.
	Header http.Header
	// URL is the scraped target (ScrapeConfig.URL); "" for a plugin result.
	URL string
	// contains filtered or unexported fields
}

Result is a successfully delivered scrape (or plugin) response. The body is already transparently decompressed. StatusCode is the *target's* status (a scrape of a 404 page is a successful Result with StatusCode == 404).

func (*Result) Bytes 

func (r *Result) Bytes() []byte

Bytes returns the raw, decompressed payload.

func (*Result) ContentType 

func (r *Result) ContentType() string

ContentType returns the response Content-Type.

func (*Result) Cookies 

func (r *Result) Cookies() string

Cookies returns the raw session cookies the gateway reported, or "" if absent.

func (*Result) Cost 

func (r *Result) Cost() (int, bool)

Cost returns the credits this request cost (Scrape.do-Request-Cost, the authoritative figure) and whether the header was present as a single non-negative integer of at most 2^53-1.

func (*Result) InitialStatusCode 

func (r *Result) InitialStatusCode() (int, bool)

InitialStatusCode returns the target's first-hop status (before any gateway retry/redirect handling) and whether the header was present.

func (*Result) JSON 

func (r *Result) JSON(v any) error

JSON unmarshals the body into v. The body is UTF-8 (a leading BOM ignored, invalid bytes read as U+FFFD); NaN, Infinity and numbers beyond the range of a float64 are rejected. Any failure is a *TransportError, like every other SDK failure (and like the Python and JS SDKs); the decoder's error stays reachable through errors.As.

func (*Result) RemainingCredits 

func (r *Result) RemainingCredits() (int, bool)

RemainingCredits returns the remaining credits and whether the header was present as a single integer.

func (*Result) ResolvedURL 

func (r *Result) ResolvedURL() string

ResolvedURL returns the final URL the gateway resolved to, or "" if absent.

func (*Result) TargetRedirectedLocation 

func (r *Result) TargetRedirectedLocation() string

TargetRedirectedLocation returns the Location the target answered with when redirects were not followed (ScrapeConfig.DisableRedirection), from the Scrape.do-Target-Redirected-Location header, or "" if absent.

func (*Result) TargetURL 

func (r *Result) TargetURL() string

TargetURL returns the originally requested target URL, or "" if absent.

func (*Result) Text 

func (r *Result) Text() string

Text decodes the body per the WHATWG Encoding Standard: a BOM wins and is consumed; then the Content-Type charset (resolved through the WHATWG label table — an unknown label, or one that maps to "replacement", is ignored); JSON media types are UTF-8; HTML (or an unlabelled body) may name its charset in a <meta> within the first 1024 bytes; otherwise UTF-8. Invalid sequences become U+FFFD, one per maximal subpart, so the result is always valid UTF-8 and identical to what the Python and JS SDKs return.

type ScrapeConfig 

type ScrapeConfig struct {
	// URL is the target: an absolute http or https URL. It is sent exactly as
	// given.
	URL string

	// Render enables JS rendering (render=true).
	Render bool
	// RenderMode is the render enum: "true" or "false" (matched
	// case-insensitively, sent lowercase). When set it wins over Render.
	// "false" disables rendering and, like a false Render, is not sent.
	// Render: true together with RenderMode "false" is a *ConfigError.
	RenderMode string
	SuperProxy bool   // wire: super
	Device     string // desktop | mobile | tablet

	Timeout int // 5000–120000 ms
	// RetryTimeout is 5000–55000 ms and cannot be combined with rendering
	// (the gateway rejects render with retryTimeout).
	RetryTimeout int
	DisableRetry bool

	// geo
	GeoCode string
	// RegionalGeoCode routes through a random country of a region (europe,
	// asia, africa, oceania, northamerica, southamerica). It requires
	// SuperProxy and cannot be combined with GeoCode: the gateway rejects
	// both, so they are a *ConfigError here.
	RegionalGeoCode string
	// ZipCode targets a postal code. It requires SuperProxy and GeoCode (the
	// gateway silently ignores it without super, routing through datacenter
	// proxies), so a ZipCode without them is a *ConfigError.
	ZipCode string

	// SessionID keeps the same proxy (IP) across requests: an integer
	// 0-1000000 in decimal digits (at most 7 of them, the gateway's limit).
	// Anything else is a *ConfigError. "" is unset; "0" is a valid id.
	SessionID string
	// SetCookies is the RAW cookie string ("name=value; name2=value2"). The
	// gateway expects it URL-encoded and the SDK encodes it as a query value, so
	// pre-encoding it yourself double-encodes and breaks it.
	SetCookies string
	// CustomHeaders, ExtraHeaders and ForwardHeaders choose how Headers reach
	// the target; at most one may be set (the gateway rejects any two).
	CustomHeaders  bool
	ExtraHeaders   bool
	ForwardHeaders bool
	// Headers are sent with the request; Request.Headers win over them.
	// Whether the gateway forwards them to the target is controlled by
	// CustomHeaders / ExtraHeaders / ForwardHeaders. Names are
	// case-insensitive and must be RFC 9110 tokens; values printable ASCII
	// plus space and tab. Host, Content-Length, Transfer-Encoding and
	// Connection cannot be set. Anything else is a *ConfigError before the
	// request is sent.
	Headers map[string]string

	// render-only behaviour: each requires rendering (Render or RenderMode)
	CustomWait   int    // 0–35000 ms
	WaitSelector string // CSS selector
	WaitUntil    string // domcontentloaded | load | networkidle0 | networkidle2
	// BlockResources is tri-state. nil leaves it unset and sends nothing, so
	// the gateway applies its default of true (resources blocked). A *false
	// is sent explicitly (blockResources=false) to load resources — omitting
	// it would leave them wrongly blocked by that default. A *true is sent as
	// blockResources=true and, being render-only, requires rendering; a *false
	// does not. Use sdo.Bool to take the address of a literal.
	BlockResources  *bool
	PlayWithBrowser []map[string]any
	// ShowWebsocketRequests and ShowFrames add WebSocket messages and iframe
	// content to the JSON response: each requires ReturnJSON as well.
	ShowWebsocketRequests bool
	ShowFrames            bool

	// screenshots (render-only, and each requires ReturnJSON: the image comes
	// back base64-encoded inside the JSON response). At most one of the three
	// may be set, and ParticularScreenshot cannot be combined with
	// PlayWithBrowser (take the screenshot as a ScreenShot action instead).
	Screenshot           bool
	FullScreenshot       bool
	ParticularScreenshot string // CSS selector
	// Height and Width set the browser viewport in px (render-only).
	Height int
	Width  int

	// output / behaviour
	Output string // raw | markdown
	// ReturnJSON returns the network requests and page content as JSON. With
	// rendering it carries the browser's XHR/fetch requests (and screenshots,
	// frames, WebSocket messages when asked for); without rendering the gateway
	// wraps the plain response in the same JSON envelope.
	ReturnJSON bool
	// PureCookies returns the target's original Set-Cookie header instead of
	// the Scrape.do-Cookies header. The body is unaffected.
	PureCookies         bool
	DisableRedirection  bool
	TransparentResponse bool

	// misc
	// ExtraParams is the escape hatch for gateway parameters without a typed
	// field. Keys are sent after the typed params, sorted, with values
	// verbatim. "token" and "url" are reserved (case-insensitively), and a key
	// must be an ASCII letter followed by letters, digits, '_', '.' or '-'.
	ExtraParams map[string]string
}

ScrapeConfig is the single, serializable request spec.

A zero value means "unset" and is omitted from the wire: false, 0, "" and an empty slice or map. Unset values are never validated. Use ExtraParams to send a literal the typed fields cannot express (for example key=false).

Example

ExampleScrapeConfig demonstrates the request spec and error inspection. A gateway-origin failure is an *sdo.APIError whose Kind classifies the cause. 

package main

import (
	"context"
	"errors"
	"fmt"

	sdo "github.com/scrape-do/sdo-go"
)

func main() {
	client, err := sdo.New("your-token")
	if err != nil {
		return
	}
	cfg := &sdo.ScrapeConfig{
		URL:        "https://httpbin.org/json",
		SuperProxy: true,
		Device:     "desktop",
	}
	if _, err := client.Scrape(context.Background(), cfg); err != nil {
		var apiErr *sdo.APIError
		// An exhausted credit balance arrives as an auth error: the gateway has
		// no 402 tier, so the message is what tells the two apart.
		if errors.As(err, &apiErr) && apiErr.IsAuth() {
			fmt.Println("auth failed or out of monthly credits:", apiErr.Message)
		}
	}
}

type Task 

type Task struct {
	TaskID string `json:"TaskID"`
	URL    string `json:"URL"`
	Status string `json:"Status"`
}

Task is one target within a job.

func (*Task) UnmarshalJSON 

func (t *Task) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes a task leniently.

type TaskResult 

type TaskResult struct {
	TaskID string `json:"TaskID"`
	JobID  string `json:"JobID"`
	URL    string `json:"URL"`
	// Status is one of queuing, queued, pending, rotating, success, error,
	// canceled.
	Status     string `json:"Status"`
	StartTime  string `json:"StartTime"`
	EndTime    string `json:"EndTime"`
	ExpiresAt  string `json:"ExpiresAt"`
	UpdateTime string `json:"UpdateTime"`
	// Base64EncodedContent reports that Content is base64 (a binary target);
	// ContentBytes decodes it.
	Base64EncodedContent bool              `json:"Base64EncodedContent"`
	StatusCode           int               `json:"StatusCode"`
	ResponseHeaders      map[string]string `json:"ResponseHeaders"`
	// ScrapeDo is the gateway's metadata map (the "Scrape.do" key).
	ScrapeDo     map[string]string `json:"Scrape.do"`
	Content      string            `json:"Content"`
	ErrorMessage string            `json:"ErrorMessage"`
}

TaskResult is the delivered content of a single task. Absent or mistyped fields are zero. Times are the API's RFC 3339 strings.

func (*TaskResult) ContentBytes 

func (t *TaskResult) ContentBytes() ([]byte, error)

ContentBytes returns the task's content as bytes: base64-decoded when Base64EncodedContent is set, otherwise the UTF-8 bytes of Content. Invalid base64 is a *TransportError.

func (*TaskResult) UnmarshalJSON 

func (t *TaskResult) UnmarshalJSON(data []byte) error

UnmarshalJSON decodes a task result leniently.

type TransportError 

type TransportError struct {
	Message string
	Timeout bool
	Err     error
}

TransportError is returned when no usable HTTP response arrived: DNS, TCP, TLS, timeouts, a body that broke off or could not be decompressed, or a success body that was not the JSON the API promised. Timeout reports whether the cause was a deadline (the client timeout or the context's).

Err holds the cause with credentials already redacted: the raw *url.Error Go produces embeds the request URL — and therefore the token — so wrapping it unchanged would leak through errors.Unwrap and %+v as well as Error().

func (*TransportError) Error 

func (e *TransportError) Error() string

func (*TransportError) Unwrap 

func (e *TransportError) Unwrap() error

type Usage 

type Usage struct {
	// IsActive reports whether the subscription is active.
	IsActive bool
	// ConcurrentRequest is the plan's concurrent-request limit.
	ConcurrentRequest int
	// MaxMonthlyRequest is the plan's monthly request limit.
	MaxMonthlyRequest int
	// RemainingConcurrentRequest is how many more requests may run right now.
	RemainingConcurrentRequest int
	// RemainingMonthlyRequest is how many requests are left this month.
	RemainingMonthlyRequest int
}

Usage is the account's usage snapshot from the /info endpoint. When the subscription is not active, the gateway reports every count as 0.

Source Files

View all Source files

Directories

Path Synopsis
Package html parses a scrape result as an HTML document for CSS selection with goquery.
 Package html parses a scrape result as an HTML document for CSS selection with goquery.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.