Help us shape the future of Go! Take the 2026 Go Developer Survey

dawgsdk

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 3, 2026 License: MIT Imports: 11 Imported by: 0

README

dawg-sdk-go

Go client for the dawg API. Two clients in one package:

  • Scraper — REST scraping with markdown/text/html extraction, batch and crawl jobs.
  • Baas — cloud headless browser pool. Allocates a Chromium instance and returns a CDP WebSocket URL ready for playwright-go, chromedp, rod, or any other CDP client.

Zero external dependencies. Go 1.22+.

Install

go get github.com/dawgswarm/dawg-sdk-go@latest

Quickstart — Scraper

package main

import (
    "context"
    "fmt"

    dawgsdk "github.com/dawgswarm/dawg-sdk-go"
)

func main() {
    s := dawgsdk.NewScraper("YOUR_API_KEY")
    res, _ := s.Scrape(context.Background(), dawgsdk.ScrapeRequest{
        URL:         "https://example.com",
        Format:      dawgsdk.FormatMarkdown,
        MainContent: true,
    })
    fmt.Println(res.Content)
}

Quickstart — Baas

b := dawgsdk.NewBaas("YOUR_API_KEY")
sess, _ := b.Create(ctx, dawgsdk.CreateRequest{})
defer func() {
    rctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    _ = b.Release(rctx)
}()

// sess.WSURL is a ready-to-use CDP WebSocket URL.
// Feed it to your CDP client of choice:
//   - playwright-go: pw.Chromium.ConnectOverCDP(sess.WSURL)
//   - chromedp:      chromedp.NewRemoteAllocator(ctx, withPort(sess.WSURL), chromedp.NoModifyURL)
//   - rod, raw devtools, anything that speaks CDP

defer b.Release plays the role of Python's with Baas() as ws_url. A fresh context.Background() for the release ensures it survives parent cancellation.

Full runnable integrations: examples/baas_with_playwright and examples/baas_with_chromedp. The chromedp one is slightly more involved — chromedp expects an explicit host:port and needs chromedp.NoModifyURL so its forceIP rewrite doesn't break TLS SNI. Playwright accepts the SDK's URL as-is.

Scraper API

Method Purpose
Scrape(ctx, ScrapeRequest) Single URL → *ScrapeResult
Crawl(ctx, CrawlRequest) Start crawl → *Job (poll with Wait)
Batch(ctx, BatchRequest) Start batch → *Job
GetJob(ctx, jobID) Fetch existing job state
CancelJob(ctx, jobID) Cancel without fetching state

Constructor options: ScraperBaseURL, ScraperHTTPClient, ScraperTimeout.

ScrapeRequest fields:

Field Type Default Notes
URL string required
Format Format "markdown" also FormatText, FormatHTML
Headers map[string]string omitted custom request headers
TimeoutMS int 30000 per-page fetch timeout
MainContent bool false strip nav/footer/ads
IncludeLinks bool false populate result.Links
Render RenderMode server-pick RenderAuto / RenderHTTP / RenderBrowser

Job.Wait(ctx, pollInterval) blocks until status leaves JobRunning or ctx is done:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
if err := job.Wait(ctx, 2*time.Second); err != nil { /* ... */ }

Baas API

Method Purpose
Create(ctx, CreateRequest) Allocate browser + poll until ready → *Session
Release(ctx) Return active browser to pool
Session() Snapshot of the current session, or nil

Constructor options: BaasBaseURL, BaasHTTPClient, BaasReadyTimeout, BaasPollInterval.

CreateRequest carries optional Proxy and Geo:

proxy, _ := dawgsdk.ParseProxyURL("socks5://user:pass@host:1080")
geo := dawgsdk.GeoSlug("moskva") // or dawgsdk.GeoCoords(55.75, 37.61)
sess, _ := b.Create(ctx, dawgsdk.CreateRequest{Proxy: &proxy, Geo: &geo})

The returned sess.WSURL is a ws://-or-wss:// URL with the appropriate auth suffix (?token=... when the server issued an ephemeral token, else ?apiKey=...).

Errors

All errors typed and errors.As-friendly:

_, err := s.Scrape(ctx, req)

var rl *dawgsdk.RateLimitError
if errors.As(err, &rl) {
    time.Sleep(rl.RetryAfter)
}

var auth *dawgsdk.AuthError
if errors.As(err, &auth) {
    log.Fatal("bad API key")
}

var api *dawgsdk.APIError
if errors.As(err, &api) {
    fmt.Printf("API error %d: %s", api.StatusCode, api.Message)
}

Types:

Error When
*dawgsdk.APIError base type; any HTTP error or wrapped network error
*dawgsdk.AuthError HTTP 401
*dawgsdk.RateLimitError HTTP 429; RetryAfter from detail.retry_after_seconds (default 60s)
*dawgsdk.BrowserNotReadyError Baas.Create exceeded its ready timeout
*dawgsdk.JobTimeoutError Job.Wait exhausted its context

Concurrency

*Scraper is safe for concurrent use by multiple goroutines.

*Baas tracks one active session at a time — to drive multiple browsers in parallel, construct multiple *Baas instances (cheap; share a *http.Client via BaasHTTPClient for connection-pool reuse).

Examples

Runnable examples live in examples/. Each subdir is its own go run-able program:

  • scrape_simple/, scrape_with_options/, batch/, crawl/
  • baas_simple/, baas_defer/, baas_with_proxy/, baas_with_geo/
  • baas_with_chromedp/, baas_with_playwright/
  • error_handling/

All require DAWG_API_KEY in the environment. The Playwright example additionally needs the driver: go run github.com/playwright-community/playwright-go/cmd/playwright install chromium.

License

MIT

Documentation

Overview

Package dawgsdk is the Go client for the dawg API.

Two clients live in this package, mirroring the Python SDK layout:

  • Scraper — REST scraping with content extraction (markdown/text/html), batch and crawl jobs.
  • Baas — cloud headless browser pool; returns a CDP WebSocket URL for use with chromedp, Playwright, Puppeteer, or any other CDP client.

Both clients authenticate via the X-API-Key header. Default backend is https://dawgswarm.ru; override via options.

Quickstart: 

s := dawgsdk.NewScraper("YOUR_KEY")
res, err := s.Scrape(ctx, dawgsdk.ScrapeRequest{URL: "https://example.com"})

b := dawgsdk.NewBaas("YOUR_KEY")
sess, err := b.Create(ctx, dawgsdk.CreateRequest{})
defer b.Release(context.Background())
// use sess.WSURL with your CDP client

All errors are typed and switchable with errors.As — see APIError, AuthError, RateLimitError, BrowserNotReadyError, JobTimeoutError.

Index

Constants

View Source 
const DefaultBaseURL = "https://dawgswarm.ru"

DefaultBaseURL is the production backend.

View Source 
const Version = "0.2.0"

Variables

This section is empty.

Functions

This section is empty.

Types

type APIError 

type APIError struct {
	StatusCode int
	Message    string
	Err        error
}

APIError is the base error type returned by every SDK call that hits the backend. It carries the HTTP status code (zero for non-HTTP failures, e.g. connection errors), a short human-readable message, and the optional underlying error (network error, JSON decode error, etc).

Specialised errors below embed *APIError so callers can do: 

var apiErr *dawgsdk.APIError
if errors.As(err, &apiErr) && apiErr.StatusCode == 502 {
    // ...
}

or match a more specific type when the API contract distinguishes it: 

var rl *dawgsdk.RateLimitError
if errors.As(err, &rl) { ... }

func (*APIError) Error 

func (e *APIError) Error() string

func (*APIError) Unwrap 

func (e *APIError) Unwrap() error

type AuthError 

type AuthError struct {
	*APIError
}

AuthError is returned on HTTP 401 — the API key was missing or rejected.

func (*AuthError) Error 

func (e *AuthError) Error() string

type Baas 

type Baas struct {
	// contains filtered or unexported fields
}

Baas allocates and releases cloud browser instances. A single Baas tracks at most one active session — create multiple Baas instances to run browsers in parallel.

Methods that touch the active session (Create, Release, Session) are guarded by an internal mutex, so concurrent use does not race. Callers should still serialise their own logic: e.g. don't call Create and Release concurrently — the result is unpredictable.

func NewBaas 

func NewBaas(apiKey string, opts ...BaasOption) *Baas

NewBaas constructs a Baas. apiKey is the X-API-Key header value.

func (*Baas) Close 

func (c *Baas) Close() error

Close is currently a no-op. Reserved for future state teardown.

func (*Baas) Create 

func (c *Baas) Create(ctx context.Context, req CreateRequest) (*Session, error)

Create allocates a browser and blocks until it reports a usable CDP endpoint. Returns a Session whose WSURL is ready to hand to a CDP client.

Polling honours the parent ctx (cancellation aborts immediately) as well as the client's ReadyTimeout/PollInterval. On timeout returns *BrowserNotReadyError.

Calling Create when a session is already active overwrites it; the previous session is not released automatically. Call Release first.

func (*Baas) ListSessions added in v0.2.0 

func (c *Baas) ListSessions(ctx context.Context) ([]SessionInfo, error)

ListSessions returns the API key owner's live browser sessions (ACTIVE and IDLE), newest first. 

sessions, _ := c.ListSessions(ctx)
for _, s := range sessions {
    _ = c.ReleaseSession(ctx, s.ID)
}

func (*Baas) Release 

func (c *Baas) Release(ctx context.Context) error

Release returns the active browser to the pool. No-op (returns nil) if there is no active session. After a successful Release, Session returns nil.

func (*Baas) ReleaseSession added in v0.2.0 

func (c *Baas) ReleaseSession(ctx context.Context, sessionID string) error

ReleaseSession releases a browser by session id, whether or not this client created it. Pair it with ListSessions to reclaim sessions whose id was lost (crashed worker, restarted process) instead of waiting out the farm's inactivity timeout. Clears the active session if the id matches it.

func (*Baas) Session 

func (c *Baas) Session() *Session

Session returns a copy of the most recently created session, or nil if none is active.

type BaasOption 

type BaasOption func(*Baas)

BaasOption configures a Baas at construction time.

func BaasBaseURL 

func BaasBaseURL(u string) BaasOption

BaasBaseURL overrides the default backend (DefaultBaseURL). The WS URL returned by Create derives its ws://-or-wss:// scheme from this.

func BaasHTTPClient 

func BaasHTTPClient(h *http.Client) BaasOption

BaasHTTPClient sets a custom *http.Client.

func BaasPollInterval 

func BaasPollInterval(d time.Duration) BaasOption

BaasPollInterval overrides the gap between warmup polls. Default 2s.

func BaasReadyTimeout 

func BaasReadyTimeout(d time.Duration) BaasOption

BaasReadyTimeout overrides how long Create polls for a usable CDP endpoint before returning *BrowserNotReadyError. Default 60s.

type BatchRequest 

type BatchRequest struct {
	URLs        []string
	Format      Format
	Concurrency int
	TimeoutMS   int
	MainContent bool
}

BatchRequest is the input to Scraper.Batch.

type BrowserNotReadyError 

type BrowserNotReadyError struct {
	*APIError
	Waited time.Duration
}

BrowserNotReadyError is returned by the BaaS client when a newly-allocated browser doesn't report a usable CDP endpoint within the configured ready timeout. Waited records how long the polling loop ran before giving up.

func (*BrowserNotReadyError) Error 

func (e *BrowserNotReadyError) Error() string

type CrawlRequest 

type CrawlRequest struct {
	URL             string
	Format          Format
	MaxDepth        int
	MaxPages        int
	Concurrency     int
	IncludePatterns []string
	ExcludePatterns []string
	TimeoutMS       int
	MainContent     bool
}

CrawlRequest is the input to Scraper.Crawl.

type CreateRequest 

type CreateRequest struct {
	Proxy *Proxy
	Geo   *Geo
}

CreateRequest is the input to Baas.Create. All fields are optional.

type Format 

type Format string

Format selects the output format for scraped content. 

const (
	FormatMarkdown Format = "markdown"
	FormatText     Format = "text"
	FormatHTML     Format = "html"
)

type Geo 

type Geo struct {
	Slug string
	Lat  float64
	Lon  float64
	// contains filtered or unexported fields
}

Geo pins the browser's apparent location. Use GeoSlug or GeoCoords to construct.

func GeoCoords 

func GeoCoords(lat, lon float64) Geo

GeoCoords returns a Geo with explicit coordinates.

func GeoSlug 

func GeoSlug(slug string) Geo

GeoSlug returns a Geo selecting a server-known city by slug, e.g. "moskva".

type Job 

type Job struct {
	JobID       string
	Type        string
	Status      JobStatus
	Progress    Progress
	Pages       []JobPage
	Errors      []string
	CreatedAt   time.Time
	CompletedAt *time.Time
	ElapsedMS   int
	// contains filtered or unexported fields
}

Job is a handle to a crawl or batch job. Construct via Scraper.Crawl, Scraper.Batch, or Scraper.GetJob.

All fields reflect the most recent server snapshot. Call Refresh to update in-place, or Wait to block until Status transitions out of JobRunning.

func (*Job) Cancel 

func (j *Job) Cancel(ctx context.Context) error

Cancel cancels the job server-side and refreshes local state.

func (*Job) Refresh 

func (j *Job) Refresh(ctx context.Context) error

Refresh polls the server once and updates the Job in place.

func (*Job) Wait 

func (j *Job) Wait(ctx context.Context, pollInterval time.Duration) error

Wait blocks until Status leaves JobRunning, the context is cancelled, or its deadline expires. pollInterval defaults to 2s when zero. On ctx cancellation/timeout returns *JobTimeoutError wrapping ctx.Err().

type JobPage 

type JobPage struct {
	URL        string         `json:"url"`
	StatusCode int            `json:"status_code"`
	Content    string         `json:"content"`
	Metadata   map[string]any `json:"metadata"`
	Error      string         `json:"error,omitempty"`
}

JobPage is one URL's result inside a Job.

type JobStatus 

type JobStatus string

JobStatus is the lifecycle state of a Crawl or Batch job. 

const (
	JobRunning   JobStatus = "running"
	JobCompleted JobStatus = "completed"
	JobFailed    JobStatus = "failed"
	JobCancelled JobStatus = "cancelled"
)

type JobTimeoutError 

type JobTimeoutError struct {
	*APIError
	JobID  string
	Waited time.Duration
}

JobTimeoutError is returned by Job.Wait when the supplied context expires before the server reports a terminal job status.

func (*JobTimeoutError) Error 

func (e *JobTimeoutError) Error() string

type Progress 

type Progress struct {
	Total     int `json:"total"`
	Completed int `json:"completed"`
}

Progress is the running-progress counter inside a Job.

type Proxy 

type Proxy struct {
	Server   string // e.g. "socks5://host:port" or "http://host:port"
	Username string // optional
	Password string // optional
}

Proxy describes an upstream proxy the cloud browser should route through. Either build directly or use ParseProxyURL.

func ParseProxyURL 

func ParseProxyURL(raw string) (Proxy, error)

ParseProxyURL parses a proxy URL of the form "scheme://[user:pass@]host:port" into a Proxy. Auth is optional; when present it is stripped from Server and moved into Username/Password. 

ParseProxyURL("socks5://u:p@1.2.3.4:1080") →
    Proxy{Server: "socks5://1.2.3.4:1080", Username: "u", Password: "p"}

ParseProxyURL("http://1.2.3.4:8080") →
    Proxy{Server: "http://1.2.3.4:8080"}

type RateLimitError 

type RateLimitError struct {
	*APIError
	RetryAfter time.Duration
}

RateLimitError is returned on HTTP 429. RetryAfter carries the server-hinted wait duration (parsed from response body field detail.retry_after_seconds, defaulting to 60s when absent).

func (*RateLimitError) Error 

func (e *RateLimitError) Error() string

type RenderMode 

type RenderMode string

RenderMode controls how a page is fetched and rendered.

  • RenderAuto — server auto-detects, falls back to a browser for SPAs
  • RenderHTTP — HTTP-only, no browser fallback (cheaper)
  • RenderBrowser — force browser rendering (more expensive)

The zero value is omitted from the request, which lets the server pick its default (currently equivalent to RenderAuto). 

const (
	RenderAuto    RenderMode = "auto"
	RenderHTTP    RenderMode = "http"
	RenderBrowser RenderMode = "browser"
)

type ScrapeRequest 

type ScrapeRequest struct {
	URL          string
	Format       Format
	Headers      map[string]string
	TimeoutMS    int
	MainContent  bool
	IncludeLinks bool
	Render       RenderMode
}

ScrapeRequest is the input to Scraper.Scrape. Only URL is required.

Zero-value handling:

  • Format == "" → server default (markdown)
  • TimeoutMS == 0 → server default (30000)
  • Render == "" → field omitted, server picks
  • Headers == nil → field omitted
  • MainContent / IncludeLinks: sent as-is (false is a valid intentional value, matching Python's default of False)

type ScrapeResult 

type ScrapeResult struct {
	Success    bool           `json:"success"`
	URL        string         `json:"url"`
	FinalURL   string         `json:"final_url"`
	StatusCode int            `json:"status_code"`
	Content    string         `json:"content"`
	Metadata   map[string]any `json:"metadata"`
	Links      []string       `json:"links,omitempty"`
	Error      string         `json:"error,omitempty"`
	ElapsedMS  int            `json:"elapsed_ms"`
}

ScrapeResult is the output of Scraper.Scrape.

type Scraper 

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper is a client for the dawg HTTP scraping API.

Construct with NewScraper; safe for concurrent use by multiple goroutines.

func NewScraper 

func NewScraper(apiKey string, opts ...ScraperOption) *Scraper

NewScraper constructs a Scraper. apiKey is required (the X-API-Key header).

func (*Scraper) Batch 

func (c *Scraper) Batch(ctx context.Context, r BatchRequest) (*Job, error)

Batch starts a batch scrape over r.URLs.

func (*Scraper) CancelJob 

func (c *Scraper) CancelJob(ctx context.Context, jobID string) error

CancelJob cancels a running job. Returns nil on success even if the job had already reached a terminal state.

func (*Scraper) Close 

func (c *Scraper) Close() error

Close is currently a no-op. Reserved for future state teardown.

func (*Scraper) Crawl 

func (c *Scraper) Crawl(ctx context.Context, r CrawlRequest) (*Job, error)

Crawl starts a crawl job seeded from r.URL. The returned Job is ready to be polled via Refresh/Wait/Cancel.

func (*Scraper) GetJob 

func (c *Scraper) GetJob(ctx context.Context, jobID string) (*Job, error)

GetJob fetches the current state of a job by ID.

func (*Scraper) Scrape 

func (c *Scraper) Scrape(ctx context.Context, r ScrapeRequest) (*ScrapeResult, error)

Scrape fetches a single URL and returns the extracted content.

type ScraperOption 

type ScraperOption func(*Scraper)

ScraperOption configures a Scraper at construction time.

func ScraperBaseURL 

func ScraperBaseURL(u string) ScraperOption

ScraperBaseURL overrides the default backend (DefaultBaseURL).

func ScraperHTTPClient 

func ScraperHTTPClient(h *http.Client) ScraperOption

ScraperHTTPClient sets a custom *http.Client. When set, ScraperTimeout is ignored — configure the client's own Timeout instead.

func ScraperTimeout 

func ScraperTimeout(d time.Duration) ScraperOption

ScraperTimeout sets the per-request HTTP timeout. Default 60s.

type Session 

type Session struct {
	BrowserID      string
	SessionID      string
	EphemeralToken string
	WSURL          string
}

Session holds the identifiers of an active browser plus the ready-to-use CDP WebSocket URL.

type SessionInfo added in v0.2.0 

type SessionInfo struct {
	ID                string     `json:"id"`
	BrowserInstanceID string     `json:"browserInstanceId"`
	Status            string     `json:"status"`
	StartedAt         time.Time  `json:"startedAt"`
	LastActivityAt    time.Time  `json:"lastActivityAt"`
	TerminatedAt      *time.Time `json:"terminatedAt"`
	TotalMinutes      float64    `json:"totalMinutes"`
}

SessionInfo describes one server-side browser session. This is account state, not client state: sessions started by other processes appear here too.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.