crawlsnap

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 7, 2026 License: MIT Imports: 17 Imported by: 0

README

CrawlSnap Go SDK

Official Go client for the CrawlSnap data intelligence platform. Typed responses, typed errors, automatic retries, and per-product API versioning.

Install

go get github.com/crawlsnap/crawlsnap-go

Requires Go 1.23+ (the paginator uses range-over-func).

Authentication

Get an API key from your dashboard and pass it to NewClient, or set CRAWLSNAP_API_KEY and pass an empty string.

client, err := crawlsnap.NewClient("sk-cs-...")   // or NewClient("") with the env var
if err != nil {
    log.Fatal(err)
}

Quick start

package main

import (
    "context"
    "fmt"
    "log"

    crawlsnap "github.com/crawlsnap/crawlsnap-go"
)

func main() {
    client, err := crawlsnap.NewClient("")
    if err != nil {
        log.Fatal(err)
    }

    ip, err := client.VectorSnap.IP(context.Background(), "8.8.8.8")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(ip.GetReputation(), ip.GetAsOwner())
}

Every call returns the unwrapped, typed payload on success and a typed error otherwise — you never inspect the response envelope yourself.

Resources

Call Returns
client.VectorSnap.URL(ctx, q) reputation enrichment for a URL
client.VectorSnap.Hash(ctx, q) reputation enrichment for a file hash
client.VectorSnap.IP(ctx, q) reputation enrichment for an IP
client.VectorSnap.Domain(ctx, q) reputation enrichment for a domain
client.PulseSnap.URL / Hash / IP / Domain(ctx, q) threat-intelligence pulse enrichment
client.SubdoSnap.Scan(ctx, q) one page of subdomains
client.SubdoSnap.ScanIter(ctx, q) every subdomain across all pages
client.SportSnap.Channel(ctx, slug) TV channel metadata + broadcast rights
client.SportSnap.ChannelSchedule(ctx, slug) channel broadcast schedule
client.SportSnap.Match(ctx, id) match details, per-country coverage, result data
client.SportSnap.CountryChannels(ctx, country) TV channels known for a country
client.SportSnap.DailySchedule(ctx, "2026-07-05") daily schedule grouped by competition
client.SportSnap.DailyScheduleTime(ctx, t) same, from a time.Time

SportSnap covers live football (soccer) TV listings. Match status is scheduled, live, or finished and discriminates how much of the payload is populated (score, events, statistics, lineups). Match ids and channel slugs are discovered via DailySchedule and ChannelSchedule entries.

Errors

Failures return typed errors discoverable with errors.As:

ip, err := client.VectorSnap.IP(ctx, "8.8.8.8")
if err != nil {
    var nf *crawlsnap.NotFoundError
    switch {
    case errors.As(err, &nf):
        // no data for this indicator
    default:
        var status *crawlsnap.APIStatusError // any HTTP status error
        if errors.As(err, &status) {
            log.Printf("status=%d request_id=%s", status.StatusCode, status.RequestID)
        }
    }
}
Type Status Meaning
BadRequestError 400 invalid indicator
AuthenticationError 401 missing / invalid key
QuotaExceededError 402 out of credits or monthly quota
SubscriptionInactiveError 403 subscription not active
NotFoundError 404 no data for the indicator
RateLimitError 429 request limit (carries RetryAfter)
ServerError 5xx server / upstream failure
APIConnectionError — transport failure (DNS / connection / TLS)
APITimeoutError — request timed out (also an APIConnectionError)

APIStatusError matches any status error; the CrawlSnapError interface matches anything the SDK returns.

Pagination

ScanIter follows the cursor for you. Range over it with Go 1.23+:

for sub, err := range client.SubdoSnap.ScanIter(ctx, "example.com") {
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(sub["subdomain"])
}

For manual paging, use Scan then ScanWithCursor with the returned Cursor.

API versioning

Each product is versioned independently. A direct call uses that product's stable default version (crawlsnap.DefaultVersion), which never moves on its own when you upgrade the SDK. Pin one product to a specific version without affecting the others:

client.VectorSnap.IP(ctx, "8.8.8.8")        // stable default version
client.VectorSnap.V1().IP(ctx, "8.8.8.8")   // explicitly VectorSnap v1
client.PulseSnap.Domain(ctx, "example.com") // unaffected — PulseSnap default

Configuration

client, err := crawlsnap.NewClient("sk-cs-...",
    crawlsnap.WithBaseURL("https://api.crawlsnap.com"), // or $CRAWLSNAP_BASE_URL
    crawlsnap.WithTimeout(30*time.Second),
    crawlsnap.WithMaxRetries(3),                        // 429 / 5xx / connection
    crawlsnap.WithHTTPClient(myHTTPClient),             // advanced
)

Retries use exponential backoff with jitter and honor the Retry-After header on 429s.

Raw response

Capture the full envelope (status, headers, request id) alongside the typed return:

var raw crawlsnap.RawResponse
ip, err := client.VectorSnap.IP(ctx, "8.8.8.8", crawlsnap.WithRawResponse(&raw))
// raw.StatusCode, raw.RequestID, raw.Headers, raw.IsSuccess, raw.Data

Development

The typed models in models/ are generated from the public OpenAPI contract; the facade is hand-written. Regenerate the models after a contract change:

./scripts/regenerate.sh   # re-bundles the contract and regenerates models/ only
go test ./...

Releasing

Releases are git tags — pushing a vX.Y.Z tag publishes the module to the Go proxy (no separate registry upload). Use the helper:

./scripts/release.sh 0.2.0   # bumps version.go, runs checks, commits, tags, pushes, gh release

Tags are immutable: to fix a bad release, cut a new patch. A v2+ release additionally requires the /v2 module-path suffix in go.mod.

License

MIT

Documentation

Overview

Package crawlsnap is the official Go client for the CrawlSnap data intelligence platform. 

client, err := crawlsnap.NewClient("sk-cs-...")   // or set CRAWLSNAP_API_KEY
if err != nil { log.Fatal(err) }

ip, err := client.VectorSnap.IP(ctx, "8.8.8.8")
if err != nil { log.Fatal(err) }
fmt.Println(ip.GetReputation(), ip.GetAsOwner())

Every call returns the unwrapped, typed payload on success and a typed error otherwise (see errors.go) — the caller never inspects the response envelope.

Package crawlsnap — error types.

Hierarchy (mirrors the other CrawlSnap SDKs): 

CrawlSnapError                      marker interface for everything the SDK returns
├── ConfigError                     misconfiguration (e.g. missing API key)
├── DecodeError                     a 2xx body could not be decoded
├── APIConnectionError              transport failed (no HTTP response)
│   └── APITimeoutError             the request timed out
└── APIStatusError                  the API returned a non-success status
    ├── BadRequestError             400  invalid input
    ├── AuthenticationError         401  missing / invalid API key
    ├── QuotaExceededError          402  out of credits / monthly quota
    ├── SubscriptionInactiveError   403  subscription not active
    ├── NotFoundError               404  no data for the indicator
    ├── RateLimitError              429  request limit exceeded
    └── ServerError                 5xx  server / upstream failure

Discover the concrete type with errors.As: 

if _, err := client.VectorSnap.IP(ctx, "8.8.8.8"); err != nil {
    var nf *crawlsnap.NotFoundError
    if errors.As(err, &nf) { /* no data for this indicator */ }

    var status *crawlsnap.APIStatusError   // catches any HTTP status error
    if errors.As(err, &status) { log.Println(status.StatusCode, status.RequestID) }

    var any crawlsnap.CrawlSnapError       // catches anything the SDK returns
    if errors.As(err, &any) { /* ... */ }
}

Index

Constants

View Source 
const (
	DefaultBaseURL    = "https://api.crawlsnap.com"
	DefaultTimeout    = 30 * time.Second
	DefaultMaxRetries = 2
)

Defaults for client configuration.

View Source 
const DefaultVersion = "v1"

DefaultVersion is the stable, per-release default API version used by unpinned calls. It does not move on its own: upgrading the SDK never silently retargets your calls at a newer API version. Promoting a product's default to a new version is a deliberate, changelogged release change.

Pin a single product to a specific version with its version accessor (e.g. VectorSnap.V1()) — this affects only that product, leaving the others on their own defaults.

View Source 
const Version = "0.4.0"

Version is the SDK release version, sent in the User-Agent header.

Variables

View Source 
var ErrNoAPIKey = &ConfigError{Message: "no API key provided: pass it to NewClient or set CRAWLSNAP_API_KEY"}

ErrNoAPIKey is returned by NewClient when no API key is supplied and CRAWLSNAP_API_KEY is unset.

Functions

This section is empty.

Types

type APIConnectionError 

type APIConnectionError struct {
	Message string
	Err     error
}

APIConnectionError is returned when the request never reached the API (DNS, connection refused, TLS, ...). Unwrap to inspect the underlying cause.

func (*APIConnectionError) As 

func (e *APIConnectionError) As(target any) bool

As lets errors.As extract the embedded *APIConnectionError from *APITimeoutError.

func (*APIConnectionError) Error 

func (e *APIConnectionError) Error() string

func (*APIConnectionError) Unwrap 

func (e *APIConnectionError) Unwrap() error

type APIStatusError 

type APIStatusError struct {
	// StatusCode is the effective HTTP status (the body response_code when it
	// disagrees with and supersedes the transport status).
	StatusCode int
	// Message is the human-readable error from the response envelope.
	Message string
	// RequestID is the x-request-id response header, useful for support.
	RequestID string
	// Body is the raw response body (the BaseResponse envelope), when available.
	Body []byte
}

APIStatusError is the base error for any non-success HTTP status. The seven status-specific types embed it; errors.As(err, &(*APIStatusError)(nil)) on any of them yields this base value.

func (*APIStatusError) As 

func (e *APIStatusError) As(target any) bool

As lets errors.As extract the embedded *APIStatusError from any of the status-specific subtypes (they embed APIStatusError and promote this method).

func (*APIStatusError) Error 

func (e *APIStatusError) Error() string

type APITimeoutError 

type APITimeoutError struct {
	*APIConnectionError
}

APITimeoutError is returned when the request timed out. It embeds *APIConnectionError, so it is also matched by errors.As as a connection error.

type AuthenticationError 

type AuthenticationError struct{ APIStatusError }

AuthenticationError is returned for HTTP 401 (missing / invalid API key).

type BadRequestError 

type BadRequestError struct{ APIStatusError }

BadRequestError is returned for HTTP 400 (invalid indicator).

type Client 

type Client struct {

	// Resource groups.
	VectorSnap *VectorSnap
	PulseSnap  *PulseSnap
	SubdoSnap  *SubdoSnap
	SportSnap  *SportSnap
	// contains filtered or unexported fields
}

Client is the CrawlSnap API client. It is safe for concurrent use.

func NewClient 

func NewClient(apiKey string, opts ...Option) (*Client, error)

NewClient creates a client. The API key falls back to $CRAWLSNAP_API_KEY when empty; if neither is set, ErrNoAPIKey is returned.

type ConfigError 

type ConfigError struct{ Message string }

ConfigError is returned for SDK misconfiguration detected before any request is made (e.g. a missing API key). It is a config-time error, distinct from the transport-level APIConnectionError.

func (*ConfigError) Error 

func (e *ConfigError) Error() string

type CrawlSnapError 

type CrawlSnapError interface {
	error
	// contains filtered or unexported methods
}

CrawlSnapError is implemented by every error the SDK returns. Use it with errors.As to catch anything originating from this package.

type DecodeError 

type DecodeError struct {
	Message string
	Err     error
}

DecodeError is returned when a successful (2xx) response body could not be decoded into the expected typed payload — an application-layer failure, not a transport one.

func (*DecodeError) Error 

func (e *DecodeError) Error() string

func (*DecodeError) Unwrap 

func (e *DecodeError) Unwrap() error

type IocDomainScanData 

type IocDomainScanData = models.IocDomainScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type IocHashScanData 

type IocHashScanData = models.IocHashScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type IocIPScanData 

type IocIPScanData = models.IocIpScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type IocURLScanData 

type IocURLScanData = models.IocUrlScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type NotFoundError 

type NotFoundError struct{ APIStatusError }

NotFoundError is returned for HTTP 404 (no data for the supplied indicator).

type Option 

type Option func(*clientConfig)

Option configures a Client in NewClient.

func WithBaseURL 

func WithBaseURL(baseURL string) Option

WithBaseURL overrides the API host (default https://api.crawlsnap.com, or $CRAWLSNAP_BASE_URL).

func WithHTTPClient 

func WithHTTPClient(hc *http.Client) Option

WithHTTPClient supplies your own *http.Client (advanced). Takes precedence over WithTimeout and WithTransport.

func WithMaxRetries 

func WithMaxRetries(n int) Option

WithMaxRetries sets retries for 429 / 5xx / connection errors (default 2).

func WithTimeout 

func WithTimeout(d time.Duration) Option

WithTimeout sets the per-request timeout (default 30s). Ignored when WithHTTPClient is supplied.

func WithTransport 

func WithTransport(rt http.RoundTripper) Option

WithTransport supplies a custom http.RoundTripper, e.g. a mock for testing.

type PulseDomainScanData 

type PulseDomainScanData = models.PulseDomainScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type PulseHashScanData 

type PulseHashScanData = models.PulseHashScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type PulseIPScanData 

type PulseIPScanData = models.PulseIpScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type PulseSnap 

type PulseSnap struct {
	// contains filtered or unexported fields
}

PulseSnap is the threat-intelligence pulse enrichment resource. A direct call uses the stable default API version; pin explicitly with V1().

func (*PulseSnap) Domain 

func (r *PulseSnap) Domain(ctx context.Context, query string, opts ...RequestOption) (models.PulseDomainScanData, error)

Domain returns pulse enrichment for a domain.

func (*PulseSnap) Hash 

func (r *PulseSnap) Hash(ctx context.Context, query string, opts ...RequestOption) (models.PulseHashScanData, error)

Hash returns pulse enrichment for a file hash.

func (*PulseSnap) IP 

func (r *PulseSnap) IP(ctx context.Context, query string, opts ...RequestOption) (models.PulseIpScanData, error)

IP returns pulse enrichment for an IP address.

func (*PulseSnap) URL 

func (r *PulseSnap) URL(ctx context.Context, query string, opts ...RequestOption) (models.PulseUrlScanData, error)

URL returns pulse enrichment for a URL.

func (*PulseSnap) V1 

func (r *PulseSnap) V1() *PulseSnap

V1 pins PulseSnap to API version v1.

type PulseURLScanData 

type PulseURLScanData = models.PulseUrlScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type QuotaExceededError 

type QuotaExceededError struct{ APIStatusError }

QuotaExceededError is returned for HTTP 402 (out of credits or monthly quota).

type RateLimitError 

type RateLimitError struct {
	APIStatusError
	// RetryAfter is the Retry-After header value in seconds, if the server sent one.
	RetryAfter float64
}

RateLimitError is returned for HTTP 429 (request limit exceeded).

type RawResponse 

type RawResponse struct {
	StatusCode   int
	IsSuccess    bool
	Data         any
	Message      string
	ResponseCode *int
	RequestID    string
	Headers      http.Header
}

RawResponse is the full envelope, populated when a call is made with WithRawResponse. Data holds the parsed typed payload.

type RequestOption 

type RequestOption func(*requestConfig)

RequestOption customizes a single request.

func WithRawResponse 

func WithRawResponse(dst *RawResponse) RequestOption

WithRawResponse populates dst with the full response envelope (status, headers, request id, is_success, message) alongside the normal typed return.

type ServerError 

type ServerError struct{ APIStatusError }

ServerError is returned for HTTP 5xx (server or upstream failure).

type SportSnap added in v0.3.0 

type SportSnap struct {
	// contains filtered or unexported fields
}

SportSnap is the live football TV listings resource: channel metadata and broadcast schedules, match details with per-country coverage (score, events, statistics, and lineups for finished matches), country channel directories, and daily schedules. A direct call uses the stable default API version; pin explicitly with V1().

func (*SportSnap) Channel added in v0.3.0 

func (r *SportSnap) Channel(ctx context.Context, slug string, opts ...RequestOption) (models.ChannelData, error)

Channel returns TV channel metadata and competition broadcast rights for a channel slug (e.g. "bein-connect-turkey").

func (*SportSnap) ChannelSchedule added in v0.3.0 

func (r *SportSnap) ChannelSchedule(ctx context.Context, slug string, opts ...RequestOption) (models.ChannelScheduleData, error)

ChannelSchedule returns the channel's upcoming broadcast listings (day, kickoff, match, competition). Entries may be empty when the channel has no upcoming listings — that is a valid result, not an error.

func (*SportSnap) CountryChannels added in v0.3.0 

func (r *SportSnap) CountryChannels(ctx context.Context, country string, opts ...RequestOption) (models.CountryChannelsData, error)

CountryChannels returns the TV channels known for a country (slugified name, e.g. "turkey", "united-states").

func (*SportSnap) DailySchedule added in v0.3.0 

func (r *SportSnap) DailySchedule(ctx context.Context, date string, opts ...RequestOption) (models.DailyScheduleData, error)

DailySchedule returns the full broadcast schedule for a date (YYYY-MM-DD), grouped by competition, with kickoff times normalized to UTC. Use DailyScheduleTime to pass a time.Time instead.

func (*SportSnap) DailyScheduleTime added in v0.3.0 

func (r *SportSnap) DailyScheduleTime(ctx context.Context, date time.Time, opts ...RequestOption) (models.DailyScheduleData, error)

DailyScheduleTime is DailySchedule with the date taken from a time.Time (formatted as YYYY-MM-DD in the time's location).

func (*SportSnap) Match added in v0.3.0 

func (r *SportSnap) Match(ctx context.Context, id int64, opts ...RequestOption) (models.MatchData, error)

Match returns match details keyed by the numeric match id. Status discriminates the payload: "scheduled" carries meta + broadcasts only; "live" and "finished" additionally carry score, events, statistics, and lineups as available. Match ids are discovered via DailySchedule and ChannelSchedule entries.

func (*SportSnap) V1 added in v0.3.0 

func (r *SportSnap) V1() *SportSnap

V1 pins SportSnap to API version v1.

type SubdoSnap 

type SubdoSnap struct {
	// contains filtered or unexported fields
}

SubdoSnap is the subdomain enumeration resource. A direct call uses the stable default API version; pin explicitly with V1().

func (*SubdoSnap) Scan 

func (r *SubdoSnap) Scan(ctx context.Context, query string, opts ...RequestOption) (models.SubdoSnapScanData, error)

Scan fetches the first page of subdomains for a domain. Use the returned Cursor with ScanWithCursor to page manually, or ScanIter to stream every subdomain automatically.

func (*SubdoSnap) ScanIter 

func (r *SubdoSnap) ScanIter(ctx context.Context, query string) iter.Seq2[map[string]any, error]

ScanIter streams every subdomain across all pages, following the cursor for you. Range over it with Go 1.23+ range-over-func: 

for sub, err := range client.SubdoSnap.ScanIter(ctx, "example.com") {
    if err != nil { return err }
    fmt.Println(sub["subdomain"])
}

Iteration stops at the first error (yielded once with a nil subdomain) or when the cursor is exhausted.

func (*SubdoSnap) ScanWithCursor 

func (r *SubdoSnap) ScanWithCursor(ctx context.Context, query, cursor string, opts ...RequestOption) (models.SubdoSnapScanData, error)

ScanWithCursor fetches one page of subdomains starting at the given cursor.

func (*SubdoSnap) V1 

func (r *SubdoSnap) V1() *SubdoSnap

V1 pins SubdoSnap to API version v1.

type SubdoSnapScanData 

type SubdoSnapScanData = models.SubdoSnapScanData

Convenience aliases for the typed response payloads, so callers can refer to them without importing the models package directly. The canonical definitions (generated from the OpenAPI contract) live in package models.

type SubscriptionInactiveError 

type SubscriptionInactiveError struct{ APIStatusError }

SubscriptionInactiveError is returned for HTTP 403 (subscription not active).

type VectorSnap 

type VectorSnap struct {
	// contains filtered or unexported fields
}

VectorSnap is the IoC reputation enrichment resource. A direct call uses the stable default API version; pin explicitly with V1().

func (*VectorSnap) Domain 

func (r *VectorSnap) Domain(ctx context.Context, query string, opts ...RequestOption) (models.IocDomainScanData, error)

Domain returns reputation enrichment for a domain.

func (*VectorSnap) Hash 

func (r *VectorSnap) Hash(ctx context.Context, query string, opts ...RequestOption) (models.IocHashScanData, error)

Hash returns reputation enrichment for a file hash.

func (*VectorSnap) IP 

func (r *VectorSnap) IP(ctx context.Context, query string, opts ...RequestOption) (models.IocIpScanData, error)

IP returns reputation enrichment for an IP address.

func (*VectorSnap) URL 

func (r *VectorSnap) URL(ctx context.Context, query string, opts ...RequestOption) (models.IocUrlScanData, error)

URL returns reputation enrichment for a URL.

func (*VectorSnap) V1 

func (r *VectorSnap) V1() *VectorSnap

V1 pins VectorSnap to API version v1.

Source Files

View all Source files

Directories

Path Synopsis
examples
quickstart command
Command quickstart demonstrates the CrawlSnap Go SDK.
 Command quickstart demonstrates the CrawlSnap Go SDK.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.