kloakd

package module
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 2, 2026 License: MIT Imports: 14 Imported by: 0

README

kloakd-go

Official Go SDK for the KLOAKD API — the anti-bot intelligence platform.

Requirements

  • Go 1.18+
  • KLOAKD API key and organization ID

Installation

go get github.com/kloakd-dev/kloakd-go

Quickstart

package main

import (
    "context"
    "fmt"
    "log"

    kloakd "github.com/kloakd-dev/kloakd-go"
)

func main() {
    client, err := kloakd.New(kloakd.Config{
        APIKey:         "sk-live-...",
        OrganizationID: "your-org-uuid",
    })
    if err != nil {
        log.Fatal(err)
    }

    ctx := context.Background()

    // Anti-bot fetch
    fetch, err := client.Evadr.Fetch(ctx, "https://books.toscrape.com", nil)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Fetched via tier %d\n", fetch.TierUsed)

    // Site crawl (waits for the crawl, then returns every page)
    crawl, err := client.Webgrph.Crawl(ctx, "https://books.toscrape.com", &kloakd.CrawlOptions{
        MaxDepth: 2,
        MaxPages: 50,
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Found %d pages\n", crawl.TotalPages)

    // Extract structured data ({field: "css:selector"} is converted to the
    // API's {"fields": [{"name", "selector"}]})
    data, err := client.Kolektr.Extract(ctx, "https://books.toscrape.com", &kloakd.ExtractOptions{
        SchemaSpec: map[string]interface{}{
            "title": "css:h3 a",
            "price": "css:p.price_color",
        },
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Extracted %d records\n", data.TotalRecords)
}

Configuration

client, err := kloakd.New(kloakd.Config{
    APIKey:         "sk-live-...",   // required
    OrganizationID: "org-uuid",      // required
    BaseURL:        "https://api.kloakd.dev", // optional, default shown
    Timeout:        100 * time.Second,        // optional, default 100 s (per attempt; not applied to SSE streams)
    MaxRetries:     3,                        // optional, default 3
    HTTPClient:     customHTTPClient,         // optional, for testing/proxies
    PollInterval:   3 * time.Second,          // optional: first wait when following a 202
    PollTimeout:    15 * time.Minute,         // optional: longest wait for a handed-off job
})
Long requests (HTTP 202 hand-off)

The API answers synchronous requests within about 85 s (it sits behind Cloudflare, which closes requests at 100 s). A request still running then continues as a job: the API answers 202 with a job_id and poll_url, and the SDK polls poll_url (waits grow 1.5× from PollInterval, at most 15 s) and returns the same typed result a direct answer would have. This applies to Evadr.Fetch, Kolektr.Extract, Fetchyr.Authenticate, DetectForms, FillForm, DetectMFA and Nexus.Analyze, Synthesize, Execute. A failed or expired job is an *kloakd.ApiError with Code "job_failed"; a job still running at PollTimeout has Code "job_timeout"; JobID names the job. Cancelling ctx stops the wait.

The default Timeout (100 s) is above the API's 85 s wait, so the 202 arrives before the client gives up. To resume a wait that ended with job_timeout, call client.WaitForHandoff(ctx, apiErr.JobID, nil); client.GetHandoff(ctx, jobID) reads the job's state once (GET handoff/{job_id}).

Streams

Evadr.FetchStream / JobEvents, Webgrph.CrawlStream / CrawlEvents, Parlyr.ChatStream and Client.CrawlStream return a channel. Streams are not bound by Timeout (use ctx). If reading a stream fails, the last value on the channel carries Err:

for ev := range ch {
    if ev.Err != nil {
        log.Fatal(ev.Err)
    }
    // ...
}

Modules

client.Evadr — Anti-Bot Intelligence
// Fetch a URL through anti-bot bypass
result, err := client.Evadr.Fetch(ctx, url, nil)

// A fetch handed off as a job (HTTP 202) is followed to its result.
// FetchOptions overrides Config.PollInterval / PollTimeout for one call:
fetch, err = client.Evadr.Fetch(ctx, url, &kloakd.FetchOptions{
    PollInterval: 3 * time.Second, PollTimeout: 15 * time.Minute,
})

// Async job
jobID, err := client.Evadr.FetchAsync(ctx, url, nil)

// Stream a fetch's progress (starts an async job, streams its events).
// A job with a live pipeline streams pipeline events (event.Pipeline);
// otherwise Type is STATUS, COMPLETE (TierUsed, ArtifactID), FAILED or ERROR.
ch, err := client.Evadr.FetchStream(ctx, url, nil)
for event := range ch {
    if event.Pipeline != nil {
        fmt.Println(event.Pipeline.EventType, event.Pipeline.Message)
    } else {
        fmt.Println(event.Type, event.Status)
    }
}

// Analyze anti-bot defenses
analysis, err := client.Evadr.Analyze(ctx, url, nil)
client.Webgrph — Site Mapping
// Crawl a site: starts the crawl, waits for it, returns every page
crawl, err := client.Webgrph.Crawl(ctx, url, &kloakd.CrawlOptions{MaxDepth: 3, MaxPages: 100})
// crawl.TotalPages and len(crawl.Pages) count fetched pages only

// Or start it and follow it yourself
started, err := client.Webgrph.StartCrawl(ctx, url, nil)
status, err := client.Webgrph.GetCrawlStatus(ctx, started.CrawlID)

// Crawl policy: scope, budget and capture mode
crawl, err = client.Webgrph.Crawl(ctx, url, &kloakd.CrawlOptions{
    CaptureMode: "content",
    Scope:       map[string]interface{}{"same_domain": true},
    Budget:      map[string]interface{}{"max_resources": 500},
})

// Stream a crawl's pipeline events
ch, err := client.Webgrph.CrawlStream(ctx, url, nil)
for event := range ch {
    fmt.Println(event.EventType, event.StageID, event.Message)
}

// The pages listing, one page at a time (Limit 1-1000, default 100). Only
// fetched pages are listed; IncludeGrouping also returns the tree's
// grouping folders (PageNode.IsGrouping), which were not fetched.
pages, err := client.Webgrph.ListCrawlPages(ctx, crawl.CrawlID, &kloakd.CrawlPagesOptions{Limit: 500})
for pages.NextCursor != "" {
    pages, err = client.Webgrph.ListCrawlPages(ctx, crawl.CrawlID,
        &kloakd.CrawlPagesOptions{Cursor: pages.NextCursor, Limit: 500})
}

// Discovery sessions: active only (true) or all (false)
sessions, err := client.Webgrph.ListSessions(ctx, true)

// Reader view, from a crawl's cache or with an authenticated session
view, err := client.Webgrph.ReaderView(ctx, url, &kloakd.ReaderViewOptions{CrawlID: crawl.CrawlID})
client.Crawl — discover, fetch and extract in one call
site, err := client.Crawl(ctx, url, &kloakd.CrawlOrchestratorOptions{
    ExtractSchema: map[string]string{"title": "css:h1"},
})
fmt.Println(site.Success, site.PagesFetched, site.PagesFailed)

// One record per repeated item on each page (CrawlPage.Records)
site, err = client.Crawl(ctx, url, &kloakd.CrawlOrchestratorOptions{
    ExtractItemSelector: "article.product_pod",
    ExtractSchema:       map[string]string{"title": "css:h3 a", "price": "css:.price_color"},
})

A page that fails is recorded on the result; billing, entitlement, authentication and rate-limit errors stop the crawl and are returned.

client.Skanyr — Site Knowledge
site, err := client.Skanyr.CreateSiteFromCrawl(ctx, crawl.CrawlID)
siteID := site["site_id"].(string)
sync, err := client.Skanyr.Sync(ctx, siteID, &kloakd.SyncOptions{Policy: "incremental"})
_, err = client.Skanyr.UpdateSite(ctx, siteID, "conditional") // default sync policy
hits, err := client.Skanyr.Search(ctx, "pricing", &kloakd.SearchOptions{SiteID: siteID})

resources, err := client.Skanyr.SiteResources(ctx, siteID, &kloakd.ResourceListOptions{Limit: 100})
techs, err := client.Skanyr.GetTechnologies(ctx, siteID)
graph, err := client.Skanyr.GetGraph(ctx, siteID, nil)
diff, err := client.Skanyr.Diff(ctx, siteID, "", "")
zip, err := client.Skanyr.ExportSite(ctx, siteID, nil) // ZIP bytes

Sync policies are full, incremental, conditional and head_first.

Detected APIs, the reader view and discovery sessions are on client.Webgrph.

client.Nexus — Strategy Engine
page, _       := client.Evadr.Fetch(ctx, url, nil)
perception, _ := client.Nexus.Analyze(ctx, url, *page.HTML, nil)
strategy, _   := client.Nexus.Synthesize(ctx, url, *page.HTML, "product prices")
verified, _   := client.Nexus.Verify(ctx, strategy.StrategyID)
result, _     := client.Nexus.Execute(ctx, strategy.StrategyID, url, nil)
if result.ExecutionResultID != nil {
    learned, _ := client.Nexus.Knowledge(ctx, *result.ExecutionResultID)
    fmt.Println(learned.TotalPatterns)
}
client.Parlyr — Conversational NLP
parse, err := client.Parlyr.Parse(ctx, "scrape example.com", nil)

// Blocking chat: collects the turn's events until "end"
turn, err := client.Parlyr.Chat(ctx, sessionID, "scrape example.com")
fmt.Println(turn.Response)       // the text and clarification content
fmt.Println(turn.Interpretation) // url, entity, fields, summary, ...
for _, r := range turn.Results {
    fmt.Println(r["count"], r["items"])
}
if !turn.OK {
    fmt.Println(*turn.Errors[0].Content) // error events are collected, not returned
}

// SSE streaming: thinking, interpretation, progress, results, error,
// warning, auth_required, mfa_required, clarification, export_ready, text
ch, err := client.Parlyr.ChatStream(ctx, sessionID, message)
for event := range ch {
    fmt.Println(event.Type, event.Content, event.Data)
}
client.Fetchyr — RPA & Authentication
_, err := client.Fetchyr.StoreCredentials(ctx, "my-login", map[string]interface{}{"username": "u", "password": "p"})
session, err := client.Fetchyr.Authenticate(ctx, "my-login", loginURL)
page, err    := client.Evadr.Fetch(ctx, protectedURL, &kloakd.FetchOptions{SessionArtifactID: *session.ArtifactID})

// A fetch that meets a login page reports it and starts an auth-chain discovery
if page.AuthRequired {
    status, _ := client.Fetchyr.GetAuthChainStatus(ctx, *page.DiscoveryID)
    _, _ = client.Fetchyr.ResumeAuthChainCredentials(ctx, *page.DiscoveryID, "user", "password", false)
    _, _ = client.Fetchyr.ResumeAuthChainMFA(ctx, *page.DiscoveryID, "123456")
    fmt.Println(status)
}
client.Kolektr — Data Extraction
data, err := client.Kolektr.Extract(ctx, url, &kloakd.ExtractOptions{
    SchemaSpec: map[string]interface{}{"title": "css:h1", "price": "css:.price"},
})

// The explicit field list
data, err = client.Kolektr.Extract(ctx, url, &kloakd.ExtractOptions{
    Fields: []kloakd.SchemaField{{Name: "image", Selector: "img", Attribute: "src"}},
})

// A list page: one record per repeated item. Each field's selector is read
// inside the item: the first match (every match for Type "list"), its text
// or Attribute; no match is nil; a field without Selector reads the item.
books, err := client.Kolektr.Extract(ctx, "https://books.toscrape.com", &kloakd.ExtractOptions{
    ItemSelector: "article.product_pod",
    Fields: []kloakd.SchemaField{
        {Name: "title", Selector: "h3 a", Attribute: "title"},
        {Name: "price", Selector: ".price_color"},
        {Name: "rating", Selector: "p.star-rating", Attribute: "class"},
    },
})
fmt.Println(books.TotalRecords) // 20 records, one per book

// The shorthand takes the item selector under "item_selector"
books, err = client.Kolektr.Extract(ctx, "https://books.toscrape.com", &kloakd.ExtractOptions{
    SchemaSpec: map[string]interface{}{
        "item_selector": "article.product_pod",
        "title":         "css:h3 a",
        "price":         "css:.price_color",
    },
})

// From raw HTML (ExtractHTMLOptions takes ItemSelector too)
data, err = client.Kolektr.ExtractHTML(ctx, html, url, nil)

// Lists take paging options
jobs, err := client.Kolektr.ListJobs(ctx, &kloakd.ListJobsOptions{Page: 1, Limit: 50, StatusFilter: "completed"})

Error Handling

import "errors"

result, err := client.Evadr.Fetch(ctx, url, nil)
if err != nil {
    var rlErr *kloakd.RateLimitError
    var quotaErr *kloakd.QuotaExceededError
    var payErr *kloakd.PaymentRequiredError
    var authErr *kloakd.AuthenticationError
    var concErr *kloakd.ConcurrencyLimitError
    switch {
    case errors.As(err, &quotaErr):
        log.Printf("GB used up (%.1f of %.1f): buy a top-up", quotaErr.GBUsed, quotaErr.GBLimit)
    case errors.As(err, &payErr):
        log.Printf("payment failed: update it at %s", payErr.UpgradeURL)
    case errors.As(err, &rlErr) && rlErr.QuotaExhausted:
        log.Printf("playground quota used (%d/%d): %s", rlErr.QuotaUsed, rlErr.QuotaLimit, rlErr.UpgradeURL)
    case errors.As(err, &rlErr):
        time.Sleep(time.Duration(rlErr.RetryAfter) * time.Second)
    case errors.As(err, &concErr):
        log.Printf("all %d work slots in use (%d crawls, %d requests); try again when one finishes",
            concErr.ConcurrentLimit, concErr.ActiveCrawls, concErr.ActiveRequests)
    case errors.As(err, &authErr):
        log.Fatal("invalid API key")
    default:
        log.Printf("error: %v", err)
    }
}

Error types: KloakdError (with Code, the API's error word), AuthenticationError (401), PaymentRequiredError (402), NotEntitledError (403 not_entitled / tier_access_denied, with Module, Feature, CurrentTier, RequiredTier), ForbiddenError (any other 403; errors.As with a *ForbiddenError also matches a NotEntitledError), ConcurrencyLimitError (409 concurrent_limit, with Tier, ConcurrentLimit, ActiveCrawls, ActiveRequests; not retried), QuotaExceededError (429 gb_exhausted), RateLimitError (429; QuotaExhausted for the playground quota), UpstreamError (502), BillingUnavailableError (503 billing_unavailable), ApiError (other 4xx/5xx; JobID on job_failed / job_timeout / crawl_timeout). A body that is not JSON becomes the message; a 422's detail[] is kept as field: message; ....

Concurrency (work slots)

Every metered call holds one of the organisation's work slots while it runs; a crawl or batch that is running or queued holds one too. The plan sets how many run at once: Free 2, Starter 5, Pro 10, Pro+ 20, Developer 50, Enterprise 100. A call made when every slot is in use is refused with 409 concurrent_limit (ConcurrencyLimitError: ActiveCrawls are the crawls and batches, ActiveRequests the other calls in flight). The SDK does not retry it; keep your own concurrency at or below the plan's limit, or try again when a call finishes.

Retry Policy

  • 5xx answers (500, 502, 503, 504) and network errors: retried for GET, HEAD and DELETE only; a POST, PUT or PATCH may already have done its work, so it is not sent again
  • 429 rate_limited: retried on any method, waiting for the Retry-After header, else retry_after, else reset_at
  • 503 billing_unavailable: retried on any method (the request was not started)
  • Not retried: 4xx other than the above (including 409 concurrent_limit), 429 gb_exhausted and the playground quota (429 with retry_after 0 and no reset; waiting does not clear either)
  • Backoff without a hint: 1s × 2^attempt, capped at 60 s
  • Default max retries: 3 (override via Config.MaxRetries; negative for none)
  • Config.Timeout applies to each attempt

Running the Quickstart

export KLOAKD_API_KEY=sk-live-...
export KLOAKD_ORG_ID=your-org-uuid
go run ./examples/quickstart/main.go

Running Tests

go vet ./...
go test ./...

License

MIT

Documentation

Overview

Package kloakd provides the Go SDK for the KLOAKD API.

Index

Constants

View Source 
const (
	DefaultPollInterval = 3 * time.Second
	DefaultPollTimeout  = 15 * time.Minute

	DefaultFetchPollInterval = DefaultPollInterval
	DefaultFetchPollTimeout  = DefaultPollTimeout
)

Defaults for following a request that the API hands off as a job.

View Source 
const (
	DefaultCrawlPollInterval = 3 * time.Second
	DefaultCrawlPollTimeout  = 60 * time.Minute
)

Defaults for waiting on a crawl (crawls run asynchronously on the crawlers).

View Source 
const (
	CrawlPagesDefaultLimit = 100
	CrawlPagesMaxLimit     = 1000
)

Bounds of the pages listing's limit (GET webgrph/crawl/{id}/pages).

Variables

View Source 
var ErrInvalidPagesLimit = errors.New("invalid crawl pages limit")

ErrInvalidPagesLimit is wrapped by the error of a pages listing whose limit is outside 1–CrawlPagesMaxLimit; no request is sent (the API would answer 422).

Functions

This section is empty.

Types

type APIDataOptions added in v0.5.0 

type APIDataOptions struct {
	// EndpointHash selects the endpoint by its hash (endpoint_hash).
	EndpointHash string
}

APIDataOptions configures optional parameters for the api-data calls.

type AnalyzeOptions 

type AnalyzeOptions struct {
	StatusCode  int
	Headers     map[string]string
	BodySnippet string
}

AnalyzeOptions configures optional parameters for Evadr.Analyze.

type AnalyzeResult 

type AnalyzeResult struct {
	Blocked            bool
	Vendor             *string
	Confidence         float64
	RecommendedActions []string
}

AnalyzeResult is returned by Evadr.Analyze.

type ApiError 

type ApiError struct {
	KloakdError
	JobID string
}

ApiError is returned for any other 4xx/5xx response. It is also the error of a handed-off job that failed or expired (Code "job_failed") or was still running at the poll timeout (Code "job_timeout", and "crawl_timeout" for Webgrph.Crawl); JobID names the job (or crawl), so the wait can be resumed with Client.WaitForHandoff (or Webgrph.GetCrawlStatus).

func (*ApiError) Error 

func (e *ApiError) Error() string

type AuthenticationError 

type AuthenticationError struct {
	KloakdError
}

AuthenticationError is returned for HTTP 401 — invalid or expired API key.

func (*AuthenticationError) Error 

func (e *AuthenticationError) Error() string

type BillingUnavailableError added in v0.3.0 

type BillingUnavailableError struct {
	KloakdError
	RetryAfter int
}

BillingUnavailableError is returned for HTTP 503 "billing_unavailable": usage could not be checked, so the request was not started. The SDK retries it itself (on any method) up to Config.MaxRetries.

func (*BillingUnavailableError) Error added in v0.3.0 

func (e *BillingUnavailableError) Error() string

type ChatEvent 

type ChatEvent struct {
	Type      string
	Content   *string
	Data      map[string]interface{}
	Actions   []map[string]interface{}
	Timestamp string
	Raw       map[string]interface{}
	Err       error
}

ChatEvent is one event of a Parlyr chat turn (Parlyr.ChatStream), with the API's payload fields {type, content, data, actions, timestamp} (application/services/chat_orchestrator.py). Type is thinking, interpretation, progress, results, error, warning, auth_required, mfa_required, clarification, export_ready or text. Err is set only on the last value of a stream whose reading failed.

type ChatTurn 

type ChatTurn struct {
	SessionID string
	// OK is false when the turn emitted an "error" event (see Errors).
	OK bool
	// Response is the content of the turn's "text" and "clarification"
	// events, in order, joined with "\n".
	Response string
	// Intent is the "intent" of the interpretation event's data, when the
	// API sends one; "" otherwise. (The API's interpretation data has no
	// intent key today; Interpretation holds what it does send.)
	Intent string
	// Interpretation is the data of the last "interpretation" event
	// (url, entity, fields, summary, ...), nil when there was none.
	Interpretation map[string]interface{}
	// Results holds the data of each "results" event ({items, count, url,
	// job_id, ...}).
	Results []map[string]interface{}
	// ClarificationNeeded is the content of the last "clarification" event.
	ClarificationNeeded *string
	// Warnings holds the content of each "warning" event.
	Warnings []string
	// Errors holds each "error" event (Content is the message, Data and
	// Actions the details and recovery options).
	Errors []ChatEvent
	// Actions are the actions of the last event that offered any.
	Actions []map[string]interface{}
	// Events is every event of the turn, in order.
	Events []ChatEvent
}

ChatTurn is returned by Parlyr.Chat: one turn's events, collected.

type CheckDuplicateOptions added in v0.3.0 

type CheckDuplicateOptions struct {
	ContentHash string
	SessionID   string
}

CheckDuplicateOptions configures optional parameters for CheckDuplicate.

type Client 

type Client struct {
	Evadr   *EvadrNamespace
	Webgrph *WebgrphNamespace
	Skanyr  *SkanyrNamespace
	Nexus   *NexusNamespace
	Parlyr  *ParlyrNamespace
	Fetchyr *FetchyrNamespace
	Kolektr *KolektrNamespace
	// contains filtered or unexported fields
}

Client is the KLOAKD SDK client. Create one with New(). 

client := kloakd.New(kloakd.Config{
    APIKey:         "sk-live-...",
    OrganizationID: "your-org-uuid",
})

func MustNew 

func MustNew(cfg Config) *Client

MustNew is like New but panics on configuration error.

func New 

func New(cfg Config) (*Client, error)

New creates a new KLOAKD client with the given configuration. Returns an error if APIKey or OrganizationID are missing.

func (*Client) Crawl added in v0.3.0 

func (c *Client) Crawl(ctx context.Context, targetURL string, opts *CrawlOrchestratorOptions) (*SiteCrawlResult, error)

Crawl is the high-level orchestrator that discovers, fetches, and optionally extracts structured data from all pages on a site in a single call.

It internally:

  1. Discovers pages via a Webgrph crawl (Webgrph.Crawl)
  2. Fetches each page through the anti-bot engine (Evadr.Fetch)
  3. Extracts structured data if ExtractSchema or ExtractFields is given (Kolektr)

A page whose fetch or extraction fails is recorded (Success=false, or ExtractionError) and the crawl goes on. Errors that would fail every page stop it and are returned: billing (PaymentRequiredError, QuotaExceededError, BillingUnavailableError), entitlement and authentication (NotEntitledError, AuthenticationError), RateLimitError, ConcurrencyLimitError and ctx's error. The result's Success and Error are the discovery crawl's.

func (*Client) CrawlStream added in v0.3.0 

func (c *Client) CrawlStream(ctx context.Context, targetURL string, opts *CrawlOrchestratorOptions) (<-chan CrawlProgressEvent, error)

CrawlStream is the streaming version of Crawl. It sends CrawlProgressEvent values on the returned channel as pages are fetched; the last is crawl_complete, or crawl_error (with Err) when an error stops the crawl (see Crawl). The channel closes after it, or when ctx is cancelled.

func (*Client) GetHandoff added in v0.5.0 

func (c *Client) GetHandoff(ctx context.Context, jobID string) (*HandoffJob, error)

GetHandoff gets the state of a handed-off job once (no polling). A job not found or expired (kept one hour) is an *ApiError with StatusCode 404.

func (*Client) WaitForHandoff added in v0.5.0 

func (c *Client) WaitForHandoff(ctx context.Context, jobID string, poll *PollOptions) (map[string]interface{}, error)

WaitForHandoff polls a handed-off job until it ends and returns its result (the response the route would have given directly), as the methods that follow a 202 do. Use it to resume a wait that ended with Code "job_timeout" (ApiError.JobID). poll nil means Config.PollInterval / PollTimeout. A failed or expired job is Code "job_failed". (An Evadr.FetchAsync job is polled with Evadr.GetJob instead.)

type ConcurrencyLimitError added in v0.5.0 

type ConcurrencyLimitError struct {
	KloakdError
	// Tier is the organisation's plan (detail.tier).
	Tier string
	// ConcurrentLimit is how many slots the plan has (detail.concurrent_limit).
	ConcurrentLimit int
	// ActiveCrawls is the crawls and batches running or queued
	// (detail.active_crawls).
	ActiveCrawls int
	// ActiveRequests is the other metered requests in flight
	// (detail.active_requests; 0.6.0).
	ActiveRequests int
	// TestKeyID is set for a playground test key's own limit.
	TestKeyID string
}

ConcurrencyLimitError is returned for HTTP 409 "concurrent_limit": all of the organisation's work slots are in use. Every metered call holds one slot while it runs (crawls and batches running or queued hold one each); the plan sets how many run at once (Free 2, Starter 5, Pro 10, Pro+ 20, Developer 50, Enterprise 100). The SDK does not retry it: try again when one of the running calls or crawls finishes.

func (*ConcurrencyLimitError) Error added in v0.5.0 

func (e *ConcurrencyLimitError) Error() string

type Config 

type Config struct {
	// APIKey is the KLOAKD API key (required). Starts with "sk-".
	APIKey string

	// OrganizationID is the UUID of the organization (required).
	OrganizationID string

	// BaseURL overrides the API base URL. Defaults to https://api.kloakd.dev.
	BaseURL string

	// Timeout is the per-request timeout (http.Client.Timeout). Defaults to
	// 100 s, above the API's 85 s synchronous wait, so a long request
	// receives its 202 hand-off. SSE streams (ChatStream, CrawlStream,
	// FetchStream, JobEvents, CrawlEvents) are not bound by it; ctx ends them.
	Timeout time.Duration

	// MaxRetries sets the number of retry attempts for retryable errors.
	// 0 (unset) means the default, 3; a negative value means no retries.
	MaxRetries int

	// HTTPClient allows injecting a custom *http.Client (e.g. for testing or
	// a proxy). Its own Timeout then applies instead of Timeout, except to
	// SSE streams, which use a copy of it without a Timeout.
	HTTPClient *http.Client

	// PollInterval and PollTimeout govern following a request the API hands
	// off as a job (HTTP 202 after about 85 s): Evadr.Fetch, Kolektr.Extract,
	// Fetchyr.Authenticate / DetectForms / FillForm / DetectMFA and
	// Nexus.Analyze / Synthesize / Execute. The first poll waits
	// PollInterval (default 3 s); each later wait is 1.5 times longer, up to
	// 15 s. PollTimeout (default 15 min) bounds the whole wait; ctx
	// cancellation also stops it.
	PollInterval time.Duration
	PollTimeout  time.Duration
}

Config holds all configuration for the KLOAKD client.

type CrawlEvent 

type CrawlEvent = PipelineEvent

CrawlEvent is yielded by Webgrph.CrawlStream and Webgrph.CrawlEvents: a crawl streams pipeline events.

type CrawlOptions 

type CrawlOptions struct {
	MaxDepth             int
	MaxPages             int
	IncludeExternalLinks bool
	SessionArtifactID    string
	FastMode             bool
	// CaptureMode is "structure", "content" or "visual". Scope is a
	// CrawlScope (same_host, same_domain, allowed_domains,
	// allowed_subdomains, excluded_paths, include_patterns,
	// exclude_patterns); Budget a CrawlBudget (max_resources, max_depth,
	// max_file_size, max_total_bytes, ...). All optional.
	CaptureMode string
	Scope       map[string]interface{}
	Budget      map[string]interface{}
	// PollInterval and PollTimeout govern waiting for the crawl to finish
	// (defaults 3 s and 60 min; ctx cancellation also stops the wait).
	PollInterval time.Duration
	PollTimeout  time.Duration
	// IncludeGrouping makes Crawl also list the tree's grouping folders
	// (PageNode.IsGrouping), which were not fetched. By default only
	// fetched pages are listed, so len(Pages) matches TotalPages (0.6.0).
	IncludeGrouping bool
}

CrawlOptions configures optional parameters for Webgrph.Crawl.

type CrawlOrchestratorOptions added in v0.3.0 

type CrawlOrchestratorOptions struct {
	// MaxDepth and MaxPages are sent only when set; otherwise the API
	// applies the tier's defaults.
	MaxDepth int
	MaxPages int
	// ExtractSchema is the shorthand {field: "css:selector"} ("css:" is
	// optional); ExtractFields is the explicit list. Either one extracts
	// each fetched page with Kolektr.
	ExtractSchema map[string]string
	ExtractFields []SchemaField
	// ExtractItemSelector extracts one record per repeated item on each
	// page (schema_spec.item_selector; see ExtractOptions.ItemSelector).
	// The records are in CrawlPage.Records (0.6.0).
	ExtractItemSelector  string
	IncludeExternalLinks bool
	SessionArtifactID    string
	// Scope, Budget and CaptureMode are the crawl policy (see CrawlOptions).
	Scope       map[string]interface{}
	Budget      map[string]interface{}
	CaptureMode string
}

CrawlOrchestratorOptions configures the high-level Crawl() and CrawlStream() methods.

type CrawlPage added in v0.3.0 

type CrawlPage struct {
	Success    bool
	URL        string
	StatusCode int
	TierUsed   int
	HTML       *string
	// StructuredData is the page's first extracted record.
	StructuredData map[string]interface{}
	// Records is every extracted record: one per item with
	// ExtractItemSelector, otherwise at most one (0.6.0).
	Records    []map[string]interface{}
	ArtifactID *string
	Error      *string
	// ExtractionError is the error of the page's extraction, when
	// ExtractSchema / ExtractFields was given and it failed.
	ExtractionError *string
}

CrawlPage represents a single page result from the high-level Crawl() orchestrator.

func (CrawlPage) Ok added in v0.3.0 

func (p CrawlPage) Ok() bool

Ok returns true if the page was fetched successfully.

type CrawlPagesOptions added in v0.6.0 

type CrawlPagesOptions struct {
	// Cursor is the previous page's next_cursor ("" for the first page).
	// The API answers 422 for a cursor that is not a non-negative integer.
	Cursor string
	// Limit is the nodes per page, 1 to CrawlPagesMaxLimit; 0 sends none
	// (the API's default, CrawlPagesDefaultLimit).
	Limit int
	// IncludeGrouping also lists the tree's grouping folders
	// (metadata.is_grouping), which were not fetched. By default only
	// fetched pages are listed, each under the page the crawl found it on,
	// and total matches the crawl status's total_pages.
	IncludeGrouping bool
}

CrawlPagesOptions configures Webgrph.ListCrawlPages.

type CrawlPagesResult added in v0.6.0 

type CrawlPagesResult struct {
	CrawlID string
	Nodes   []PageNode
	// NextCursor is the cursor of the next page; "" on the last.
	NextCursor string
	// Total is the number of nodes the listing has across all pages.
	Total int
	// Raw is the answer as received.
	Raw map[string]interface{}
}

CrawlPagesResult is one page of a crawl's pages listing.

type CrawlProgressEvent added in v0.3.0 

type CrawlProgressEvent struct {
	Type     string
	URL      *string
	Page     *int
	Total    *int
	Success  *bool
	Error    *string
	Metadata map[string]interface{}
	Err      error
}

CrawlProgressEvent is yielded by Client.CrawlStream. Type is discovery_complete, page_fetching, page_fetched, page_failed, crawl_complete or, when the crawl stops on an error (billing, auth or rate limit, ctx), crawl_error with Err set.

type CrawlResult 

type CrawlResult struct {
	Success         bool
	CrawlID         string
	URL             string
	Status          string
	TotalPages      int
	MaxDepthReached int
	Pages           []PageNode
	Total           int
	ArtifactID      *string
	Error           *string
}

CrawlResult is returned by Webgrph.Crawl.

type CrawlStreamOptions 

type CrawlStreamOptions struct {
	MaxDepth             int
	MaxPages             int
	IncludeExternalLinks bool
	SessionArtifactID    string
	CaptureMode          string
	Scope                map[string]interface{}
	Budget               map[string]interface{}
}

CrawlStreamOptions configures optional parameters for Webgrph.CrawlStream.

type CreateWorkflowOptions added in v0.3.0 

type CreateWorkflowOptions struct {
	Description  string
	WorkflowType string
}

CreateWorkflowOptions configures optional parameters for CreateWorkflow.

type DeduplicationResult 

type DeduplicationResult struct {
	IsDuplicate     bool
	OriginalURL     *string
	SimilarityScore float64
	MatchType       *string
}

DeduplicationResult is returned by Fetchyr.CheckDuplicate.

type EvadrNamespace 

type EvadrNamespace struct {
	// contains filtered or unexported fields
}

EvadrNamespace exposes the Anti-Bot Intelligence module. Access via client.Evadr.

func (*EvadrNamespace) Analyze 

func (e *EvadrNamespace) Analyze(ctx context.Context, targetURL string, opts *AnalyzeOptions) (*AnalyzeResult, error)

Analyze inspects the anti-bot defenses of the given URL.

func (*EvadrNamespace) DeleteProxy added in v0.2.0 

func (e *EvadrNamespace) DeleteProxy(ctx context.Context, name string) error

DeleteProxy deletes a stored proxy configuration by name.

func (*EvadrNamespace) Fetch 

func (e *EvadrNamespace) Fetch(ctx context.Context, targetURL string, opts *FetchOptions) (*FetchResult, error)

Fetch performs an anti-bot fetch of the given URL.

func (*EvadrNamespace) FetchAsync 

func (e *EvadrNamespace) FetchAsync(ctx context.Context, targetURL string, opts *FetchAsyncOptions) (string, error)

FetchAsync enqueues an anti-bot fetch job and returns the job ID.

func (*EvadrNamespace) FetchStream 

func (e *EvadrNamespace) FetchStream(ctx context.Context, targetURL string, opts *FetchStreamOptions) (<-chan FetchEvent, error)

FetchStream starts an asynchronous fetch and streams its progress events (GET evadr/jobs/{job_id}/events). The channel closes when the job ends or ctx is cancelled. There is no POST stream route; this is the live route.

func (*EvadrNamespace) GetJob added in v0.2.0 

func (e *EvadrNamespace) GetJob(ctx context.Context, jobID string) (map[string]interface{}, error)

GetJob polls an async fetch job by ID.

func (*EvadrNamespace) JobEvents added in v0.3.0 

func (e *EvadrNamespace) JobEvents(ctx context.Context, jobID string) (<-chan FetchEvent, error)

JobEvents streams an async fetch job's progress events (SSE). The channel closes when the job ends or ctx is cancelled; if reading the stream failed, the last event carries Err.

func (*EvadrNamespace) ListProxies added in v0.2.0 

func (e *EvadrNamespace) ListProxies(ctx context.Context) (map[string]interface{}, error)

ListProxies lists stored proxy configurations (names only, no secrets).

func (*EvadrNamespace) ListVendors added in v0.2.0 

func (e *EvadrNamespace) ListVendors(ctx context.Context) (map[string]interface{}, error)

ListVendors lists all known anti-bot detection vendors.

func (*EvadrNamespace) Profile added in v0.3.0 

func (e *EvadrNamespace) Profile(ctx context.Context, targetURL string) (map[string]interface{}, error)

Profile analyses a URL and returns its canonical site profile.

func (*EvadrNamespace) Scan added in v0.2.0 

func (e *EvadrNamespace) Scan(ctx context.Context, targetURL string) (map[string]interface{}, error)

Scan fetches a URL once (plain HTTP) and reports its anti-bot infrastructure, vendors and challenges.

func (*EvadrNamespace) StoreProxy 

func (e *EvadrNamespace) StoreProxy(ctx context.Context, name, proxyURL string) error

StoreProxy registers a named proxy in the organization's proxy pool.

type ExecuteWorkflowOptions added in v0.3.0 

type ExecuteWorkflowOptions struct {
	SessionArtifactID string
	InputData         map[string]interface{}
}

ExecuteWorkflowOptions configures optional parameters for ExecuteWorkflow.

type ExecutionResult 

type ExecutionResult struct {
	ExecutionID string
	WorkflowID  string
	Status      string
	StartedAt   *string
	CompletedAt *string
	Result      map[string]interface{}
	Error       *string
}

ExecutionResult is returned by Fetchyr.ExecuteWorkflow / GetExecution.

type ExportOptions added in v0.5.0 

type ExportOptions struct {
	// ResourceIDs selects the resources; nil exports all with content.
	ResourceIDs []string
	// MaxBytes bounds the archive (0: the API's default, 500 MiB).
	MaxBytes int
}

ExportOptions configures Skanyr.ExportSite.

type ExtractAllAPIDataOptions added in v0.5.0 

type ExtractAllAPIDataOptions struct {
	EndpointHash string
	// MaxRecords bounds the records extracted; 0 means no bound.
	MaxRecords int
}

ExtractAllAPIDataOptions configures optional parameters for ExtractAllAPIData.

type ExtractHTMLOptions 

type ExtractHTMLOptions struct {
	// SchemaSpec, Fields and ItemSelector are as in ExtractOptions.
	SchemaSpec   map[string]interface{}
	Fields       []SchemaField
	ItemSelector string
}

ExtractHTMLOptions configures optional parameters for Kolektr.ExtractHTML.

type ExtractOptions added in v0.3.0 

type ExtractOptions struct {
	// SchemaSpec describes the fields to extract (sent as schema_spec):
	// either the API's own spec, {"fields": [{"name", "selector", ...}],
	// "item_selector", "pagination", "max_pages", ...}, or the shorthand
	// {field: "css:selector"} ("css:" is optional), which is converted. In
	// the shorthand the key "item_selector" is the item selector, not a
	// field.
	SchemaSpec map[string]interface{}
	// Fields is the explicit field list; it replaces SchemaSpec's fields.
	Fields []SchemaField
	// ItemSelector (schema_spec.item_selector, 0.6.0) is the CSS selector of
	// the repeated item on a list page, e.g. "article.product_pod". Each
	// matching element is one record and each field's selector is read
	// inside it: the first match gives the value (every match for a field
	// of Type "list"), its text or the field's Attribute; a field with no
	// match in an item is nil; a field with no Selector is read from the
	// item element itself. It needs at least one field; an empty or invalid
	// CSS selector, or one without fields, is a 422 (ApiError).
	// Without it the fields run against the whole page and give one
	// record, as before. It overrides an item_selector in SchemaSpec.
	ItemSelector      string
	Level             string
	FetchArtifactID   string
	SessionArtifactID string
	ApiMapArtifactID  string
	Options           map[string]interface{}
}

ExtractOptions configures optional parameters for Kolektr.Extract.

type ExtractionResult 

type ExtractionResult struct {
	Success      bool
	URL          string
	Method       string
	Records      []map[string]interface{}
	TotalRecords int
	PagesScraped int
	ArtifactID   *string
	JobID        *string
	Error        *string
	// AuthRequired is true when a login page was detected at the URL;
	// DiscoveryID is then the auth-chain discovery started for it.
	AuthRequired bool
	DiscoveryID  *string
}

ExtractionResult is returned by Kolektr.Extract and Kolektr.ExtractHTML.

type FetchAsyncOptions 

type FetchAsyncOptions struct {
	ForceBrowser      bool
	UseProxy          bool
	SessionArtifactID string
	WebhookURL        string
}

FetchAsyncOptions configures optional parameters for Evadr.FetchAsync.

type FetchEvent 

type FetchEvent struct {
	Type       string
	JobID      string
	Status     string
	TierUsed   *int
	ArtifactID *string
	Error      *string
	Pipeline   *PipelineEvent
	Raw        map[string]interface{}
	Err        error
}

FetchEvent is yielded by Evadr.FetchStream and Evadr.JobEvents. A job with a live pipeline streams pipeline events (Pipeline is set). Otherwise the API polls the job and streams {type, job_id, status | tier_used, artifact_id | error}: Type is "STATUS", "COMPLETE", "FAILED" or "ERROR" (modules/evadr/router.py). Err is set only on the last value of a stream whose reading failed.

type FetchOptions 

type FetchOptions struct {
	ForceBrowser      bool
	UseProxy          bool
	SessionArtifactID string
	// PollInterval and PollTimeout override Config.PollInterval /
	// PollTimeout for following a fetch the API hands off as a job (HTTP
	// 202, after about 85 s); ctx cancellation also stops the wait.
	PollInterval time.Duration
	PollTimeout  time.Duration
}

FetchOptions configures optional parameters for Evadr.Fetch.

type FetchResult 

type FetchResult struct {
	Success    bool
	URL        string
	StatusCode int
	TierUsed   int
	// HTML is nil when the API returned none (for example when
	// AuthRequired is true).
	HTML            *string
	VendorDetected  *string
	AntiBotBypassed bool
	ArtifactID      *string
	Error           *string
	// AuthRequired is true when a login page was detected at the URL;
	// DiscoveryID is then the auth-chain discovery started for it
	// (Fetchyr.GetAuthChainStatus).
	AuthRequired bool
	DiscoveryID  *string
}

FetchResult is returned by Evadr.Fetch.

type FetchStreamOptions 

type FetchStreamOptions struct {
	ForceBrowser      bool
	UseProxy          bool
	SessionArtifactID string
}

FetchStreamOptions configures optional parameters for Evadr.FetchStream.

type FetchyrNamespace 

type FetchyrNamespace struct {
	// contains filtered or unexported fields
}

FetchyrNamespace exposes the RPA & Authentication module. Access via client.Fetchyr.

func (*FetchyrNamespace) Authenticate added in v0.3.0 

func (f *FetchyrNamespace) Authenticate(ctx context.Context, credentialName, targetURL string) (*SessionResult, error)

Authenticate logs in to a site with credentials stored under credentialName (StoreCredentials) and returns the authenticated session. It replaces Login, whose route was removed: the API no longer takes selectors or a password per call. Like DetectForms, FillForm and DetectMFA, a request the API hands off as a job (202, after about 85 s) is followed to its result; see Config.PollInterval.

func (*FetchyrNamespace) CheckDuplicate added in v0.3.0 

func (f *FetchyrNamespace) CheckDuplicate(ctx context.Context, targetURL string, opts *CheckDuplicateOptions) (*DeduplicationResult, error)

CheckDuplicate reports whether a URL (or its content hash) was already seen, within a deduplication session if given.

func (*FetchyrNamespace) CreateDedupSession added in v0.2.0 

func (f *FetchyrNamespace) CreateDedupSession(ctx context.Context, config map[string]interface{}) (map[string]interface{}, error)

CreateDedupSession creates a deduplication session.

func (*FetchyrNamespace) CreateMultiSiteWorkflow added in v0.2.0 

func (f *FetchyrNamespace) CreateMultiSiteWorkflow(ctx context.Context, name string, sites []map[string]interface{}, opts *MultiSiteWorkflowOptions) (map[string]interface{}, error)

CreateMultiSiteWorkflow creates a multi-site orchestration workflow. Each site is {site_id, name, url, authentication_config?, workflow_steps?, timeout?, retry_count?, priority?, metadata?}.

func (*FetchyrNamespace) CreateWorkflow 

func (f *FetchyrNamespace) CreateWorkflow(ctx context.Context, name string, steps []map[string]interface{}, opts *CreateWorkflowOptions) (*WorkflowResult, error)

CreateWorkflow creates a new RPA workflow.

func (*FetchyrNamespace) DeleteCredentials added in v0.2.0 

func (f *FetchyrNamespace) DeleteCredentials(ctx context.Context, name string) error

DeleteCredentials deletes stored credentials by name.

func (*FetchyrNamespace) DeleteWorkflow added in v0.2.0 

func (f *FetchyrNamespace) DeleteWorkflow(ctx context.Context, workflowID string) error

DeleteWorkflow deletes a workflow definition.

func (*FetchyrNamespace) DetectForms 

func (f *FetchyrNamespace) DetectForms(ctx context.Context, targetURL string, sessionArtifactID string) (*FormDetectionResult, error)

DetectForms detects forms on a page.

func (*FetchyrNamespace) DetectMFA 

func (f *FetchyrNamespace) DetectMFA(ctx context.Context, targetURL string, sessionArtifactID string) (*MfaDetectionResult, error)

DetectMFA detects MFA challenges on a page.

func (*FetchyrNamespace) DiscoverAuthChain added in v0.3.0 

func (f *FetchyrNamespace) DiscoverAuthChain(ctx context.Context, targetURL string) (map[string]interface{}, error)

DiscoverAuthChain starts discovering how a site authenticates.

func (*FetchyrNamespace) ExecuteWorkflow 

func (f *FetchyrNamespace) ExecuteWorkflow(ctx context.Context, workflowID, targetURL string, opts *ExecuteWorkflowOptions) (*ExecutionResult, error)

ExecuteWorkflow runs a workflow against targetURL.

func (*FetchyrNamespace) FillForm added in v0.2.0 

func (f *FetchyrNamespace) FillForm(ctx context.Context, targetURL string, formData map[string]string, sessionArtifactID string) (map[string]interface{}, error)

FillForm fills and submits a form on a page.

func (*FetchyrNamespace) GetAuthChainStatus added in v0.3.0 

func (f *FetchyrNamespace) GetAuthChainStatus(ctx context.Context, discoveryID string) (map[string]interface{}, error)

GetAuthChainStatus gets an auth-chain discovery's status.

func (*FetchyrNamespace) GetDedupDomainStatistics added in v0.2.0 

func (f *FetchyrNamespace) GetDedupDomainStatistics(ctx context.Context, domain string) (map[string]interface{}, error)

GetDedupDomainStatistics gets deduplication statistics for a domain.

func (*FetchyrNamespace) GetDedupSession added in v0.2.0 

func (f *FetchyrNamespace) GetDedupSession(ctx context.Context, sessionID string) (map[string]interface{}, error)

GetDedupSession gets a deduplication session by ID.

func (*FetchyrNamespace) GetDedupSessionStatistics added in v0.2.0 

func (f *FetchyrNamespace) GetDedupSessionStatistics(ctx context.Context, sessionID string) (map[string]interface{}, error)

GetDedupSessionStatistics gets statistics for a deduplication session.

func (*FetchyrNamespace) GetExecution 

func (f *FetchyrNamespace) GetExecution(ctx context.Context, workflowID, executionID string) (*ExecutionResult, error)

GetExecution retrieves the result of a workflow execution.

func (*FetchyrNamespace) GetMFAChallenge added in v0.2.0 

func (f *FetchyrNamespace) GetMFAChallenge(ctx context.Context, challengeID string) (map[string]interface{}, error)

GetMFAChallenge gets a specific MFA challenge by ID.

func (*FetchyrNamespace) GetMFAStatistics added in v0.2.0 

func (f *FetchyrNamespace) GetMFAStatistics(ctx context.Context, domain string) (map[string]interface{}, error)

GetMFAStatistics gets MFA statistics for a domain.

func (*FetchyrNamespace) GetWorkflow added in v0.2.0 

func (f *FetchyrNamespace) GetWorkflow(ctx context.Context, workflowID string) (map[string]interface{}, error)

GetWorkflow gets a workflow definition by ID.

func (*FetchyrNamespace) GetWorkflowStatistics added in v0.2.0 

func (f *FetchyrNamespace) GetWorkflowStatistics(ctx context.Context, workflowID string) (map[string]interface{}, error)

GetWorkflowStatistics gets execution statistics for a workflow.

func (*FetchyrNamespace) ListCredentials added in v0.2.0 

func (f *FetchyrNamespace) ListCredentials(ctx context.Context) (map[string]interface{}, error)

ListCredentials lists stored credential names (no secret values).

func (*FetchyrNamespace) ListDedupSessions added in v0.2.0 

func (f *FetchyrNamespace) ListDedupSessions(ctx context.Context) (map[string]interface{}, error)

ListDedupSessions lists active deduplication sessions.

func (*FetchyrNamespace) ListMFAChallenges added in v0.2.0 

func (f *FetchyrNamespace) ListMFAChallenges(ctx context.Context) (map[string]interface{}, error)

ListMFAChallenges lists pending MFA challenges.

func (*FetchyrNamespace) ListSessions added in v0.2.0 

func (f *FetchyrNamespace) ListSessions(ctx context.Context) (map[string]interface{}, error)

ListSessions lists active browser sessions.

func (*FetchyrNamespace) ListWorkflows added in v0.2.0 

func (f *FetchyrNamespace) ListWorkflows(ctx context.Context, opts *ListWorkflowsOptions) (map[string]interface{}, error)

ListWorkflows lists RPA workflow definitions.

func (*FetchyrNamespace) ResolveMFA added in v0.3.0 

func (f *FetchyrNamespace) ResolveMFA(ctx context.Context, challengeID, resolution string) (map[string]interface{}, error)

ResolveMFA resolves a queued MFA challenge (the code, or "approved"). It replaces SubmitMFA, whose route was removed.

func (*FetchyrNamespace) ResumeAuthChainCredentials added in v0.5.0 

func (f *FetchyrNamespace) ResumeAuthChainCredentials(ctx context.Context, discoveryID, username, password string, remember bool) (map[string]interface{}, error)

ResumeAuthChainCredentials gives the credentials a paused auth-chain discovery asked for. With remember true the API stores them encrypted for later workflow runs.

func (*FetchyrNamespace) ResumeAuthChainMFA added in v0.5.0 

func (f *FetchyrNamespace) ResumeAuthChainMFA(ctx context.Context, discoveryID, mfaCode string) (map[string]interface{}, error)

ResumeAuthChainMFA gives the MFA code a paused auth-chain discovery asked for.

func (*FetchyrNamespace) ResumeExecution added in v0.3.0 

func (f *FetchyrNamespace) ResumeExecution(ctx context.Context, workflowID, executionID string, resume map[string]interface{}) (*ExecutionResult, error)

ResumeExecution resumes a paused execution (after MFA).

func (*FetchyrNamespace) StoreCredentials added in v0.2.0 

func (f *FetchyrNamespace) StoreCredentials(ctx context.Context, name string, credentials map[string]interface{}) (map[string]interface{}, error)

StoreCredentials stores login credentials securely.

func (*FetchyrNamespace) TerminateSession added in v0.2.0 

func (f *FetchyrNamespace) TerminateSession(ctx context.Context, artifactID string) error

TerminateSession terminates a session and releases Chrome slot.

func (*FetchyrNamespace) UpdateWorkflow added in v0.2.0 

func (f *FetchyrNamespace) UpdateWorkflow(ctx context.Context, workflowID string, updates map[string]interface{}) (map[string]interface{}, error)

UpdateWorkflow updates an existing workflow.

type FindingsOptions added in v0.5.0 

type FindingsOptions struct {
	FindingKind string
	Limit       int
	Offset      int
}

FindingsOptions filters Skanyr.GetFindings.

type ForbiddenError added in v0.5.0 

type ForbiddenError struct {
	KloakdError
}

ForbiddenError is returned for any other HTTP 403: the key may not do this (for example another organisation's resource). errors.As with a *ForbiddenError also matches a NotEntitledError.

func (*ForbiddenError) Error added in v0.5.0 

func (e *ForbiddenError) Error() string

type FormDetectionResult 

type FormDetectionResult struct {
	URL               string
	FormsFound        int
	LoginFormDetected bool
	Fields            []map[string]interface{}
}

FormDetectionResult is returned by Fetchyr.DetectForms.

type GraphOptions added in v0.5.0 

type GraphOptions struct {
	ResourceType string
	MaxDepth     int
	EdgeKind     string
	Limit        int
}

GraphOptions filters Skanyr.GetGraph.

type HandedOff added in v0.3.0 

type HandedOff struct {
	JobID         string
	Reason        string
	WaitedSeconds float64
	PollURL       string
}

HandedOff describes a 202 from a synchronous route: still running, continuing as the job JobID, whose state is at PollURL.

type HandoffJob added in v0.5.0 

type HandoffJob struct {
	JobID  string
	Kind   string // the route's kind, e.g. "kolektr_extract"
	Status string // "running", "completed" or "failed"
	// Result is the route's own response, set when Status is "completed".
	Result map[string]interface{}
	// Error is set when Status is "failed".
	Error *string
	Raw   map[string]interface{}
}

HandoffJob is the state of a request the API handed off as a job (GET handoff/{job_id}, kernel/sync_handoff.py).

type KloakdError 

type KloakdError struct {
	StatusCode int
	Message    string
	// Code is the API's error word (detail.error), e.g. "gb_exhausted".
	Code string
}

KloakdError is the base error type for all SDK errors. Every error carries StatusCode and Message.

func (*KloakdError) Error 

func (e *KloakdError) Error() string

type KolektrNamespace 

type KolektrNamespace struct {
	// contains filtered or unexported fields
}

KolektrNamespace exposes the Data Extraction module. Access via client.Kolektr.

func (*KolektrNamespace) CreateScraper added in v0.2.0 

func (k *KolektrNamespace) CreateScraper(ctx context.Context, config map[string]interface{}) (map[string]interface{}, error)

CreateScraper creates a scraper configuration.

func (*KolektrNamespace) DeleteContent added in v0.2.0 

func (k *KolektrNamespace) DeleteContent(ctx context.Context, itemID string) error

DeleteContent deletes a content item.

func (*KolektrNamespace) DeleteScraper added in v0.2.0 

func (k *KolektrNamespace) DeleteScraper(ctx context.Context, scraperID string) error

DeleteScraper deletes a scraper configuration.

func (*KolektrNamespace) Extract added in v0.3.0 

func (k *KolektrNamespace) Extract(ctx context.Context, targetURL string, opts *ExtractOptions) (*ExtractionResult, error)

Extract extracts structured data from a URL with the full Kolektr pipeline (POST kolektr/extract). A request the API hands off as a job (202, after about 85 s) is followed to its result; see Config.PollInterval.

func (*KolektrNamespace) ExtractAllAPIData added in v0.2.0 

func (k *KolektrNamespace) ExtractAllAPIData(ctx context.Context, apiEndpoint string, opts *ExtractAllAPIDataOptions) (map[string]interface{}, error)

ExtractAllAPIData starts a job extracting all data from a discovered API endpoint.

func (*KolektrNamespace) ExtractHTML 

func (k *KolektrNamespace) ExtractHTML(ctx context.Context, html, targetURL string, opts *ExtractHTMLOptions) (*ExtractionResult, error)

ExtractHTML extracts structured data from raw HTML without making a network request.

func (*KolektrNamespace) GetAPIData added in v0.2.0 

func (k *KolektrNamespace) GetAPIData(ctx context.Context, apiEndpoint string, opts *APIDataOptions) (map[string]interface{}, error)

GetAPIData gets all extracted data for a discovered API endpoint. apiEndpoint is the endpoint's URL; it is sent escaped as one path segment.

func (*KolektrNamespace) GetAPIDataPaginated added in v0.2.0 

func (k *KolektrNamespace) GetAPIDataPaginated(ctx context.Context, apiEndpoint string, offset, limit int, opts *APIDataOptions) (map[string]interface{}, error)

GetAPIDataPaginated gets API data with pagination.

func (*KolektrNamespace) GetContent added in v0.2.0 

func (k *KolektrNamespace) GetContent(ctx context.Context, itemID string) (map[string]interface{}, error)

GetContent gets a content item by ID.

func (*KolektrNamespace) GetJob added in v0.2.0 

func (k *KolektrNamespace) GetJob(ctx context.Context, jobID string) (map[string]interface{}, error)

GetJob gets an extraction job by ID.

func (*KolektrNamespace) GetJobStatus added in v0.2.0 

func (k *KolektrNamespace) GetJobStatus(ctx context.Context, jobID string) (map[string]interface{}, error)

GetJobStatus gets extraction job status.

func (*KolektrNamespace) GetScraper added in v0.2.0 

func (k *KolektrNamespace) GetScraper(ctx context.Context, scraperID string) (map[string]interface{}, error)

GetScraper gets a scraper configuration by ID.

func (*KolektrNamespace) ListContent added in v0.2.0 

func (k *KolektrNamespace) ListContent(ctx context.Context, opts *ListContentOptions) (map[string]interface{}, error)

ListContent lists content items.

func (*KolektrNamespace) ListJobs added in v0.2.0 

func (k *KolektrNamespace) ListJobs(ctx context.Context, opts *ListJobsOptions) (map[string]interface{}, error)

ListJobs lists extraction jobs.

func (*KolektrNamespace) ListScrapers added in v0.2.0 

func (k *KolektrNamespace) ListScrapers(ctx context.Context, opts *ListScrapersOptions) (map[string]interface{}, error)

ListScrapers lists scraper configurations.

func (*KolektrNamespace) Page deprecated 

func (k *KolektrNamespace) Page(ctx context.Context, targetURL string, opts *ExtractOptions) (*ExtractionResult, error)

Page is the 0.2 name of Extract.

Deprecated: use Extract.

func (*KolektrNamespace) UpdateScraper added in v0.2.0 

func (k *KolektrNamespace) UpdateScraper(ctx context.Context, scraperID string, updates map[string]interface{}) (map[string]interface{}, error)

UpdateScraper updates a scraper configuration.

type ListContentOptions added in v0.5.0 

type ListContentOptions struct {
	Page     int
	PageSize int
	Search   string
}

ListContentOptions configures Kolektr.ListContent (zero values: the API's defaults).

type ListJobsOptions added in v0.5.0 

type ListJobsOptions struct {
	Page         int
	Limit        int
	StatusFilter string
}

ListJobsOptions configures Kolektr.ListJobs (zero values: the API's defaults).

type ListPage added in v0.5.0 

type ListPage struct {
	Limit  int
	Offset int
}

ListPage bounds a Skanyr list (zero values: the API's defaults).

type ListScrapersOptions added in v0.5.0 

type ListScrapersOptions struct {
	Page     int
	PageSize int
	// IsTemplate filters on templates (true) or non-templates (false); nil for all.
	IsTemplate *bool
}

ListScrapersOptions configures Kolektr.ListScrapers (zero values: the API's defaults).

type ListWorkflowsOptions added in v0.5.0 

type ListWorkflowsOptions struct {
	Page     int
	PageSize int
}

ListWorkflowsOptions pages Fetchyr.ListWorkflows (zero values: the API's defaults).

type MfaDetectionResult 

type MfaDetectionResult struct {
	MfaDetected bool
	ChallengeID *string
	MfaType     *string
	Message     *string
}

MfaDetectionResult is returned by Fetchyr.DetectMFA.

type MultiSiteWorkflowOptions added in v0.5.0 

type MultiSiteWorkflowOptions struct {
	Description  string
	Dependencies []SiteDependency
	GlobalConfig map[string]interface{}
}

MultiSiteWorkflowOptions configures optional parameters for CreateMultiSiteWorkflow.

type NexusAnalyzeOptions 

type NexusAnalyzeOptions struct {
	Constraints map[string]interface{}
}

NexusAnalyzeOptions configures optional parameters for Nexus.Analyze.

type NexusAnalyzeResult 

type NexusAnalyzeResult struct {
	URL          string
	PerceptionID string
	ReasoningID  string
	Strategy     map[string]interface{}
	PageType     string
	Confidence   float64
	ArtifactID   *string
}

NexusAnalyzeResult is returned by Nexus.Analyze.

type NexusExecuteResult 

type NexusExecuteResult struct {
	StrategyID string
	Success    bool
	Records    []map[string]interface{}
	ArtifactID *string
	DurationMs int
	Error      *string
	// ExecutionResultID identifies this execution; pass it to
	// Nexus.Knowledge. Nil when the API did not return one.
	ExecutionResultID *string
}

NexusExecuteResult is returned by Nexus.Execute.

type NexusKnowledgeResult 

type NexusKnowledgeResult struct {
	LearnedConcepts []map[string]interface{}
	Patterns        []map[string]interface{}
	TotalPatterns   int
	DurationMs      int
}

NexusKnowledgeResult is returned by Nexus.Knowledge.

type NexusNamespace 

type NexusNamespace struct {
	// contains filtered or unexported fields
}

NexusNamespace exposes the Strategy Engine module (5-layer cognitive pipeline). Access via client.Nexus.

func (*NexusNamespace) Analyze 

func (n *NexusNamespace) Analyze(ctx context.Context, targetURL, html string, opts *NexusAnalyzeOptions) (*NexusAnalyzeResult, error)

Analyze classifies a page and proposes an extraction strategy. The API needs the page's HTML (fetch it first, e.g. with Evadr.Fetch).

func (*NexusNamespace) Execute 

func (n *NexusNamespace) Execute(ctx context.Context, strategyID, targetURL string, options map[string]interface{}) (*NexusExecuteResult, error)

Execute runs a verified strategy against a URL. The result's ExecutionResultID is the id to pass to Knowledge.

Analyze, Synthesize and Execute follow a request the API hands off as a job (202, after about 85 s) to its result; see Config.PollInterval.

func (*NexusNamespace) Knowledge 

func (n *NexusNamespace) Knowledge(ctx context.Context, executionResultID string) (*NexusKnowledgeResult, error)

Knowledge stores learned concepts and patterns from an execution result. executionResultID is NexusExecuteResult.ExecutionResultID, returned by Execute (execution_result_id).

func (*NexusNamespace) Reason added in v0.2.0 

func (n *NexusNamespace) Reason(ctx context.Context, queryType string, parameters []string, opts *ReasonOptions) (*NexusReasonResult, error)

Reason runs a logical query on the Strategy Engine: queryType with its parameters (a list of strings; nil sends []).

func (*NexusNamespace) Synthesize 

func (n *NexusNamespace) Synthesize(ctx context.Context, targetURL, html, goal string) (*NexusSynthesisResult, error)

Synthesize generates an extraction program for a page and a goal.

func (*NexusNamespace) Verify 

func (n *NexusNamespace) Verify(ctx context.Context, strategyID string) (*NexusVerifyResult, error)

Verify verifies the safety of a strategy before execution.

type NexusReasonResult added in v0.5.0 

type NexusReasonResult struct {
	Success     bool
	Results     []string
	Confidence  float64
	ReasoningID *string
}

NexusReasonResult is returned by Nexus.Reason.

type NexusSynthesisResult 

type NexusSynthesisResult struct {
	StrategyID      string
	StrategyName    string
	StrategyType    string
	Program         string
	Confidence      float64
	Steps           []map[string]interface{}
	SynthesisTimeMs int
}

NexusSynthesisResult is returned by Nexus.Synthesize.

type NexusVerifyResult 

type NexusVerifyResult struct {
	StrategyID  string
	Verified    bool
	SafetyScore float64
	Violations  []string
}

NexusVerifyResult is returned by Nexus.Verify.

type NotEntitledError 

type NotEntitledError struct {
	KloakdError
	Module       string
	Feature      string
	CurrentTier  string
	RequiredTier string
	UpgradeURL   string
}

NotEntitledError is returned for HTTP 403 with code "not_entitled" (the organisation's plan does not include the module) or "tier_access_denied" (the tier does not include the feature; Feature, CurrentTier and RequiredTier say which).

func (*NotEntitledError) As added in v0.5.0 

func (e *NotEntitledError) As(target interface{}) bool

As makes a NotEntitledError also a *ForbiddenError for errors.As: every NotEntitledError is a 403.

func (*NotEntitledError) Error 

func (e *NotEntitledError) Error() string

type PageNode 

type PageNode struct {
	ID        string
	URL       string
	ParentURL *string
	Depth     int
	Category  string
	Title     *string
	Children  []string
	Metadata  map[string]interface{}
	// IsGrouping is true for a grouping folder of the displayed tree
	// (metadata.is_grouping): a display node that was never fetched. The
	// pages listing returns them only with IncludeGrouping (0.6.0).
	IsGrouping bool
	// CrawlParentURL is the page the crawl found this page on, when the
	// tree shows it under a grouping folder (crawl_parent_url). Only the
	// stored page records carry it (a crawl stream's discovery_response
	// pages); the pages listing does not, and by default already puts each
	// page under that parent (ParentURL). nil when absent (0.6.0).
	CrawlParentURL *string
}

PageNode represents a single page in the site hierarchy.

type PageOptions deprecated 

type PageOptions = ExtractOptions

PageOptions is the 0.2 name of ExtractOptions.

Deprecated: use ExtractOptions.

type ParlyrNamespace 

type ParlyrNamespace struct {
	// contains filtered or unexported fields
}

ParlyrNamespace exposes the Conversational NLP module. Access via client.Parlyr.

func (*ParlyrNamespace) Chat 

func (p *ParlyrNamespace) Chat(ctx context.Context, sessionID, message string) (*ChatTurn, error)

Chat sends a message to a conversation session and collects the turn's SSE events until "end" (see ChatTurn). An "error" event does not make Chat fail: it is collected in Errors and OK is false. The returned error is a request or stream error.

func (*ParlyrNamespace) ChatStream 

func (p *ParlyrNamespace) ChatStream(ctx context.Context, sessionID, message string) (<-chan ChatEvent, error)

ChatStream opens an SSE stream for a conversation turn and yields its events as they arrive (see ChatEvent). The channel closes at the API's "end" event (not yielded), when the stream ends, or when ctx is cancelled; if reading the stream failed, the last event carries Err.

func (*ParlyrNamespace) DeleteSession 

func (p *ParlyrNamespace) DeleteSession(ctx context.Context, sessionID string) error

DeleteSession removes a conversation session.

func (*ParlyrNamespace) Parse 

func (p *ParlyrNamespace) Parse(ctx context.Context, message string, opts *ParseOptions) (*ParseResult, error)

Parse interprets a natural language message.

type ParseOptions 

type ParseOptions struct {
	SessionID string
}

ParseOptions configures optional parameters for Parlyr.Parse.

type ParseResult 

type ParseResult struct {
	Intent              string
	Confidence          float64
	Tier                int
	Source              string
	Entities            map[string]interface{}
	RequiresAction      bool
	ClarificationNeeded *string
	Reasoning           *string
	DetectedURL         *string
}

ParseResult is returned by Parlyr.Parse.

type PaymentRequiredError added in v0.3.0 

type PaymentRequiredError struct {
	KloakdError
	UpgradeURL string
}

PaymentRequiredError is returned for HTTP 402: the organisation's paid plan has a failed payment; new billable work is refused until it is paid.

func (*PaymentRequiredError) Error added in v0.3.0 

func (e *PaymentRequiredError) Error() string

type PipelineEvent added in v0.5.0 

type PipelineEvent struct {
	PipelineID  string
	EventType   string // e.g. "stage_started", "pipeline_complete", "pipeline_error"
	StageID     string
	Status      string
	Phase       string
	Message     string
	SubStatus   string
	ProgressPct *float64
	DurationMs  *float64
	Metadata    map[string]interface{}
	// Error is the event's PipelineError ({error_code, category, severity,
	// user_message, technical_message, recovery_actions, ...}), nil when none.
	Error     map[string]interface{}
	Timestamp string
	Sequence  int
	// Raw is the event's JSON as received.
	Raw map[string]interface{}
	Err error
}

PipelineEvent is one event of a pipeline's SSE stream (a crawl's webgrph/crawl/{id}/events, and an Evadr job's events when the job has a live pipeline). Its fields are those of the API's PipelineEvent (core/pipeline/models.py). Err is set only on the last value of a stream whose reading failed; the other fields are then empty.

type PollOptions added in v0.4.0 

type PollOptions struct {
	Interval time.Duration
	Timeout  time.Duration
}

PollOptions bounds the following of a handed-off request. Zero values mean the defaults (3 s first interval, 15 min overall).

type QuotaExceededError added in v0.3.0 

type QuotaExceededError struct {
	KloakdError
	Tier               string
	UpgradeURL         string
	GBUsed             float64
	GBLimit            float64
	PurchasedGBBalance float64
}

QuotaExceededError is returned for HTTP 429 "gb_exhausted": the plan's GB allocation and this period's top-ups are used up. Retrying does not help; buy a top-up or upgrade.

func (*QuotaExceededError) Error added in v0.3.0 

func (e *QuotaExceededError) Error() string

type RateLimitError 

type RateLimitError struct {
	KloakdError
	RetryAfter     int    // seconds
	ResetAt        string // ISO timestamp
	QuotaExhausted bool
	QuotaUsed      int
	QuotaLimit     int
	UpgradeURL     string
}

RateLimitError is returned for HTTP 429 "rate_limited". For the tier's request rate, RetryAfter (seconds) and ResetAt say when to retry; the SDK retries it itself up to Config.MaxRetries. QuotaExhausted is true for the playground key's permanent quota (retry_after 0, no reset): waiting does not clear it, so it is not retried; QuotaUsed, QuotaLimit and UpgradeURL describe it.

func (*RateLimitError) Error 

func (e *RateLimitError) Error() string

type ReaderViewOptions added in v0.5.0 

type ReaderViewOptions struct {
	// CrawlID reads the page from that crawl's cache when it has it.
	CrawlID string
	// SessionArtifactID fetches with an authenticated Fetchyr session.
	SessionArtifactID string
}

ReaderViewOptions configures optional parameters for Webgrph.ReaderView.

type ReasonOptions added in v0.5.0 

type ReasonOptions struct {
	// TimeoutSecs bounds the reasoning call (timeout_secs); 0 for the API's default.
	TimeoutSecs float64
	// PerceptionID is a perception from a prior Analyze.
	PerceptionID string
}

ReasonOptions configures optional parameters for Nexus.Reason.

type ResourceListOptions added in v0.5.0 

type ResourceListOptions struct {
	ResourceType string
	Limit        int
	Offset       int
}

ResourceListOptions filters a resource list.

type ResourcesOptions added in v0.5.0 

type ResourcesOptions struct {
	SiteID       string
	ResourceType string
	Limit        int
	Offset       int
}

ResourcesOptions filters Skanyr.ListResources.

type SchemaField added in v0.5.0 

type SchemaField struct {
	Name      string
	Selector  string // CSS selector
	Attribute string // attribute to read; "" for the text
	Type      string // "" means the API's default, "text"
}

SchemaField is one field of an extraction schema (schema_spec.fields).

type SearchOptions added in v0.3.0 

type SearchOptions struct {
	SiteID   string
	MimeType string
	Limit    int
	Offset   int
}

SearchOptions configures optional parameters for Skanyr.Search.

type SessionResult 

type SessionResult struct {
	Success      bool
	SessionID    string
	ArtifactID   *string
	TargetDomain *string
	AuthMethod   *string
	Error        *string
}

SessionResult is returned by Fetchyr.Authenticate.

type SiteCrawlResult added in v0.3.0 

type SiteCrawlResult struct {
	Success              bool
	URL                  string
	TotalPagesDiscovered int
	PagesFetched         int
	PagesFailed          int
	Pages                []CrawlPage
	CrawlArtifactID      *string
	Error                *string
}

SiteCrawlResult is returned by Client.Crawl. Success and Error are the discovery crawl's own.

type SiteDependency added in v0.5.0 

type SiteDependency struct {
	DependentSite    string
	DependencySite   string
	DependencyType   string
	Condition        string
	DataRequirements []string
	// TimeoutSecs is the longest wait for the dependency; 0 for none.
	TimeoutSecs int
}

SiteDependency orders two sites of a multi-site workflow: DependentSite runs after DependencySite (site_id values). DependencyType is "sequential" (the API's default), "data" or "conditional".

type SiteSearchOptions added in v0.5.0 

type SiteSearchOptions struct {
	Limit    int
	Offset   int
	MimeType string
}

SiteSearchOptions configures Skanyr.SearchSite.

type SkanyrNamespace 

type SkanyrNamespace struct {
	// contains filtered or unexported fields
}

SkanyrNamespace exposes the Site Knowledge module: sites built from crawls, their resources, search, change history, the resource graph and technology findings (modules/skanyr/router.py).

Page discovery is Webgrph.Crawl; the session, detected-API and reader-view methods are on Webgrph. Access via client.Skanyr.

func (*SkanyrNamespace) CreateSiteFromCrawl added in v0.3.0 

func (s *SkanyrNamespace) CreateSiteFromCrawl(ctx context.Context, crawlID string) (map[string]interface{}, error)

CreateSiteFromCrawl builds (or updates) a site from a finished Webgrph crawl.

func (*SkanyrNamespace) DeleteSite added in v0.3.0 

func (s *SkanyrNamespace) DeleteSite(ctx context.Context, siteID string) error

DeleteSite deletes a site.

func (*SkanyrNamespace) Diff added in v0.5.0 

func (s *SkanyrNamespace) Diff(ctx context.Context, siteID, snapshotA, snapshotB string) (map[string]interface{}, error)

Diff compares two snapshots of a site ("" for the API's defaults).

func (*SkanyrNamespace) ExportSite added in v0.5.0 

func (s *SkanyrNamespace) ExportSite(ctx context.Context, siteID string, opts *ExportOptions) ([]byte, error)

ExportSite exports a site's captured content and returns the ZIP archive.

func (*SkanyrNamespace) Extract added in v0.3.0 

func (s *SkanyrNamespace) Extract(ctx context.Context, contentHash, mimeType string, metadata map[string]interface{}) (map[string]interface{}, error)

Extract extracts text and structure from stored content (by its content hash).

func (*SkanyrNamespace) GetExtractionStructure added in v0.5.0 

func (s *SkanyrNamespace) GetExtractionStructure(ctx context.Context, contentHash string) (map[string]interface{}, error)

GetExtractionStructure gets an extraction's structured content (headings, tables, links, metadata).

func (*SkanyrNamespace) GetExtractionText added in v0.5.0 

func (s *SkanyrNamespace) GetExtractionText(ctx context.Context, contentHash string) (string, error)

GetExtractionText gets an extraction's plain text (the API answers text/plain).

func (*SkanyrNamespace) GetFindings added in v0.5.0 

func (s *SkanyrNamespace) GetFindings(ctx context.Context, siteID string, opts *FindingsOptions) ([]map[string]interface{}, error)

GetFindings lists a site's findings.

func (*SkanyrNamespace) GetGraph added in v0.5.0 

func (s *SkanyrNamespace) GetGraph(ctx context.Context, siteID string, opts *GraphOptions) (map[string]interface{}, error)

GetGraph gets a site's resource graph.

func (*SkanyrNamespace) GetGraphStats added in v0.5.0 

func (s *SkanyrNamespace) GetGraphStats(ctx context.Context, siteID string) (map[string]interface{}, error)

GetGraphStats gets a site's graph statistics.

func (*SkanyrNamespace) GetResource added in v0.5.0 

func (s *SkanyrNamespace) GetResource(ctx context.Context, resourceID string) (map[string]interface{}, error)

GetResource gets one resource.

func (*SkanyrNamespace) GetSite added in v0.3.0 

func (s *SkanyrNamespace) GetSite(ctx context.Context, siteID string) (map[string]interface{}, error)

GetSite gets a site.

func (*SkanyrNamespace) GetSubgraph added in v0.5.0 

func (s *SkanyrNamespace) GetSubgraph(ctx context.Context, siteID, resourceID string, hops int) (map[string]interface{}, error)

GetSubgraph gets the graph around one resource, hops away (0: the API's default, 2).

func (*SkanyrNamespace) GetTechnologies added in v0.5.0 

func (s *SkanyrNamespace) GetTechnologies(ctx context.Context, siteID string) ([]map[string]interface{}, error)

GetTechnologies lists the technologies detected on a site, aggregated.

func (*SkanyrNamespace) GetTechnologyEvidence added in v0.5.0 

func (s *SkanyrNamespace) GetTechnologyEvidence(ctx context.Context, siteID, subject string, opts *TechnologyEvidenceOptions) (map[string]interface{}, error)

GetTechnologyEvidence lists the evidence for one technology (subject, e.g. "React") on a site: {total, subject, evidence}.

func (*SkanyrNamespace) GetTimeline added in v0.5.0 

func (s *SkanyrNamespace) GetTimeline(ctx context.Context, siteID string, limit int) (map[string]interface{}, error)

GetTimeline gets a site's change timeline (limit 0: the API's default).

func (*SkanyrNamespace) GetTimelineSnapshot added in v0.5.0 

func (s *SkanyrNamespace) GetTimelineSnapshot(ctx context.Context, siteID, snapshotID string) (map[string]interface{}, error)

GetTimelineSnapshot gets one snapshot of a site's timeline, with its entries.

func (*SkanyrNamespace) ListResources added in v0.5.0 

func (s *SkanyrNamespace) ListResources(ctx context.Context, opts *ResourcesOptions) ([]map[string]interface{}, error)

ListResources lists the organisation's resources (of one site when SiteID is set).

func (*SkanyrNamespace) ListSites added in v0.3.0 

func (s *SkanyrNamespace) ListSites(ctx context.Context) ([]map[string]interface{}, error)

ListSites lists the organisation's sites.

func (*SkanyrNamespace) ListSnapshots added in v0.5.0 

func (s *SkanyrNamespace) ListSnapshots(ctx context.Context, siteID string, limit int) (map[string]interface{}, error)

ListSnapshots lists a site's sync snapshots (limit 0: the API's default).

func (*SkanyrNamespace) Search added in v0.3.0 

func (s *SkanyrNamespace) Search(ctx context.Context, query string, opts *SearchOptions) (map[string]interface{}, error)

Search searches the organisation's extracted content.

func (*SkanyrNamespace) SearchSite added in v0.5.0 

func (s *SkanyrNamespace) SearchSite(ctx context.Context, siteID, query string, opts *SiteSearchOptions) (map[string]interface{}, error)

SearchSite searches one site's content.

func (*SkanyrNamespace) SetSitePolicy deprecated added in v0.3.0 

func (s *SkanyrNamespace) SetSitePolicy(ctx context.Context, siteID, policy string) (map[string]interface{}, error)

SetSitePolicy is UpdateSite.

Deprecated: use UpdateSite.

func (*SkanyrNamespace) SiteAPIs added in v0.3.0 

func (s *SkanyrNamespace) SiteAPIs(ctx context.Context, siteID string, page *ListPage) ([]map[string]interface{}, error)

SiteAPIs lists the APIs found on a site.

func (*SkanyrNamespace) SiteResources added in v0.3.0 

func (s *SkanyrNamespace) SiteResources(ctx context.Context, siteID string, opts *ResourceListOptions) ([]map[string]interface{}, error)

SiteResources lists a site's resources.

func (*SkanyrNamespace) Sync added in v0.3.0 

func (s *SkanyrNamespace) Sync(ctx context.Context, siteID string, opts *SyncOptions) (map[string]interface{}, error)

Sync re-fetches a site's changed resources.

func (*SkanyrNamespace) UpdateSite added in v0.5.0 

func (s *SkanyrNamespace) UpdateSite(ctx context.Context, siteID, policy string) (map[string]interface{}, error)

UpdateSite sets a site's default sync policy ("full", "incremental", "conditional" or "head_first"), the one setting the API lets you change.

type SyncOptions added in v0.3.0 

type SyncOptions struct {
	URLs []string
	// Policy is "full", "incremental", "conditional" or "head_first"
	// (default: the site's policy, else incremental).
	Policy string
}

SyncOptions configures optional parameters for Skanyr.Sync.

type TechnologyEvidenceOptions added in v0.5.0 

type TechnologyEvidenceOptions struct {
	Detector string
	Limit    int
	Offset   int
}

TechnologyEvidenceOptions filters Skanyr.GetTechnologyEvidence.

type UpstreamError 

type UpstreamError struct {
	KloakdError
}

UpstreamError is returned for HTTP 502 — upstream site fetch failed.

func (*UpstreamError) Error 

func (e *UpstreamError) Error() string

type WebgrphNamespace 

type WebgrphNamespace struct {
	// contains filtered or unexported fields
}

WebgrphNamespace exposes the Site Mapping & Discovery module. Access via client.Webgrph.

func (*WebgrphNamespace) CancelCrawl added in v0.3.0 

func (w *WebgrphNamespace) CancelCrawl(ctx context.Context, crawlID string) (map[string]interface{}, error)

CancelCrawl cancels a running crawl.

func (*WebgrphNamespace) Crawl 

func (w *WebgrphNamespace) Crawl(ctx context.Context, targetURL string, opts *CrawlOptions) (*CrawlResult, error)

Crawl starts a site crawl, waits for it to finish, and returns every discovered page (GET webgrph/crawl/{id}/pages, followed to the end).

func (*WebgrphNamespace) CrawlAll 

func (w *WebgrphNamespace) CrawlAll(ctx context.Context, targetURL string, opts *CrawlOptions) ([]PageNode, error)

CrawlAll is Crawl, returning only the pages.

func (*WebgrphNamespace) CrawlEvents added in v0.3.0 

func (w *WebgrphNamespace) CrawlEvents(ctx context.Context, crawlID string) (<-chan CrawlEvent, error)

CrawlEvents streams a crawl's pipeline events (SSE). The channel closes when the stream ends or ctx is cancelled; if reading the stream failed, the last event carries Err.

func (*WebgrphNamespace) CrawlStream 

func (w *WebgrphNamespace) CrawlStream(ctx context.Context, targetURL string, opts *CrawlStreamOptions) (<-chan CrawlEvent, error)

CrawlStream starts a crawl and streams its events (GET webgrph/crawl/{crawl_id}/events). The channel closes when the crawl ends or ctx is cancelled; if reading the stream failed, the last event carries Err.

func (*WebgrphNamespace) DeleteSession added in v0.3.0 

func (w *WebgrphNamespace) DeleteSession(ctx context.Context, sessionID string) error

DeleteSession deletes a discovery session.

func (*WebgrphNamespace) DetectedAPIs added in v0.3.0 

func (w *WebgrphNamespace) DetectedAPIs(ctx context.Context, pageURL string) (map[string]interface{}, error)

DetectedAPIs lists the APIs detected on a page.

func (*WebgrphNamespace) EndSession added in v0.3.0 

func (w *WebgrphNamespace) EndSession(ctx context.Context, sessionID string) (map[string]interface{}, error)

EndSession ends a discovery session.

func (*WebgrphNamespace) GetCrawlPages added in v0.2.0 

func (w *WebgrphNamespace) GetCrawlPages(ctx context.Context, crawlID, cursor string, limit int) (map[string]interface{}, error)

GetCrawlPages gets one page of discovered pages (cursor "" for the first; limit 0 for the API's default of 100, else 1–1000). Only fetched pages are listed; ListCrawlPages has IncludeGrouping and typed nodes.

func (*WebgrphNamespace) GetCrawlStatus added in v0.2.0 

func (w *WebgrphNamespace) GetCrawlStatus(ctx context.Context, crawlID string) (map[string]interface{}, error)

GetCrawlStatus polls crawl status and stats.

func (*WebgrphNamespace) GetSession added in v0.3.0 

func (w *WebgrphNamespace) GetSession(ctx context.Context, sessionID string) (map[string]interface{}, error)

GetSession gets a discovery session.

func (*WebgrphNamespace) ListCrawlPages added in v0.6.0 

func (w *WebgrphNamespace) ListCrawlPages(ctx context.Context, crawlID string, opts *CrawlPagesOptions) (*CrawlPagesResult, error)

ListCrawlPages gets one page of a crawl's pages (GET webgrph/crawl/{id}/pages) as PageNode values (0.6.0). A Limit outside 1–CrawlPagesMaxLimit is refused before any request is sent.

func (*WebgrphNamespace) ListSessions added in v0.3.0 

func (w *WebgrphNamespace) ListSessions(ctx context.Context, activeOnly bool) (map[string]interface{}, error)

ListSessions lists discovery sessions (items under "items"): only the active ones when activeOnly is true, all of them when false. (The API's default is active only; active_only is always sent.)

func (*WebgrphNamespace) ReaderView added in v0.3.0 

func (w *WebgrphNamespace) ReaderView(ctx context.Context, targetURL string, opts *ReaderViewOptions) (map[string]interface{}, error)

ReaderView returns a page's readable content.

func (*WebgrphNamespace) SaveSession added in v0.3.0 

func (w *WebgrphNamespace) SaveSession(ctx context.Context, session map[string]interface{}) (map[string]interface{}, error)

SaveSession creates or updates a discovery session ("url" is required).

func (*WebgrphNamespace) StartCrawl added in v0.3.0 

func (w *WebgrphNamespace) StartCrawl(ctx context.Context, targetURL string, opts *CrawlOptions) (*CrawlResult, error)

StartCrawl starts a site crawl and returns at once (HTTP 202) with its crawl ID; follow it with GetCrawlStatus, CrawlEvents or Crawl.

func (*WebgrphNamespace) UpdateSessionJob added in v0.3.0 

func (w *WebgrphNamespace) UpdateSessionJob(ctx context.Context, sessionID, jobID string) (map[string]interface{}, error)

UpdateSessionJob links a job to a discovery session.

type WorkflowResult 

type WorkflowResult struct {
	WorkflowID   string
	Name         string
	Description  *string
	WorkflowType string
	Steps        []map[string]interface{}
	IsActive     bool
	CreatedAt    string
	Error        *string
}

WorkflowResult is returned by Fetchyr.CreateWorkflow.

Source Files

View all Source files

Directories

Path Synopsis
examples
quickstart command
Quickstart: Discover and extract from a site in ~10 lines.
 Quickstart: Discover and extract from a site in ~10 lines.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.