urlpipe

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 27, 2026 License: MIT Imports: 18 Imported by: 0

README

urlpipe-go

Turn any URL into clean data from Go: Markdown, rendered HTML, a full-page screenshot, metadata, a summary, keywords, console errors or a Lighthouse audit, each page rendered in real Chrome.

This is the official Go client for URLpipe. It uses the standard library only.

Install

go get github.com/URLpipe/urlpipe-go

Requires Go 1.21 or later.

Quickstart

Grab a project API key from your dashboard and put it in URLPIPE_API_KEY. The Free plan gives you 1,000 credits a month, no card needed.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	client, err := urlpipe.NewClient() // reads URLPIPE_API_KEY
	if err != nil {
		log.Fatal(err)
	}

	res, err := client.Markdown(context.Background(), "https://example.com", nil)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(res.Data)
}

Every call waits for the result and hands it back typed. The API itself defaults to async; the client sends sync: true for you, because a library caller almost always wants the answer. See Async and Wait to opt out.

The client

client, err := urlpipe.NewClient(
	urlpipe.WithAPIKey("..."),              // default: $URLPIPE_API_KEY
	urlpipe.WithBaseURL("https://urlpipe.dev"),
	urlpipe.WithTimeout(90*time.Second),    // per HTTP request
	urlpipe.WithMaxRetries(2),              // transient failures
	urlpipe.WithWaitTimeout(5*time.Minute), // how long a long analysis is polled for
	urlpipe.WithHTTPClient(&http.Client{}), // your own transport
)

NewClient returns urlpipe.ErrMissingAPIKey when there is no key. A *Client is safe to share between goroutines.

Methods

Every method takes a context.Context first and the URL second, and returns a *urlpipe.Response[T]:

type Response[T any] struct {
	Status Status            // "completed", "accepted" or "processing"
	Data   T                 // the typed result, set when Status is "completed"
	Token  string            // fetch the result again for free, for 30 days
	Labels map[string]string // the labels the request was made with
	Meta   Meta              // cache, credits, timing: see below
}

ctx := context.Background()

md, _ := client.Markdown(ctx, url, nil)     // Data: string
html, _ := client.HTML(ctx, url, nil)       // Data: string, the HTML after JavaScript ran
sum, _ := client.Summarize(ctx, url, nil)   // Data: string, Markdown
meta, _ := client.Meta(ctx, url, nil)       // Data: *urlpipe.Metadata
kw, _ := client.Keywords(ctx, url, nil)     // Data: []string
logs, _ := client.Console(ctx, url, nil)    // Data: []urlpipe.ConsoleEntry

fmt.Println(meta.Data.Title, meta.Data.Language, meta.Data.PublicationDate)
for _, e := range logs.Data {
	fmt.Println(e.Type, e.Text) // "error", "warning" or "exception"
}
Screenshot

The image arrives decoded, with its format read off the bytes and a link that needs no API key:

shot, err := client.Screenshot(ctx, "https://example.com", &urlpipe.ScreenshotOptions{
	Screenshot: map[string]any{"format": "webp", "viewport_width": 390, "device_scale_factor": 2},
})
if err != nil {
	log.Fatal(err)
}
fmt.Println(shot.Data.MIMEType)  // image/webp
fmt.Println(shot.Data.ResultURL) // put it straight in an <img>
err = shot.Data.Save("example.webp")
Lighthouse
audit, err := client.Lighthouse(ctx, "https://example.com", &urlpipe.LighthouseOptions{
	Device:        "desktop", // default "mobile"
	IncludeAudits: true,      // add all 150+ audits
})
fmt.Println(*audit.Data.Categories["performance"].Score)
fmt.Println(audit.Data.Metrics["largest-contentful-paint"].DisplayValue)
// audit.Data.Raw holds the whole JSON for anything else.
Scrape: several results off one page visit
res, err := client.Scrape(ctx, "https://example.com",
	[]urlpipe.Operation{urlpipe.OperationMarkdown, urlpipe.OperationMeta, urlpipe.OperationScreenshot},
	nil)
if err != nil {
	log.Fatal(err)
}
md, err := res.Data.Markdown()
meta, err := res.Data.Meta()
shot, err := res.Data.Screenshot() // decoded, like Client.Screenshot

res.Data.Order lists the operations in the order you asked for them. Operations fail independently. An accessor returns a *urlpipe.OperationError for an operation that failed; its Pending() is true for a Lighthouse audit that was still running (fetch the scrape again with client.Result(ctx, res.Token, urlpipe.OperationScrape)). When every operation fails, Scrape returns an *urlpipe.AnalysisFailedError whose Scrape field holds each operation's error.

Options

Every method takes an options struct; nil means the defaults, and only the fields you set are sent.

res, err := client.Markdown(ctx, url, &urlpipe.Options{
	MaxAge:         urlpipe.MaxAgeString("3 days"), // or MaxAgeSeconds(3600), MaxAgeDuration(time.Hour)
	Labels:         map[string]string{"client": "acme"},
	Residential:    true,
	PageOptions:    map[string]any{"block_ads": true, "block_cookie_banners": true, "remove_selectors": []string{".newsletter"}},
	IdempotencyKey: "order-1234",
	Extra:          map[string]any{"some_new_param": true}, // merged into the body as-is
})

ScreenshotOptions, LighthouseOptions and ScrapeOptions embed Options and add their own fields. Cached results are free: MaxAgeSeconds(0) forces a fresh analysis, and a wider MaxAge lowers your bill.

Meta

res.Meta is what the response headers said. A header the API did not send is nil (or ""), never a panic.

m := res.Meta
m.Cache              // "hit", "miss" or "partial"
m.CacheAge           // *int seconds, on a hit
m.ProcessingTimeMs   // *int
m.Quota.Cost         // *int credits this call spent
m.Quota.Remaining    // *Amount: .Value, or .Unlimited on an unlimited plan
m.Quota.Overage      // *int
m.Quota.ResetsAt     // *time.Time
m.ConcurrencyLimit   // *Amount: size your worker pool from it
m.IdempotentReplayed // true when an Idempotency-Key replayed an earlier answer

Async and Wait

Set Async: true to get a token back at once and collect the result later, from a webhook or with Wait:

accepted, err := client.Lighthouse(ctx, url, &urlpipe.LighthouseOptions{
	Options: urlpipe.Options{Async: true, ReportTo: "https://example.com/webhooks/urlpipe"},
})
// accepted.Status == urlpipe.StatusAccepted, accepted.Token is set

res, err := client.Wait(ctx, accepted.Token, urlpipe.OperationLighthouse, &urlpipe.WaitOptions{
	Timeout: 5 * time.Minute, Interval: 2 * time.Second,
})
audit := res.Data.(*urlpipe.Lighthouse)

GET /result/:token does not say which operation made a token, so Result and Wait take the operation as a hint and type Data as that operation's method would. Pass "" and a JSON result decodes into any while a text result stays a string, so a screenshot stays Base64: use client.Wait(ctx, token, urlpipe.OperationScreenshot, nil) to get a decoded *Screenshot.

client.Result(ctx, token, op) is a single fetch: a finished result comes back with StatusCompleted, one still running with StatusProcessing and no error.

You rarely need Wait for sync calls. The API holds a sync request for 60 seconds; when an analysis takes longer, the client polls for the result every 2 seconds for up to the wait timeout, so you still see one call that returned the result. If the wait runs out, the error matches urlpipe.ErrWaitTimeout and carries the token to collect it later.

Webhooks

VerifyWebhook checks a signed delivery and returns its payload. It needs no client:

http.HandleFunc("/webhooks/urlpipe", func(w http.ResponseWriter, r *http.Request) {
	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "unreadable body", http.StatusBadRequest)
		return
	}
	event, err := urlpipe.VerifyWebhook(body, r.Header, os.Getenv("URLPIPE_WEBHOOK_SECRET"), 5*time.Minute)
	if err != nil {
		http.Error(w, err.Error(), http.StatusUnauthorized)
		return
	}
	data, _ := event.Data() // typed by event.Operation: string, *Screenshot, *Metadata, ...
	log.Println(event.Token, event.Success, data)
	w.WriteHeader(http.StatusNoContent)
})

Give it the raw request body, the exact bytes received. Decoding the JSON and encoding it again changes the bytes, and the signature no longer matches. It accepts a delivery when any v1= signature matches (so a secret rotation verifies with either secret), ignores other schemes, and refuses a timestamp further than the tolerance from now (zero means 5 minutes). A refusal is a *urlpipe.WebhookVerificationError with a Reason. Holding the two header values instead of an http.Header? Use VerifyWebhookSignature(body, timestamp, signature, secret, tolerance).

Errors

Every API and transport error is an *urlpipe.Error with Kind, Status, Code, Message, Body and Token. Test for a kind with errors.Is, and reach the extra fields with errors.As:

_, err := client.Summarize(ctx, url, nil)

var quota *urlpipe.QuotaExceededError
var apiErr *urlpipe.Error
switch {
case errors.As(err, &quota):
	fmt.Printf("needs %d credits; the allowance resets %s\n", quota.Needed, quota.ResetsAt)
case errors.Is(err, urlpipe.ErrAnalysisFailed):
	fmt.Println("the page could not be analysed:", err)
case errors.As(err, &apiErr):
	fmt.Println(apiErr.Kind, apiErr.Status, apiErr.Code, apiErr.Message)
}
Sentinel When Detailed type
ErrAuthentication 401: the key is missing or not active
ErrEmailUnverified 403: confirm the email on the account
ErrInvalidRequest 422 with a code: invalid_url, invalid_options, ...
ErrAnalysisFailed 422: the page could not be analysed; the message says why *AnalysisFailedError (Scrape)
ErrQuotaExceeded 429: the Free plan's credits are spent *QuotaExceededError (Limit, Used, Needed, ResetsAt)
ErrConcurrencyLimit 429: every parallel request is running *ConcurrencyLimitError (Limit, Running)
ErrRateLimited 429: sending too fast *RateLimitedError (RetryAfter)
ErrNotFound 404: no result for this token
ErrStaleResult 410: the result is past its 30 days
ErrServer 5xx
ErrConnection the API could not be reached (Unwrap gives the cause)
ErrWaitTimeout the analysis was still running when the wait ended (Token)

Any other answer, such as a 403 other than email_unverified or a 429 with an unknown code, is a plain *urlpipe.Error with Kind urlpipe.KindUnexpected. Code is set only when the body's error is a code like invalid_url; when it is a sentence, the sentence is in Message. When every operation of a scrape fails, Message reads Every operation failed: <op>: <error>; ....

errors.As(err, &apiErr) also works on the detailed types. A cancelled context comes back as the context's own error.

Retries and idempotency

Connection errors, 500/502/503 and concurrency_limit are retried after 1 s, 2 s, 4 s, ... (at most 60 s); rate_limited after the Retry-After it names (else 1 s, at most 60 s). That happens up to WithMaxRetries times (default 2). A 401, 403, 404, 410, 422, 504, quota_exceeded or any other 429 is never retried.

A retry is only safe if it cannot run the work twice, so every analysis the client might retry carries an Idempotency-Key: yours from Options.IdempotencyKey, or a UUID the client generates for that call and reuses for each of its retries. The API answers a repeated key with the first request's token and result, charged once and delivered to your webhook once. WithMaxRetries(0) turns retries off, and the generated key with them.

License

MIT, see LICENSE.

Documentation

Overview

Package urlpipe is the official Go client for the URLpipe API (https://urlpipe.dev), which turns a URL into clean data: Markdown, rendered HTML, a full-page screenshot, metadata, a summary, keywords, console errors, a Lighthouse audit, or several of these off one page visit.

Create a client once and share it; it is safe for concurrent use: 

client, err := urlpipe.NewClient() // reads URLPIPE_API_KEY
if err != nil {
	log.Fatal(err)
}
res, err := client.Markdown(ctx, "https://example.com", nil)
if err != nil {
	log.Fatal(err)
}
fmt.Println(res.Data)

Every analysis method waits for the result by default (the API itself defaults to async). Set Options.Async to get a token back straight away and collect the result later with Client.Wait or a webhook.

Requests that fail for a transient reason are retried, and every retried analysis carries the same Idempotency-Key, so a retry never runs or bills the work twice.

Homepage: https://urlpipe.dev. Documentation: https://urlpipe.dev/docs. Source: https://github.com/URLpipe/urlpipe-go. Contact: contact@urlpipe.dev.

Example (Errors) 
package main

import (
	"context"
	"errors"
	"fmt"
	"log"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	client, err := urlpipe.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	_, err = client.Summarize(context.Background(), "https://example.com", nil)

	var quota *urlpipe.QuotaExceededError
	var apiErr *urlpipe.Error
	switch {
	case err == nil:
		fmt.Println("done")
	case errors.As(err, &quota):
		fmt.Printf("needs %d credits, %d left until %s\n", quota.Needed, quota.Limit-quota.Used, quota.ResetsAt)
	case errors.Is(err, urlpipe.ErrAnalysisFailed):
		fmt.Println("the page could not be analysed:", err)
	case errors.As(err, &apiErr):
		fmt.Println(apiErr.Kind, apiErr.Status, apiErr.Code, apiErr.Message)
	default:
		log.Fatal(err)
	}
}

Index

Examples

Constants

View Source 
const (
	ConsoleError     = "error"
	ConsoleWarning   = "warning"
	ConsoleException = "exception"
)

Console message types.

View Source 
const (
	// DefaultTimeout bounds each HTTP request. The API holds a sync request
	// for up to 60 seconds, so this sits above that.
	DefaultTimeout = 90 * time.Second
	// DefaultMaxRetries is how many times a transient failure is retried.
	DefaultMaxRetries = 2
	// DefaultWaitTimeout bounds how long a long analysis is polled for.
	DefaultWaitTimeout = 300 * time.Second
	// DefaultPollInterval is the pause between two polls of GET /result/:token.
	DefaultPollInterval = 2 * time.Second
)

Defaults used by NewClient when the matching option is not given.

View Source 
const (
	TimestampHeader = "X-URLpipe-Timestamp"
	SignatureHeader = "X-URLpipe-Signature"
)

Webhook signature headers.

View Source 
const APIKeyEnv = "URLPIPE_API_KEY"

APIKeyEnv is the environment variable read when no key is passed to NewClient.

View Source 
const DefaultBaseURL = "https://urlpipe.dev"

DefaultBaseURL is where the URLpipe API lives.

View Source 
const DefaultWebhookTolerance = 5 * time.Minute

DefaultWebhookTolerance is how old (or how far in the future) a delivery's timestamp may be when VerifyWebhook is given a tolerance of zero.

View Source 
const Version = "0.1.0"

Version is the version of this library.

Variables

View Source 
var (
	// ErrAuthentication: the API key is missing or not an active project key (401).
	ErrAuthentication error = &kindSentinel{KindAuthentication}
	// ErrEmailUnverified: the key is valid, but the account's email address
	// has not been confirmed yet (403).
	ErrEmailUnverified error = &kindSentinel{KindEmailUnverified}
	// ErrInvalidRequest: a parameter was refused (422 with a code).
	ErrInvalidRequest error = &kindSentinel{KindInvalidRequest}
	// ErrAnalysisFailed: the page could not be analysed (422); the message says why.
	ErrAnalysisFailed error = &kindSentinel{KindAnalysisFailed}
	// ErrQuotaExceeded: the Free plan's monthly credits are spent (429).
	ErrQuotaExceeded error = &kindSentinel{KindQuotaExceeded}
	// ErrConcurrencyLimit: the plan's parallel requests are all running (429).
	ErrConcurrencyLimit error = &kindSentinel{KindConcurrencyLimit}
	// ErrRateLimited: requests are arriving too fast (429).
	ErrRateLimited error = &kindSentinel{KindRateLimited}
	// ErrNotFound: no result for this token under the project (404).
	ErrNotFound error = &kindSentinel{KindNotFound}
	// ErrStaleResult: the result is past the 30-day retention window (410).
	ErrStaleResult error = &kindSentinel{KindStaleResult}
	// ErrServer: the API answered with a 5xx.
	ErrServer error = &kindSentinel{KindServer}
	// ErrConnection: the API could not be reached.
	ErrConnection error = &kindSentinel{KindConnection}
	// ErrWaitTimeout: the analysis was still running when the wait ended.
	ErrWaitTimeout error = &kindSentinel{KindWaitTimeout}
)

Sentinels to test an error's kind with errors.Is: 

if errors.Is(err, urlpipe.ErrQuotaExceeded) { ... }
View Source 
var ErrMissingAPIKey = errors.New("urlpipe: an API key is required: pass urlpipe.WithAPIKey or set " + APIKeyEnv + " (find it in your project's settings at https://urlpipe.dev)")

ErrMissingAPIKey is returned by NewClient when no API key was given and URLPIPE_API_KEY is empty.

View Source 
var ErrWebhookVerification error = &kindSentinel{"webhook_verification"}

ErrWebhookVerification matches every *WebhookVerificationError.

Functions

This section is empty.

Types

type Amount 

type Amount struct {
	// Value is the number, when there is one.
	Value int
	// Unlimited is true on an unlimited plan.
	Unlimited bool
	// Raw is the value exactly as the API sent it.
	Raw string
}

Amount is a quota figure that is either a number or "unlimited".

func (*Amount) String 

func (a *Amount) String() string

func (*Amount) UnmarshalJSON 

func (a *Amount) UnmarshalJSON(b []byte) error

UnmarshalJSON reads a number or a string such as "unlimited".

type AnalysisFailedError 

type AnalysisFailedError struct {

	// Scrape is set when a /scrape failed in every operation: the body is
	// the scrape object, with each operation's error.
	Scrape *ScrapeResult
	// contains filtered or unexported fields
}

AnalysisFailedError is a 422 where the page could not be analysed; the message is the API's sentence saying why.

func (*AnalysisFailedError) Unwrap 

func (e *AnalysisFailedError) Unwrap() error

Unwrap returns the underlying *Error.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client talks to the URLpipe API. Build one with NewClient; it is safe for concurrent use by multiple goroutines.

func NewClient 

func NewClient(opts ...Option) (*Client, error)

NewClient builds a client. It returns ErrMissingAPIKey when no API key was given with WithAPIKey and URLPIPE_API_KEY is empty.

Example 
package main

import (
	"log"
	"os"
	"time"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	// Reads the key from URLPIPE_API_KEY.
	client, err := urlpipe.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	_ = client

	// Or pass everything explicitly.
	client, err = urlpipe.NewClient(
		urlpipe.WithAPIKey(os.Getenv("MY_URLPIPE_KEY")),
		urlpipe.WithTimeout(2*time.Minute),
		urlpipe.WithMaxRetries(3),
	)
	if err != nil {
		log.Fatal(err)
	}
	_ = client
}

func (*Client) Console 

func (c *Client) Console(ctx context.Context, url string, opts *Options) (*Response[[]ConsoleEntry], error)

Console returns the errors, warnings and uncaught exceptions the page logs while it loads (1 credit). An empty list means a clean page.

func (*Client) HTML 

func (c *Client) HTML(ctx context.Context, url string, opts *Options) (*Response[string], error)

HTML returns the page's HTML after Chrome has rendered it (1 credit).

func (*Client) Keywords 

func (c *Client) Keywords(ctx context.Context, url string, opts *Options) (*Response[[]string], error)

Keywords returns 5 to 15 keywords for the page, most relevant first (15 credits).

func (*Client) Lighthouse 

func (c *Client) Lighthouse(ctx context.Context, url string, opts *LighthouseOptions) (*Response[*Lighthouse], error)

Lighthouse runs a Lighthouse audit of the page (2 credits). Audits can outlast the API's 60-second sync window; the client then polls for the result for you, up to the wait timeout.

func (*Client) Markdown 

func (c *Client) Markdown(ctx context.Context, url string, opts *Options) (*Response[string], error)

Markdown converts the page at url to Markdown (1 credit).

Example 
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	client, err := urlpipe.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	res, err := client.Markdown(context.Background(), "https://example.com", &urlpipe.Options{
		MaxAge:      urlpipe.MaxAgeString("1 hour"),
		PageOptions: map[string]any{"block_cookie_banners": true},
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(res.Data)
	fmt.Println("cache:", res.Meta.Cache, "credits left:", res.Meta.Quota.Remaining)
}

func (*Client) Meta 

func (c *Client) Meta(ctx context.Context, url string, opts *Options) (*Response[*Metadata], error)

Meta extracts the page's metadata: title, description, language, author, publication date, feed and images (5 credits).

Example 
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	client, err := urlpipe.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	res, err := client.Meta(context.Background(), "https://example.com", nil)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(res.Data.Title, res.Data.Language)
}

func (*Client) Result 

func (c *Client) Result(ctx context.Context, token string, op Operation) (*Response[any], error)

Result fetches the result behind a token once, with GET /result/:token. A finished analysis comes back with Status StatusCompleted; one still running comes back with StatusProcessing and no error.

GET /result does not say which operation produced a token, so op types Data the way that operation's method would (a screenshot is decoded into a *Screenshot, meta into a *Metadata, ...). With op "", a JSON result is decoded into any and a text result stays a string, so a screenshot stays Base64.

func (*Client) Scrape 

func (c *Client) Scrape(ctx context.Context, url string, operations []Operation, opts *ScrapeOptions) (*Response[*ScrapeResult], error)

Scrape runs several operations off one visit to the page. Each operation is billed as usual; the page is loaded once, so the whole set arrives much sooner. Operations fail independently: read each one with the ScrapeResult accessors. When every operation fails, the error is an *AnalysisFailedError whose Scrape field holds each one's error.

Example 
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	client, err := urlpipe.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	res, err := client.Scrape(context.Background(), "https://example.com",
		[]urlpipe.Operation{urlpipe.OperationMarkdown, urlpipe.OperationMeta, urlpipe.OperationScreenshot}, nil)
	if err != nil {
		log.Fatal(err)
	}
	md, err := res.Data.Markdown()
	if err != nil {
		log.Println(err) // this operation failed; the others are still there
	}
	meta, _ := res.Data.Meta()
	fmt.Println(meta.Title, len(md))
}

func (*Client) Screenshot 

func (c *Client) Screenshot(ctx context.Context, url string, opts *ScreenshotOptions) (*Response[*Screenshot], error)

Screenshot captures the page, full-page PNG unless opts.Screenshot says otherwise (1 credit). Data holds the decoded image.

Example 
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	client, err := urlpipe.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	res, err := client.Screenshot(context.Background(), "https://example.com", &urlpipe.ScreenshotOptions{
		Screenshot: map[string]any{"format": "webp", "viewport_width": 390},
	})
	if err != nil {
		log.Fatal(err)
	}
	if err := res.Data.Save("example.webp"); err != nil {
		log.Fatal(err)
	}
	fmt.Println(res.Data.MIMEType, res.Data.ResultURL)
}

func (*Client) Summarize 

func (c *Client) Summarize(ctx context.Context, url string, opts *Options) (*Response[string], error)

Summarize returns a summary of the page, as Markdown (17 credits).

func (*Client) Wait 

func (c *Client) Wait(ctx context.Context, token string, op Operation, opts *WaitOptions) (*Response[any], error)

Wait polls GET /result/:token until the analysis finishes, fails, or the wait times out with ErrWaitTimeout. Use it for the token of an async call. op types Data as in Client.Result; pass OperationScreenshot to get a decoded *Screenshot. A nil opts waits for the client's wait timeout, polling every 2 seconds.

Example 
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	client, err := urlpipe.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	ctx := context.Background()

	// Accepted straight away; the audit runs in the background.
	accepted, err := client.Lighthouse(ctx, "https://example.com", &urlpipe.LighthouseOptions{
		Options: urlpipe.Options{Async: true},
		Device:  "desktop",
	})
	if err != nil {
		log.Fatal(err)
	}

	// Later, or in another process: the operation hint types Data.
	res, err := client.Wait(ctx, accepted.Token, urlpipe.OperationLighthouse, nil)
	if err != nil {
		log.Fatal(err)
	}
	audit := res.Data.(*urlpipe.Lighthouse)
	fmt.Println(*audit.Categories["performance"].Score)
}

type ConcurrencyLimitError 

type ConcurrencyLimitError struct {
	Limit   int
	Running int
	// contains filtered or unexported fields
}

ConcurrencyLimitError is a 429 concurrency_limit: the plan's parallel requests are all running.

func (*ConcurrencyLimitError) Unwrap 

func (e *ConcurrencyLimitError) Unwrap() error

Unwrap returns the underlying *Error.

type ConsoleEntry 

type ConsoleEntry struct {
	// Type is "error", "warning" or "exception".
	Type string `json:"type"`
	Text string `json:"text"`
}

ConsoleEntry is one message from Client.Console.

type Error 

type Error struct {
	Kind ErrorKind
	// Status is the HTTP status, 0 when there was no response.
	Status int
	// Code is the body's machine-readable error code, such as invalid_url:
	// its error field when that matches ^[a-z][a-z0-9_]*$. Empty when the
	// error is a human sentence.
	Code string
	// Message says what went wrong.
	Message string
	// Body is the response body: decoded JSON (usually map[string]any) or,
	// when it is not JSON, the raw text.
	Body any
	// Token identifies the request, when the answer carried one.
	Token string
	// Err is the underlying cause of a connection error.
	Err error
}

Error is every error the API or the transport produces. Some kinds come wrapped in a type with extra fields (*QuotaExceededError, *ConcurrencyLimitError, *RateLimitedError, *AnalysisFailedError); errors.As with an *Error target reaches the Error inside those too.

func (*Error) Error 

func (e *Error) Error() string

func (*Error) Is 

func (e *Error) Is(target error) bool

Is matches the sentinel of e's kind.

func (*Error) Unwrap 

func (e *Error) Unwrap() error

Unwrap returns the underlying cause, if any.

type ErrorKind 

type ErrorKind string

ErrorKind classifies an *Error. 

const (
	KindAuthentication   ErrorKind = "authentication"
	KindEmailUnverified  ErrorKind = "email_unverified"
	KindInvalidRequest   ErrorKind = "invalid_request"
	KindAnalysisFailed   ErrorKind = "analysis_failed"
	KindQuotaExceeded    ErrorKind = "quota_exceeded"
	KindConcurrencyLimit ErrorKind = "concurrency_limit"
	KindRateLimited      ErrorKind = "rate_limited"
	KindNotFound         ErrorKind = "not_found"
	KindStaleResult      ErrorKind = "stale_result"
	KindServer           ErrorKind = "server"
	KindConnection       ErrorKind = "connection"
	KindWaitTimeout      ErrorKind = "wait_timeout"
	// KindUnexpected is an answer the API does not document, such as a 403
	// other than email_unverified or a 429 with an unknown code.
	KindUnexpected ErrorKind = "unexpected"
)

The kinds of error the client returns. Each has a sentinel (ErrXxx) to test for with errors.Is.

type Lighthouse 

type Lighthouse struct {
	URL       string `json:"url"`
	FetchTime string `json:"fetchTime"`
	Device    string `json:"device"`
	// Categories has performance, accessibility, best-practices and seo.
	// A category Lighthouse no longer reports (pwa) is nil.
	Categories map[string]*LighthouseCategory `json:"categories"`
	// Metrics has first-contentful-paint, largest-contentful-paint,
	// cumulative-layout-shift, total-blocking-time and the rest. A metric
	// Lighthouse could not compute is nil.
	Metrics map[string]*LighthouseMetric `json:"metrics"`
	// Audits is only present with IncludeAudits.
	Audits map[string]json.RawMessage `json:"audits"`
	// Raw is the result exactly as the API sent it.
	Raw json.RawMessage `json:"-"`
}

Lighthouse is the result of Client.Lighthouse. The documented fields are decoded; Raw holds the whole JSON object for anything else.

func (*Lighthouse) UnmarshalJSON 

func (l *Lighthouse) UnmarshalJSON(b []byte) error

UnmarshalJSON decodes the documented fields and keeps the raw object.

type LighthouseCategory 

type LighthouseCategory struct {
	Score *float64 `json:"score"`
	Title string   `json:"title"`
}

LighthouseCategory is one category score, from 0 to 1.

type LighthouseMetric 

type LighthouseMetric struct {
	Score        *float64 `json:"score"`
	DisplayValue string   `json:"displayValue"`
	NumericValue *float64 `json:"numericValue"`
	NumericUnit  string   `json:"numericUnit"`
}

LighthouseMetric is one measured metric.

type LighthouseOptions 

type LighthouseOptions struct {
	Options

	// Device is "mobile" (the API's default) or "desktop".
	Device string

	// IncludeAudits adds the full audits object (150+ audits).
	IncludeAudits bool
}

LighthouseOptions are the settings of Client.Lighthouse.

type MaxAge 

type MaxAge struct {
	// contains filtered or unexported fields
}

MaxAge is how fresh a cached result must be to be accepted. The zero value leaves it to the API (7 days). Build one with MaxAgeSeconds, MaxAgeDuration or MaxAgeString; MaxAgeSeconds(0) always runs a fresh analysis.

func MaxAgeDuration 

func MaxAgeDuration(d time.Duration) MaxAge

MaxAgeDuration is a max_age of d, sent as whole seconds.

func MaxAgeSeconds 

func MaxAgeSeconds(n int) MaxAge

MaxAgeSeconds is a max_age of n seconds.

func MaxAgeString 

func MaxAgeString(s string) MaxAge

MaxAgeString is a max_age in the API's duration syntax, such as "3 days" or "2 hours". It is sent unchanged.

func (MaxAge) IsSet 

func (m MaxAge) IsSet() bool

IsSet reports whether m carries a value.

type Meta 

type Meta struct {
	// Cache is "hit", "miss" or, for a scrape, "partial".
	Cache string
	// CacheAge is the served result's age in seconds, on a hit.
	CacheAge *int
	// ProcessingTimeMs is the total processing time, once the work is done.
	ProcessingTimeMs *int
	Quota            Quota
	// ConcurrencyLimit is how many requests the plan runs in parallel.
	ConcurrencyLimit *Amount
	// ResultURL is a link to a screenshot that needs no API key.
	ResultURL string
	// IdempotentReplayed is true when this answer belongs to an earlier
	// request with the same Idempotency-Key.
	IdempotentReplayed bool
}

Meta is the request metadata the API sends in its X- headers (and, on a webhook, in the payload's meta object). A value the API did not send is nil, or "" for strings.

func (*Meta) UnmarshalJSON 

func (m *Meta) UnmarshalJSON(b []byte) error

UnmarshalJSON reads the meta object of a webhook payload, where the concurrency limit sits inside quota.

type Metadata 

type Metadata struct {
	Title       string `json:"title"`
	Description string `json:"description"`
	// Language is an ISO 639-1 code such as "en".
	Language     string `json:"language"`
	MainImageURL string `json:"main_image_url"`
	FaviconURL   string `json:"favicon_url"`
	AuthorName   string `json:"author_name"`
	FeedURL      string `json:"feed_url"`
	// PublicationDate is the date of first publication, ISO 8601, as given.
	PublicationDate string `json:"publication_date"`
	// AdditionalAuthorInformation holds extra author details such as social
	// handles or an email address.
	AdditionalAuthorInformation map[string]any `json:"additional_author_information"`
}

Metadata is the result of Client.Meta. A field the page does not carry is empty (nil for AdditionalAuthorInformation). URL fields are absolute.

type Operation 

type Operation string

Operation names one of the API's analyses. It is what Client.Scrape takes a list of, and the hint Client.Result and Client.Wait use to type the data they return. 

const (
	OperationMarkdown   Operation = "markdown"
	OperationHTML       Operation = "html"
	OperationSummarize  Operation = "summarize"
	OperationScreenshot Operation = "screenshot"
	OperationMeta       Operation = "meta"
	OperationKeywords   Operation = "keywords"
	OperationConsole    Operation = "console"
	OperationLighthouse Operation = "lighthouse"
	OperationScrape     Operation = "scrape"
)

The operations the API runs.

type OperationError 

type OperationError struct {
	Operation Operation
	Message   string
}

OperationError is returned by the ScrapeResult accessors when that operation failed, is still running, or was not requested.

func (*OperationError) Error 

func (e *OperationError) Error() string

func (*OperationError) Pending 

func (e *OperationError) Pending() bool

Pending reports whether the operation (a lighthouse audit) was still running when the scrape answered. Fetch the scrape again with Client.Result and its token to pick it up.

type Option 

type Option func(*Client)

Option configures a Client.

func WithAPIKey 

func WithAPIKey(key string) Option

WithAPIKey sets the project API key. Without it, NewClient reads URLPIPE_API_KEY.

func WithBaseURL 

func WithBaseURL(baseURL string) Option

WithBaseURL points the client at another host, such as a local stub server in tests. The default is DefaultBaseURL.

func WithHTTPClient 

func WithHTTPClient(hc *http.Client) Option

WithHTTPClient sets the *http.Client requests are sent with, for custom transports, proxies or instrumentation. The per-request timeout set with WithTimeout still applies on top of it.

func WithMaxRetries 

func WithMaxRetries(n int) Option

WithMaxRetries sets how many times a transient failure is retried (default DefaultMaxRetries). Zero turns retries off, and with them the Idempotency-Key the client otherwise generates for every analysis.

func WithTimeout 

func WithTimeout(d time.Duration) Option

WithTimeout bounds each HTTP request (default DefaultTimeout). Zero or less leaves requests bounded only by their context.

func WithWaitTimeout 

func WithWaitTimeout(d time.Duration) Option

WithWaitTimeout bounds how long a long analysis is polled for before giving up with ErrWaitTimeout (default DefaultWaitTimeout). It applies to the polling a sync call falls back to, and is the default for Client.Wait.

type Options 

type Options struct {
	// Async returns as soon as the API accepts the request, with a token and
	// Status "accepted", instead of waiting for the result. The default
	// (false) sends sync: true and hands back the result.
	Async bool

	// MaxAge is how fresh a cached result must be. Cached results are free.
	MaxAge MaxAge

	// Labels are your own keys for the request, returned with the result.
	Labels map[string]string

	// Residential fetches the page from a home broadband address.
	Residential bool

	// ReportTo is a webhook URL the result is POSTed to (async requests).
	ReportTo string

	// PageOptions is sent as page_options unchanged: wait_for_selector,
	// delay, block_ads, block_cookie_banners, remove_selectors.
	PageOptions map[string]any

	// IdempotencyKey is sent as the Idempotency-Key header. Leave it empty
	// and the client generates one per call whenever retries are on.
	IdempotencyKey string

	// Extra is merged into the JSON body last, for API parameters this
	// version of the library has no field for.
	Extra map[string]any
}

Options are the settings every analysis takes. The zero value, or a nil *Options, is a synchronous request with the API's defaults; only the fields you set are sent.

type Quota 

type Quota struct {
	Cost      *int
	Limit     *Amount
	Remaining *Amount
	Overage   *int
	ResetsAt  *time.Time
}

Quota is what a request cost and what is left of the monthly allowance.

type QuotaExceededError 

type QuotaExceededError struct {
	Limit  int
	Used   int
	Needed int
	// ResetsAt is when the allowance rolls over; zero if the API sent none.
	ResetsAt time.Time
	// contains filtered or unexported fields
}

QuotaExceededError is a 429 quota_exceeded: the Free plan's credits for the month are spent. Never retried.

func (*QuotaExceededError) Unwrap 

func (e *QuotaExceededError) Unwrap() error

Unwrap returns the underlying *Error.

type RateLimitedError 

type RateLimitedError struct {

	// RetryAfter is how long the API asked to wait; zero when it did not say.
	RetryAfter time.Duration
	// contains filtered or unexported fields
}

RateLimitedError is a 429 rate_limited: requests are arriving too fast.

func (*RateLimitedError) Unwrap 

func (e *RateLimitedError) Unwrap() error

Unwrap returns the underlying *Error.

type Response 

type Response[T any] struct {
	Status Status
	Data   T
	// Token identifies the request; GET /result/:token with it is free for
	// 30 days.
	Token string
	// Labels are the labels the request was made with; an empty map when
	// there are none.
	Labels map[string]string
	// Meta is what the response headers said about the request.
	Meta Meta
}

Response is what every method returns. Data is typed by the operation: string for Markdown, HTML and Summarize, *Screenshot, *Metadata, []string for Keywords, []ConsoleEntry, *Lighthouse and *ScrapeResult. Data is only set when Status is StatusCompleted; otherwise it is the zero value.

func (*Response[T]) Completed 

func (r *Response[T]) Completed() bool

Completed reports whether Data holds the result.

type ScrapeOperation 

type ScrapeOperation struct {
	Success bool `json:"success"`
	// Result is the operation's result in its usual format, as JSON.
	Result json.RawMessage `json:"result"`
	// Error is the failure message when Success is false.
	Error string `json:"error"`
	// Cached is true when the operation was served from cache (free).
	Cached bool `json:"cached"`
}

ScrapeOperation is one operation's outcome inside a scrape.

type ScrapeOptions 

type ScrapeOptions struct {
	Options

	Device        string
	IncludeAudits bool
	Screenshot    map[string]any
}

ScrapeOptions are the settings of Client.Scrape. Device and IncludeAudits apply to a lighthouse operation, Screenshot to a screenshot operation.

type ScrapeResult 

type ScrapeResult struct {
	URL        string                        `json:"url"`
	Operations map[Operation]ScrapeOperation `json:"operations"`
	// Order lists the operations in the order the API returned them, which
	// is the order they were asked for.
	Order []Operation `json:"-"`
}

ScrapeResult is the result of Client.Scrape: one entry per requested operation. Use the typed accessors (Markdown, Meta, Screenshot, ...) to read an operation's result.

func (*ScrapeResult) Console 

func (s *ScrapeResult) Console() ([]ConsoleEntry, error)

Console is the console operation's result.

func (*ScrapeResult) HTML 

func (s *ScrapeResult) HTML() (string, error)

HTML is the html operation's result.

func (*ScrapeResult) Keywords 

func (s *ScrapeResult) Keywords() ([]string, error)

Keywords is the keywords operation's result.

func (*ScrapeResult) Lighthouse 

func (s *ScrapeResult) Lighthouse() (*Lighthouse, error)

Lighthouse is the lighthouse operation's result.

func (*ScrapeResult) Markdown 

func (s *ScrapeResult) Markdown() (string, error)

Markdown is the markdown operation's result.

func (*ScrapeResult) Meta 

func (s *ScrapeResult) Meta() (*Metadata, error)

Meta is the meta operation's result.

func (*ScrapeResult) Screenshot 

func (s *ScrapeResult) Screenshot() (*Screenshot, error)

Screenshot is the screenshot operation's image, decoded.

func (*ScrapeResult) Summary 

func (s *ScrapeResult) Summary() (string, error)

Summary is the summarize operation's result.

func (*ScrapeResult) UnmarshalJSON 

func (s *ScrapeResult) UnmarshalJSON(b []byte) error

UnmarshalJSON decodes the scrape object and records the operations' order.

type Screenshot 

type Screenshot struct {
	// Data is the image itself.
	Data []byte
	// MIMEType is image/png, image/jpeg or image/webp, read off the bytes.
	MIMEType string
	// ResultURL is a link to the image that needs no API key, valid for the
	// 30 days the result is kept. Empty when the response carried none.
	ResultURL string
}

Screenshot is the decoded image from Client.Screenshot.

func (*Screenshot) Save 

func (s *Screenshot) Save(path string) error

Save writes the image to path.

type ScreenshotOptions 

type ScreenshotOptions struct {
	Options

	// Screenshot is sent as screenshot_options unchanged: full_page,
	// viewport_width, device_scale_factor, format, selector, dark_mode,
	// hide_selectors and the rest.
	Screenshot map[string]any
}

ScreenshotOptions are the settings of Client.Screenshot.

type Status 

type Status string

Status says where a request stands. 

const (
	// StatusCompleted means Data holds the result.
	StatusCompleted Status = "completed"
	// StatusAccepted means an async request was accepted; collect the
	// result with the Token.
	StatusAccepted Status = "accepted"
	// StatusProcessing means GET /result/:token found the analysis still
	// running.
	StatusProcessing Status = "processing"
)

The statuses a Response can carry.

type WaitOptions 

type WaitOptions struct {
	// Timeout bounds the wait. Zero uses the client's wait timeout.
	Timeout time.Duration
	// Interval is the pause between polls. Zero uses 2 seconds.
	Interval time.Duration
}

WaitOptions tune Client.Wait.

type WebhookEvent 

type WebhookEvent struct {
	Token     string    `json:"token"`
	Operation Operation `json:"operation"`
	// Labels are the request's labels; an empty map when it had none.
	Labels  map[string]string `json:"labels"`
	Success bool              `json:"success"`
	// Result is the operation's result as JSON; Data decodes it.
	Result json.RawMessage `json:"result"`
	// ResultURL links to a screenshot with no API key needed.
	ResultURL string `json:"result_url"`
	// Error is the failure message when Success is false.
	Error string `json:"error"`
	Meta  Meta   `json:"meta"`
}

WebhookEvent is the payload of a webhook delivery.

func VerifyWebhook 

func VerifyWebhook(rawBody []byte, header http.Header, secret string, tolerance time.Duration) (*WebhookEvent, error)

VerifyWebhook checks a signed delivery and returns its payload. It needs no Client.

rawBody must be the request body exactly as received: parsing the JSON and serializing it again changes the bytes, and the signature with them. header supplies X-URLpipe-Timestamp and X-URLpipe-Signature. secret is the project's signing secret (whsec_...), used as-is. A delivery whose timestamp is further than tolerance from now, in either direction, is refused; zero means DefaultWebhookTolerance.

Example 
package main

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"fmt"
	"net/http"
	"strconv"
	"time"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	secret := "whsec_example"
	body := []byte(`{"token":"tok_1","operation":"markdown","labels":{},"success":true,"result":"# Example Domain","result_url":null,"error":null,"meta":{}}`)

	// What URLpipe sends: a timestamp and an HMAC of "<timestamp>.<body>".
	ts := strconv.FormatInt(time.Now().Unix(), 10)
	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write([]byte(ts + "." + string(body)))
	header := http.Header{}
	header.Set("X-URLpipe-Timestamp", ts)
	header.Set("X-URLpipe-Signature", "v1="+hex.EncodeToString(mac.Sum(nil)))

	event, err := urlpipe.VerifyWebhook(body, header, secret, 5*time.Minute)
	if err != nil {
		fmt.Println(err)
		return
	}
	data, _ := event.Data()
	fmt.Println(event.Token, event.Operation, data)
}

Output:
tok_1 markdown # Example Domain
Example (Handler) 
package main

import (
	"io"
	"log"
	"net/http"
	"os"

	"github.com/URLpipe/urlpipe-go"
)

func main() {
	secret := os.Getenv("URLPIPE_WEBHOOK_SECRET")
	http.HandleFunc("/webhooks/urlpipe", func(w http.ResponseWriter, r *http.Request) {
		body, err := io.ReadAll(r.Body) // the raw bytes, not re-encoded JSON
		if err != nil {
			http.Error(w, "unreadable body", http.StatusBadRequest)
			return
		}
		event, err := urlpipe.VerifyWebhook(body, r.Header, secret, 0)
		if err != nil {
			http.Error(w, err.Error(), http.StatusUnauthorized)
			return
		}
		log.Printf("result %s for %s ready", event.Token, event.Operation)
		w.WriteHeader(http.StatusNoContent)
	})
}

func VerifyWebhookSignature 

func VerifyWebhookSignature(rawBody []byte, timestamp, signature, secret string, tolerance time.Duration) (*WebhookEvent, error)

VerifyWebhookSignature is VerifyWebhook for callers that hold the two header values rather than an http.Header.

func (*WebhookEvent) Data 

func (e *WebhookEvent) Data() (any, error)

Data decodes Result the way the operation's method types it: a string for markdown, a *Screenshot, a *Metadata, and so on. It is nil for a failed analysis.

type WebhookVerificationError 

type WebhookVerificationError struct {
	Reason string
}

WebhookVerificationError says why a delivery failed verification. It matches ErrWebhookVerification with errors.Is.

func (*WebhookVerificationError) Error 

func (e *WebhookVerificationError) Error() string

func (*WebhookVerificationError) Is 

func (e *WebhookVerificationError) Is(target error) bool

Is matches ErrWebhookVerification.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.