fetch

Documentation

Overview

Package fetch provides URL fetching backends for the url tool. Supports DefuddleCLI (local mode) and BrowserGateway (remote mode) with daily caching.

Plane: shared

Index

• Constants
• func BinaryFetchError(url, contentType string) error
• func IsBinaryBody(data []byte) bool
• func IsBinaryContentType(mediatype string) bool
• func TruncateContent(s string) string
• type Backend
  • func NewBrowserGatewayBackend(gatewayURL string, client *http.Client) Backend
  • func NewCachedFetchBackend(cacheDir string, fallback Backend) Backend
  • func NewDefuddleCLIBackend() Backend
  • func Resolve() Backend
• type CachedFetchBackend
  • func (b *CachedFetchBackend) Fetch(ctx context.Context, rawURL string) (string, error)

Constants

const MaxContentChars = 30_000

MaxContentChars is the rune limit applied by TruncateContent.

Functions

func BinaryFetchError

func BinaryFetchError(url, contentType string) error

BinaryFetchError returns an error telling the agent to use a download tool. Suggests aria2c if available, then wget, then curl.

func IsBinaryBody

func IsBinaryBody(data []byte) bool

IsBinaryBody checks the first 8KB for null bytes.

func IsBinaryContentType

func IsBinaryContentType(mediatype string) bool

IsBinaryContentType returns true for content types that are never human-readable text.

func TruncateContent

func TruncateContent(s string) string

TruncateContent truncates content to MaxContentChars runes.

Types

type Backend

type Backend interface {
	Fetch(ctx context.Context, url string) (content string, err error)
}

Backend controls how HTML is fetched and converted to markdown.

func NewBrowserGatewayBackend

func NewBrowserGatewayBackend(gatewayURL string, client *http.Client) Backend

NewBrowserGatewayBackend creates a backend that fetches via browser-gateway.

func NewCachedFetchBackend

func NewCachedFetchBackend(cacheDir string, fallback Backend) Backend

NewCachedFetchBackend creates a CachedFetchBackend wrapping the given fallback. If the cache directory cannot be created, returns the fallback directly (caching disabled).

func NewDefuddleCLIBackend

func NewDefuddleCLIBackend() Backend

NewDefuddleCLIBackend creates a backend that fetches URLs via Go's HTTP client. HTML pages are passed to defuddle for parsing; non-HTML content is returned raw. Requires defuddle to be installed and on PATH.

func Resolve

func Resolve() Backend

Resolve returns the appropriate backend based on environment. If BROWSER_GATEWAY_URL is set, uses BrowserGatewayBackend (no cache). Otherwise, uses DefuddleCLIBackend wrapped with daily file cache.

type CachedFetchBackend

type CachedFetchBackend struct {
	// contains filtered or unexported fields
}

CachedFetchBackend wraps a Backend with a file-based daily cache. Cache key: sanitized URL + today's date. Cache dir: ~/.cache/organon/scrapes/.

func (*CachedFetchBackend) Fetch

func (b *CachedFetchBackend) Fetch(ctx context.Context, rawURL string) (string, error)

Fetch returns cached content if available for today, otherwise delegates to fallback and caches.