web

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 16, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package tools implements the raw-content cache used by WebFetch.

Mirrors the behaviour of URL_CACHE in src/tools/WebFetchTool/utils.ts: entries are keyed by the original URL (not the prompt), contain fetched markdown plus HTTP/binary metadata, expire after 15 minutes, and share a 50MB byte budget. Prompt application deliberately happens after lookup so two prompts reuse one network response while invoking the model twice.

Package tools — webfetch-domain-blocklist-preflight.

Mirrors src/tools/WebFetchTool/utils.ts:171-203,386-398. Before any outbound fetch we ask Anthropic's domain_info endpoint whether the hostname is on the brand/security blocklist. The result is cached per-hostname for 5 minutes so the preflight cost is paid at most once per allowed hostname per window. Blocked and failed verdicts are never cached, matching the TS fail-closed security boundary.

Package tools — DOM-aware HTML→Markdown converter.

The walker uses golang.org/x/net/html for nested lists, tables, code-block language hints, and escaped preformatted content.

Package tools — preapproved-host allowlist for WebFetch.

Mirrors src/tools/WebFetchTool/preapproved.ts verbatim: GET-only fetches to these hosts skip per-domain user confirmation. Path-prefix entries (e.g., "github.com/anthropics") only match the prefix or a path under it, never an unrelated path that happens to share the prefix string.

SECURITY: This list is exclusively for WebFetch GET requests. The sandbox system intentionally does NOT inherit this list — many entries (huggingface, kaggle, nuget) accept uploads, so unrestricted egress would enable exfil.

Package tools — local LLM summariser used by WebFetch.

Mirrors applyPromptToMarkdown in src/tools/WebFetchTool/utils.ts: feed the truncated markdown plus the user's prompt to a small/fast model (Haiku class) and return the model's text response. The summariser is pluggable so providers can supply the active small-model client.

Package tools — Sources/citations rendering for WebSearch.

Mirrors mapToolResultToToolResultBlockParam from src/tools/WebSearchTool/WebSearchTool.ts. The Go side also has to produce a "Web search results for query: ..." block with a JSON-encoded list of links and the SOURCES reminder at the tail so the model reliably emits a Sources: section.

Package tools — canonical WebSearch server-tool result types.

Package tools — Anthropic web_search_20250305 server-tool integration.

Mirrors src/tools/WebSearchTool/WebSearchTool.ts which uses the Anthropic-hosted web_search server tool to deliver normalised, citation-attached results.

Index

Constants

View Source 
const MaxFetchURLLength = 2000

MaxFetchURLLength caps the raw URL length accepted by validateURL. The Anthropic web_fetch_20250910 server tool rejects anything longer; matching the limit here keeps Go in sync (mirrors WEB_FETCH_MAX_URL_LENGTH=2000 in src/tools/WebFetchTool/validateUrl.ts).

View Source 
const MaxMarkdownBytes = 100 * 1024
View Source 
const WebFetchCacheMaxBytes = 50 * 1024 * 1024

WebFetchCacheMaxBytes mirrors the lru-cache `maxSize` ceiling on the TS side. Once the sum of cached entry sizes exceeds this we begin evicting the oldest entries until the new entry fits.

View Source 
const WebFetchCacheTTL = 15 * time.Minute

WebFetchCacheTTL is the lifetime of a cached fetch result. Mirrors TS CACHE_TTL_MS = 15 minutes.

View Source 
const WebFetchSummariserMaxTokens = 4096

WebFetchSummariserMaxTokens caps the secondary-model output to keep the returned summary bounded. Mirrors the behaviour described by the task spec (4096 tokens).

View Source 
const WebFetchTimeout = 60 * time.Second

WebFetchTimeout aligns with TS WebFetchTool.fetchTimeout=60s. Servers behind CDN edge caches sometimes need >30s for first-byte; matching the TS budget avoids spurious "fetch failed" surfacing for slow but reachable targets.

View Source 
const WebSearchServerToolName = "web_search_20250305"

WebSearchServerToolName matches the SDK BetaWebSearchTool20250305 type.

View Source 
const WebSearchSnippetCap = 280

WebSearchSnippetCap mirrors the 280-char limit specified in the task acceptance criteria. Snippets longer than this are cut and suffixed with an ellipsis.

Variables

View Source 
var ErrSummariserUnavailable = errors.New("WebFetch summariser is not configured")

ErrSummariserUnavailable is returned by RunWebFetchSummariser when no client has been configured. Callers should fall back to the structured payload built from the markdown directly.

View Source 
var ErrWebSearchServerToolUnavailable = errors.New("web_search server tool is unavailable")

ErrWebSearchServerToolUnavailable signals that the server tool cannot service the request.

Functions

func FormatWebSearchToolResultError 

func FormatWebSearchToolResultError(err WebSearchToolResultError) string

FormatWebSearchToolResultError renders the error envelope into the human-readable string the model sees on a transient web_search failure (rate limit, upstream error, etc.).

func HTMLToMarkdownDOM 

func HTMLToMarkdownDOM(htmlInput string) string

HTMLToMarkdownDOM converts the supplied HTML fragment to Markdown via a structural DOM walk. The output is post-processed through the same truncation logic the regex converter uses so callers can interchange implementations.

func IsPreapprovedHost 

func IsPreapprovedHost(rawURL string) bool

IsPreapprovedHost reports whether a URL points at a preapproved host. Hostname matching is case-insensitive and IDN/punycode-normalised. Bare hostname entries are exact: the TS allowlist does not grant implicit access to subdomains. Path-prefix entries enforce segment boundaries: "/anthropics" matches itself or "/anthropics/foo", never "/anthropics-evil".

func PreflightDomainInfoEnabled 

func PreflightDomainInfoEnabled(w *WebFetchTool) bool

PreflightDomainInfoEnabled reports whether a WebFetchTool instance has the preflight wired up. Used so callers (and tests) can skip the preflight when no endpoint is configured.

func ResetDomainInfoCache 

func ResetDomainInfoCache()

ResetDomainInfoCache clears the per-hostname verdict cache. Tests use this to ensure each invocation talks to the (fake) endpoint.

func RunWebFetchSummariser 

func RunWebFetchSummariser(
	ctx context.Context,
	client SummariserClient,
	url, userPrompt, markdown string,
	isPreapprovedDomain bool,
) (string, error)

RunWebFetchSummariser truncates `markdown` to MaxMarkdownBytes, formats the secondary-model prompt via SecondaryModelPrompt, and invokes the supplied client. Returns the model's text or an error.

func SecondaryModelPrompt 

func SecondaryModelPrompt(markdownContent, prompt string, isPreapprovedDomain bool) string

SecondaryModelPrompt formats markdown content + user prompt into the exact wrapper that the TS reference uses.

func SourcesReminder 

func SourcesReminder() string

SourcesReminder returns the localized trailing instruction appended to every web search result block. Its function form follows runtime language switches.

func WebFetchCacheKey 

func WebFetchCacheKey(rawURL string) string

WebFetchCacheKey computes a raw URL cache key. URL case is preserved: TS Map/LRU keys use the exact original URL string, so case-sensitive paths and escaped octets must not collapse into the same entry.

Types

type SummariserClient 

type SummariserClient interface {
	// Summarise sends a single user message + system prompt to a fast model
	// and returns the assistant text. ctx, maxTokens, and the system prompt
	// are honoured by the implementation; cancellation surfaces as
	// ctx.Err().
	Summarise(ctx context.Context, req SummariserRequest) (string, error)
}

SummariserClient is the minimal interface the summariser needs. It lets us swap the production Anthropic client for fakes in tests without pulling the SDK into this file.

type SummariserFunc 

type SummariserFunc func(ctx context.Context, req SummariserRequest) (string, error)

SummariserFunc adapts a function to the SummariserClient interface so tests can supply an inline closure.

func (SummariserFunc) Summarise 

func (f SummariserFunc) Summarise(ctx context.Context, req SummariserRequest) (string, error)

Summarise implements SummariserClient.

type SummariserRequest 

type SummariserRequest struct {
	SystemPrompt string
	UserPrompt   string
	MaxTokens    int
	// Source URL — used by the implementation for telemetry/auth scoping.
	URL string
	// Original user prompt (without markdown wrapper) — passed through so
	// providers that prefer message arrays can build them.
	Prompt string
}

SummariserRequest captures the inputs to a single summariser call.

type WebFetchCache 

type WebFetchCache struct {
	// contains filtered or unexported fields
}

WebFetchCache is a self-cleaning sha256(originalURL)-keyed cache with a hard byte-cap ceiling. Entries are evicted in oldest-first order once the total cached bytes exceed WebFetchCacheMaxBytes.

func NewWebFetchCache 

func NewWebFetchCache() *WebFetchCache

NewWebFetchCache creates a cache with the default 15-minute TTL and starts the background purge goroutine. Call Stop when no longer needed.

func (*WebFetchCache) Clear 

func (c *WebFetchCache) Clear()

Clear removes every cached response without stopping the cache. It is the state-lifecycle hook used by /clear and session-reset integrations.

func (*WebFetchCache) Get 

func (c *WebFetchCache) Get(key string) (WebFetchCacheEntry, bool)

Get returns the cached entry for key. The second value is false when no entry is present or the existing entry has expired (in which case the stale entry is also evicted).

func (*WebFetchCache) Len 

func (c *WebFetchCache) Len() int

Len returns the number of live entries (not counting expired ones still physically present until the next purge or Get).

func (*WebFetchCache) MakeKey 

func (c *WebFetchCache) MakeKey(rawURL string) string

MakeKey returns the canonical sha256(originalURL) digest.

func (*WebFetchCache) Set 

func (c *WebFetchCache) Set(key string, value WebFetchCacheEntry)

Set writes value at key. If the cache has been Stop()ped the call is a no-op so callers don't need to track lifecycle. Once the cache exceeds WebFetchCacheMaxBytes, the oldest entries are evicted until the new value fits (mirrors TS lru-cache `maxSize` accounting).

func (*WebFetchCache) Stop 

func (c *WebFetchCache) Stop()

Stop terminates the background purge goroutine. It is idempotent.

func (*WebFetchCache) TotalBytes 

func (c *WebFetchCache) TotalBytes() int

TotalBytes returns the running byte total used by the byte-cap eviction policy. Exposed so tests and metrics can verify the accounting.

type WebFetchCacheEntry 

type WebFetchCacheEntry struct {
	Body          string
	ContentType   string
	CacheSize     int
	StatusCode    int
	StatusText    string
	Bytes         int
	PersistedPath string
	PersistedSize int
}

WebFetchCacheEntry is the raw fetched-content value stored per original URL.

type WebFetchInput 

type WebFetchInput struct {
	URL    string `json:"url"`
	Prompt string `json:"prompt"`
}

WebFetchInput is the typed input for WebFetchTool.

type WebFetchOutput 

type WebFetchOutput struct {
	Bytes      int    `json:"bytes"`
	Code       int    `json:"code"`
	CodeText   string `json:"codeText"`
	Result     string `json:"result"`
	DurationMs int64  `json:"durationMs"`
	URL        string `json:"url"`
}

WebFetchOutput mirrors WebFetchTool.outputSchema in the TS implementation.

type WebFetchTool 

type WebFetchTool struct {

	// Summariser is the small/fast secondary model used to apply the user
	// prompt to fetched markdown (mirrors src/tools/WebFetchTool/utils.ts:484-530
	// applyPromptToMarkdown).
	Summariser SummariserClient

	// DomainInfoEndpoint is the optional base URL for the brand/security
	// blocklist preflight (api.anthropic.com/api/web/domain_info). When
	// empty the preflight is skipped. DomainInfoClient overrides the
	// default http.Client used for the preflight.
	DomainInfoEndpoint string
	DomainInfoClient   *http.Client
	// SkipWebFetchPreflight is the Go settings equivalent of the TS
	// skipWebFetchPreflight escape hatch for restricted enterprise networks.
	SkipWebFetchPreflight bool

	// Domain restrictions (Task 7: security hardening).
	AllowedDomains    []string // nil = all allowed (whitelist)
	DisallowedDomains []string // these domains always blocked (blacklist)
	// contains filtered or unexported fields
}

WebFetchTool fetches a URL and returns its text content.

func NewWebFetchTool 

func NewWebFetchTool(cache *WebFetchCache) *WebFetchTool

NewWebFetchTool creates the canonical local-HTTP WebFetch tool.

func (*WebFetchTool) CheckPermissions 

func (w *WebFetchTool) CheckPermissions(_ context.Context, input map[string]any, request types.ToolPermissionRequest) (types.ToolPermissionResult, error)

func (*WebFetchTool) ClearWebFetchCache 

func (w *WebFetchTool) ClearWebFetchCache()

ClearWebFetchCache clears fetched content and allowed domain verdicts while keeping the tool usable for the rest of the session.

func (*WebFetchTool) Description 

func (w *WebFetchTool) Description() string

func (*WebFetchTool) Execute 

func (w *WebFetchTool) Execute(ctx context.Context, input map[string]any) (types.ToolResult, error)

func (*WebFetchTool) FetchCache 

func (w *WebFetchTool) FetchCache() *WebFetchCache

FetchCache exposes the session-owned cache for lifecycle wiring.

func (*WebFetchTool) MapToolResultToToolResultBlock 

func (w *WebFetchTool) MapToolResultToToolResultBlock(data any, toolUseID string) types.ToolResultBlock

func (*WebFetchTool) Name 

func (w *WebFetchTool) Name() string

func (*WebFetchTool) Schema 

func (w *WebFetchTool) Schema() types.JSONSchema

func (*WebFetchTool) ToolMetadata 

func (w *WebFetchTool) ToolMetadata(map[string]any) types.ToolMetadata

func (*WebFetchTool) WithSummariser 

func (w *WebFetchTool) WithSummariser(client SummariserClient) *WebFetchTool

WithSummariser plugs a fast-model summariser into the WebFetchTool and returns the receiver so callers can chain. Mirrors the TS pattern of passing a Haiku-class client into the local fetch path.

type WebSearchInput 

type WebSearchInput struct {
	Query          string   `json:"query"`
	AllowedDomains []string `json:"allowed_domains"`
	BlockedDomains []string `json:"blocked_domains"`
}

WebSearchInput is the typed input for WebSearchTool.

type WebSearchOutput 

type WebSearchOutput struct {
	Query           string  `json:"query"`
	Results         []any   `json:"results"`
	DurationSeconds float64 `json:"durationSeconds"`
}

WebSearchOutput matches the TS tool output. Results contains string commentary/error entries or WebSearchOutputSearchResult values. 

type WebSearchOutputLink struct {
	Title string `json:"title"`
	URL   string `json:"url"`
}

type WebSearchOutputSearchResult 

type WebSearchOutputSearchResult struct {
	ToolUseID string                `json:"tool_use_id"`
	Content   []WebSearchOutputLink `json:"content"`
}

type WebSearchProgressEvent 

type WebSearchProgressEvent struct {
	Type        string `json:"type"`
	ToolUseID   string `json:"toolUseID,omitempty"`
	Query       string `json:"query,omitempty"`
	ResultCount int    `json:"resultCount,omitempty"`
	Count       int    `json:"count,omitempty"`
}

WebSearchProgressEvent is a single progress event emitted to the optional WebSearchTool.OnProgress callback.

type WebSearchResult 

type WebSearchResult struct {
	Title     string `json:"title"`
	URL       string `json:"url"`
	Snippet   string `json:"snippet,omitempty"`
	CitedText string `json:"cited_text,omitempty"`
	PageAge   string `json:"page_age,omitempty"`
}

WebSearchResult is the canonical provider-native result.

type WebSearchResultBlock 

type WebSearchResultBlock struct {
	URL       string `json:"url"`
	Title     string `json:"title"`
	Snippet   string `json:"snippet,omitempty"`
	CitedText string `json:"cited_text,omitempty"`
	PageAge   string `json:"page_age,omitempty"`
}

WebSearchResultBlock is a single normalised citation block.

type WebSearchServerToolEntry 

type WebSearchServerToolEntry struct {
	Text   string
	Result *WebSearchServerToolResult
}

WebSearchServerToolEntry preserves the API block order used by the TS output builder: text commentary and result blocks may be interleaved.

type WebSearchServerToolFunc 

type WebSearchServerToolFunc func(ctx context.Context, req WebSearchServerToolRequest) (WebSearchServerToolResponse, error)

WebSearchServerToolFunc adapts a closure to the provider interface.

func (WebSearchServerToolFunc) SearchViaServerTool 

func (f WebSearchServerToolFunc) SearchViaServerTool(ctx context.Context, req WebSearchServerToolRequest) (WebSearchServerToolResponse, error)

SearchViaServerTool implements WebSearchServerToolProvider.

type WebSearchServerToolProvider 

type WebSearchServerToolProvider interface {
	SearchViaServerTool(ctx context.Context, req WebSearchServerToolRequest) (WebSearchServerToolResponse, error)
}

WebSearchServerToolProvider executes a single web_search_20250305 call against the Anthropic API and returns normalised search results.

type WebSearchServerToolRequest 

type WebSearchServerToolRequest struct {
	Query          string
	AllowedDomains []string
	BlockedDomains []string
	MaxUses        int
	OnProgress     func(WebSearchProgressEvent)
}

WebSearchServerToolRequest captures the inputs needed to invoke the server tool. AllowedDomains/BlockedDomains are passed through unchanged.

type WebSearchServerToolResponse 

type WebSearchServerToolResponse struct {
	Results      []WebSearchResult
	ResultBlocks []WebSearchServerToolResult
	Entries      []WebSearchServerToolEntry
	Citations    []string
	DurationMs   int64
	Usage        types.Usage

	// websearch-tool-result-error-rendering: when the upstream
	// web_search server returns an error envelope (rate-limited, upstream
	// failure, etc.) instead of an array, the provider sets ErrorCode so
	// the WebSearch executor can render "Web search error: <code>" rather
	// than "no results".
	ErrorCode string
}

WebSearchServerToolResponse is the parsed result set. Citations is raw block JSON propagated to downstream consumers without parsing so citation metadata round-trips cleanly.

type WebSearchServerToolResult 

type WebSearchServerToolResult struct {
	ToolUseID string
	Results   []WebSearchResult
	ErrorCode string
}

WebSearchServerToolResult is one web_search_tool_result block. Content is either Results or ErrorCode, preserving the TS mixed output array.

type WebSearchTool 

type WebSearchTool struct {

	// websearch-streaming-progress-events: optional callback the harness
	// wires to surface live progress chips ("Searching for: <query>", "N
	// results received") in TUI clients. Mirrors the TS onProgress
	// callback at WebSearchTool.ts:295-388. Nil = no progress events.
	OnProgress func(event WebSearchProgressEvent)

	// Domain restrictions (Task 7: security hardening).
	AllowedDomains    []string // nil = all allowed (whitelist)
	DisallowedDomains []string // these domains always blocked (blacklist)
	// contains filtered or unexported fields
}

WebSearchTool searches through the active provider's native server tool.

func NewWebSearchTool 

func NewWebSearchTool() *WebSearchTool

func (*WebSearchTool) CheckPermissions 

func (w *WebSearchTool) CheckPermissions(_ context.Context, _ map[string]any, _ types.ToolPermissionRequest) (types.ToolPermissionResult, error)

func (*WebSearchTool) Description 

func (w *WebSearchTool) Description() string

func (*WebSearchTool) Execute 

func (w *WebSearchTool) Execute(ctx context.Context, input map[string]any) (types.ToolResult, error)

func (*WebSearchTool) HasWebSearchServerTool 

func (w *WebSearchTool) HasWebSearchServerTool() bool

HasWebSearchServerTool reports whether a provider has been configured.

func (*WebSearchTool) IsEnabled 

func (w *WebSearchTool) IsEnabled(runtime types.ToolRuntimeContext) bool

func (*WebSearchTool) MapToolResultToToolResultBlock 

func (w *WebSearchTool) MapToolResultToToolResultBlock(data any, toolUseID string) types.ToolResultBlock

func (*WebSearchTool) Name 

func (w *WebSearchTool) Name() string

func (*WebSearchTool) Schema 

func (w *WebSearchTool) Schema() types.JSONSchema

func (*WebSearchTool) SetWebSearchServerToolProvider 

func (w *WebSearchTool) SetWebSearchServerToolProvider(p WebSearchServerToolProvider)

SetWebSearchServerToolProvider wires the canonical provider-native executor.

func (*WebSearchTool) ToolMetadata 

func (w *WebSearchTool) ToolMetadata(map[string]any) types.ToolMetadata

type WebSearchToolResultBlock 

type WebSearchToolResultBlock struct {
	Type      string                 `json:"type"`
	ToolUseID string                 `json:"tool_use_id,omitempty"`
	Content   []WebSearchResultBlock `json:"content"`
}

WebSearchToolResultBlock matches the TS server-tool block shape `{ type: 'web_search_tool_result', content: [...] }` so the Go renderer can emit identical citation blocks.

func (WebSearchToolResultBlock) GetType 

func (b WebSearchToolResultBlock) GetType() types.ContentType

GetType reports the ContentBlock type marker. The Anthropic server tool emits `web_search_tool_result` blocks; we surface that string verbatim so the typed block round-trips through ContentBlock-aware code.

type WebSearchToolResultError 

type WebSearchToolResultError struct {
	Type      string `json:"type"`
	ErrorCode string `json:"error_code"`
}

WebSearchToolResultError is the alternate envelope the Anthropic server returns when web_search_tool_result.content is an error object instead of an array. Mirrors src/tools/WebSearchTool/WebSearchTool.ts:115-122 where the TS reference detects this shape and surfaces a distinct "Web search error: <error_code>" message instead of crashing the array-only parser.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.