Documentation
¶
Overview ¶
Package webfetch implements the SSRF-safe HTTP fetch primitive used by the web_fetch tool. It provides configurable options, an IP guard that blocks loopback/private/link-local addresses, a custom dialer, redirect validation, content-type gating, and a bounded body reader.
Index ¶
Constants ¶
const ToolName = "web_fetch"
ToolName is the model-facing name of the web_fetch tool.
Variables ¶
var ApprovalDescriptor = approval.Descriptor{
ID: "web:fetch",
CommandPrefix: "",
Notice: "The model wants to fetch this URL.",
HarnessVerb: "fetch",
EvaluatorPrompt: evaluatorPrompt,
}
ApprovalDescriptor describes how a web_fetch approval request is presented and evaluated. It is the single source of truth for this tool's approval metadata: the approval overlay, harness logger, and AI evaluator all read it instead of switching on a category enum.
var DefaultOptions = Options{ Timeout: 15 * time.Second, MaxBodyBytes: 2 << 20, MaxRedirects: 5, AllowedContentTypes: []string{"text/html", "text/plain", "application/xhtml+xml"}, }
DefaultOptions are conservative defaults suitable for production use.
Functions ¶
func ToolDef ¶
func ToolDef() tooldef.Definition
ToolDef returns the jungi tool definition for web_fetch.
Types ¶
type Extracted ¶
type Extracted struct {
// Title is the page title extracted by the readability parser.
Title string
// Markdown is the lean markdown conversion of the main content.
Markdown string
}
Extracted holds the result of extracting and converting a web page.
func DefaultFetcher ¶
DefaultFetcher performs a real SSRF-safe fetch followed by readability extraction and markdown conversion.
func ExtractMarkdown ¶
ExtractMarkdown extracts the main content from htmlBytes (the raw HTTP body fetched from pageURL) and converts it to lean markdown. It:
- runs go-readability to isolate the main article content and title,
- converts the resulting HTML to markdown (headings, lists, links, code, blockquote, tables only),
- strips data: URIs, inline SVG blocks, and collapses redundant blank lines.
type Fetcher ¶
Fetcher is the function signature for the HTTP fetch + extraction pipeline. It is injectable so tests can substitute a stub without hitting the network.
type Options ¶
type Options struct {
// Timeout is the total request timeout including reading the body.
Timeout time.Duration
// MaxBodyBytes caps the number of body bytes read. Content beyond this
// limit is silently discarded (not an error); the caller receives a
// partial body.
MaxBodyBytes int64
// MaxRedirects is the maximum number of redirects to follow. Exceeding
// this limit returns an error.
MaxRedirects int
// AllowedContentTypes is the set of MIME type prefixes that are
// accepted. Responses whose Content-Type does not start with one of
// these values are rejected. A nil or empty slice falls back to
// DefaultOptions.AllowedContentTypes; use a non-empty slice to narrow
// the set. There is no way to accept all content types.
AllowedContentTypes []string
// OnBlocked, if non-nil, is called whenever a request or redirect is
// blocked by the SSRF guard or the content-type gate. Use this to emit
// diagnostic logs without importing a specific logger into this package.
OnBlocked func(msg string)
// contains filtered or unexported fields
}
Options configures the fetch behaviour.
type Result ¶
type Result struct {
// Body contains up to MaxBodyBytes of the response body.
Body []byte
// ContentType is the raw Content-Type header value.
ContentType string
// StatusCode is the HTTP response status code.
StatusCode int
}
Result holds the response body and metadata from a successful fetch.
func Fetch ¶
Fetch performs an SSRF-safe HTTP GET of url using opts. It:
- rejects non-http/https schemes,
- resolves the target host and rejects private/loopback/link-local IPs,
- follows up to opts.MaxRedirects hops, re-validating each destination,
- gates the response on opts.AllowedContentTypes,
- reads at most opts.MaxBodyBytes of the response body.
type Tool ¶
type Tool struct {
// contains filtered or unexported fields
}
Tool implements tool.Tool for the web_fetch tool. It binds the sandbox (for the output directory), a lazy approval accessor, and an injectable fetcher.
func New ¶
New returns a web_fetch Tool bound to sb, the lazy approval accessor, and the fetcher. Pass DefaultFetcher for production; pass a stub in tests.
func (*Tool) Definition ¶
func (*Tool) Definition() tooldef.Definition
Definition delegates to the package-level ToolDef.
func (*Tool) Execute ¶
Execute approval-gates the fetch, runs the fetcher on approval, writes the result to the pages directory, and returns the path, title, line count, and a capped first-window preview.
Source Files
¶
- approval.go
- content.go
- fetch.go
- tool.go