webfetch

Documentation

Overview

Package webfetch provides URL fetching with HTML-to-markdown conversion.

Index

• Constants
• func ConvertHTMLToMarkdown(htmlContent string) string
• func ExtractTextFromHTML(htmlContent string) string
• func NewTool(client Client) tool.Tool
• type Client
  • func NewClient() Client
• type DirectClient
  • func (c *DirectClient) Fetch(ctx context.Context, req Request) (*Response, error)
• type Request
• type Response
• type ToolInput

Constants

const (
	MaxResponseSize = 5 * 1024 * 1024 // 5MB
	DefaultTimeout  = 30 * time.Second
	MaxTimeout      = 120 * time.Second
	UserAgent       = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36"
)

Functions

func ConvertHTMLToMarkdown

func ConvertHTMLToMarkdown(htmlContent string) string

ConvertHTMLToMarkdown converts HTML to simplified Markdown.

func ExtractTextFromHTML

func ExtractTextFromHTML(htmlContent string) string

ExtractTextFromHTML extracts plain text from HTML, skipping scripts/styles.

func NewTool

func NewTool(client Client) tool.Tool

NewTool creates a webfetch tool backed by the given Client.

Types

type Client

type Client interface {
	Fetch(ctx context.Context, req Request) (*Response, error)
}

Client fetches URLs and converts content.

func NewClient

func NewClient() Client

NewClient creates a new direct fetch client.

type DirectClient

type DirectClient struct{}

DirectClient fetches URLs directly via HTTP.

func (*DirectClient) Fetch

func (c *DirectClient) Fetch(ctx context.Context, req Request) (*Response, error)

type Request

type Request struct {
	URL     string `json:"url"`
	Format  string `json:"format,omitempty"`  // "markdown" (default), "text", "html"
	Timeout int    `json:"timeout,omitempty"` // seconds, max 120
}

Request is the input for a web fetch operation.

type Response

type Response struct {
	Content     string `json:"content"`
	ContentType string `json:"contentType"`
	URL         string `json:"url"`
	StatusCode  int    `json:"statusCode"`
}

Response is the result of a web fetch operation.

type ToolInput

type ToolInput struct {
	URL     string `json:"url" description:"The URL to fetch content from"`
	Format  string `` /* 157-byte string literal not displayed */
	Timeout int    `json:"timeout,omitempty" description:"Optional timeout in seconds (max 120)"`
}

ToolInput is the input schema for the webfetch tool.