pagemark

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 24, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

README

pagemark

Go Reference

Pagemark extracts useful web page content as compact, safe Markdown. It supports prose and structured pages. It uses block classification and can keep multiple content regions.

Pagemark does not fetch pages or run JavaScript. Supply rendered HTML when a page needs JavaScript.

Install

go get github.com/ryanfowler/pagemark

Use

package main

import (
    "fmt"
    "strings"

    "github.com/ryanfowler/pagemark"
)

func main() {
    source := `<main><h1>Guide</h1><p>Install the tool.</p></main>`
    doc, err := pagemark.Extract(strings.NewReader(source), "https://example.com/guide")
    if err != nil {
        panic(err)
    }
    fmt.Println(doc.Markdown)
}

You can also use ExtractBytes or ExtractNode. ExtractNode does not change the supplied tree.

The Document result contains metadata, a page type, Markdown, plain text, sections, safe links, useful images, a quality score, warnings, and extraction statistics. Useful images are included by default as Markdown image syntax and in Document.Images; Pagemark records their safe remote URLs but does not fetch them. Pass WithIncludeImages(false) for text-only output.

Enable diagnostics only when you need block scores and rejected-link details:

doc, err := pagemark.ExtractBytes(source, pageURL, pagemark.WithDiagnostics(true))

Safety

The Markdown has no raw HTML. The default policy permits only HTTP and HTTPS links. Pagemark rejects link credentials, control characters, unsafe schemes, and long URLs.

The extracted words are untrusted source data. Pagemark does not protect an agent from prompt injection. Supply the result to an agent as data, not as privileged instructions. See the contract.

Limits

Pagemark has default limits for input size, DOM size, depth, text, output, links, images, tables, and repeated items. Use options such as WithMaxInputBytes, WithMaxElements, and WithMaxOutputBytes to change these limits. Output truncation occurs only at a block boundary and adds a warning.

Page types

Pagemark detects article, documentation, discussion, product, listing, collection, service, and generic pages. Use WithPageType when the caller has a reliable type. Type profiles change scores. They do not change parser limits or URL safety.

Command-line tool

The repository has one optional command. It fetches a page and writes Markdown to standard output. Page metadata is included as YAML frontmatter, followed by an empty line and the extracted content. Fetching remains outside the library API.

go install github.com/ryanfowler/pagemark/cmd/pagemark@latest
pagemark https://example.com/page > page.md

Use pagemark -help to see timeout, input limit, and User-Agent options. The command accepts HTTP and HTTPS URLs without credentials. It resolves each initial and redirect host itself and connects to the resolved public IP address. It rejects loopback, private, link-local, multicast, reserved, and documentation address ranges. It does not use environment proxies. These rules also prevent a DNS change between validation and connection from redirecting the connection to a private address. The command also rejects non-HTML responses and unsuccessful HTTP status codes.

Normal tests do not access the external network.

Benchmark method

The benchmark uses multiset word precision, recall, and F1. It also checks required and forbidden snippets. The WCXB version file pins the source commit and archive SHA-256 checksum.

On WCXB v1.0 development data, an implementation run on the initial profile gave 0.760 overall F1. Results vary only when code or profiles change. Use the held-out split only for a release review. Do not compare this number with a benchmark that uses a different normalization method.

The project includes synthetic safety and structure tests. It also keeps a small, checksummed real-world regression corpus with declarative page-type and content expectations. Tests use frozen snapshots and never refresh them from the network.

The full WCXB data is not in this repository because it is large. WCXB uses the CC BY 4.0 license.

Difference from Readability

Readability is an article specialist that usually selects one prose region. Pagemark uses its own page-type-aware extraction pipeline and can keep distributed sections, discussion posts, code, tables, specifications, and linked records.

Development

gofmt -w .
go test ./...
go test -race ./...
staticcheck ./...

Run the end-to-end benchmarks against the frozen real-world corpus with:

go test -run '^$' -bench '^BenchmarkExtractRealWorld$' -benchmem

The benchmark includes HTML parsing and reports time, throughput, bytes, and allocations for representative articles, documentation, discussions, products, listings, services, and generic pages. CPU and allocation profiles can be captured by adding -cpuprofile cpu.out -memprofile mem.out and inspected with go tool pprof.

The package has no mutable global extraction state. Concurrent calls are safe.

Documentation

Overview

Package pagemark extracts useful page content as safe Markdown.

The output contains untrusted source data. It does not protect an agent from prompt injection. The package does not fetch pages or run JavaScript.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	ErrNoContent  = errors.New("pagemark: no useful content")
	ErrInvalidURL = errors.New("pagemark: invalid page URL")
	ErrLimit      = errors.New("pagemark: resource limit exceeded")
)

Functions

This section is empty.

Types

type BlockDiagnostic 

type BlockDiagnostic struct {
	ID       int      `json:"id"`
	Kind     string   `json:"kind"`
	Text     string   `json:"text"`
	Score    float64  `json:"score"`
	Selected bool     `json:"selected"`
	Reasons  []string `json:"reasons,omitempty"`
}

BlockDiagnostic explains one content block.

type Diagnostics 

type Diagnostics struct {
	ProfileVersion string            `json:"profile_version"`
	Fallback       string            `json:"fallback"`
	PageCandidates []PageCandidate   `json:"page_candidates,omitempty"`
	Blocks         []BlockDiagnostic `json:"blocks,omitempty"`
	RejectedLinks  []string          `json:"rejected_links,omitempty"`
}

Diagnostics explains selection decisions. Its format can grow in minor releases.

type Document 

type Document struct {
	URL           string       `json:"url,omitempty"`
	CanonicalURL  string       `json:"canonical_url,omitempty"`
	Title         string       `json:"title,omitempty"`
	Description   string       `json:"description,omitempty"`
	Author        string       `json:"author,omitempty"`
	SiteName      string       `json:"site_name,omitempty"`
	Language      string       `json:"language,omitempty"`
	PublishedTime string       `json:"published_time,omitempty"`
	PageType      PageType     `json:"page_type"`
	PageTypeScore float64      `json:"page_type_score"`
	Markdown      string       `json:"markdown"`
	Text          string       `json:"text"`
	Sections      []Section    `json:"sections,omitempty"`
	Links         []Link       `json:"links,omitempty"`
	Images        []Image      `json:"images,omitempty"`
	Quality       float64      `json:"quality"`
	Diagnostics   *Diagnostics `json:"diagnostics,omitempty"`
	Warnings      []Warning    `json:"warnings,omitempty"`
	Stats         Stats        `json:"stats"`
}

Document contains safe Markdown and metadata from one HTML document. Markdown is untrusted source data. Do not use it as privileged instructions.

func Extract 

func Extract(input io.Reader, pageURL string, opts ...Option) (*Document, error)

Extract reads UTF-8 HTML and extracts useful content. Callers must decode input in other character encodings before calling Extract.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/ryanfowler/pagemark"
)

func main() {
	source := `<main><h1>Guide</h1><p>Install the tool.</p></main>`
	doc, err := pagemark.Extract(strings.NewReader(source), "https://example.com/guide")
	if err != nil {
		panic(err)
	}
	fmt.Println(doc.Markdown)
}

Output:
# Guide

Install the tool.

func ExtractBytes 

func ExtractBytes(input []byte, pageURL string, opts ...Option) (*Document, error)

ExtractBytes extracts useful content from UTF-8 HTML bytes.

func ExtractNode 

func ExtractNode(root *html.Node, pageURL string, opts ...Option) (*Document, error)

ExtractNode extracts useful content from a parsed HTML tree. It does not change root. The caller must not change root during extraction.

Example (UntrustedContent) 
package main

import (
	"fmt"

	"github.com/ryanfowler/pagemark"
)

func main() {
	// Keep doc.Markdown in an untrusted data channel when you supply it to an agent.
	doc, err := pagemark.ExtractBytes([]byte(`<main><p>Source data for an agent.</p></main>`), "")
	if err != nil {
		panic(err)
	}
	fmt.Println(doc.Text)
}

Output:
Source data for an agent.

type Image 

type Image struct {
	Alt string `json:"alt"`
	URL string `json:"url,omitempty"`
}

Image describes a useful source image.

type LimitError 

type LimitError struct {
	Resource string
	Count    int64
	Max      int64
}

LimitError reports a resource limit.

func (*LimitError) Error 

func (e *LimitError) Error() string

func (*LimitError) Unwrap 

func (e *LimitError) Unwrap() error
type Link struct {
	Text string `json:"text"`
	URL  string `json:"url"`
}

Link is a safe link that occurs in Markdown.

type Option 

type Option func(*options)

Option changes extraction. Options are safe for concurrent reuse.

func WithDiagnostics 

func WithDiagnostics(v bool) Option

func WithFavorPrecision 

func WithFavorPrecision(v bool) Option

func WithFavorRecall 

func WithFavorRecall(v bool) Option

func WithIncludeImages 

func WithIncludeImages(v bool) Option

WithIncludeImages controls useful images in Markdown and Document.Images. Images are included by default; pass false for text-only output. 

func WithIncludeLinks(v bool) Option

func WithIncludeMetadata 

func WithIncludeMetadata(v bool) Option

func WithIncludeTables 

func WithIncludeTables(v bool) Option

func WithLogger 

func WithLogger(v *slog.Logger) Option

func WithMaxDepth 

func WithMaxDepth(v int) Option

func WithMaxElements 

func WithMaxElements(v int) Option

func WithMaxImages 

func WithMaxImages(v int) Option

func WithMaxInputBytes 

func WithMaxInputBytes(v int64) Option
func WithMaxLinks(v int) Option

func WithMaxOutputBytes 

func WithMaxOutputBytes(v int) Option

func WithMaxRepeatedItems 

func WithMaxRepeatedItems(v int) Option

func WithMaxTableCells 

func WithMaxTableCells(v int) Option

func WithPageType 

func WithPageType(v PageType) Option

func WithProfile 

func WithProfile(v Profile) Option

func WithURLPolicy 

func WithURLPolicy(v URLPolicy) Option

type PageCandidate 

type PageCandidate struct {
	Type  PageType `json:"type"`
	Score float64  `json:"score"`
}

PageCandidate is a possible page type.

type PageType 

type PageType string

PageType identifies the main shape of a page. 

const (
	PageTypeArticle       PageType = "article"
	PageTypeDocumentation PageType = "documentation"
	PageTypeDiscussion    PageType = "discussion"
	PageTypeProduct       PageType = "product"
	PageTypeListing       PageType = "listing"
	PageTypeCollection    PageType = "collection"
	PageTypeService       PageType = "service"
	PageTypeGeneric       PageType = "generic"
)

type Profile 

type Profile struct {
	PageType PageType
}

Profile selects a page profile. Use WithPageType for normal overrides.

type Section 

type Section struct {
	Heading string `json:"heading,omitempty"`
	Text    string `json:"text"`
}

Section identifies a retained section.

type Stats 

type Stats struct {
	InputBytes     int `json:"input_bytes"`
	Elements       int `json:"elements"`
	TextBytes      int `json:"text_bytes"`
	Blocks         int `json:"blocks"`
	SelectedBlocks int `json:"selected_blocks"`
	OutputBytes    int `json:"output_bytes"`
}

Stats contains bounded extraction counts.

type URLPolicy 

type URLPolicy struct {
	Schemes       []string
	AllowMailto   bool
	MaxLength     int
	StripTracking bool
}

URLPolicy controls URLs in output.

type Warning 

type Warning struct {
	Code    string `json:"code"`
	Message string `json:"message"`
}

Warning reports a nonfatal extraction condition.

Source Files

View all Source files

Directories

Path Synopsis
cmd
diagnose-blocks command
Command diagnose-blocks prints scoring diagnostics for snippets in a local HTML file.
 Command diagnose-blocks prints scoring diagnostics for snippets in a local HTML file.
pagemark command
Command pagemark fetches a web page and writes extracted Markdown.
 Command pagemark fetches a web page and writes extracted Markdown.
internal
dom
Package dom contains shared HTML tree rules.
 Package dom contains shared HTML tree rules.
markdown
Package markdown converts selected HTML nodes to a safe Markdown tree.
 Package markdown converts selected HTML nodes to a safe Markdown tree.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.