pagemark

package module
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 27, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

README

Pagemark

Go Reference

Pagemark extracts the useful content from an HTML page. It returns compact Markdown, plain text, and page metadata.

Pagemark supports articles, documentation, discussions, products, listings, collections, and service pages. It can keep content from more than one page region.

Pagemark does not fetch pages. It does not run JavaScript. If a page needs JavaScript, supply the rendered HTML.

Installation

Pagemark requires Go 1.25 or a later version.

go get github.com/ryanfowler/pagemark

Quick start

package main

import (
	"fmt"
	"strings"

	"github.com/ryanfowler/pagemark"
)

func main() {
	source := `<main><h1>Guide</h1><p>Install the tool.</p></main>`
	doc, err := pagemark.Extract(strings.NewReader(source), "https://example.com/guide")
	if err != nil {
		panic(err)
	}

	fmt.Println(doc.Title)
	fmt.Println(doc.Markdown)
}

Use one of these functions:

  • Extract reads UTF-8 HTML from an io.Reader.
  • ExtractBytes reads UTF-8 HTML from a byte slice.
  • ExtractNode reads a parsed html.Node tree. It does not change the tree.

The page URL is optional. If you set it, use an absolute HTTP or HTTPS URL. Pagemark uses it to resolve relative links.

Result

Extract returns a Document. The main fields are:

  • Title: the document title.
  • Markdown: the selected content as Markdown.
  • Text: the selected content as plain text.
  • Sections: a plain-text view of the selected sections.
  • Links and Images: the safe resources that occur in the output.
  • PageType and PageTypeScore: the detected page shape and its confidence score.
  • Quality: a score for the observable quality of the output.
  • Warnings: nonfatal conditions, such as output truncation.
  • Stats: input, tree, selection, and output counts.

The title is separate from the content. Pagemark does not repeat it in Markdown, Text, or Sections.

Images are enabled by default. Pagemark records image URLs, but it does not fetch the images. Use WithIncludeImages(false) for text-only output.

Options

Pass options after the page URL:

doc, err := pagemark.ExtractBytes(
	source,
	pageURL,
	pagemark.WithPageType(pagemark.PageTypeDocumentation),
	pagemark.WithMaxOutputBytes(512<<10),
	pagemark.WithIncludeImages(false),
)

Pagemark detects the page type by default. Use WithPageType only when you know the page type. The page type changes content scores. It does not change safety rules or parser limits.

Use these options to control output:

  • WithIncludeLinks
  • WithIncludeImages
  • WithIncludeTables
  • WithIncludeMetadata
  • WithFavorPrecision
  • WithFavorRecall

Use WithFavorPrecision(true) to select less content. Use WithFavorRecall(true) to select more content. Do not enable both options. Their score changes cancel each other.

Diagnostics can use much more memory. Enable them only when you must inspect page-type scores, block scores, or rejected links:

doc, err := pagemark.ExtractBytes(source, pageURL, pagemark.WithDiagnostics(true))

Limits

Pagemark limits resource use. The default public limits are:

Resource Default Option
Input 10 MiB WithMaxInputBytes
DOM elements 200,000 WithMaxElements
DOM depth 256 WithMaxDepth
Markdown output 2 MiB WithMaxOutputBytes
Links 1,000 WithMaxLinks
Images 100 WithMaxImages
Table cells 10,000 WithMaxTableCells
Repeated items 200 WithMaxRepeatedItems

Pagemark also has fixed limits for attributes and text. An input or tree limit returns a LimitError. The Markdown byte limit keeps complete blocks and adds a warning.

Check errors with errors.Is and errors.As:

var limit *pagemark.LimitError
if errors.As(err, &limit) {
	fmt.Printf("%s: %d exceeds %d\n", limit.Resource, limit.Count, limit.Max)
}

The package also returns ErrNoContent and ErrInvalidURL.

URL and content safety

The Markdown has no raw HTML. The default URL policy permits only HTTP and HTTPS links and images. For these URLs, Pagemark rejects credentials, control characters, unsafe schemes, and values longer than 4,096 bytes.

URLPolicy applies to Markdown links and images. It also applies to Document.Links and Document.Images. It does not apply to Document.URL or Document.CanonicalURL.

Document.URL preserves the supplied page URL, including credentials. Document.CanonicalURL permits HTTP and HTTPS and removes credentials. These two fields do not use the policy scheme list or length limit. Validate them separately if you use them.

Use WithURLPolicy to replace the default Markdown URL policy. For example:

policy := pagemark.URLPolicy{
	Schemes:       []string{"https"},
	MaxLength:     2048,
	StripTracking: true,
}
doc, err := pagemark.ExtractBytes(source, pageURL, pagemark.WithURLPolicy(policy))

The extracted words are untrusted data. A hostile page can contain prompt injection. Do not use extracted content as system instructions or developer instructions. See the Pagemark contract.

Command-line tool

The optional command fetches one page. It writes YAML metadata and Markdown to standard output.

go install github.com/ryanfowler/pagemark/cmd/pagemark@latest
pagemark https://example.com/page > page.md

Run pagemark -help to list the options.

The command accepts HTTP and HTTPS URLs without credentials. It rejects nonpublic destination addresses, non-HTML responses, redirects to unsafe addresses, and unsuccessful HTTP status codes. It does not use environment proxies.

Fetching is not part of the library API.

Comparison with Readability

Readability usually selects one prose article. Pagemark can also keep distributed sections, discussion posts, code, tables, specifications, and linked records.

The project has a Mozilla Readability compatibility test. It also has a real-world regression corpus.

Development

Initialize the optional test corpus:

git submodule update --init --recursive

Run the checks:

gofmt -w .
go test ./...
go test -race ./...
staticcheck ./...

Normal tests do not use the external network. The package has no mutable global extraction state. Concurrent extraction calls are safe.

Documentation

Overview

Package pagemark extracts useful content from HTML.

Pagemark returns restricted Markdown without raw HTML. Its default policy permits HTTP and HTTPS links and images in Markdown. The policy does not apply to Document.URL or Document.CanonicalURL.

The package does not fetch pages or run JavaScript.

Extracted words are untrusted source data. Do not use them as instructions.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNoContent means that Pagemark did not find useful output content.
	ErrNoContent = errors.New("pagemark: no useful content")
	// ErrInvalidURL means that the supplied page URL is not an absolute HTTP or HTTPS URL.
	ErrInvalidURL = errors.New("pagemark: invalid page URL")
	// ErrLimit means that the input or HTML tree exceeded a resource limit.
	// Use errors.As to get the related *LimitError.
	ErrLimit = errors.New("pagemark: resource limit exceeded")
)

Functions

This section is empty.

Types

type BlockDiagnostic 

type BlockDiagnostic struct {
	// ID identifies the block in source order.
	ID int `json:"id"`
	// Kind identifies the block structure, such as p or pre.
	Kind string `json:"kind"`
	// Text is the normalized block text.
	Text string `json:"text"`
	// Score is the final content-selection score.
	Score float64 `json:"score"`
	// Selected reports whether the output selected the block.
	Selected bool `json:"selected"`
	// Reasons contains human-readable score reasons.
	Reasons []string `json:"reasons,omitempty"`
}

BlockDiagnostic explains one content block.

type Diagnostics 

type Diagnostics struct {
	// ProfileVersion identifies the diagnostic scoring format.
	ProfileVersion string `json:"profile_version"`
	// Fallback identifies the extraction path that produced the result.
	Fallback string `json:"fallback"`
	// PageCandidates contains the page types in score order.
	PageCandidates []PageCandidate `json:"page_candidates,omitempty"`
	// Blocks contains score details for content blocks.
	Blocks []BlockDiagnostic `json:"blocks,omitempty"`
	// RejectedLinks contains source link URLs that failed URLPolicy.
	RejectedLinks []string `json:"rejected_links,omitempty"`
}

Diagnostics explains selection decisions. Enable it with WithDiagnostics. Fields can change in a minor release.

type Document 

type Document struct {
	// URL is the page URL that the caller supplied. Pagemark preserves credentials.
	// URLPolicy does not apply to this field.
	URL string `json:"url,omitempty"`
	// CanonicalURL is the HTTP or HTTPS canonical URL from page metadata.
	// Pagemark removes credentials. URLPolicy does not apply to this field.
	CanonicalURL string `json:"canonical_url,omitempty"`
	// Title is the document title.
	Title string `json:"title,omitempty"`
	// Description is the page description from metadata.
	Description string `json:"description,omitempty"`
	// Author is the author name from metadata.
	Author string `json:"author,omitempty"`
	// SiteName is the site or publication name from metadata.
	SiteName string `json:"site_name,omitempty"`
	// Language is the language value from the HTML lang attribute.
	Language string `json:"language,omitempty"`
	// PublishedTime is the publication value from page metadata.
	PublishedTime string `json:"published_time,omitempty"`
	// PageType is the detected or specified page shape.
	PageType PageType `json:"page_type"`
	// PageTypeScore is the page-type confidence from 0 through 1.
	// An explicit page type has a score of 1.
	PageTypeScore float64 `json:"page_type_score"`
	// Markdown is the selected content as restricted Markdown.
	Markdown string `json:"markdown"`
	// Text is the selected content as plain text.
	Text string `json:"text"`
	// Sections contains a plain-text view of the selected sections.
	Sections []Section `json:"sections,omitempty"`
	// Links contains the links that occur in Markdown.
	Links []Link `json:"links,omitempty"`
	// Images contains the useful images that occur in Markdown.
	Images []Image `json:"images,omitempty"`
	// Quality measures observable output properties from 0 through 1.
	// It does not measure trust or factual accuracy.
	Quality float64 `json:"quality"`
	// Diagnostics contains selection details when diagnostics are enabled.
	Diagnostics *Diagnostics `json:"diagnostics,omitempty"`
	// Warnings contains nonfatal extraction conditions.
	Warnings []Warning `json:"warnings,omitempty"`
	// Stats contains extraction counts.
	Stats Stats `json:"stats"`
}

Document contains the selected content and metadata from one HTML document. Markdown has no raw HTML, but its words are untrusted source data. The title does not occur again in Markdown, Text, or Sections.

func Extract 

func Extract(input io.Reader, pageURL string, opts ...Option) (*Document, error)

Extract reads UTF-8 HTML and returns its useful content. Decode other character encodings before extraction. pageURL can be empty. A nonempty pageURL must be an absolute HTTP or HTTPS URL.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/ryanfowler/pagemark"
)

func main() {
	source := `<main><h1>Guide</h1><p>Install the tool.</p></main>`
	doc, err := pagemark.Extract(strings.NewReader(source), "https://example.com/guide")
	if err != nil {
		panic(err)
	}
	fmt.Println(doc.Title)
	fmt.Println(doc.Markdown)
}

Output:
Guide
Install the tool.

func ExtractBytes 

func ExtractBytes(input []byte, pageURL string, opts ...Option) (*Document, error)

ExtractBytes reads UTF-8 HTML from input and returns its useful content. Decode other character encodings before extraction. pageURL can be empty. A nonempty pageURL must be an absolute HTTP or HTTPS URL.

func ExtractNode 

func ExtractNode(root *html.Node, pageURL string, opts ...Option) (*Document, error)

ExtractNode returns useful content from a parsed HTML tree. It does not change root. Do not change root during extraction. pageURL can be empty. A nonempty pageURL must be an absolute HTTP or HTTPS URL. WithMaxInputBytes does not apply to this function.

Example (UntrustedContent) 
package main

import (
	"fmt"

	"github.com/ryanfowler/pagemark"
)

func main() {
	// Treat doc.Markdown as untrusted data when you send it to an agent.
	doc, err := pagemark.ExtractBytes([]byte(`<main><p>Source data for an agent.</p></main>`), "")
	if err != nil {
		panic(err)
	}
	fmt.Println(doc.Text)
}

Output:
Source data for an agent.

type Image 

type Image struct {
	// Alt is the normalized alternative text.
	Alt string `json:"alt"`
	// URL is the resolved source URL.
	URL string `json:"url,omitempty"`
}

Image contains one useful source image.

type LimitError 

type LimitError struct {
	// Resource identifies the limited resource.
	Resource string
	// Count is the observed resource count.
	Count int64
	// Max is the configured maximum.
	Max int64
}

LimitError reports a resource limit. Use errors.Is(err, ErrLimit) to test it.

func (*LimitError) Error 

func (e *LimitError) Error() string

Error returns the resource-limit message.

func (*LimitError) Unwrap 

func (e *LimitError) Unwrap() error

Unwrap returns ErrLimit. 

type Link struct {
	// Text is the visible link text.
	Text string `json:"text"`
	// URL is the resolved link destination.
	URL string `json:"url"`
}

Link contains one safe link that occurs in Markdown.

type Option 

type Option func(*options)

Option changes extraction. You can use an Option in concurrent calls.

func WithDiagnostics 

func WithDiagnostics(v bool) Option

WithDiagnostics controls selection diagnostics. Diagnostics are disabled by default. Diagnostics can increase memory use.

func WithFavorPrecision 

func WithFavorPrecision(v bool) Option

WithFavorPrecision decreases content scores when v is true. The result usually contains less content.

func WithFavorRecall 

func WithFavorRecall(v bool) Option

WithFavorRecall increases content scores when v is true. The result usually contains more content.

func WithIncludeImages 

func WithIncludeImages(v bool) Option

WithIncludeImages controls useful images in Markdown and Document.Images. Images are enabled by default. Set v to false for text-only output. 

func WithIncludeLinks(v bool) Option

WithIncludeLinks controls links in Markdown and Document.Links. If v is false, visible link text remains without a destination.

func WithIncludeMetadata 

func WithIncludeMetadata(v bool) Option

WithIncludeMetadata controls metadata fields such as Document.Title. The title does not occur in the content when metadata is disabled.

func WithIncludeTables 

func WithIncludeTables(v bool) Option

WithIncludeTables controls Markdown table syntax. Tables are enabled by default. If v is false, Pagemark keeps table content without table syntax.

func WithLogger 

func WithLogger(v *slog.Logger) Option

WithLogger sets a logger for extraction debug messages. A nil logger disables messages.

func WithMaxDepth 

func WithMaxDepth(v int) Option

WithMaxDepth sets the maximum DOM depth. The default is 256. A nonpositive value disables this limit.

func WithMaxElements 

func WithMaxElements(v int) Option

WithMaxElements sets the maximum number of HTML elements. The default is 200,000. A nonpositive value disables this limit.

func WithMaxImages 

func WithMaxImages(v int) Option

WithMaxImages sets the maximum number of images in Markdown. The default is 100. A nonpositive value removes all images.

func WithMaxInputBytes 

func WithMaxInputBytes(v int64) Option

WithMaxInputBytes sets the maximum HTML input size. The default is 10 MiB. A nonpositive value disables this limit. This option does not apply to ExtractNode. 

func WithMaxLinks(v int) Option

WithMaxLinks sets the maximum number of links in Markdown. The default is 1,000. Link text remains when the output reaches the limit. A nonpositive value removes all link destinations.

func WithMaxOutputBytes 

func WithMaxOutputBytes(v int) Option

WithMaxOutputBytes sets the maximum Markdown size. The default is 2 MiB. A nonpositive value disables this limit. Truncation occurs at a block boundary.

func WithMaxRepeatedItems 

func WithMaxRepeatedItems(v int) Option

WithMaxRepeatedItems sets the item limit for listings and collections. The default is 200. A nonpositive value disables this limit.

func WithMaxTableCells 

func WithMaxTableCells(v int) Option

WithMaxTableCells sets the total table-cell limit. The default is 10,000. Pagemark converts a table that exceeds the limit to fallback content. A nonpositive value converts all tables to fallback content.

func WithPageType 

func WithPageType(v PageType) Option

WithPageType overrides page-type detection. The selected type changes content scores. It does not change limits or URL rules.

func WithProfile 

func WithProfile(v Profile) Option

WithProfile overrides page-type detection with v.PageType.

func WithURLPolicy 

func WithURLPolicy(v URLPolicy) Option

WithURLPolicy replaces the default Markdown link and image URL policy. It does not change Document.URL or Document.CanonicalURL.

type PageCandidate 

type PageCandidate struct {
	// Type is the possible page type.
	Type PageType `json:"type"`
	// Score is the raw classification score. It is not a confidence value.
	Score float64 `json:"score"`
}

PageCandidate contains one possible page type and its raw score.

type PageType 

type PageType string

PageType identifies the main shape of a page. 

const (
	// PageTypeArticle identifies a page with one main prose work.
	PageTypeArticle PageType = "article"
	// PageTypeDocumentation identifies a guide or reference page.
	PageTypeDocumentation PageType = "documentation"
	// PageTypeDiscussion identifies a question, thread, or conversation.
	PageTypeDiscussion PageType = "discussion"
	// PageTypeProduct identifies one product detail page.
	PageTypeProduct PageType = "product"
	// PageTypeListing identifies a page with repeated linked records.
	PageTypeListing PageType = "listing"
	// PageTypeCollection identifies a page with repeated related items.
	PageTypeCollection PageType = "collection"
	// PageTypeService identifies a page that describes a service.
	PageTypeService PageType = "service"
	// PageTypeGeneric identifies a page with no more specific shape.
	PageTypeGeneric PageType = "generic"
)

type Profile 

type Profile struct {
	// PageType is the page shape that the profile uses.
	PageType PageType
}

Profile specifies a page profile. Use WithPageType when you only need to override the detected page type.

type Section 

type Section struct {
	// Heading is the section heading. It is empty for content before a heading.
	Heading string `json:"heading,omitempty"`
	// Text is the plain text in the section.
	Text string `json:"text"`
}

Section contains one selected section as plain text.

type Stats 

type Stats struct {
	// InputBytes is the HTML byte count for Extract or ExtractBytes.
	// It is zero for ExtractNode.
	InputBytes int `json:"input_bytes"`
	// Elements is the number of indexed HTML elements.
	Elements int `json:"elements"`
	// TextBytes is the number of indexed source-text bytes.
	TextBytes int `json:"text_bytes"`
	// Blocks is the number of content blocks that Pagemark scored.
	Blocks int `json:"blocks"`
	// SelectedBlocks is the number of blocks in the output.
	SelectedBlocks int `json:"selected_blocks"`
	// OutputBytes is the Markdown byte count.
	OutputBytes int `json:"output_bytes"`
}

Stats contains extraction counts.

type URLPolicy 

type URLPolicy struct {
	// Schemes contains the permitted link and image URL schemes.
	// Scheme checks ignore case.
	Schemes []string
	// AllowMailto permits mailto links in addition to Schemes.
	AllowMailto bool
	// MaxLength is the maximum source link or image URL length in bytes.
	// A nonpositive value does not set a length limit.
	MaxLength int
	// StripTracking removes utm_*, fbclid, and gclid parameters from links and images.
	StripTracking bool
}

URLPolicy controls link and image URLs from the Markdown converter. It applies to Markdown links and images, Document.Links, and Document.Images. It does not apply to Document.URL or Document.CanonicalURL.

type Warning 

type Warning struct {
	// Code is a short machine-readable identifier.
	Code string `json:"code"`
	// Message describes the condition.
	Message string `json:"message"`
}

Warning reports a nonfatal extraction condition.

Source Files

View all Source files

Directories

Path Synopsis
cmd
diagnose-blocks command
Command diagnose-blocks prints scoring diagnostics for snippets in a local HTML file.
 Command diagnose-blocks prints scoring diagnostics for snippets in a local HTML file.
pagemark command
Command pagemark fetches a web page and writes extracted Markdown.
 Command pagemark fetches a web page and writes extracted Markdown.
internal
dom
Package dom contains shared HTML tree rules.
 Package dom contains shared HTML tree rules.
markdown
Package markdown converts selected HTML nodes to a safe Markdown tree.
 Package markdown converts selected HTML nodes to a safe Markdown tree.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.