header

package module

v0.0.3 Latest Latest Go to latest Published: Oct 30, 2026 License: MIT Imports: 15 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/arran4/header

Links

Open Source Insights

README ¶

header

Header Project Card

header is a Go command-line application and library that extracts human-readable metadata from a given URL. It scrapes OpenGraph information, Twitter cards, JSON-LD, AppLinks, and YouTube specifics (watch time, video IDs, titles), displaying them cleanly in the terminal.

Documentation

Developer documentation — getting started, how-to guides, concepts, CLI reference and contribution guidance. The site deploys from main through GitHub Pages Actions; see docs/ for its source.

Prerequisites and Supported Platforms

Build targets: Linux and Windows on amd64, arm64 and 386; macOS on amd64 and arm64.
Building from Source: Requires Go 1.26 or higher.
Go Library: Importable as github.com/arran4/header for embedding metadata extraction directly into Go applications.

Installation

GitHub Releases

No GitHub releases have been published yet. After the first release, binaries will be available at https://github.com/arran4/header/releases.

Homebrew (after the first release)

brew tap arran4/homebrew-tap
brew install --cask header

Go install

go install github.com/arran4/header/cmd/header@latest

Commands

`show` / `extract` (default)

Extracts human-readable metadata from a given URL.

header https://youtube.com/watch?v=dQw4w9WgXcQ
header extract https://youtube.com/watch?v=dQw4w9WgXcQ

Example Output:

URL: https://youtube.com/watch?v=dQw4w9WgXcQ
Title: Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster) - YouTube
Description: The official video for “Never Gonna Give You Up” by Rick Astley. Never: The Autobiography 📚 OUT NOW! Follow this link to get your copy and listen to Rick’s ...
Cover Images:
  - https://i.ytimg.com/vi/dQw4w9WgXcQ/maxresdefault.jpg

Output Formats The tool supports multiple output formatting templates using the --style flag:

default: The default format printing concise, highlighted essential fields (URL, title, description, cover images).
all_fields: A detailed format printing all captured metadata grouped logically.
all_fields_plain: A detailed plain text format.
json: Prints the raw page metadata as JSON.
discord: A format tailored for Discord metadata embedding.
jsonld: Prints human readable JSON-LD metadata.
opengraph: Prints OpenGraph metadata only.
twitter: Prints Twitter card metadata only.

Example of retrieving all fields in JSON:

header extract --style json https://example.com

Display Cover Images The --show-image flag will download the image and use your system's default image viewer (xdg-open, open, or start) to display it.

header extract --show-image https://example.com/some/article

Experimental Shiny UI: You can build the app with the shiny tag to display the image using an experimental UI window:

go build -tags shiny ./cmd/header
./header extract --show-image https://example.com/some/article

Test Diagnostics If you want to debug what metadata a site is missing, you can use --test:

header extract --test https://example.com

`edit`

Edits HTML file head/meta fields.

header edit head -S '<link rel="canonical" href="https://example.com/"/>' index.html
header edit meta -s "new value" "description" index.html

The edit command operates in two modes: head and meta.

header edit head [flags] <file> targets the root <head> block, allowing you to manipulate standard HTML links and elements.
header edit meta [flags] <name> <file> specifically targets a <meta name="..."> or <meta property="..."> tag for modification.

Options:

-s: set (removes all existing occurrences of this tag and inserts the new one).
-S: add to set (appends a new one, does not remove existing tags).
-r: replace (replaces exactly 1 existing value).
-n: replace number x (only works with -r, specifies which index to replace).

Use - as the filename to read from stdin and write to stdout, which is useful for chaining in pipes.

`trace-redirects`

Traces URL HTTP standard redirects as well as HTML meta-refresh redirects.

header trace-redirects https://example.com/redirect

Customization flags include --store-cookies, --redirect-limit, --update-referer, and --redirect-delay.

`verify`

Checks all URLs and other details; a subset of lint rules mostly concerned with valid info (URL reachability and image validity).

header verify https://example.com

`lint`

Lints the metadata of a URL.

header lint https://example.com
header lint --list all https://example.com
header lint --rule twitter,opengraph https://example.com

`lint-rule-lists`

Lists available lint rule lists.

header lint-rule-lists

`lint-rules`

Lists available lint rules.

header lint-rules

`card`

Exports a "card" (as an image) including the image, description, and metadata from a URL. Output can be to a file, clipboard, or stdout (specify an output destination subcommand: file, clipboard, or stdout).

header card file --out card.png https://example.com
header card clipboard https://example.com

`view`

Views images or icons from a URL (positional syntax: header view <viewType> <url>).

header view image https://example.com/image.png
header view icon https://example.com

`httpretty`

Fetches a URL and prints HTTP requests and responses using httpretty.

header httpretty https://example.com

Development & Testing

See AGENTS.md for contribution and generated-file contracts.

Run unit, integration, and race tests using:

go test -race ./...

Use golangci-lint v2.15.0, matching CI, to run lint checks:

golangci-lint run

To regenerate CLI command definitions:

go generate ./...

The generator is pinned as a Go tool dependency in go.mod. The generation notes explain the small main-template overlay and deterministic provenance settings.

License

This project is licensed under the MIT License; see LICENSE and third-party notices. Fetched website content and third-party assets retain their owners’ rights. See SECURITY.md before using the fetcher in a privileged service.

Documentation ¶

Overview ¶

Package header provides tools for extracting metadata from web pages.

Index ¶

func ApplyEdit(r io.Reader, opts EditOptions) (*html.Node, error)
func FindNodes(n *html.Node, tag string, match func(*html.Node) bool) []*html.Node
func GetOrCreateHead(doc *html.Node) *html.Node
func RegisterExtractor(e Extractor)
func RemoveNodes(nodes []*html.Node)
func SetContent(n *html.Node, text string)
type AlternateLink
type EditOptions
type Extractor
type FeedLink
type FetchOptions
type GroupedProperty
type Icon
type KnownMetaTracker
- func (t *KnownMetaTracker) AddRaw(key, value string)
- func (t *KnownMetaTracker) MarkKnown(key string)
- func (t *KnownMetaTracker) ProcessUnknown(excludedKeys []string) []UnknownMeta
type PageInfo
- func FetchURL(urlStr string, opts FetchOptions) (*PageInfo, error)
- func (p *PageInfo) CoverImages() []string
- func (p *PageInfo) GroupedProperties() []GroupedProperty
- func (p *PageInfo) Mutate(fn func(*PageInfo))
type Processor
type UnknownMeta

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func ApplyEdit ¶

func ApplyEdit(r io.Reader, opts EditOptions) (*html.Node, error)

ApplyEdit parses an HTML document from the reader, applies the edit options, and returns the modified document.

func FindNodes ¶

func FindNodes(n *html.Node, tag string, match func(*html.Node) bool) []*html.Node

FindNodes finds all HTML nodes with the given tag that match the predicate.

func GetOrCreateHead ¶

func GetOrCreateHead(doc *html.Node) *html.Node

GetOrCreateHead finds the <head> element or creates it if it doesn't exist.

func RegisterExtractor ¶

func RegisterExtractor(e Extractor)

RegisterExtractor adds an extractor to the list of extractors used when fetching a URL.

func RemoveNodes ¶

func RemoveNodes(nodes []*html.Node)

RemoveNodes removes the specified nodes from their parents.

func SetContent ¶

func SetContent(n *html.Node, text string)

SetContent removes all children of a node and appends a single text node.

Types ¶

type AlternateLink ¶

type AlternateLink struct {
	URL   string `json:",omitempty"`
	Media string `json:",omitempty"`
}

AlternateLink represents a parsed alternate link.

type EditOptions ¶

type EditOptions struct {
	CommandType  string // "head" or "meta"
	MetaName     string
	SetVal       *string
	AddVal       *string
	ReplaceVal   *string
	ReplaceIndex int
}

EditOptions configures the edit operation.

type Extractor ¶

type Extractor interface {
	Name() string
	Init(info *PageInfo) Processor
}

Extractor creates a stateful Processor for extracting metadata from a stream.

type FeedLink ¶

type FeedLink struct {
	URL   string `json:",omitempty"`
	Title string `json:",omitempty"`
	Type  string `json:",omitempty"`
}

FeedLink represents a parsed RSS, Atom or JSON feed link.

type FetchOptions ¶

type FetchOptions struct {
	UserAgent         string
	TraceRedirects    bool
	FollowMetaRefresh bool
	UpdateReferer     bool
	StoreCookies      bool
	RedirectLimit     int
	RedirectDelay     time.Duration
	Timeout           time.Duration
	MaxBodySize       int64
	Httpretty         bool
}

FetchOptions contains configuration for fetching a URL.

type GroupedProperty ¶

type GroupedProperty struct {
	Label string
	Value string
	Count int
	Order int
}

GroupedProperty represents a dynamically grouped metadata field for output.

type Icon ¶

type Icon struct {
	URL   string `json:",omitempty"`
	Sizes string `json:",omitempty"`
}

Icon represents a parsed icon link.

type KnownMetaTracker ¶

type KnownMetaTracker struct {
	RawTags   []UnknownMeta
	KnownKeys map[string]bool
	// contains filtered or unexported fields
}

KnownMetaTracker tracks all seen and known metadata tags during processing.

func (*KnownMetaTracker) AddRaw ¶

func (t *KnownMetaTracker) AddRaw(key, value string)

AddRaw adds a raw metadata tag that has been seen.

func (*KnownMetaTracker) MarkKnown ¶

func (t *KnownMetaTracker) MarkKnown(key string)

MarkKnown registers a tag key as explicitly handled by a processor.

func (*KnownMetaTracker) ProcessUnknown ¶

func (t *KnownMetaTracker) ProcessUnknown(excludedKeys []string) []UnknownMeta

ProcessUnknown returns a list of tags that are raw but have not been marked known.

type PageInfo ¶

type PageInfo struct {

	// Tracker for unknown metadata
	KnownMetaTracker KnownMetaTracker `json:"-"`

	// General Information
	URL         string   `json:",omitempty"`
	Trace       []string `json:",omitempty"`
	MetaRefresh string   `json:",omitempty"`
	Title       string   `json:",omitempty"`
	Keywords    string   `json:",omitempty"`

	// OpenGraph
	OGTitle          string `json:",omitempty"`
	OGType           string `json:",omitempty"`
	OGImage          string `json:",omitempty"`
	OGImageSecureURL string `json:",omitempty"`
	OGImageType      string `json:",omitempty"`
	OGImageWidth     string `json:",omitempty"`
	OGImageHeight    string `json:",omitempty"`
	OGDescription    string `json:",omitempty"`
	OGImageAlt       string `json:",omitempty"`
	OGVideo          string `json:",omitempty"`
	OGVideoSecureURL string `json:",omitempty"`
	OGVideoType      string `json:",omitempty"`
	OGVideoWidth     string `json:",omitempty"`
	OGVideoHeight    string `json:",omitempty"`
	OGVideoAlt       string `json:",omitempty"`

	// Twitter
	TwitterCard         string `json:",omitempty"`
	TwitterImage        string `json:",omitempty"`
	TwitterTitle        string `json:",omitempty"`
	TwitterDescription  string `json:",omitempty"`
	TwitterImageAlt     string `json:",omitempty"`
	TwitterPlayer       string `json:",omitempty"`
	TwitterPlayerWidth  string `json:",omitempty"`
	TwitterPlayerHeight string `json:",omitempty"`

	// YouTube Specific
	YouTubeVideoID       string        `json:",omitempty"`
	YouTubeLengthSeconds int           `json:",omitempty"`
	YouTubeDuration      time.Duration `json:",omitempty"`

	// App Links
	AppLinksIOSName        string `json:",omitempty"`
	AppLinksIOSAppStoreID  string `json:",omitempty"`
	AppLinksIOSURL         string `json:",omitempty"`
	AppLinksAndroidName    string `json:",omitempty"`
	AppLinksAndroidPackage string `json:",omitempty"`
	AppLinksAndroidURL     string `json:",omitempty"`

	// Instagram Specific
	InstagramOwnerUserID string `json:",omitempty"`

	// HTML Application
	ApplicationTitle string `json:",omitempty"`
	ThemeColor       string `json:",omitempty"`

	// JSON-LD (First found entity or merged data)
	JSONLDType             string        `json:",omitempty"`
	JSONLDID               string        `json:",omitempty"`
	JSONLDName             string        `json:",omitempty"`
	JSONLDDescription      string        `json:",omitempty"`
	JSONLDUploadDate       string        `json:",omitempty"`
	JSONLDUploadDateParsed time.Time     `json:",omitempty"`
	JSONLDDuration         string        `json:",omitempty"`
	JSONLDDurationTime     time.Duration `json:",omitempty"`
	JSONLDThumbnailURL     string        `json:",omitempty"`
	JSONLDAuthor           string        `json:",omitempty"`
	JSONLDPublisher        string        `json:",omitempty"`

	// HTML Links
	CanonicalLink  string          `json:",omitempty"`
	ShortLink      string          `json:",omitempty"`
	AlternateLinks []AlternateLink `json:",omitempty"`
	FeedLinks      []FeedLink      `json:",omitempty"`
	Icons          []Icon          `json:",omitempty"`

	// Unknown Metadata
	UnknownMeta []UnknownMeta `json:",omitempty"`
	// contains filtered or unexported fields
}

PageInfo represents the extracted information from a URL.

func FetchURL ¶

func FetchURL(urlStr string, opts FetchOptions) (*PageInfo, error)

FetchURL retrieves a URL and extracts metadata from it.

func (*PageInfo) CoverImages ¶

func (p *PageInfo) CoverImages() []string

CoverImages returns a list of potential image URLs.

func (*PageInfo) GroupedProperties ¶

func (p *PageInfo) GroupedProperties() []GroupedProperty

GroupedProperties returns fields grouped by value and sorted by number of grouped sources.

func (*PageInfo) Mutate ¶

func (p *PageInfo) Mutate(fn func(*PageInfo))

Mutate executes the given function while holding the PageInfo mutex lock, allowing thread-safe modifications to the PageInfo fields.

type Processor ¶

type Processor interface {
	// ProcessToken processes a single HTML token. It returns true if the processor
	// has finished and no longer needs to receive tokens.
	ProcessToken(t html.Token) (stop bool)
}

Processor receives a stream of HTML tokens and extracts metadata into PageInfo.

type UnknownMeta ¶

type UnknownMeta struct {
	Key   string `json:",omitempty"`
	Value string `json:",omitempty"`
}

UnknownMeta represents metadata that was found but not extracted into specific fields.

Source Files ¶

View all Source files

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
header command
header/templates
diagnostic Package diagnostic provides rules to check for missing metadata.	Package diagnostic provides rules to check for missing metadata.
internal
cli Package cli provides the command line interface subcommands.	Package cli provides the command line interface subcommands.
cli/cards Package cards provides generators for social media summary cards.	Package cards provides generators for social media summary cards.
ui Package ui provides image display utilities.	Package ui provides image display utilities.
lint Package lint provides a framework for analyzing extracted HTML metadata.	Package lint provides a framework for analyzing extracted HTML metadata.
lists Package lists provides a registry for grouped collections of lint rules.	Package lists provides a registry for grouped collections of lint rules.
lists/all Package all provides a list containing all available lint rules.	Package all provides a list containing all available lint rules.
lists/diagnose Package diagnose provides a list of rules for diagnosing issues.	Package diagnose provides a list of rules for diagnosing issues.
lists/recommended Package recommended provides the default recommended rule list.	Package recommended provides the default recommended rule list.
lists/verify Package verify provides a list of rules for verifying metadata.	Package verify provides a list of rules for verifying metadata.
rules/cloudflare Package cloudflare provides lint rules to detect Cloudflare blocking.	Package cloudflare provides lint rules to detect Cloudflare blocking.
rules/consistency Package consistency provides rules checking metadata consistency.	Package consistency provides rules checking metadata consistency.
rules/cors Package cors provides lint rules for Cross-Origin Resource Sharing.	Package cors provides lint rules for Cross-Origin Resource Sharing.
rules/facebook Package facebook contains lint rules for Facebook specific tags.	Package facebook contains lint rules for Facebook specific tags.
rules/html Package html contains lint rules for standard HTML metadata.	Package html contains lint rules for standard HTML metadata.
rules/images Package images provides lint rules for validating referenced images.	Package images provides lint rules for validating referenced images.
rules/opengraph Package opengraph contains lint rules for OpenGraph tags.	Package opengraph contains lint rules for OpenGraph tags.
rules/twitter Package twitter contains lint rules for Twitter Card metadata.	Package twitter contains lint rules for Twitter Card metadata.
rules/urls Package urls provides lint rules for validating URLs referenced in metadata.	Package urls provides lint rules for validating URLs referenced in metadata.
templates Package templates contains HTML templates for output rendering.	Package templates contains HTML templates for output rendering.

README ¶

header

Documentation

Prerequisites and Supported Platforms

Installation

GitHub Releases

Homebrew (after the first release)

Go install

Commands

show / extract (default)

edit

trace-redirects

verify

lint

lint-rule-lists

lint-rules

card

view

httpretty

Development & Testing

License

Documentation ¶

Overview ¶

Index ¶

Constants ¶

Variables ¶

Functions ¶

func ApplyEdit ¶

func FindNodes ¶

func GetOrCreateHead ¶

func RegisterExtractor ¶

func RemoveNodes ¶

func SetContent ¶

Types ¶

type AlternateLink ¶

type EditOptions ¶

type Extractor ¶

type FeedLink ¶

type FetchOptions ¶

type GroupedProperty ¶

type Icon ¶

type KnownMetaTracker ¶

func (*KnownMetaTracker) AddRaw ¶

func (*KnownMetaTracker) MarkKnown ¶

func (*KnownMetaTracker) ProcessUnknown ¶

type PageInfo ¶

func FetchURL ¶

func (*PageInfo) CoverImages ¶

func (*PageInfo) GroupedProperties ¶

func (*PageInfo) Mutate ¶

type Processor ¶

type UnknownMeta ¶

Source Files ¶

Directories ¶

`show` / `extract` (default)

`edit`

`trace-redirects`

`verify`

`lint`

`lint-rule-lists`

`lint-rules`

`card`

`view`

`httpretty`