metadata

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

README

metadata

Sherlock's extraction engine: fetch a web page once and return everything it says about itself as a Preview — title, description, canonical URL, thumbnail, embed, provider, authors, dates, and a semantic Kind.

The shape is oEmbed's field names on an ActivityStreams-shaped object — the same hybrid as Mastodon's PreviewCard. It is a rebuildable value, never persisted or federated, and it is not ActivityStreams: callers who need AS2 use sherlock.Client.Load instead.

Usage

preview, err := metadata.Get(ctx, "https://example.com/article",
    metadata.WithUserAgent("my-app/1.0"),
)
if err != nil {
    return err
}

fmt.Println(preview.Title, preview.Kind, preview.Thumbnail.URL)

sherlock.Client.Metadata is a thin wrapper that threads the Client's configuration (User-Agent, Authorized Fetch signatures, SSRF policy, body cap) into Get. Use Get directly when you have no Client.

How it works

One request negotiates ActivityStreams and HTML together (Accept: application/activity+json, text/html;q=0.9). The response is byte-capped (truncated, not rejected, so the <head> of a huge page survives), charset-decoded to UTF-8, and parsed exactly once. Every in-page extractor reads that one tree and returns a sparse partial; the engine merges them in a fixed precedence order:

ActivityStreams → Open Graph → Twitter Cards → oEmbed → HTML natives

One field breaks the global order on purpose: the canonical URL prefers rel=canonical over og:url (same-origin required). Everything else — the embed included — is decided by the merge order above and nothing else. Correlated fields (thumbnail, embed, provider, authors) fill as whole groups from one source, past a validity floor — one source's image URL is never paired with another's dimensions.

The oEmbed endpoint is the only extra network hop, and it is gated: called only when a registry match or player tags suggest an embed, or when a field oEmbed can carry (title, thumbnail, provider, author) is still empty. In practice the second half of that gate is generous — the author field is empty on most pages, since og:author is non-standard and a bare article:author URL fails the author floor — so a page that advertises an endpoint usually gets called.

Options

Option Default Purpose
WithUserAgent empty User-Agent sent on every request
WithRemoteOptions none extra remote.Options (signatures, instrumentation)
WithAllowPrivateIPs false SSRF guard — set TRUE only for httptest servers or intentional LAN targets
WithAllowSandboxEmbeds false let non-extractable oEmbed HTML become an EmbedProviderHTML embed for a sandboxed iframe
WithMaxBodySize 1 MB response body cap; oversize bodies are truncated

Rendering embeds

Preview.Embed is never raw provider markup. Its Mode tells you how to render it: EmbedIframe (build your own iframe from IframeURL), EmbedStream (a direct media file at StreamURL), or EmbedProviderHTML (only when WithAllowSandboxEmbeds(true); place HTML inside an iframe with sandbox="allow-scripts allow-popups" — never allow-same-origin). AllowSandboxEmbeds is a product decision applied to every provider uniformly, not a per-provider trust ranking.

Design notes

The full design, decisions, and precedence rationale live in the Emissary spec LINK-METADATA-CONSUMER.md. The rules an editor of this package needs — the ones that are easy to break and hard to notice — are collected in AGENTS.md.

Tolerances toward sloppy peers are audited in POSTEL.md: spec-mandated variants are cited in place, bounds and security decisions are fenced off from Postel's law entirely, and the tolerances that still lack a real-peer fixture are listed as open work. oEmbed response tolerance lives upstream in benpate/oembed, which already carries named-provider fixtures.

Documentation

Overview

Package metadata fetches a web page once and extracts an oEmbed-adjacent Preview from every source the page carries: ActivityStreams, Open Graph, Twitter Cards, oEmbed, and HTML natives.

Fetching

One request negotiates ActivityStreams and HTML together. The response is byte-capped (truncated, not rejected, so the head of a huge page survives), charset-decoded to UTF-8, and parsed exactly once. Every in-page extractor reads that single tree.

Precedence

Each source returns a sparse partial, and those partials merge in a fixed order: ActivityStreams, Open Graph, Twitter Cards, oEmbed, HTML natives. Call order IS the precedence: merge never overwrites a field that an earlier source already supplied. One field breaks that order on purpose — the canonical URL prefers rel=canonical over og:url, and requires the same origin.

Correlated fields — thumbnail, embed, provider, authors — fill as whole groups from one source, past a validity floor. One source's image URL is never paired with another source's dimensions.

The oEmbed endpoint is the only extra network hop, and it is gated: called only when an embed is plausible and not already found, or when a field oEmbed can carry is still empty.

Boundaries

This package is sherlock's extraction engine and deliberately knows nothing about ActivityStreams result types. A Preview is never AS2; the Preview-to-ActivityStreams projection lives in the parent package, which imports this one and never the reverse. Callers who need AS2 use sherlock.Client.Load instead.

Index

Examples

Constants

View Source 
const DefaultMaxBodySize = 1 << 20

DefaultMaxBodySize is the default cap (1MB) on response bodies read by the metadata engine. Bodies larger than this are truncated, not rejected.

Variables

This section is empty.

Functions

This section is empty.

Types

type Author 

type Author struct {
	Name string // display name
	URL  string // profile URL
}

Author is one creator of the resource. Name and URL travel together as a group so one source's name is never paired with another source's URL.

type Embed 

type Embed struct {
	Mode          EmbedMode // how this embed renders
	IframeURL     string    // twitter:player, VideoObject.embedUrl
	StreamURL     string    // twitter:player:stream, og:video with a file MIME type
	HTML          string    // provider markup; only meaningful when Mode == EmbedProviderHTML
	Width, Height int       // player dimensions; zero means unknown
}

Embed is a playable/interactive representation of the resource. Its dimensions are the PLAYER's, never the thumbnail's.

type EmbedMode 

type EmbedMode uint8

EmbedMode identifies how an Embed can be rendered. 

const (
	EmbedIframe       EmbedMode = iota + 1 // we construct the iframe from IframeURL
	EmbedStream                            // StreamURL is a direct media file
	EmbedProviderHTML                      // HTML is sanitized provider markup — last resort
)

EmbedMode values, ordered by preference: build our own iframe wherever possible, trust provider markup last.

type GetOption 

type GetOption func(*config)

GetOption configures one Get call.

func WithAllowPrivateIPs 

func WithAllowPrivateIPs(allow bool) GetOption

WithAllowPrivateIPs controls whether fetches may reach non-public IP addresses (loopback, private ranges); the default FALSE is an SSRF guard.

func WithAllowSandboxEmbeds 

func WithAllowSandboxEmbeds(allow bool) GetOption

WithAllowSandboxEmbeds controls whether oEmbed provider HTML that cannot be reduced to a single clean iframe may render inside a sandboxed srcdoc iframe.

func WithMaxBodySize 

func WithMaxBodySize(maxBytes int64) GetOption

WithMaxBodySize caps how many bytes are read from any response body; values of zero or less restore DefaultMaxBodySize.

func WithRemoteOptions 

func WithRemoteOptions(options ...remote.Option) GetOption

WithRemoteOptions adds remote.Options that are passed to the remote library when making requests (signatures, caching, instrumentation).

func WithUserAgent 

func WithUserAgent(userAgent string) GetOption

WithUserAgent sets the User-Agent string sent with every request.

type Kind 

type Kind uint8

Kind identifies what a resource IS, independent of how it renders. 

const (
	KindUnknown Kind = iota
	KindWebsite
	KindArticle
	KindVideo
	KindAudio
	KindImage
	KindProfile
)

Kind values, ordered roughly from least to most specific. oEmbed's `type` string conflates semantics with rendering; consumers compute a wire `type` from Kind+Embed at their own boundary instead.

type Preview 

type Preview struct {
	RequestURL string // the URL the caller asked for
	URL        string // canonical URL: AS id, rel=canonical, or og:url — same-origin verified
	Kind       Kind   // what the resource is (not how it renders)

	Title       string // scalars: filled individually, first source wins
	Description string // oEmbed's missing field
	Language    string // BCP-47, normalized at extraction

	Thumbnail *Thumbnail // groups: nil, or filled whole from ONE source
	Embed     *Embed
	Provider  *Provider
	Authors   []Author

	PublishedAt   *time.Time
	ModifiedAt    *time.Time
	CreatorHandle string // FEP-2345 fediverse:creator, e.g. "@user@example.social"

	FetchedAt time.Time // bookkeeping — cache TTL only, never rendered
}

Preview is everything the engine could learn about a web page, in an oEmbed-adjacent shape: a rebuildable cache entry, never persisted or federated.

func Get 

func Get(ctx context.Context, url string, options ...GetOption) (Preview, error)

Get fetches a URL once and returns everything the engine could learn about it, as an oEmbed-adjacent Preview.

Example

ExampleGet fetches a page and prints the extracted preview. The page is served locally so the example is hermetic; real callers pass a public URL and omit WithAllowPrivateIPs. 

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/benpate/sherlock/metadata"
)

func main() {

	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.Header().Set("Content-Type", "text/html; charset=utf-8")
		_, _ = fmt.Fprint(w, `<html><head>
			<title>Silver Blaze | The Strand</title>
			<meta property="og:type" content="article">
			<meta property="og:title" content="The Adventure of Silver Blaze">
			<meta property="og:site_name" content="The Strand">
		</head><body></body></html>`)
	}))
	defer server.Close()

	preview, err := metadata.Get(context.Background(), server.URL,
		metadata.WithUserAgent("example/1.0"),
		metadata.WithAllowPrivateIPs(true), // httptest is loopback
	)

	if err != nil {
		fmt.Println("error:", err)
		return
	}

	fmt.Println(preview.Title)
	fmt.Println(preview.Kind == metadata.KindArticle)
	fmt.Println(preview.Provider.Name)

}

Output:
The Adventure of Silver Blaze
true
The Strand

type Provider 

type Provider struct {
	Name    string // display name, e.g. "YouTube"
	URL     string // site root
	IconURL string // favicon or touch icon
}

Provider identifies the site or service hosting the resource.

type Thumbnail 

type Thumbnail struct {
	URL           string // image location
	Width, Height int    // declared dimensions; zero means unknown
	Alt           string // alt text, where a source carries it
}

Thumbnail is a preview image for the resource, mirroring oEmbed's thumbnail_* fields. It never shares a dimension with Embed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.