embisto

README

embisto

Link previews you host yourself. Give embisto a URL; get the title, description, image, icon and player a client needs to draw a card.

_{The card above is embisto's own format=html, ready for an iframe or an email.}

Quickstart

Run the server and ask it about a page:

docker run --rm -p 8080:8080 ghcr.io/pantafive/embisto:main
curl 'localhost:8080/v1/preview?url=https://go.dev/blog/'

{
  "url": "https://go.dev/blog/",
  "canonical_url": "https://go.dev/blog/",
  "type": "link",
  "title": "The Go Blog - The Go Programming Language",
  "icon": {"url": "https://go.dev/images/favicon-gopher-plain.png", "width": 180, "height": 180},
  "image": {"url": "https://go.dev/doc/gopher/runningsquare.jpg", "width": 299, "height": 281},
  "sources": ["opengraph", "twitter", "html"]
}

_{Shortened: the full card also lists every picture and link the page names. See the HTTP API.}

Or use it as a Go library:

go get github.com/pantafive/embisto

core := embisto.New(embisto.Defaults())

result, err := core.Extract(ctx, card.Request{URL: "https://go.dev/blog/"})
if err != nil {
	log.Fatal(card.CodeOf(err)) // invalid_url, blocked_url, fetch_failed, timeout, not_found, …
}
fmt.Println(result.Preview.Title)

Features

• Reads what pages say: oEmbed, Open Graph, Twitter Cards, JSON-LD and plain HTML, merged field by field.
• Knows the hard sites: YouTube, X, Reddit and TikTok providers, honest cards for login walls, and every site on the oembed.com list.
• Answers in the format you need: the JSON card, plain text, a self-contained HTML card, Iframely-compatible /iframely and /oembed, or your own template.
• Proxies the pictures: signed, resized JPEG, PNG and WebP variants, so clients never load them from other sites (details).
• Caches by the origin's rules: keeps a card as long as its page allows, makes one fetch for many identical requests, and shares the cache between instances through Redis or Valkey.
• Safe to point at the internet: refuses private addresses at connect time, and caps sizes, time and memory (details).
• Easy to run: a single static binary or a multi-arch image, Prometheus metrics, OpenAPI at /openapi.json with a reference at /docs, and a debug page that shows how a card was made.

Documentation

• HTTP API: the card, routes and error codes
• Configuration: environment variables and memory
• Caching and operations: metrics and the debug page
• Go library: the pipeline, site providers and layers
• Templates, image proxy, Iframely compatibility, hardening
• Contributing and security

License

MIT © Mikhail Solomenik

Documentation

Overview

Package embisto turns a URL into the data for a link preview: title, description, image, icon, video and the provider's embed HTML.

A preview goes through a fixed pipeline: fetch the page, run the generic sources over it, ask the site provider that knows its address, merge their facts field by field, enrich the result. New builds the pipeline from Options, which start from Defaults. The Core it returns is a card.Extractor, as is every layer around it, such as the cache in package cache, so layers stack in any order:

core := embisto.New(embisto.Defaults())
previews := cache.New(core, cache.NewMemory(64<<20, nil), cache.Options{TTL: time.Hour, FailureTTL: time.Minute})
result, err := previews.Extract(ctx, card.Request{URL: "https://github.com"})

The modules of each stage are in their own packages: site providers in sites, output formats in render, the resized pictures in thumbs. The contract they share is package card.

Index

• Constants
• func NewFetcher(opts Options) card.Fetcher
• type Core
  • func New(opts Options) *Core
  • func (c *Core) Extract(ctx context.Context, req card.Request) (card.Result, error)
  • func (c *Core) Purge(ctx context.Context, req card.Request) (int, error)
• type Observer
• type Options
  • func Defaults() Options

Constants

const Version = "0.1.0"

Version is the library's release.

Functions

func NewFetcher

func NewFetcher(opts Options) card.Fetcher

NewFetcher is the client the options configure, which refuses addresses that are not public; a fetcher of your own can wrap it, such as to record or to route through a proxy.

Types

type Core

type Core struct {
	// contains filtered or unexported fields
}

Core is the pipeline. It keeps nothing between previews but what Options.HomePages stores.

func New

func New(opts Options) *Core

New builds the pipeline.

func (*Core) Extract

func (c *Core) Extract(ctx context.Context, req card.Request) (card.Result, error)

Extract makes the preview of the request's URL. It fails with a *card.Error, or with the context's error when the caller gave up; a module that panics fails its preview as internal, never the program.

func (*Core) Purge

func (c *Core) Purge(ctx context.Context, req card.Request) (int, error)

Purge forgets what the core remembers of the URL's site: what its home page says.

type Observer

type Observer interface {
	// Fetched is one request a stage made, with its outcome: ok, a card code, or canceled. The stage is page, or the
	// name of the source, provider or enricher that asked.
	Fetched(stage, outcome string, took time.Duration)
	// Matched is a provider asked about a URL it knows.
	Matched(provider string)
	// Made is a card made afresh, with its class (full, partial, minimal, wall or error) and the provider that shaped
	// it, or generic.
	Made(class, source string)
}

Observer hears what the pipeline does, for metrics. It is told names and codes, never URLs.

type Options

type Options struct {
	// Timeout bounds one preview, every request it makes included.
	Timeout time.Duration
	// MaxBodySize is the most bytes read of one response; a larger one fails with too_large.
	MaxBodySize  int64
	MaxRedirects int
	UserAgent    string
	// DefaultAcceptLanguage is the Accept-Language a page is asked in, unless a request names its own language; sites
	// otherwise answer in the language of wherever the server runs.
	DefaultAcceptLanguage string
	// AllowedNetworks are reachable although not public, for a deployment that previews its own intranet.
	AllowedNetworks []netip.Prefix
	// Fetcher replaces the client the fields above configure, and with it the guard against non-public addresses.
	Fetcher card.Fetcher
	Sources []card.Source
	// HomePages remembers what each site's home page says, so that a description or a picture a site puts on every
	// page is not taken for one page's own. Nil reads no home page. A store shared between instances shares what
	// they learn; purging a URL forgets its site.
	HomePages cache.Store
	// Providers know particular sites and are asked first, in order: the first that knows a URL is the only one asked.
	Providers []card.Provider
	// Fallbacks are asked only about a URL no provider knows, such as the provider for every site on the oEmbed
	// list; so a provider added to Providers always wins over them.
	Fallbacks []card.Provider
	Enrichers []card.Enricher
	// Log hears about sources and providers that failed; the preview goes on without them. Nil logs nothing.
	Log *slog.Logger
	// Observer hears what the pipeline does, for metrics; nil hears nothing.
	Observer Observer
}

Options configure the pipeline. A zero number takes its default; a nil list takes the default modules, an empty one runs none.

func Defaults

func Defaults() Options

Defaults are the options embisto ships with; the home pages are kept in this process's memory.