urlform

README

urlform

Classify raw untrusted URL strings by structural form: the browser-vs-net/url parse quirks that decide whether a string really carries a host

A standalone, stdlib-only Go library for programs that PUBLISH untrusted URLs to humans or extract the host a browser would navigate to. Go's net/url and a browser's WHATWG parser read several string shapes differently — a browser strips embedded tabs/newlines (https://anime\tbytes.tv navigates to animebytes.tv), treats \ as /, reads an authority through any run of slashes after https:, navigates host/x to host, resolves //host/x against the ambient scheme, and shows the post-@ host for a user@host authority — so code that trusts the Go parse alone can publish a link whose real destination it never saw. urlform names those quirk classes once, extracts the browser-visible facts, and leaves the fail direction to each consumer.

The covered divergence set is bounded and enumerated (see the package docs), pinned by a conformance-fixture corpus derived from web-platform-tests — urlform is a classifier with the WHATWG readings layered on, not a full WHATWG parser. Out of scope by design: IDNA/punycode mapping (non-ASCII host evidence survives raw for the fail-closed gates), percent-encoding normalization, and port range checks (the facts are reported; the publisher validates).

This is deliberately NOT an SSRF guard. Validating a URL your own process will fetch answers to net/url and the dialer (the parser of record for the request); use an SSRF library for that. urlform models classify-for-publish, where the parser of record is the reader's browser.

Install

go get github.com/cplieger/urlform@latest

Usage

f := urlform.Classify(raw)
switch f.Class {
case urlform.ClassAbsolute:
	if f.HasUserInfo || f.HasBackslash {
		// visual-spoofing vectors a publisher typically rejects
		return "", false
	}
	return f.Trimmed, true
case urlform.ClassRelative:
	return base + f.Trimmed, true // rooted path, no host of its own
default:
	return "", false // protocol-relative, schemeless, hidden-host, malformed
}

Host evidence for matching against known domains:

f := urlform.Classify(raw)
if f.Host == "" || !urlform.IsASCIIHost(f.Host) {
	// no host evidence, or homograph territory: fail closed
	return nil, false
}
return domainTable[f.Host]

API

• Classify(raw string) Form — total classification: every input lands in exactly one class, never an error. Input gets the WHATWG preprocessing first (edges trimmed, embedded tab/newline removed — the same hardening CPython adopted for CVE-2022-0391); backslashes are canonicalized to slashes for the parse exactly where a browser reads them that way (special schemes and schemeless forms, ahead of the query) while Trimmed keeps the emit form.
• Form — the extracted facts: Class, Trimmed (preprocessed, emit-safe), Host (lowercased ASCII-only fold; non-ASCII homograph bytes survive unfolded for fail-closed gates), Scheme, Port (extracted, deliberately not range-checked), HasBackslash, HasTabOrNewline (a whitespace-smuggling attempt was removed), HasUserInfo, HostUnrecoverable.
• Class — ClassEmpty, ClassMalformed, ClassAbsolute, ClassHiddenHost (a scheme-bearing parse hiding host evidence; for the authority-carrying special schemes the browser's reading is recovered into the facts: https:/host/x and https:host/x expose host, while host:443/x and https://:443/x stay evidence-free like the browser's own reading), ClassProtocolRelative (//host/x and the ambiguous ///x), ClassSchemelessHost (host/x, where a browser navigates to host), ClassRelative (/x).
• IsASCIIHost(host string) bool — the fail-closed companion gate: reports whether every byte is ASCII, so a homograph host (Cyrillic lookalikes, fold-laundering U+0130/U+212A) never string-matches a canonical domain. Consumers that must accept international hosts convert punycode explicitly instead of relaxing the gate.

Design notes

• Judgment-free classification. The library names facts; policy stays with the caller. One consumer publishes-or-drops, another extracts-evidence-or-hides — both branch on the same classes and can never drift on what the string structurally is.
• WHATWG input preprocessing. Browsers delete embedded tab/newline wherever they appear and trim C0-control/space edges before parsing, so a string-level gate that skips this reads a different URL than the reader's browser will. Classify runs both steps first; HasTabOrNewline records a removed smuggling attempt, and Trimmed is already clean to emit. Edge trimming is deliberately widened to all Unicode whitespace (an NBSP-wrapped link still classifies; over-trimming errs fail-safe).
• ASCII-only host fold. strings.ToLower has ASCII-producing mappings (U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE folds to i, U+212A KELVIN SIGN to k) that would launder a homograph host into a matchable ASCII domain before any gate sees it. Host folds only A-Z, and IsASCIIHost rejects what survives.
• Backslash canonicalization is read-only and spec-scoped. The parsed facts describe the WHATWG reading (/\host/x classifies protocol-relative) for special-scheme and schemeless forms ahead of the query — for a non-special scheme a backslash is an ordinary character and rewriting it would fabricate host evidence a browser never sees. HasBackslash lets a publisher that must emit the raw string reject it outright; the raw form is never rewritten.
• Bounded and total. Allocation is bounded and linear in the input: the trimmed copy, plus at most one same-sized copy each for tab/newline stripping and backslash canonicalization when those bytes are present. Unparseable input is a class (ClassMalformed), not an error. Fuzz targets pin the exactly-one-class, nil-parse-only-for-no-facts, tab/newline-stripping, and metamorphic backslash invariants; rapid properties cover the canonicalization and smuggling-insensitivity laws on every PR; and a WPT-derived fixture corpus (testdata/whatwg-fixtures.json) pins the covered divergence set against the browser's documented reading.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

GPL-3.0 — see LICENSE.

Documentation

Overview

Package urlform classifies the structural form of a raw, untrusted URL string — specifically the forms where a BROWSER's reading (the WHATWG URL parser) diverges from net/url's: whitespace-smuggled URLs, backslash authorities, slash-count fixups after a special scheme, protocol-relative and schemeless-host forms, hidden-host parses, userinfo spoofing.

The parser of record is the divide that places this package. Validating a URL the PROCESS will fetch is a different concern (there net/url and the dialer are authoritative; use an SSRF guard). urlform models classify-for-PUBLISH: the string is destined for a human whose browser will read it, and the quirk classes exist precisely where that reading and Go's disagree.

The covered divergence set is bounded and enumerated — urlform is a classifier over net/url with the WHATWG readings layered on, not a conformant WHATWG parser. What it models, pinned by the conformance fixtures in testdata/whatwg-fixtures.json (WPT-derived plus hand-derived address-bar rows):

• Input preprocessing: leading/trailing C0-control-or-space trimming (widened to all Unicode whitespace, a documented fail-safe superset) and embedded ASCII tab/newline removal, recorded by HasTabOrNewline — so "https://anime\tbytes.tv/x" classifies with its real host.
• Backslash-as-slash for special schemes (http, https, ws, wss, ftp, file) and schemeless forms, ahead of the query/fragment; a non-special scheme's backslashes stay ordinary characters.
• Slash-count fixups: after an authority-carrying special scheme the browser reads an authority through ANY run of slashes, so "https:/host/x" and "https:host/x" expose their hidden host evidence (ClassHiddenHost with recovered facts).
• Address-bar forms outside the URL spec: "host/x" navigates to host (ClassSchemelessHost), "//host/x" resolves against the ambient scheme (ClassProtocolRelative).

Deliberately NOT modeled (the boundary of the contract): IDNA/UTS46 host mapping and punycode (non-ASCII host evidence survives raw for the fail-closed gates — see IsASCIIHost), percent-encoding normalization, port range checking (the fact is reported, the publisher validates), full host validation (net/url's acceptance stands in for it), interior non-tab C0 controls (net/url rejects them; where they sit in a host the browser rejects them too), and the file scheme's drive-letter quirks. Future WHATWG changes land here only when enumerated — the fixtures name the supported set, and drift against them fails the build.

Classify never errors: every input lands in exactly one Class with the extractable semantic facts (Host, Scheme, Port, HasUserInfo, HasBackslash, HasTabOrNewline) alongside. The classification is deliberately judgment-free; each consumer applies its own fail direction over the same facts — a publisher drops what it cannot vouch for, an evidence gate hides what it cannot classify.

Host evidence folds ASCII-only (a full-Unicode fold would launder homograph bytes such as U+0130 or U+212A into ASCII), and IsASCIIHost is the fail-closed companion gate for consumers matching hosts against known ASCII domains.

Index

• func IsASCIIHost(host string) bool
• type Class
• type Form
  • func Classify(raw string) Form

Functions

func IsASCIIHost

func IsASCIIHost(host string) bool

IsASCIIHost reports whether every byte of host is ASCII (below utf8.RuneSelf). It is the fail-closed companion of Form.Host's ASCII-only fold: a consumer matching host evidence against known ASCII domains gates on it first, so a homograph host (Cyrillic lookalikes, a fold-laundering U+0130 or U+212A) never string-matches a canonical domain. Callers that must ACCEPT international hosts convert punycode explicitly instead of relaxing this predicate.

Types

type Class

type Class int

Class names the structural form of a raw, untrusted URL string - specifically the browser-vs-net/url parse quirks that decide whether the string really carries a host. It is the single home of that quirk vocabulary; see Form.

const (
	// ClassEmpty is a string that is empty after the input preprocessing
	// (edge trimming plus tab/newline removal; see Classify).
	ClassEmpty Class = iota
	// ClassMalformed is a string the canonicalized parse rejected; no
	// structural facts (and no host evidence) can be extracted from it.
	ClassMalformed
	// ClassAbsolute is a scheme-and-host absolute URL ("https://host/x");
	// Host carries the parsed hostname.
	ClassAbsolute
	// ClassHiddenHost is a scheme-bearing parse with no hostname, where
	// net/url sees no host but a browser may navigate to one. For the
	// authority-carrying special schemes (http, https, ws, wss, ftp) the
	// WHATWG parser skips ANY run of slashes after the scheme - zero
	// ("https:host/x"), one ("https:/host/x"), or many - and reads the
	// authority, so the classifier runs the same authority reparse it uses
	// for schemeless forms and recovers the browser's reading into
	// Host/Port/HasUserInfo (HostUnrecoverable marks a failed recovery; a
	// port-only authority such as "https://:443/x" recovers no host, which
	// matches the browser - the WHATWG parser fails on an empty special
	// host). For every other scheme ("host:443/x" parsing the host as an
	// opaque scheme, "javascript:alert(1)", "mailto:x") the browser reads an
	// opaque path with no authority, so the facts stay empty - there the
	// host evidence, if any, is genuinely hidden.
	ClassHiddenHost
	// ClassProtocolRelative is a network-path reference: "//host/x" (Host
	// carries the parsed host a browser would resolve against the ambient
	// scheme) or a leading-"//" form with no extractable host evidence - a
	// three-or-more-slash form ("///x": Go parses a rooted path while
	// browsers read an authority) or an empty-authority form ("//", "//?q").
	// Host is the discriminator between the two sub-forms: consumers that
	// need host evidence treat an empty Host here as ambiguous and fail
	// closed.
	ClassProtocolRelative
	// ClassSchemelessHost is a scheme-free, non-rooted form ("host/x"):
	// net/url parses a bare path, but a browser address bar navigates to the
	// first segment as a host. Host carries that authority-reparse evidence
	// (empty for a query- or fragment-only form such as "?x:y");
	// HostUnrecoverable marks a failed reparse.
	ClassSchemelessHost
	// ClassRelative is a rooted, host-free relative path ("/x").
	ClassRelative
)

type Form

type Form struct {

	// Trimmed is the preprocessed raw string the classification read: edges
	// trimmed and embedded ASCII tab/newline removed (the WHATWG input
	// preprocessing, recorded by HasTabOrNewline), with backslashes NOT
	// canonicalized. It is what a publisher emits or prefixes - already free
	// of the whitespace-smuggling bytes a browser would silently drop, and
	// never otherwise rewritten.
	Trimmed string
	// Host is the lowercased host evidence a browser would navigate to, when
	// extractable: the parsed hostname of an absolute or protocol-relative
	// form, or the authority reparse of a schemeless-host or recoverable
	// hidden-host form. Empty when the string carries none (or the form
	// hides it; see Class). The fold is ASCII-only by design (see
	// asciiLower): a full-Unicode fold would launder homograph bytes
	// (U+0130 -> 'i', U+212A -> 'k') into ASCII, so non-ASCII host evidence
	// survives here unfolded for a consumer's fail-closed ASCII-only host
	// gates.
	Host string
	// Scheme is the canonicalized parse's scheme, which url.Parse folds to
	// lowercase (an "HTTPS://" source reads "https", RFC 3986 canonical
	// form), so the value is already case-folded; empty when the string
	// carries none or did not parse. Case-insensitive comparison by consumers
	// remains correct as defense in depth.
	Scheme string
	// Port is the canonicalized parse's port string; empty when none is
	// present or the string did not parse. net/url only accepts an
	// all-digit port, but it does not range-check it - consumers that need
	// a valid 16-bit port (a link publisher) validate the range. (The WHATWG
	// parser rejects an out-of-range port outright; reporting the fact and
	// leaving the fail direction to the consumer is this package's model.)
	Port string
	// Class is the structural form.
	Class Class
	// HasBackslash records a '\' anywhere in the trimmed string. Browsers
	// (WHATWG URL parser) treat '\' as '/' for the special schemes (http,
	// https, ws, wss, ftp, file) and for schemeless forms (where the address
	// bar's ambient scheme is special), so for those the parsed facts
	// (Scheme/Host/Port/Class) describe the canonicalized reading - a
	// `/\host/x` form classifies protocol-relative, not as a host-less
	// rooted path. For a non-special scheme a backslash is an ordinary
	// character and is NOT canonicalized (a browser reads an opaque path).
	// Either way the flag lets a publisher that must emit the raw string
	// reject it outright.
	HasBackslash bool
	// HasTabOrNewline records that the edge-trimmed string contained
	// embedded ASCII tab or newline (U+0009, U+000A, U+000D), which the
	// WHATWG parser - and therefore this classification - removes wherever
	// they appear ("https://anime\tbytes.tv/x" navigates to animebytes.tv).
	// Trimmed already has them removed, so emitting Trimmed is safe; the
	// flag records the smuggling attempt for publishers that treat the
	// ORIGINAL string as emittable or want to reject de-smuggled input
	// outright.
	HasTabOrNewline bool
	// HostUnrecoverable marks a ClassSchemelessHost or recoverable
	// ClassHiddenHost whose authority reparse failed (e.g. a space before an
	// "@"): browser-visible host evidence may exist but cannot be extracted,
	// so evidence-driven consumers treat the form like a parse failure.
	HostUnrecoverable bool
	// HasUserInfo records a userinfo authority component ("user@host") in
	// the canonicalized parse - a visual-spoofing vector
	// ("https://trusted@evil/") a link publisher typically rejects. For a
	// ClassSchemelessHost or recovered ClassHiddenHost the fact comes from
	// the same authority reparse that supplies Host (so "user@host/x"
	// reports it alongside the recovered host). Always false when the
	// string did not parse.
	HasUserInfo bool
	// contains filtered or unexported fields
}

Form is the structural classification of one raw, untrusted URL string (an upstream API field, a scraped link, operator input). It names the browser-vs-net/url parse-quirk classes ONCE - backslash authorities, protocol-relative and schemeless-host forms, hidden-host parses - so every consumer branches on the same facts while keeping its own fail direction as policy: a publisher drops what it cannot vouch for (publish-or-drop), while an evidence gate hides what it cannot classify (extract-evidence-or-hide). Fields are ordered for govet fieldalignment.

func Classify

func Classify(raw string) Form

Classify classifies a raw URL string into its structural Form. It never errors: every input lands in exactly one class, and unparseable input is ClassMalformed. Consumers apply their own policy over the returned facts (see Form).

Classification starts with the WHATWG basic parser's input preprocessing, so string-level whitespace smuggling cannot hide a URL from the facts: leading/trailing C0 controls and whitespace are trimmed (trimEdges), and embedded ASCII tab/newline are removed everywhere (recorded by HasTabOrNewline). This is the same hardening CPython adopted for urllib.parse (CVE-2022-0391).