urlform

README

urlform

Classify raw untrusted URL strings by structural form: the browser-vs-net/url parse quirks that decide whether a string really carries a host

A standalone, stdlib-only Go library for programs that PUBLISH untrusted URLs to humans or extract the host a browser would navigate to. Go's net/url and a browser's WHATWG parser read several string shapes differently. A browser strips embedded tabs and newlines (https://anime\tbytes.tv navigates to animebytes.tv), treats \ as /, reads an authority through any run of slashes after https:, navigates host/x to host, resolves //host/x against the ambient scheme, and shows the post-@ host for a user@host authority. Code that trusts the Go parse alone can publish a link whose real destination it never saw. urlform names those quirk classes once, extracts the browser-visible facts, and leaves the fail direction to each consumer.

The covered divergence set is bounded and enumerated (see the package docs), pinned by a conformance-fixture corpus derived from web-platform-tests; urlform is a classifier with the WHATWG readings layered on, not a full WHATWG parser. Out of scope by design: IDNA/punycode mapping (non-ASCII host evidence survives raw for the fail-closed gates), percent-encoding normalization, and port range checks (the facts are reported; the publisher validates).

This is deliberately NOT an SSRF guard. Validating a URL your own process will fetch answers to net/url and the dialer (the parser of record for the request); use an SSRF library for that. urlform models classify-for-publish, where the parser of record is the reader's browser.

Install

go get github.com/cplieger/urlform@latest

Usage

f := urlform.Classify(raw)
switch f.Class {
case urlform.ClassAbsolute:
	if f.HasUserInfo || f.HasBackslash {
		// visual-spoofing vectors a publisher typically rejects
		return "", false
	}
	return f.Trimmed, true
case urlform.ClassRelative:
	return base + f.Trimmed, true // rooted path, no host of its own
default:
	return "", false // protocol-relative, schemeless, hidden-host, malformed
}

Host evidence for matching against known domains:

f := urlform.Classify(raw)
if f.Host == "" || !urlform.IsASCIIHost(f.Host) {
	// no host evidence, or homograph territory: fail closed
	return nil, false
}
for domain, tracker := range knownDomains {
	if urlform.HostMatchesDomain(f.Host, domain) {
		return tracker, true
	}
}
return nil, false

Names of an untrusted raw query, without the evadable parsed view:

for name := range urlform.RawQueryNames(u.RawQuery) {
	if isCredentialParam(name) { // the caller's predicate, and its fail direction
		return true
	}
}

API

• Classify(raw string) Form: total classification. Every input lands in exactly one class, never an error; the WHATWG input preprocessing and backslash canonicalization run first (see Design notes).
• Form: the extracted facts: Class, Trimmed (preprocessed, emit-safe), Host (ASCII-only lowercase fold), Scheme, Port (extracted, deliberately not range-checked), HasBackslash, HasTabOrNewline (a whitespace-smuggling attempt was removed), HasUserInfo, HostUnrecoverable.
• Class: ClassEmpty, ClassMalformed, ClassAbsolute, ClassHiddenHost (a scheme-bearing parse hiding host evidence; for the authority-carrying special schemes the browser's reading is recovered into the facts, so https:/host/x and https:host/x expose host, while host:443/x and https://:443/x stay evidence-free like the browser's own reading), ClassProtocolRelative (//host/x and the ambiguous ///x), ClassSchemelessHost (host/x, where a browser navigates to host), ClassRelative (/x).
• IsASCIIHost(host string) bool: the fail-closed companion gate. It reports whether every byte is ASCII, so a homograph host (Cyrillic lookalikes, fold-laundering U+0130/U+212A) never string-matches a canonical domain. Consumers that must accept international hosts convert punycode explicitly instead of relaxing the gate.
• HostMatchesDomain(host, domain string) bool: the matcher the gate above leads to. Reports whether the host equals the domain or is a real dot-delimited subdomain of it, refusing the three readings a plain suffix test accepts: suffix confusion (evilnyaa.si), parent-domain spoofing (nyaa.si.evil.example), and empty DNS labels (.nyaa.si, a..nyaa.si). Fail-closed on an empty or empty-labelled argument; folds ASCII-only. Trimming surrounding space and the trailing root dot stays the caller's.
• RawQueryNames(rawQuery string) iter.Seq[string]: the percent-decoded parameter names of a raw query (u.RawQuery's shape, no leading ?), split on both & and ;. url.ParseQuery drops a malformed pair wholesale, so an unescaped semicolon deletes the pair it sits in while the bytes still ride every request and log line; this walk is the strict superset a gate can't be evaded on. Judgment-free: it reports names and takes no view of them, because consumers need opposite fail directions over the same walk.

Design notes

• Judgment-free classification. The library names facts; policy stays with the caller. One consumer publishes-or-drops, another extracts-evidence-or-hides; both branch on the same classes and can never drift on what the string structurally is.
• WHATWG input preprocessing. Browsers delete embedded tab/newline wherever they appear and trim C0-control/space edges before parsing (the same hardening CPython adopted for CVE-2022-0391), so a string-level gate that skips this reads a different URL than the reader's browser will. Classify runs both steps first; HasTabOrNewline records a removed smuggling attempt, and Trimmed is already clean to emit. Edge trimming is deliberately widened to all Unicode whitespace (an NBSP-wrapped link still classifies; over-trimming errs fail-safe).
• ASCII-only host fold. strings.ToLower has ASCII-producing mappings (U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE folds to i, U+212A KELVIN SIGN to k) that would launder a homograph host into a matchable ASCII domain before any gate sees it. Host folds only A-Z, and IsASCIIHost rejects what survives.
• Backslash canonicalization is read-only and spec-scoped. The parsed facts describe the WHATWG reading (/\host/x classifies protocol-relative) for special-scheme and schemeless forms ahead of the query; for a non-special scheme a backslash is an ordinary character, and rewriting it would fabricate host evidence a browser never sees. HasBackslash lets a publisher that must emit the raw string reject it outright; the raw form is never rewritten.
• Bounded and total. Allocation is bounded and linear in the input, and unparseable input is a class (ClassMalformed), not an error.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

GPL-3.0-or-later. See LICENSE.

Documentation

Overview

Package urlform classifies the structural form of a raw, untrusted URL string — specifically the forms where a BROWSER's reading (the WHATWG URL parser) diverges from net/url's: whitespace-smuggled URLs, backslash authorities, slash-count fixups after a special scheme, protocol-relative and schemeless-host forms, hidden-host parses, userinfo spoofing.

The parser of record is the divide that places this package. Validating a URL the PROCESS will fetch is a different concern (there net/url and the dialer are authoritative; use an SSRF guard). urlform models classify-for-PUBLISH: the string is destined for a human whose browser will read it, and the quirk classes exist precisely where that reading and Go's disagree.

The covered divergence set is bounded and enumerated — urlform is a classifier over net/url with the WHATWG readings layered on, not a conformant WHATWG parser. What it models, pinned by the conformance fixtures in testdata/whatwg-fixtures.json (WPT-derived plus hand-derived address-bar rows):

• Input preprocessing: leading/trailing C0-control-or-space trimming (widened to all Unicode whitespace, a documented fail-safe superset) and embedded ASCII tab/newline removal, recorded by HasTabOrNewline — so "https://anime\tbytes.tv/x" classifies with its real host.
• Backslash-as-slash for special schemes (http, https, ws, wss, ftp, file) and schemeless forms, ahead of the query/fragment; a non-special scheme's backslashes stay ordinary characters.
• Slash-count fixups: after an authority-carrying special scheme the browser reads an authority through ANY run of slashes, so "https:/host/x" and "https:host/x" expose their hidden host evidence (ClassHiddenHost with recovered facts).
• Address-bar forms outside the URL spec: "host/x" navigates to host (ClassSchemelessHost), "//host/x" resolves against the ambient scheme (ClassProtocolRelative).

Deliberately NOT modeled (the boundary of the contract): IDNA/UTS46 host mapping and punycode (non-ASCII host evidence survives raw for the fail-closed gates — see IsASCIIHost), percent-encoding normalization, port range checking (the fact is reported, the publisher validates), full host validation (net/url's acceptance stands in for it), interior non-tab C0 controls (net/url rejects them; where they sit in a host the browser rejects them too), and the file scheme's drive-letter quirks. Future WHATWG changes land here only when enumerated — the fixtures name the supported set, and drift against them fails the build.

Classify never errors: every input lands in exactly one Class with the extractable semantic facts (Host, Scheme, Port, HasUserInfo, HasBackslash, HasTabOrNewline) alongside. The classification is deliberately judgment-free; each consumer applies its own fail direction over the same facts — a publisher drops what it cannot vouch for, an evidence gate hides what it cannot classify.

Host evidence folds ASCII-only (a full-Unicode fold would launder homograph bytes such as U+0130 or U+212A into ASCII), IsASCIIHost is the fail-closed companion gate for consumers matching hosts against known ASCII domains, and HostMatchesDomain is the safe equals-or-subdomain comparison that gate leads to.

RawQueryNames sits beside those as the other raw-reading primitive: the percent-decoded parameter names of a query string, split on both '&' and ';', because url.ParseQuery drops a malformed pair wholesale and a gate built on the parsed view can therefore be evaded by a smuggled separator while the bytes still ride every request and log line. Like the Class facts it is judgment-free - it reports names, and each consumer applies its own predicate and fail direction.

Index

• func HostMatchesDomain(host, domain string) bool
• func IsASCIIHost(host string) bool
• func RawQueryNames(rawQuery string) iter.Seq[string]
• type Class
• type Form
  • func Classify(raw string) Form

Functions

func HostMatchesDomain

func HostMatchesDomain(host, domain string) bool

HostMatchesDomain reports whether host equals domain or is a real dot-delimited subdomain of it. It is the matcher IsASCIIHost's documented scenario leads to: a consumer that has gated untrusted host evidence on IsASCIIHost then has to compare it against a known ASCII domain, and plain suffix matching is wrong in three ways this closes.

• Suffix confusion: "evilnyaa.si" ends in "nyaa.si" without being under it, so the separating dot is required, not incidental.
• Parent-domain spoofing: "nyaa.si.evil.example" contains the domain but is owned by evil.example, so the domain must be the SUFFIX, not a substring.
• Empty DNS labels: a bare suffix test also accepts ".nyaa.si" (its leading dot) and "a..nyaa.si" (an inner one). No resolvable DNS name carries an empty label, so both forms are adversarial and every label of the subdomain prefix must be non-empty.

It is fail-closed and total. An empty host or domain matches nothing (an empty domain would otherwise match an empty host), and a domain that itself carries an empty label matches nothing rather than lending its malformation to the comparison. Comparison folds ASCII-only, for the same reason Form.Host does: a full-Unicode fold has ASCII-producing mappings that would launder a homograph host into a match. That fold is a convenience, NOT a substitute for the gate - a non-ASCII host can never equal an ASCII domain here, but callers still run IsASCIIHost first, because it is the check that refuses the evidence outright instead of merely failing to match one domain.

It does NOT require an ASCII host, and deliberately so: a non-ASCII byte in a SUBDOMAIN label ("\u00e9.nyaa.si" against "nyaa.si") is a truthful match, since that host really is under the domain - the spoofing risk lives in the region aligned with the domain, which is compared byte-wise against ASCII and so cannot hold laundered bytes. Refusing non-ASCII evidence outright is IsASCIIHost's separate, stricter job, which is why consumers run it first rather than expecting this to imply it.

Normalization beyond the ASCII fold stays the caller's: this compares the host it is given, so a caller holding raw evidence trims its own surrounding ASCII space and its own trailing root dot ("nyaa.si." does not match "nyaa.si") before calling. Doing it here would mean applying Unicode-aware trimming to a string whose non-ASCII bytes are exactly what the caller's gate exists to see.

func IsASCIIHost

func IsASCIIHost(host string) bool

IsASCIIHost reports whether every byte of host is ASCII (below utf8.RuneSelf). It is the fail-closed companion of Form.Host's ASCII-only fold: a consumer matching host evidence against known ASCII domains gates on it first, so a homograph host (Cyrillic lookalikes, a fold-laundering U+0130 or U+212A) never string-matches a canonical domain. Callers that must ACCEPT international hosts convert punycode explicitly instead of relaxing this predicate.

func RawQueryNames

func RawQueryNames(rawQuery string) iter.Seq[string]

RawQueryNames iterates the percent-decoded parameter NAMES of a raw query string, in order, without consulting url.Values. It exists because the parsed view can be evaded: url.ParseQuery (and therefore u.Query()) drops a malformed pair WHOLESALE, so an unescaped semicolon deletes the pair it sits in - "apikey=SECRET;foo=x" disappears from the parsed map while the bytes stay in RawQuery for every outgoing request and every logged URL. A consumer whose gate must not be evadable therefore needs the raw reading, which is a strict superset of the parsed one:

• Pairs are split on BOTH '&' and ';' (the historic separator whose removal from url.ParseQuery is what creates the gap), empty fields skipped.
• The name is the text before the first '=' (a pair with no '=' yields its whole field as a name, which is how a bare flag parameter reads).
• Each name is percent-decoded, so an encoded spelling cannot hide from a literal comparison. A name whose escapes do not decode is yielded RAW rather than skipped, so a malformed pair still reaches the caller's predicate instead of vanishing the way the parsed view vanishes it.

The iteration is judgment-free, like the Class facts: it reports names and takes no view of them, because consumers need opposite fail directions over the same walk - a credential-in-URL warning wants any suspicious name to match (over-matching is safe), while a structural identity gate wants only the name it recognizes (over-matching admits a URL it should refuse).

The argument is a raw query WITHOUT its leading '?' - u.RawQuery's shape. A '?' is a legal literal inside a query, so it is not trimmed: a caller holding a whole URL takes u.RawQuery (or cuts at the first '?') rather than passing the URL, exactly as IsASCIIHost takes a host and not a URL.

Types

type Class

type Class int

Class names the structural form of a raw, untrusted URL string - specifically the browser-vs-net/url parse quirks that decide whether the string really carries a host. It is the single home of that quirk vocabulary; see Form.

const (
	// ClassEmpty is a string that is empty after the input preprocessing
	// (edge trimming plus tab/newline removal; see Classify).
	ClassEmpty Class = iota
	// ClassMalformed is a string the canonicalized parse rejected; no
	// structural facts (and no host evidence) can be extracted from it.
	ClassMalformed
	// ClassAbsolute is a scheme-and-host absolute URL ("https://host/x");
	// Host carries the parsed hostname.
	ClassAbsolute
	// ClassHiddenHost is a scheme-bearing parse with no hostname, where
	// net/url sees no host but a browser may navigate to one. For the
	// authority-carrying special schemes (http, https, ws, wss, ftp) the
	// WHATWG parser skips ANY run of slashes after the scheme - zero
	// ("https:host/x"), one ("https:/host/x"), or many - and reads the
	// authority, so the classifier runs the same authority reparse it uses
	// for schemeless forms and recovers the browser's reading into
	// Host/Port/HasUserInfo (HostUnrecoverable marks a failed recovery; a
	// port-only authority such as "https://:443/x" recovers no host, which
	// matches the browser - the WHATWG parser fails on an empty special
	// host). For every other scheme ("host:443/x" parsing the host as an
	// opaque scheme, "javascript:alert(1)", "mailto:x") the browser reads an
	// opaque path with no authority, so the facts stay empty - there the
	// host evidence, if any, is genuinely hidden.
	ClassHiddenHost
	// ClassProtocolRelative is a network-path reference: "//host/x" (Host
	// carries the parsed host a browser would resolve against the ambient
	// scheme) or a leading-"//" form with no extractable host evidence - a
	// three-or-more-slash form ("///x": Go parses a rooted path while
	// browsers read an authority) or an empty-authority form ("//", "//?q").
	// Host is the discriminator between the two sub-forms: consumers that
	// need host evidence treat an empty Host here as ambiguous and fail
	// closed.
	ClassProtocolRelative
	// ClassSchemelessHost is a scheme-free, non-rooted form ("host/x"):
	// net/url parses a bare path, but a browser address bar navigates to the
	// first segment as a host. Host carries that authority-reparse evidence
	// (empty for a query- or fragment-only form such as "?x:y");
	// HostUnrecoverable marks a failed reparse.
	ClassSchemelessHost
	// ClassRelative is a rooted, host-free relative path ("/x").
	ClassRelative
)

type Form

type Form struct {

	// Trimmed is the preprocessed raw string the classification read: edges
	// trimmed and embedded ASCII tab/newline removed (the WHATWG input
	// preprocessing, recorded by HasTabOrNewline), with backslashes NOT
	// canonicalized. It is what a publisher emits or prefixes - already free
	// of the whitespace-smuggling bytes a browser would silently drop, and
	// never otherwise rewritten.
	Trimmed string
	// Host is the lowercased host evidence a browser would navigate to, when
	// extractable: the parsed hostname of an absolute or protocol-relative
	// form, or the authority reparse of a schemeless-host or recoverable
	// hidden-host form. Empty when the string carries none (or the form
	// hides it; see Class). The fold is ASCII-only by design (see
	// asciiLower): a full-Unicode fold would launder homograph bytes
	// (U+0130 -> 'i', U+212A -> 'k') into ASCII, so non-ASCII host evidence
	// survives here unfolded for a consumer's fail-closed ASCII-only host
	// gates.
	Host string
	// Scheme is the canonicalized parse's scheme, which url.Parse folds to
	// lowercase (an "HTTPS://" source reads "https", RFC 3986 canonical
	// form), so the value is already case-folded; empty when the string
	// carries none or did not parse. Case-insensitive comparison by consumers
	// remains correct as defense in depth.
	Scheme string
	// Port is the canonicalized parse's port string; empty when none is
	// present or the string did not parse. net/url only accepts an
	// all-digit port, but it does not range-check it - consumers that need
	// a valid 16-bit port (a link publisher) validate the range. (The WHATWG
	// parser rejects an out-of-range port outright; reporting the fact and
	// leaving the fail direction to the consumer is this package's model.)
	Port string
	// Class is the structural form.
	Class Class
	// HasBackslash records a '\' anywhere in the trimmed string. Browsers
	// (WHATWG URL parser) treat '\' as '/' for the special schemes (http,
	// https, ws, wss, ftp, file) and for schemeless forms (where the address
	// bar's ambient scheme is special), so for those the parsed facts
	// (Scheme/Host/Port/Class) describe the canonicalized reading - a
	// `/\host/x` form classifies protocol-relative, not as a host-less
	// rooted path. For a non-special scheme a backslash is an ordinary
	// character and is NOT canonicalized (a browser reads an opaque path).
	// Either way the flag lets a publisher that must emit the raw string
	// reject it outright.
	HasBackslash bool
	// HasTabOrNewline records that the edge-trimmed string contained
	// embedded ASCII tab or newline (U+0009, U+000A, U+000D), which the
	// WHATWG parser - and therefore this classification - removes wherever
	// they appear ("https://anime\tbytes.tv/x" navigates to animebytes.tv).
	// Trimmed already has them removed, so emitting Trimmed is safe; the
	// flag records the smuggling attempt for publishers that treat the
	// ORIGINAL string as emittable or want to reject de-smuggled input
	// outright.
	HasTabOrNewline bool
	// HostUnrecoverable marks a ClassSchemelessHost or recoverable
	// ClassHiddenHost whose authority reparse failed (e.g. a space before an
	// "@"): browser-visible host evidence may exist but cannot be extracted,
	// so evidence-driven consumers treat the form like a parse failure.
	HostUnrecoverable bool
	// HasUserInfo records a userinfo authority component ("user@host") in
	// the canonicalized parse - a visual-spoofing vector
	// ("https://trusted@evil/") a link publisher typically rejects. For a
	// ClassSchemelessHost or recovered ClassHiddenHost the fact comes from
	// the same authority reparse that supplies Host (so "user@host/x"
	// reports it alongside the recovered host). Always false when the
	// string did not parse.
	HasUserInfo bool
	// contains filtered or unexported fields
}

Form is the structural classification of one raw, untrusted URL string (an upstream API field, a scraped link, operator input). It names the browser-vs-net/url parse-quirk classes ONCE - backslash authorities, protocol-relative and schemeless-host forms, hidden-host parses - so every consumer branches on the same facts while keeping its own fail direction as policy: a publisher drops what it cannot vouch for (publish-or-drop), while an evidence gate hides what it cannot classify (extract-evidence-or-hide). Fields are ordered for govet fieldalignment.

func Classify

func Classify(raw string) Form

Classify classifies a raw URL string into its structural Form. It never errors: every input lands in exactly one class, and unparseable input is ClassMalformed. Consumers apply their own policy over the returned facts (see Form).

Classification starts with the WHATWG basic parser's input preprocessing, so string-level whitespace smuggling cannot hide a URL from the facts: leading/trailing C0 controls and whitespace are trimmed (trimEdges), and embedded ASCII tab/newline are removed everywhere (recorded by HasTabOrNewline). This is the same hardening CPython adopted for urllib.parse (CVE-2022-0391).