urlform

README

urlform

Classify raw untrusted URL strings by structural form: the browser-vs-net/url parse quirks that decide whether a string really carries a host

A standalone, stdlib-only Go library for programs that PUBLISH untrusted URLs to humans or extract the host a browser would navigate to. Go's net/url and a browser's WHATWG parser read several string shapes differently — a browser treats \ as /, navigates host/x to host, resolves //host/x against the ambient scheme, and shows the post-@ host for a user@host authority — so code that trusts the Go parse alone can publish a link whose real destination it never saw. urlform names those quirk classes once, extracts the browser-visible facts, and leaves the fail direction to each consumer.

This is deliberately NOT an SSRF guard. Validating a URL your own process will fetch answers to net/url and the dialer (the parser of record for the request); use an SSRF library for that. urlform models classify-for-publish, where the parser of record is the reader's browser.

Install

go get github.com/cplieger/urlform@latest

Usage

f := urlform.Classify(raw)
switch f.Class {
case urlform.ClassAbsolute:
	if f.HasUserInfo || f.HasBackslash {
		// visual-spoofing vectors a publisher typically rejects
		return "", false
	}
	return f.Trimmed, true
case urlform.ClassRelative:
	return base + f.Trimmed, true // rooted path, no host of its own
default:
	return "", false // protocol-relative, schemeless, hidden-host, malformed
}

Host evidence for matching against known domains:

f := urlform.Classify(raw)
if f.Host == "" || !urlform.IsASCIIHost(f.Host) {
	// no host evidence, or homograph territory: fail closed
	return nil, false
}
return domainTable[f.Host]

API

• Classify(raw string) Form — total classification: every input lands in exactly one class, never an error. Whitespace-trimmed; backslashes canonicalized to slashes for the parse (the WHATWG reading) while Trimmed keeps the raw form a publisher would emit.
• Form — the extracted facts: Class, Trimmed, Host (lowercased ASCII-only fold; non-ASCII homograph bytes survive unfolded for fail-closed gates), Scheme, Port (extracted, deliberately not range-checked), HasBackslash, HasUserInfo, HostUnrecoverable.
• Class — ClassEmpty, ClassMalformed, ClassAbsolute, ClassHiddenHost (a scheme-bearing parse hiding host evidence: https:/host/x, host:443/x, https://:443/x), ClassProtocolRelative (//host/x and the ambiguous ///x), ClassSchemelessHost (host/x, where a browser navigates to host), ClassRelative (/x).
• IsASCIIHost(host string) bool — the fail-closed companion gate: reports whether every byte is ASCII, so a homograph host (Cyrillic lookalikes, fold-laundering U+0130/U+212A) never string-matches a canonical domain. Consumers that must accept international hosts convert punycode explicitly instead of relaxing the gate.

Design notes

• Judgment-free classification. The library names facts; policy stays with the caller. One consumer publishes-or-drops, another extracts-evidence-or-hides — both branch on the same classes and can never drift on what the string structurally is.
• ASCII-only host fold. strings.ToLower has ASCII-producing mappings (U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE folds to i, U+212A KELVIN SIGN to k) that would launder a homograph host into a matchable ASCII domain before any gate sees it. Host folds only A-Z, and IsASCIIHost rejects what survives.
• Backslash canonicalization is read-only. The parsed facts describe the WHATWG reading (/\host/x classifies protocol-relative), while HasBackslash lets a publisher that must emit the raw string reject it outright; the raw form is never rewritten.
• Bounded and total. No allocation scales with input beyond the trimmed copy; unparseable input is a class (ClassMalformed), not an error. Fuzz targets pin the exactly-one-class, nil-parse-only-for-no-facts, and metamorphic backslash invariants; a rapid property covers the canonicalization law on every PR.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude Opus and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

GPL-3.0 — see LICENSE.

Documentation

Overview

Package urlform classifies the structural form of a raw, untrusted URL string — specifically the forms where a BROWSER's reading (the WHATWG URL parser) diverges from net/url's: backslash authorities, protocol-relative and schemeless-host forms, hidden-host parses, userinfo spoofing.

The parser of record is the divide that places this package. Validating a URL the PROCESS will fetch is a different concern (there net/url and the dialer are authoritative; use an SSRF guard). urlform models classify-for-PUBLISH: the string is destined for a human whose browser will read it, and the quirk classes exist precisely where that reading and Go's disagree — a browser treats '\' as '/', navigates "host/x" to host, and resolves "//host/x" against the ambient scheme, while net/url parses a rooted path, a bare path, and a hostless reference.

Classify never errors: every input lands in exactly one Class with the extractable semantic facts (Host, Scheme, Port, HasUserInfo, HasBackslash) alongside. The classification is deliberately judgment-free; each consumer applies its own fail direction over the same facts — a publisher drops what it cannot vouch for, an evidence gate hides what it cannot classify.

Host evidence folds ASCII-only (a full-Unicode fold would launder homograph bytes such as U+0130 or U+212A into ASCII), and IsASCIIHost is the fail-closed companion gate for consumers matching hosts against known ASCII domains.

Index

• func IsASCIIHost(host string) bool
• type Class
• type Form
  • func Classify(raw string) Form

Functions

func IsASCIIHost

func IsASCIIHost(host string) bool

IsASCIIHost reports whether every byte of host is ASCII (below utf8.RuneSelf). It is the fail-closed companion of Form.Host's ASCII-only fold: a consumer matching host evidence against known ASCII domains gates on it first, so a homograph host (Cyrillic lookalikes, a fold-laundering U+0130 or U+212A) never string-matches a canonical domain. Callers that must ACCEPT international hosts convert punycode explicitly instead of relaxing this predicate.

Types

type Class

type Class int

Class names the structural form of a raw, untrusted URL string - specifically the browser-vs-net/url parse quirks that decide whether the string really carries a host. It is the single home of that quirk vocabulary; see Form.

const (
	// ClassEmpty is a string that is empty after whitespace trimming.
	ClassEmpty Class = iota
	// ClassMalformed is a string the canonicalized parse rejected; no
	// structural facts (and no host evidence) can be extracted from it.
	ClassMalformed
	// ClassAbsolute is a scheme-and-host absolute URL ("https://host/x");
	// Host carries the parsed hostname.
	ClassAbsolute
	// ClassHiddenHost is a scheme-bearing parse with no hostname, where
	// net/url sees no host but a browser may navigate to one: a
	// path-relative scheme form ("https:/host/x" parses scheme + path), an
	// opaque host:port form ("host:443/x" parses the host as the scheme), or
	// a port-only authority ("https://:443/x"). The host evidence is hidden.
	ClassHiddenHost
	// ClassProtocolRelative is a network-path reference: "//host/x" (Host
	// carries the parsed host a browser would resolve against the ambient
	// scheme) or a three-or-more-slash form ("///x"), which Go parses as a
	// rooted path while browsers read an authority (Host stays empty - the
	// form is ambiguous).
	ClassProtocolRelative
	// ClassSchemelessHost is a scheme-free, non-rooted form ("host/x"):
	// net/url parses a bare path, but a browser address bar navigates to the
	// first segment as a host. Host carries that authority-reparse evidence
	// (empty for a query- or fragment-only form such as "?x:y");
	// HostUnrecoverable marks a failed reparse.
	ClassSchemelessHost
	// ClassRelative is a rooted, host-free relative path ("/x").
	ClassRelative
)

type Form

type Form struct {

	// Trimmed is the whitespace-trimmed raw string the classification read,
	// with backslashes NOT canonicalized: it is what a publisher emits or
	// prefixes, never a rewritten form.
	Trimmed string
	// Host is the lowercased host evidence a browser would navigate to, when
	// extractable: the parsed hostname of an absolute or protocol-relative
	// form, or the authority reparse of a schemeless-host form. Empty when
	// the string carries none (or the form hides it; see Class). The fold is
	// ASCII-only by design (see asciiLower): a full-Unicode fold would
	// launder homograph bytes (U+0130 -> 'i', U+212A -> 'k') into ASCII, so
	// non-ASCII host evidence survives here unfolded for a consumer's
	// fail-closed ASCII-only host gates.
	Host string
	// Scheme is the canonicalized parse's scheme, which url.Parse folds to
	// lowercase (an "HTTPS://" source reads "https", RFC 3986 canonical
	// form), so the value is already case-folded; empty when the string
	// carries none or did not parse. Case-insensitive comparison by consumers
	// remains correct as defense in depth.
	Scheme string
	// Port is the canonicalized parse's port string; empty when none is
	// present or the string did not parse. net/url only accepts an
	// all-digit port, but it does not range-check it - consumers that need
	// a valid 16-bit port (a link publisher) validate the range.
	Port string
	// Class is the structural form.
	Class Class
	// HasBackslash records a '\' anywhere in the trimmed string. Browsers
	// (WHATWG URL parser) treat '\' as '/', so the parsed facts (Scheme/
	// Host/Port/Class) describe the canonicalized reading - a `/\host/x`
	// form classifies protocol-relative,
	// not as a host-less rooted path - while the flag lets a publisher that
	// must emit the raw string reject it outright.
	HasBackslash bool
	// HostUnrecoverable marks a ClassSchemelessHost whose authority reparse
	// failed (e.g. a space before an "@"): browser-visible host evidence may
	// exist but cannot be extracted, so evidence-driven consumers treat the
	// form like a parse failure.
	HostUnrecoverable bool
	// HasUserInfo records a userinfo authority component ("user@host") in
	// the canonicalized parse - a visual-spoofing vector
	// ("https://trusted@evil/") a link publisher typically rejects. For a
	// ClassSchemelessHost the fact comes from the same authority reparse
	// that supplies Host (so "user@host/x" reports it alongside the
	// recovered host). Always false when the string did not parse.
	HasUserInfo bool
	// contains filtered or unexported fields
}

Form is the structural classification of one raw, untrusted URL string (an upstream API field, a scraped link, operator input). It names the browser-vs-net/url parse-quirk classes ONCE - backslash authorities, protocol-relative and schemeless-host forms, hidden-host parses - so every consumer branches on the same facts while keeping its own fail direction as policy: a publisher drops what it cannot vouch for (publish-or-drop), while an evidence gate hides what it cannot classify (extract-evidence-or-hide). Fields are ordered for govet fieldalignment.

func Classify

func Classify(raw string) Form

Classify classifies a raw URL string into its structural Form. It never errors: every input lands in exactly one class, and unparseable input is ClassMalformed. Consumers apply their own policy over the returned facts (see Form).