urlform

README

urlform

Classify raw untrusted URL strings by structural form: the browser-vs-net/url parse quirks that decide whether a string really carries a host

A standalone, stdlib-only Go library for programs that PUBLISH untrusted URLs to humans or extract the host a browser would navigate to. Go's net/url and a browser's WHATWG parser read several string shapes differently. A browser strips embedded tabs and newlines (https://anime\tbytes.tv navigates to animebytes.tv), treats \ as /, reads an authority through any run of slashes after https:, navigates host/x to host, resolves //host/x against the ambient scheme, and shows the post-@ host for a user@host authority. Code that trusts the Go parse alone can publish a link whose real destination it never saw. urlform names those quirk classes once, extracts the browser-visible facts, and leaves the fail direction to each consumer.

The covered divergence set is bounded and enumerated (see the package docs), pinned by a conformance-fixture corpus derived from web-platform-tests; urlform is a classifier with the WHATWG readings layered on, not a full WHATWG parser. Out of scope by design: IDNA/punycode mapping (non-ASCII host evidence survives raw for the fail-closed gates), percent-encoding normalization, and port range checks (the facts are reported; the publisher validates).

This is deliberately NOT an SSRF guard. Validating a URL your own process will fetch answers to net/url and the dialer (the parser of record for the request); use an SSRF library for that. urlform models classify-for-publish, where the parser of record is the reader's browser.

Install

go get github.com/cplieger/urlform@latest

Usage

f := urlform.Classify(raw)
switch f.Class {
case urlform.ClassAbsolute:
	if f.HasUserInfo || f.HasBackslash {
		// visual-spoofing vectors a publisher typically rejects
		return "", false
	}
	return f.Trimmed, true
case urlform.ClassRelative:
	return base + f.Trimmed, true // rooted path, no host of its own
default:
	return "", false // protocol-relative, schemeless, hidden-host, malformed
}

Host evidence for matching against known domains:

f := urlform.Classify(raw)
if f.Host == "" || !urlform.IsASCIIHost(f.Host) {
	// no host evidence, or homograph territory: fail closed
	return nil, false
}
for domain, tracker := range knownDomains {
	if urlform.HostMatchesDomain(f.Host, domain) {
		return tracker, true
	}
}
return nil, false

Names of an untrusted raw query, without the evadable parsed view:

for name := range urlform.RawQueryNames(u.RawQuery) {
	if isCredentialParam(name) { // the caller's predicate, and its fail direction
		return true
	}
}

Same walk when the predicate needs the value too:

for name, value := range urlform.RawQueryPairs(u.RawQuery) {
	if isCredentialParam(name) && value != "" {
		return redact(name) // a parameter carrying nothing is not a leak
	}
}

Two spellings of one destination, compared:

f := urlform.Classify(raw)
if p := f.NormalizedPath(); p == "" || !strings.HasPrefix(p, "/beat/") {
	return errOutsideNamespace // "/beat/api/../../ghost" resolves out of the namespace
}

API

• Classify(raw string) Form: total classification. Every input lands in exactly one class, never an error; the WHATWG input preprocessing and backslash canonicalization run first (see Design notes).
• Form: the extracted facts: Class, Trimmed (preprocessed, emit-safe), Host (ASCII-only lowercase fold), Scheme, Port (extracted, deliberately not range-checked), HasBackslash, HasTabOrNewline (a whitespace-smuggling attempt was removed), HasUserInfo, HostUnrecoverable.
• Class: ClassEmpty, ClassMalformed, ClassAbsolute, ClassHiddenHost (a scheme-bearing parse hiding host evidence; for the authority-carrying special schemes the browser's reading is recovered into the facts, so https:/host/x and https:host/x expose host, while host:443/x and https://:443/x stay evidence-free like the browser's own reading), ClassProtocolRelative (//host/x and the ambiguous ///x), ClassSchemelessHost (host/x, where a browser navigates to host), ClassRelative (/x).
• (*Form).NormalizedPath() string: the path a browser resolves for the classified string, dot segments removed (/view/1/../2 reads /view/2), for comparing or displaying two spellings of one destination. Rooted; resolved by net/url's own RFC 3986 §5.2.4 resolution, so repeated slashes are preserved exactly as the WHATWG parser preserves them (a consumer wanting net/http's ServeMux rewrite is asking about Go's router, not the reader's browser). Empty when no browser-resolvable path exists: no facts at all, a failed authority reparse, an opaque-path reading (javascript:alert(1)), or a //-leading form whose authority region no parse separated.
• IsASCIIHost(host string) bool: the fail-closed companion gate. It reports whether every byte is ASCII, so a homograph host (Cyrillic lookalikes, fold-laundering U+0130/U+212A) never string-matches a canonical domain. Consumers that must accept international hosts convert punycode explicitly instead of relaxing the gate.
• FoldHostASCII(host string) string: the ASCII-only case fold Form.Host applies, exported for consumers holding host evidence from elsewhere (a configured domain, a header, a host parsed directly). Folds only A-Z, which is the point: strings.ToLower maps U+0130 to i and U+212A to k, and strings.EqualFold reads U+017F as s, so either one launders a homograph into a canonical ASCII domain before the gate above can reject it.
• EqualASCIIFold(a, b string) bool: the same rule as a comparison, for the ASCII protocol tokens that are not hosts (a URL path token, a query parameter name), which a structural gate reads out of an untrusted URL and where calling a host fold would misname the operation. strings.EqualFold accepts a U+212A KELVIN SIGN spelling of apikey and a U+017F LONG S spelling of /torrents.php, and a strings.ToLower comparison accepts a U+0130 DOTTED CAPITAL I spelling of torrentid: each hands a gate a match on bytes no server routes as that token. This comparison cannot: a string that is not ASCII never equals an ASCII token.
• HostMatchesDomain(host, domain string) bool: the matcher the gate above leads to. Reports whether the host equals the domain or is a real dot-delimited subdomain of it, refusing the three readings a plain suffix test accepts: suffix confusion (evilnyaa.si), parent-domain spoofing (nyaa.si.evil.example), and empty DNS labels (.nyaa.si, a..nyaa.si). Fail-closed on an empty or empty-labelled argument; folds ASCII-only. Trimming surrounding space and the trailing root dot stays the caller's.
• RawQueryNames(rawQuery string) iter.Seq[string]: the percent-decoded parameter names of a raw query (u.RawQuery's shape, no leading ?), split on both & and ;. url.ParseQuery drops a malformed pair wholesale, so an unescaped semicolon deletes the pair it sits in while the bytes still ride every request and log line; this walk is the strict superset a gate cannot be evaded on. Judgment-free: it reports names and takes no view of them, because consumers need opposite fail directions over the same walk.
• RawQueryPairs(rawQuery string) iter.Seq2[string, string]: the same walk carrying each name's VALUE, for the consumers whose predicate reads it (a credential warning that must not fire on an empty parameter, a redaction pass locating the secret text). Name and value are percent-decoded independently, each falling back to its raw text, so one malformed half never hides the other.

Design notes

• Judgment-free classification. The library names facts; policy stays with the caller. One consumer publishes-or-drops, another extracts-evidence-or-hides; both branch on the same classes and can never drift on what the string structurally is.
• WHATWG input preprocessing. Browsers delete embedded tab/newline wherever they appear and trim C0-control/space edges before parsing (the same hardening CPython adopted for CVE-2022-0391), so a string-level gate that skips this reads a different URL than the reader's browser will. Classify runs both steps first; HasTabOrNewline records a removed smuggling attempt, and Trimmed is already clean to emit. Edge trimming is deliberately widened to all Unicode whitespace (an NBSP-wrapped link still classifies; over-trimming errs fail-safe).
• ASCII-only case folding, one byte rule. strings.ToLower has ASCII-producing mappings (U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE folds to i, U+212A KELVIN SIGN to k) and strings.EqualFold folds simple-case orbits (U+212A to k, U+017F LATIN SMALL LETTER LONG S to s), so either one launders a homograph into a matchable ASCII host or protocol token before any gate sees it. Note the two admit different inputs and are not interchangeable: strings.ToLower accepts the U+0130 spelling that strings.EqualFold refuses, and strings.EqualFold accepts the U+017F spelling that strings.ToLower refuses. Host, FoldHostASCII, HostMatchesDomain and EqualASCIIFold all read a single A-Z byte rule instead, so they can never disagree, and IsASCIIHost rejects the host evidence that survives. The rule is defined on bytes rather than runes deliberately: mapping runes would rewrite an invalid UTF-8 byte to U+FFFD, laundering distinct non-ASCII evidence onto one canonical spelling in a fold whose job is to leave it intact. Because that rule reads no Unicode table, no Unicode revision can change which bytes it folds; the package's only Unicode-table read is the edge trim's whitespace set.
• Backslash canonicalization is read-only and spec-scoped. The parsed facts describe the WHATWG reading (/\host/x classifies protocol-relative) for special-scheme and schemeless forms ahead of the query; for a non-special scheme a backslash is an ordinary character, and rewriting it would fabricate host evidence a browser never sees. HasBackslash lets a publisher that must emit the raw string reject it outright; the raw form is never rewritten.
• Dot-segment resolution reads the decoded path. NormalizedPath resolves over the decoded path, which is what makes a percent-encoded dot segment (/a/%2e%2e/b) resolve like the literal one, matching the WHATWG parser, whose single- and double-dot segment definitions include the %2e spellings. The same decoding reads a percent-encoded slash as a separator, which the parser does not, so a caller whose comparison must keep %2F distinct compares the escaped path itself. That is the same boundary as the rest of the contract: the facts model the browser's structural reading, not percent-encoding normalization.
• Bounded and total. Allocation is bounded and linear in the input, and unparseable input is a class (ClassMalformed), not an error.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

Apache-2.0. See LICENSE.

Documentation

Overview

Package urlform classifies the structural form of a raw, untrusted URL string — specifically the forms where a BROWSER's reading (the WHATWG URL parser) diverges from net/url's: whitespace-smuggled URLs, backslash authorities, slash-count fixups after a special scheme, protocol-relative and schemeless-host forms, hidden-host parses, userinfo spoofing.

The parser of record is the divide that places this package. Validating a URL the PROCESS will fetch is a different concern (there net/url and the dialer are authoritative; use an SSRF guard). urlform models classify-for-PUBLISH: the string is destined for a human whose browser will read it, and the quirk classes exist precisely where that reading and Go's disagree.

The covered divergence set is bounded and enumerated — urlform is a classifier over net/url with the WHATWG readings layered on, not a conformant WHATWG parser. What it models, pinned by the conformance fixtures in testdata/whatwg-fixtures.json (WPT-derived plus hand-derived address-bar rows):

• Input preprocessing: leading/trailing C0-control-or-space trimming (widened to all Unicode whitespace, a documented fail-safe superset) and embedded ASCII tab/newline removal, recorded by HasTabOrNewline — so "https://anime\tbytes.tv/x" classifies with its real host.
• Backslash-as-slash for special schemes (http, https, ws, wss, ftp, file) and schemeless forms, ahead of the query/fragment; a non-special scheme's backslashes stay ordinary characters.
• Slash-count fixups: after an authority-carrying special scheme the browser reads an authority through ANY run of slashes, so "https:/host/x" and "https:host/x" expose their hidden host evidence (ClassHiddenHost with recovered facts).
• Address-bar forms outside the URL spec: "host/x" navigates to host (ClassSchemelessHost), "//host/x" resolves against the ambient scheme (ClassProtocolRelative).

Deliberately NOT modeled (the boundary of the contract): IDNA/UTS46 host mapping and punycode (non-ASCII host evidence survives raw for the fail-closed gates — see IsASCIIHost), percent-encoding normalization, port range checking (the fact is reported, the publisher validates), full host validation (net/url's acceptance stands in for it), interior non-tab C0 controls (net/url rejects them; where they sit in a host the browser rejects them too), and the file scheme's drive-letter quirks. Future WHATWG changes land here only when enumerated — the fixtures name the supported set, and drift against them fails the build.

Classify never errors: every input lands in exactly one Class with the extractable semantic facts (Host, Scheme, Port, HasUserInfo, HasBackslash, HasTabOrNewline) alongside. The classification is deliberately judgment-free; each consumer applies its own fail direction over the same facts — a publisher drops what it cannot vouch for, an evidence gate hides what it cannot classify.

Form.NormalizedPath is the derived reading beside those facts: the path a browser resolves for the classified string, dot segments removed, for the consumers that must decide whether two spellings name ONE destination ("/beat/api/../ghost" leaves the namespace it appears to sit in). It resolves via net/url's own RFC 3986 resolution, preserves repeated slashes like the WHATWG parser (path.Clean, which collapses them, answers a question about Go's router instead), and reads empty wherever no browser-resolvable path exists.

Host evidence folds ASCII-only (a full-Unicode fold would launder homograph bytes such as U+0130, U+212A or U+017F into ASCII), FoldHostASCII is that fold exported for consumers holding host evidence of their own, IsASCIIHost is the fail-closed companion gate for consumers matching hosts against known ASCII domains, and HostMatchesDomain is the safe equals-or-subdomain comparison that gate leads to. EqualASCIIFold is the same rule as a comparison, for the ASCII protocol tokens that are NOT hosts — a path token, a query parameter name — which a structural gate reads out of an untrusted URL and where strings.EqualFold would let a homograph spelling match. All four read one byte rule, so a consumer folding a host, a path and a query name can never work from two ideas of what folding ASCII means. That rule reads no Unicode table, so no Unicode revision can change which bytes it folds; the package's only Unicode-table read is the edge trim's whitespace set.

RawQueryNames sits beside those as the other raw-reading primitive: the percent-decoded parameter names of a query string, split on both '&' and ';', because url.ParseQuery drops a malformed pair wholesale and a gate built on the parsed view can therefore be evaded by a smuggled separator while the bytes still ride every request and log line. RawQueryPairs is the same walk carrying each name's value, for the consumers whose predicate reads it. Like the Class facts both are judgment-free - they report what the wire carries, and each consumer applies its own predicate and fail direction.

Index

• func EqualASCIIFold(a, b string) bool
• func FoldHostASCII(host string) string
• func HostMatchesDomain(host, domain string) bool
• func IsASCIIHost(host string) bool
• func RawQueryNames(rawQuery string) iter.Seq[string]
• func RawQueryPairs(rawQuery string) iter.Seq2[string, string]
• type Class
• type Form
  • func Classify(raw string) Form
  • func (f *Form) NormalizedPath() string

Functions

func EqualASCIIFold

func EqualASCIIFold(a, b string) bool

EqualASCIIFold reports whether a and b are equal under ASCII-only case folding: A-Z fold to a-z and nothing else does. It is FoldHostASCII's rule as a comparison (both read the single byte rule, so the two can never disagree), for the strings that are NOT hosts - a URL path token, a query parameter name, any fixed ASCII protocol token an untrusted string is matched against. A structural gate reading "/torrents.php" or "torrentid" out of an untrusted URL is doing that comparison, and calling a host fold on a path would misname the operation.

The ASCII-only restriction is the whole point, and the reason not to reach for strings.EqualFold: full Unicode simple folding has ASCII-PRODUCING mappings, so it reads U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE as 'i', U+212A KELVIN SIGN as 'k' and U+017F LATIN SMALL LETTER LONG S as 's'. Under that fold a homograph spelling of a protocol token compares EQUAL to the ASCII token - "torrent\u0130d" passes a strings.EqualFold check for "torrentid" - which hands an evidence gate a match on bytes no server ever routes and no operator ever reads as that token. This fold cannot: a string that is not ASCII can never equal an ASCII token here.

It folds case and nothing else: no trimming, no percent-decoding (see RawQueryNames for that reading), no Unicode normalization. Length is compared first because the fold is byte-length-preserving - unlike a Unicode fold, where differing lengths can still fold equal, which is the same property that makes strings.EqualFold's laundering possible.

func FoldHostASCII

func FoldHostASCII(host string) string

FoldHostASCII lowercases only the ASCII letters A-Z of host, leaving every other byte untouched. It is the exported form of the fold Form.Host already applies and HostMatchesDomain already compares with, for consumers holding host evidence from somewhere else - a configured domain, a request header, a host parsed by net/url directly - that must be compared against those facts under the same rule.

The ASCII-only restriction is the whole point, and the reason to call this instead of strings.ToLower or strings.EqualFold: both have ASCII-PRODUCING mappings for a case operation on a host. strings.ToLower folds U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE to 'i' and U+212A KELVIN SIGN to 'k', and strings.EqualFold additionally reads U+017F LATIN SMALL LETTER LONG S as 's', so either one launders a homograph host into a canonical ASCII domain BEFORE the fail-closed gate (IsASCIIHost) can see the non-ASCII bytes it exists to reject. This fold cannot: a host that is not ASCII stays not ASCII.

It folds case and nothing else - no trimming, no IDNA/punycode mapping, no percent-decoding (see the package docs for that boundary) - and it is not a substitute for the gate. Callers still run IsASCIIHost first; the fold makes host evidence safe to COMPARE case-insensitively, not safe to trust.

func HostMatchesDomain

func HostMatchesDomain(host, domain string) bool

HostMatchesDomain reports whether host equals domain or is a real dot-delimited subdomain of it. It is the matcher IsASCIIHost's documented scenario leads to: a consumer that has gated untrusted host evidence on IsASCIIHost then has to compare it against a known ASCII domain, and plain suffix matching is wrong in three ways this closes.

• Suffix confusion: "evilnyaa.si" ends in "nyaa.si" without being under it, so the separating dot is required, not incidental.
• Parent-domain spoofing: "nyaa.si.evil.example" contains the domain but is owned by evil.example, so the domain must be the SUFFIX, not a substring.
• Empty DNS labels: a bare suffix test also accepts ".nyaa.si" (its leading dot) and "a..nyaa.si" (an inner one). No resolvable DNS name carries an empty label, so both forms are adversarial and every label of the subdomain prefix must be non-empty.

It is fail-closed and total. An empty host or domain matches nothing (an empty domain would otherwise match an empty host), and a domain that itself carries an empty label matches nothing rather than lending its malformation to the comparison. Comparison folds ASCII-only, for the same reason Form.Host does: a full-Unicode fold has ASCII-producing mappings that would launder a homograph host into a match. That fold is a convenience, NOT a substitute for the gate - a non-ASCII host can never equal an ASCII domain here, but callers still run IsASCIIHost first, because it is the check that refuses the evidence outright instead of merely failing to match one domain.

It does NOT require an ASCII host, and deliberately so: a non-ASCII byte in a SUBDOMAIN label ("\u00e9.nyaa.si" against "nyaa.si") is a truthful match, since that host really is under the domain - the spoofing risk lives in the region aligned with the domain, which is compared byte-wise against ASCII and so cannot hold laundered bytes. Refusing non-ASCII evidence outright is IsASCIIHost's separate, stricter job, which is why consumers run it first rather than expecting this to imply it.

Normalization beyond the ASCII fold stays the caller's: this compares the host it is given, so a caller holding raw evidence trims its own surrounding ASCII space and its own trailing root dot ("nyaa.si." does not match "nyaa.si") before calling. Doing it here would mean applying Unicode-aware trimming to a string whose non-ASCII bytes are exactly what the caller's gate exists to see.

func IsASCIIHost

func IsASCIIHost(host string) bool

IsASCIIHost reports whether every byte of host is ASCII (below utf8.RuneSelf). It is the fail-closed companion of Form.Host's ASCII-only fold: a consumer matching host evidence against known ASCII domains gates on it first, so a homograph host (Cyrillic lookalikes, a fold-laundering U+0130 or U+212A) never string-matches a canonical domain. Callers that must ACCEPT international hosts convert punycode explicitly instead of relaxing this predicate.

func RawQueryNames

func RawQueryNames(rawQuery string) iter.Seq[string]

RawQueryNames iterates the percent-decoded parameter NAMES of a raw query string, in order, without consulting url.Values. It exists because the parsed view can be evaded: url.ParseQuery (and therefore u.Query()) drops a malformed pair WHOLESALE, so an unescaped semicolon deletes the pair it sits in - "apikey=SECRET;foo=x" disappears from the parsed map while the bytes stay in RawQuery for every outgoing request and every logged URL. A consumer whose gate must not be evadable therefore needs the raw reading, which is a strict superset of the parsed one:

• Pairs are split on BOTH '&' and ';' (the historic separator whose removal from url.ParseQuery is what creates the gap), empty fields skipped.
• The name is the text before the first '=' (a pair with no '=' yields its whole field as a name, which is how a bare flag parameter reads).
• Each name is percent-decoded, so an encoded spelling cannot hide from a literal comparison. A name whose escapes do not decode is yielded RAW rather than skipped, so a malformed pair still reaches the caller's predicate instead of vanishing the way the parsed view vanishes it.

The iteration is judgment-free, like the Class facts: it reports names and takes no view of them, because consumers need opposite fail directions over the same walk - a credential-in-URL warning wants any suspicious name to match (over-matching is safe), while a structural identity gate wants only the name it recognizes (over-matching admits a URL it should refuse).

The argument is a raw query WITHOUT its leading '?' - u.RawQuery's shape. A '?' is a legal literal inside a query, so it is not trimmed: a caller holding a whole URL takes u.RawQuery (or cuts at the first '?') rather than passing the URL, exactly as IsASCIIHost takes a host and not a URL.

RawQueryPairs is the same walk carrying each name's VALUE alongside, for the consumers whose predicate reads it.

func RawQueryPairs

func RawQueryPairs(rawQuery string) iter.Seq2[string, string]

RawQueryPairs iterates the percent-decoded NAME and VALUE of each pair in a raw query string, in order, under RawQueryNames' parsing discipline and for the same reason: the parsed view can be evaded, so a gate that must not be evadable reads the wire instead. It is the companion for the consumers whose predicate needs the value too - a credential warning that must not fire on a parameter carrying nothing, a redaction pass that has to locate the secret text, an identity gate matching one expected id - and those are exactly the consumers url.ParseQuery fails hardest: it drops a malformed pair WHOLESALE, so the value disappears from the parsed map while the bytes stay in RawQuery for every outgoing request and every logged URL.

• Pairs are split on BOTH '&' and ';' (the historic separator whose removal from url.ParseQuery creates the gap), empty fields skipped.
• The name is the text before the first '=' and the value is everything after it, so a '=' inside a value is part of the value.
• Name and value are percent-decoded INDEPENDENTLY, each falling back to its own raw text when its escapes do not decode, so one malformed half never hides the other from the caller's predicate the way the parsed view hides both.
• A field with no '=' yields its whole text as the name and an empty value, which is how a bare flag parameter reads. "x" and "x=" are therefore one reading here; a caller that must tell them apart reads the raw field itself.

The iteration is judgment-free, like RawQueryNames and the Class facts: it reports pairs and takes no view of them, because consumers need opposite fail directions over the same walk. The argument is a raw query WITHOUT its leading '?' - u.RawQuery's shape; a '?' is a legal literal inside a query, so it is not trimmed.

Types

type Class

type Class int

Class names the structural form of a raw, untrusted URL string - specifically the browser-vs-net/url parse quirks that decide whether the string really carries a host. It is the single home of that quirk vocabulary; see Form.

const (
	// ClassEmpty is a string that is empty after the input preprocessing
	// (edge trimming plus tab/newline removal; see Classify).
	ClassEmpty Class = iota
	// ClassMalformed is a string the canonicalized parse rejected; no
	// structural facts (and no host evidence) can be extracted from it.
	ClassMalformed
	// ClassAbsolute is a scheme-and-host absolute URL ("https://host/x");
	// Host carries the parsed hostname.
	ClassAbsolute
	// ClassHiddenHost is a scheme-bearing parse with no hostname, where
	// net/url sees no host but a browser may navigate to one. For the
	// authority-carrying special schemes (http, https, ws, wss, ftp) the
	// WHATWG parser skips ANY run of slashes after the scheme - zero
	// ("https:host/x"), one ("https:/host/x"), or many - and reads the
	// authority, so the classifier runs the same authority reparse it uses
	// for schemeless forms and recovers the browser's reading into
	// Host/Port/HasUserInfo (HostUnrecoverable marks a failed recovery; a
	// port-only authority such as "https://:443/x" recovers no host, which
	// matches the browser - the WHATWG parser fails on an empty special
	// host). For every other scheme ("host:443/x" parsing the host as an
	// opaque scheme, "javascript:alert(1)", "mailto:x") the browser reads an
	// opaque path with no authority, so the facts stay empty - there the
	// host evidence, if any, is genuinely hidden.
	ClassHiddenHost
	// ClassProtocolRelative is a network-path reference: "//host/x" (Host
	// carries the parsed host a browser would resolve against the ambient
	// scheme) or a leading-"//" form with no extractable host evidence - a
	// three-or-more-slash form ("///x": Go parses a rooted path while
	// browsers read an authority) or an empty-authority form ("//", "//?q").
	// Host is the discriminator between the two sub-forms: consumers that
	// need host evidence treat an empty Host here as ambiguous and fail
	// closed.
	ClassProtocolRelative
	// ClassSchemelessHost is a scheme-free, non-rooted form ("host/x"):
	// net/url parses a bare path, but a browser address bar navigates to the
	// first segment as a host. Host carries that authority-reparse evidence
	// (empty for a query- or fragment-only form such as "?x:y");
	// HostUnrecoverable marks a failed reparse.
	ClassSchemelessHost
	// ClassRelative is a rooted, host-free relative path ("/x").
	ClassRelative
)

type Form

type Form struct {

	// Trimmed is the preprocessed raw string the classification read: edges
	// trimmed and embedded ASCII tab/newline removed (the WHATWG input
	// preprocessing, recorded by HasTabOrNewline), with backslashes NOT
	// canonicalized. It is what a publisher emits or prefixes - already free
	// of the whitespace-smuggling bytes a browser would silently drop, and
	// never otherwise rewritten.
	Trimmed string
	// Host is the lowercased host evidence a browser would navigate to, when
	// extractable: the parsed hostname of an absolute or protocol-relative
	// form, or the authority reparse of a schemeless-host or recoverable
	// hidden-host form. Empty when the string carries none (or the form
	// hides it; see Class). The fold is ASCII-only by design (see
	// asciiLower): a full-Unicode fold would launder homograph bytes
	// (U+0130 -> 'i', U+212A -> 'k') into ASCII, so non-ASCII host evidence
	// survives here unfolded for a consumer's fail-closed ASCII-only host
	// gates.
	Host string
	// Scheme is the canonicalized parse's scheme, which url.Parse folds to
	// lowercase (an "HTTPS://" source reads "https", RFC 3986 canonical
	// form), so the value is already case-folded; empty when the string
	// carries none or did not parse. Case-insensitive comparison by consumers
	// remains correct as defense in depth.
	Scheme string
	// Port is the canonicalized parse's port string; empty when none is
	// present or the string did not parse. net/url only accepts an
	// all-digit port, but it does not range-check it - consumers that need
	// a valid 16-bit port (a link publisher) validate the range. (The WHATWG
	// parser rejects an out-of-range port outright; reporting the fact and
	// leaving the fail direction to the consumer is this package's model.)
	Port string
	// Class is the structural form.
	Class Class
	// HasBackslash records a '\' anywhere in the trimmed string. Browsers
	// (WHATWG URL parser) treat '\' as '/' for the special schemes (http,
	// https, ws, wss, ftp, file) and for schemeless forms (where the address
	// bar's ambient scheme is special), so for those the parsed facts
	// (Scheme/Host/Port/Class) describe the canonicalized reading - a
	// `/\host/x` form classifies protocol-relative, not as a host-less
	// rooted path. For a non-special scheme a backslash is an ordinary
	// character and is NOT canonicalized (a browser reads an opaque path).
	// Either way the flag lets a publisher that must emit the raw string
	// reject it outright.
	HasBackslash bool
	// HasTabOrNewline records that the edge-trimmed string contained
	// embedded ASCII tab or newline (U+0009, U+000A, U+000D), which the
	// WHATWG parser - and therefore this classification - removes wherever
	// they appear ("https://anime\tbytes.tv/x" navigates to animebytes.tv).
	// Trimmed already has them removed, so emitting Trimmed is safe; the
	// flag records the smuggling attempt for publishers that treat the
	// ORIGINAL string as emittable or want to reject de-smuggled input
	// outright.
	HasTabOrNewline bool
	// HostUnrecoverable marks a ClassSchemelessHost or recoverable
	// ClassHiddenHost whose authority reparse failed (e.g. a space before an
	// "@"): browser-visible host evidence may exist but cannot be extracted,
	// so evidence-driven consumers treat the form like a parse failure.
	HostUnrecoverable bool
	// HasUserInfo records a userinfo authority component ("user@host") in
	// the canonicalized parse - a visual-spoofing vector
	// ("https://trusted@evil/") a link publisher typically rejects. For a
	// ClassSchemelessHost or recovered ClassHiddenHost the fact comes from
	// the same authority reparse that supplies Host (so "user@host/x"
	// reports it alongside the recovered host). Always false when the
	// string did not parse.
	HasUserInfo bool
	// contains filtered or unexported fields
}

Form is the structural classification of one raw, untrusted URL string (an upstream API field, a scraped link, operator input). It names the browser-vs-net/url parse-quirk classes ONCE - backslash authorities, protocol-relative and schemeless-host forms, hidden-host parses - so every consumer branches on the same facts while keeping its own fail direction as policy: a publisher drops what it cannot vouch for (publish-or-drop), while an evidence gate hides what it cannot classify (extract-evidence-or-hide). Fields are ordered for govet fieldalignment.

func Classify

func Classify(raw string) Form

Classify classifies a raw URL string into its structural Form. It never errors: every input lands in exactly one class, and unparseable input is ClassMalformed. Consumers apply their own policy over the returned facts (see Form).

Classification starts with the WHATWG basic parser's input preprocessing, so string-level whitespace smuggling cannot hide a URL from the facts: leading/trailing C0 controls and whitespace are trimmed (trimEdges), and embedded ASCII tab/newline are removed everywhere (recorded by HasTabOrNewline). This is the same hardening CPython adopted for urllib.parse (CVE-2022-0391).

func (*Form) NormalizedPath

func (f *Form) NormalizedPath() string

NormalizedPath returns the browser's reading of the classified string's path with its dot segments removed ("/view/1/../2" reads "/view/2"), for comparison and display. It answers the question the raw string cannot: two spellings of ONE destination must compare equal, so a gate deciding whether a path is still inside a namespace ("/beat/api/../ghost" leaves the /beat namespace every browser resolves it out of) and a display that must not show a path pointing somewhere else both need the resolved reading rather than the bytes.

The removal is net/url's own RFC 3986 section 5.2.4 resolution (ResolveReference against a rooted base), so the package carries no second dot-segment implementation, and the result is always rooted. Repeated slashes are PRESERVED ("/a//b" reads "/a//b") because the WHATWG parser preserves them too; a consumer that wants net/http's ServeMux rewrite (path.Clean, which also collapses them) is asking a different question - about Go's router, not about the reader's browser - and keeps its own helper for it.

The reading is over the DECODED path, which is what makes a percent-encoded dot segment ("/a/%2e%2e/b") resolve like the literal one, matching the WHATWG parser (its single- and double-dot segment definitions include the %2e spellings). The same decoding reads a percent-encoded SLASH as a separator, which the WHATWG parser does NOT, so a caller whose comparison must keep "%2F" distinct from a separator compares the escaped path itself. That delta has the same shape as the package's other documented boundary: the facts model the browser's structural reading, not percent-encoding normalization.

It is empty when the string carries no browser-resolvable path: ClassEmpty and ClassMalformed (no facts at all), a failed authority reparse (HostUnrecoverable), the hidden-host forms a browser reads as an OPAQUE path ("javascript:alert(1)", "mailto:x") where no dot-segment removal happens at all, and a ClassProtocolRelative form with no Host - the three-or-more-slash sub-form, where net/url read the region a browser reads as an authority as part of its path, so no parse this classification ran separated the two and any path reported would carry the browser's authority region inside it ("///a/../b" would read "///b"). That is the same fail-closed reading Host takes there.

A form carrying host evidence but no path reads "/", the browser's own resolution of an authority-only URL ("https://nyaa.si"), while a host-less form with no path (a query- or fragment-only reference such as "?x:y") reads empty, because the path such a reference resolves against is a base this classification never saw. Where an authority WAS separated but yielded no host evidence ("https://:443/x", which a browser refuses outright for its empty host), the path region is still genuinely the path and reads as one: Host stays the fact a consumer gates on. Query and fragment are never part of the reading.