normtext

README

normtext

Deterministic, composable text normalization for Go: the mechanical "clean the text before you sketch it" layer that similarity and search tooling usually punts to the caller.

Built as the preprocessing companion to semblance (deterministic text-similarity sketching):

import (
    "github.com/ophymx/normtext"
    "github.com/ophymx/semblance"
)

clean := normtext.ForShingling()
sim := semblance.Similarity(clean(a), clean(b))

Install

go get github.com/ophymx/normtext

Requires Go 1.25+. Dependencies: golang.org/x/text (and golang.org/x/net if you import the markup sub-package).

Your pipeline is part of your frozen configuration

normtext output feeds stored similarity signatures: a change in normalization is a change in every signature computed downstream, exactly like changing a MinHash seed. normtext therefore treats pipeline definitions as frozen surface:

• Presets never change behavior under their name. An improved preset ships as a new name (ForShingling2); the old name keeps working and keeps its output forever.
• Pipelines carry a storable identity. Callers persisting signatures should record it alongside the sketch parameters:

p := normtext.ForShinglingPipeline()
p.ID() // "normtext/1:stripinvisible|nfc|lower|straightenpunct|collapsespace"

Custom chains get the same treatment via normtext.NewPipeline and normtext.Named, plus golden tests on your side. Six months from now, "which normalization produced these signatures?" should be a lookup, not an archaeology dig.

Transforms

Each transform is a func(string) string, pure and safe for concurrent use. Input that would not change is returned unchanged — same backing array, zero allocations for clean ASCII (asserted in tests).

Transform	What it does
NFC / NFKC	Unicode canonical / compatibility composition
Lower	Unicode lowercasing (language-neutral)
Fold	full case folding for caseless matching (ß → ss)
CollapseSpace	whitespace runs → one space, trim ends
Newlines	CR, CRLF, NEL, LS, PS → \n
StripDiacritics	café → cafe (NFD, drop marks, NFC)
StripInvisible	remove zero-width chars, bidi controls, BOM, soft hyphens, non-whitespace controls
StraightenPunct	smart quotes/dashes/ellipsis → ASCII
FoldConfusables	UTS #39 skeleton: homoglyph spoofs (Cyrillic "раураl") fold to their prototypes ("paypal")
MaskPattern	regex → placeholder for URLs, emails, numbers

Compose with Chain/Apply, or use a frozen preset:

• ForShingling — the conservative default before word shingles: stripinvisible | nfc | lower | straightenpunct | collapsespace.
• Aggressive — recall-maximizing fuzzy matching: adds NFKC, case folding, diacritic stripping, and confusable folding. Note that UTS #39 skeletons rewrite some ASCII (m → rn, 1 → l): Aggressive output is for matching, never display.

Sub-packages

Optional imports; the root package depends on none of them.

Package	What it gives you
message	email/NNTP structural stripping: quoted replies + attribution (StripQuotes), signatures (StripSignature, plus Usenet decorative-rule sign-offs via StripSignatureRules), header blocks (StripHeaders)
markup	StripTags: HTML/XML → normalized plain text via x/net/html tokenization (entities decoded, block boundaries → newlines)
code	SplitIdentifiers: camelCase/PascalCase/snake_case → space-separated lowercase words

A reply-corpus pipeline, end to end:

p := normtext.NewPipeline(
    message.StripHeadersStep(),
    message.StripQuotesStep(),
    message.StripSignatureStep(),
    normtext.StripInvisibleStep(),
    normtext.NFCStep(),
    normtext.LowerStep(),
    normtext.StraightenPunctStep(),
    normtext.CollapseSpaceStep(),
)
sig := minhasher.Sketch(shingle.Words(p.Apply(post), 3))
// store sig together with p.ID()

Determinism

Same input + same pipeline → same output, on every platform, in every process. The one honest dependency is the Unicode data version, which comes from two places:

• golang.org/x/text — NFC/NFKC, casing, folding. x/text ships tables for more than one Unicode generation and picks one at build time with go1.N build constraints, so the active generation is selected jointly by the x/text version in go.mod and the building Go toolchain. An upgrade of either that crosses a generation is a signature-affecting change — a dedicated test probes generation-distinguishing code points so it fails CI visibly instead of drifting silently.
• Tables generated into this repository — Unicode categories (Mn, Cf) and the UTS #39 confusables table (which x/text does not ship). These shift only on a deliberate, reviewed regeneration commit.

Two facts keep that drift surface small. Unicode's stability policies make generation changes strictly additive — newly assigned code points gain behavior, assigned ones never change (measured 15.0.0 → 17.0.0: 158 code points differ, every one newly assigned in Unicode 16/17 — new letters and marks, case pairs, and compatibility-mapped symbols; zero changes to existing behavior) — so text containing only code points assigned in the active generation keeps its normalization forever. And pure-ASCII paths, whitespace, line separators, punctuation mappings, and the message package's rules are frozen in source: unconditionally stable. Behavior is pinned by golden tests (testdata/*.tsv), regenerated only via go test -update as a reviewed diff.

Invalid UTF-8 is never repaired but always handled deterministically; each transform's godoc says exactly how.

Non-goals

No stemming, stopwords, or language detection (mechanical code-point rules only); no boilerplate/readability extraction; no encoding detection (input is UTF-8 — see x/text/encoding); no tokenization for search (that's semblance's side of the boundary); nothing model-based, no I/O.

License

MIT

Documentation

Overview

Package normtext provides deterministic, composable text normalization: the mechanical "clean the text before you sketch it" layer for similarity and search tooling (built as the preprocessing companion to github.com/ophymx/semblance).

Every transform is a pure function from string to string. Transforms compose with Chain or, when a machine-readable identity is needed for persistence, with NewPipeline. Frozen presets (ForShingling, Aggressive) take raw text to sketch-ready text in one call.

Determinism

Same input + same pipeline → same output, on every platform and in every process. Unicode-dependent transforms consult either tables generated into this repository (category snapshots, the confusables table), which shift only on a deliberate regeneration commit, or golang.org/x/text. x/text ships tables for more than one Unicode generation and selects one at build time via go1.N build constraints, so its active generation is chosen jointly by the go.mod x/text version and the building Go toolchain. Unicode's stability policies make a generation change strictly additive: previously unassigned code points gain behavior, while the normalization and case behavior of assigned code points never changes. Text containing only code points assigned in the active generation is therefore stable forever, and pure-ASCII input is unconditionally stable everywhere. A test probes generation-distinguishing code points so a toolchain upgrade that would activate a newer generation fails CI — a deliberate, reviewed change instead of a silent one. Each transform's documentation states its Unicode-version dependence.

Invalid UTF-8 is never repaired but always handled deterministically: input a transform would not change passes through byte-for-byte, and when a transform rewrites, x/text-backed and rune-deleting transforms replace invalid bytes with U+FFFD (pass-through there would break idempotence); each transform's documentation says which.

Frozen pipelines

normtext output feeds stored similarity signatures, so a pipeline's definition is as load-bearing as a MinHash seed. Named presets are frozen forever under their name; improvements ship as new names. Callers persisting signatures should store the Pipeline.ID alongside them.

Index

• func Apply(s string, ts ...Transform) string
• func Bytes(t Transform, b []byte) []byte
• type Pipeline
  • func AggressivePipeline() Pipeline
  • func ForShinglingPipeline() Pipeline
  • func NewPipeline(steps ...Step) Pipeline
  • func (p Pipeline) Apply(s string) string
  • func (p Pipeline) ID() string
  • func (p Pipeline) Steps() []Step
  • func (p Pipeline) Transform() Transform
• type Step
  • func CollapseSpaceStep() Step
  • func FoldConfusablesStep() Step
  • func FoldStep() Step
  • func LowerStep() Step
  • func MaskPatternStep(re, repl string) Step
  • func NFCStep() Step
  • func NFKCStep() Step
  • func Named(name string, t Transform) Step
  • func NewlinesStep() Step
  • func StraightenPunctStep() Step
  • func StripDiacriticsStep() Step
  • func StripInvisibleStep() Step
  • func (s Step) Name() string
  • func (s Step) Transform() Transform
• type Transform
  • func Aggressive() Transform
  • func Chain(ts ...Transform) Transform
  • func CollapseSpace() Transform
  • func Fold() Transform
  • func FoldConfusables() Transform
  • func ForShingling() Transform
  • func Lower() Transform
  • func MaskPattern(re, repl string) Transform
  • func NFC() Transform
  • func NFKC() Transform
  • func Newlines() Transform
  • func StraightenPunct() Transform
  • func StripDiacritics() Transform
  • func StripInvisible() Transform

Functions

func Apply

func Apply(s string, ts ...Transform) string

Apply applies ts to s left to right; it is Chain(ts...)(s) for one-shot use.

func Bytes

func Bytes(t Transform, b []byte) []byte

Bytes adapts a Transform to []byte without copying the input: b is viewed as a string zero-copy, and fresh bytes are allocated only if t changes it; otherwise b itself is returned.

The string passed to t aliases b: t must not retain it past the call, and b must not be mutated while Bytes runs. All transforms in this package satisfy the former.

Types

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline is a Transform with a machine-readable identity, for callers who persist signatures. Two pipelines with equal IDs produce equal output for every input, forever — built-in step names are part of the frozen surface, and callers own that guarantee for Named steps.

Pipeline values are immutable and safe for concurrent use.

func AggressivePipeline

func AggressivePipeline() Pipeline

AggressivePipeline is Aggressive with its Pipeline identity.

func ForShinglingPipeline

func ForShinglingPipeline() Pipeline

ForShinglingPipeline is ForShingling with its Pipeline identity.

func NewPipeline

func NewPipeline(steps ...Step) Pipeline

NewPipeline builds a Pipeline that applies steps left to right. It panics on a zero-value Step.

func (Pipeline) Apply

func (p Pipeline) Apply(s string) string

Apply runs the pipeline over s.

func (Pipeline) ID

func (p Pipeline) ID() string

ID returns the pipeline's identity string, e.g.

normtext/1:stripinvisible|nfc|lower|straightenpunct|collapsespace

Callers persisting similarity signatures should store this alongside them (next to the MinHash seed/k): same ID ⇒ same normalization behavior.

func (Pipeline) Steps

func (p Pipeline) Steps() []Step

Steps returns a copy of the pipeline's steps.

func (Pipeline) Transform

func (p Pipeline) Transform() Transform

Transform returns the pipeline as a plain Transform.

type Step

type Step struct {
	// contains filtered or unexported fields
}

Step is a named Transform: the unit from which a Pipeline is built. Built-in transforms have Step constructors with stable, frozen names (NFCStep, LowerStep, ...); custom transforms enter a Pipeline via Named. The zero Step is invalid.

func CollapseSpaceStep

func CollapseSpaceStep() Step

CollapseSpaceStep is CollapseSpace as a Pipeline step, named "collapsespace".

func FoldConfusablesStep

func FoldConfusablesStep() Step

FoldConfusablesStep is FoldConfusables as a Pipeline step, named "foldconfusables".

func FoldStep

func FoldStep() Step

FoldStep is Fold as a Pipeline step, named "fold".

func LowerStep

func LowerStep() Step

LowerStep is Lower as a Pipeline step, named "lower".

func MaskPatternStep

func MaskPatternStep(re, repl string) Step

MaskPatternStep is MaskPattern as a Pipeline step. Its name embeds the regex source and replacement — "mask(<re>=><repl>)" — because a caller-supplied pattern cannot be frozen under a fixed name. The name is informative, not a parseable grammar (re may contain any punctuation except '|', which Named forbids; use Named with your own name for patterns containing '|').

func NFCStep

func NFCStep() Step

NFCStep is NFC as a Pipeline step, named "nfc".

func NFKCStep

func NFKCStep() Step

NFKCStep is NFKC as a Pipeline step, named "nfkc".

func Named

func Named(name string, t Transform) Step

Named creates a Step from a caller-chosen name and transform. The caller owns keeping the (name → behavior) mapping frozen: a stored Pipeline ID containing this name is only meaningful while the name denotes the same behavior.

Named panics if name is empty or contains '|' (the ID separator), or if t is nil.

func NewlinesStep

func NewlinesStep() Step

NewlinesStep is Newlines as a Pipeline step, named "newlines".

func StraightenPunctStep

func StraightenPunctStep() Step

StraightenPunctStep is StraightenPunct as a Pipeline step, named "straightenpunct".

func StripDiacriticsStep

func StripDiacriticsStep() Step

StripDiacriticsStep is StripDiacritics as a Pipeline step, named "stripdiacritics".

func StripInvisibleStep

func StripInvisibleStep() Step

StripInvisibleStep is StripInvisible as a Pipeline step, named "stripinvisible".

func (Step) Name

func (s Step) Name() string

Name returns the step's identity name as it appears in a Pipeline ID.

func (Step) Transform

func (s Step) Transform() Transform

Transform returns the step's underlying transform.

type Transform

type Transform func(string) string

Transform maps text to normalized text.

Transforms are pure and safe for concurrent use. A transform that would make no change returns its input unchanged — the same string, same backing array. For already-clean ASCII input every transform in this package does so without allocating (asserted in tests); non-ASCII no-op input may cost transient scratch allocations on some transforms but still returns the original string.

func Aggressive

func Aggressive() Transform

Aggressive is the recall-maximizing preset for fuzzy / mutation-robust matching: compatibility folding, full case folding, diacritic stripping, confusable (homoglyph) folding. Exact sequence (frozen):

stripinvisible | nfkc | fold | stripdiacritics | straightenpunct |
foldconfusables | fold | nfc | collapsespace

ID: normtext/1:stripinvisible|nfkc|fold|stripdiacritics|straightenpunct|foldconfusables|fold|nfc|collapsespace

Because UTS #39 skeletons map some ASCII ("m" → "rn", "1" → "l"), Aggressive rewrites even clean English text: its output is for matching only, never display, and the clean-ASCII free-pass-through property does not apply (output is a fixed point instead — applying Aggressive to its own output is a zero-alloc no-op).

Unicode data: x/text (see the package Determinism notes) for NFKC/NFC, folding, and diacritics; the repo-generated confusables table; the other steps are unconditionally stable.

func Chain

func Chain(ts ...Transform) Transform

Chain composes transforms left to right into a single Transform. Chain() with no arguments returns the identity transform.

func CollapseSpace

func CollapseSpace() Transform

CollapseSpace returns a Transform that replaces every run of Unicode whitespace (the White_Space set: spaces, tabs, newlines, NBSP, ideographic space, ...) with a single U+0020 and trims leading/trailing whitespace.

Idempotent. Unconditionally stable: the whitespace set is frozen in this package, no external Unicode table. Already-collapsed input passes through allocation-free. Invalid UTF-8 passes through unchanged.

func Fold

func Fold() Transform

Fold returns a Transform applying full Unicode case folding (x/text/cases), the caseless-matching normalization: ß → ss, Σ/σ/ς → σ. More aggressive than Lower; output is for matching, not display.

Idempotent. Unicode data: x/text (see the package Determinism notes). ASCII input is unconditionally stable (folding is lowercasing there) and passes through allocation-free when already lowercase. Invalid UTF-8 in the non-ASCII path may be replaced with U+FFFD (deterministically).

func FoldConfusables

func FoldConfusables() Transform

FoldConfusables returns a Transform that maps text to its UTS #39 skeleton: NFD, replace each code point with its confusable prototype, NFD again. Visually confusable strings — homoglyph spoofs like Cyrillic "раураl" for "paypal" — fold to identical skeletons.

The skeleton maps some ASCII too ("0" → "O", "1"/"I"/"|" → "l", "m" → "rn"), so this transform rewrites even plain English text. Output is for matching only, never display. For case-insensitive matching, case-fold BEFORE folding confusables and again after (prototypes include uppercase); the Aggressive preset does both. Output is NFD-decomposed (per UTS #39); follow with NFC if composed output matters.

Idempotent: the generated table is transitively closed over NFD + mapping at generation time, because the raw Unicode data is not (a handful of skeletons therefore differ from a literal UTS #39 implementation; see confusables_gen.go). Unicode data: the repository's generated confusables table (currently Unicode 15.0.0) — x/text ships no confusables data, so this repo is the sole pin — plus x/text for NFD (see the package Determinism notes). ASCII input containing no mapped characters passes through allocation-free. Invalid UTF-8 is handled deterministically (bytes may pass through or be replaced via the x/text normalization path).

func ForShingling

func ForShingling() Transform

ForShingling is the recommended default ahead of semblance shingling: conservative and meaning-preserving. Exact sequence (frozen):

stripinvisible | nfc | lower | straightenpunct | collapsespace

ID: normtext/1:stripinvisible|nfc|lower|straightenpunct|collapsespace

Unicode data: x/text (see the package Determinism notes) for NFC and lowercasing; the other steps are unconditionally stable. ASCII input already in this form passes through allocation-free.

func Lower

func Lower() Transform

Lower returns a Transform applying Unicode lowercasing (x/text/cases, language-neutral "und" rules — no Turkish/Azeri tailoring, per the mechanical-not-linguistic principle).

Idempotent. Unicode data: x/text (see the package Determinism notes). ASCII input is unconditionally stable, lowercased via a fast path, and passes through allocation-free when already lowercase. Invalid UTF-8 in the non-ASCII path may be replaced with U+FFFD (deterministically).

Note for semblance users: its word shingler already lowercases tokens (stdlib unicode.ToLower), so Lower is redundant-but-harmless ahead of word shingles; it matters ahead of char shingles and winnowing.

func MaskPattern

func MaskPattern(re, repl string) Transform

MaskPattern returns a Transform that replaces every match of the regular expression re with repl (using regexp.Regexp.ReplaceAllString semantics, so $1-style expansions apply). Typical use: URLs, email addresses, or numbers → a fixed placeholder, so volatile substrings don't dominate similarity.

MaskPattern is the escape hatch among normtext transforms, with two caveats the frozen presets avoid (no preset includes it):

• It is not necessarily idempotent — repl may itself match re.
• It cannot be frozen by name: its Pipeline identity (MaskPatternStep) embeds the regex source and replacement, and the caller owns that (pattern → behavior) contract.

Deterministic (Go regexp is), stable across Unicode versions for ASCII patterns. Input with no match passes through unchanged; the match check may allocate transiently. Panics if re does not compile, like regexp.MustCompile.

func NFC

func NFC() Transform

NFC returns a Transform applying Unicode canonical composition (NFC).

Idempotent. Unicode data: x/text (see the package Determinism notes). ASCII input is unconditionally stable and passes through allocation-free. Invalid UTF-8 passes through when no normalization is needed; when rewriting, invalid bytes become U+FFFD.

func NFKC

func NFKC() Transform

NFKC returns a Transform applying Unicode compatibility composition (NFKC): ligatures, fullwidth forms, circled digits, etc. fold to their compatibility equivalents.

Idempotent. Unicode data: x/text (see the package Determinism notes). ASCII input is unconditionally stable and passes through allocation-free. Invalid UTF-8 passes through when no normalization is needed; when rewriting, invalid bytes become U+FFFD.

func Newlines

func Newlines() Transform

Newlines returns a Transform that normalizes line breaks to \n: CRLF and lone CR become \n, as do NEL (U+0085), LINE SEPARATOR (U+2028), and PARAGRAPH SEPARATOR (U+2029). \n itself is untouched.

Idempotent. Unconditionally stable: fixed code-point set, no external Unicode table. Input with only \n line breaks passes through allocation-free. Invalid UTF-8 passes through unchanged.

func StraightenPunct

func StraightenPunct() Transform

StraightenPunct returns a Transform that replaces typographic punctuation with ASCII equivalents:

• single quotes ‘ ’ ‚ ‛ ‹ › → '
• double quotes “ ” „ ‟ « » → "
• hyphens/dashes ‐ ‑ ‒ – — ― and minus − → -
• ellipsis … → ...

The mapping is exactly the set above — frozen in this source file, no external Unicode table, so unconditionally stable. (NFKC does not fold these: smart quotes are not compatibility equivalents of ASCII ones.)

Idempotent (output is ASCII, never remapped). ASCII input passes through allocation-free. Invalid UTF-8 passes through unchanged.

func StripDiacritics

func StripDiacritics() Transform

StripDiacritics returns a Transform that removes combining marks: NFD decomposition, drop nonspacing marks (Mn), NFC recomposition. café → cafe, façade → facade. Precomposed characters without a canonical decomposition (ß, ø, đ) are unaffected.

Idempotent. Unicode data: x/text for NFD/NFC (see the package Determinism notes) and the repository's generated Mn table snapshot (see tables_gen.go). ASCII input is unconditionally stable and passes through allocation-free. Invalid UTF-8 may be replaced with U+FFFD (deterministically).

func StripInvisible

func StripInvisible() Transform

StripInvisible returns a Transform that removes characters with no visual width or meaning for text comparison: Unicode format characters (Cf — zero-width space/joiner/non-joiner, bidi controls, soft hyphen, word joiner, BOM/ZWNBSP, ...) and C0/C1 control characters.

It never removes whitespace: \t, \n, \v, \f, \r, and NEL (U+0085) are preserved even though they are control characters — they belong to Newlines/CollapseSpace. (No Cf character is whitespace.)

Idempotent. Unicode data: the repository's generated Cf table snapshot (see tables_gen.go) — pinned by this repo, not by go.mod or the toolchain. ASCII input without controls is unconditionally stable and passes through allocation-free. Invalid UTF-8: input with nothing to remove passes through unchanged; otherwise invalid bytes are replaced with U+FFFD (removal could splice invalid bytes into new code points, so passthrough would break idempotence).