norm

README

norm

norm normalizes user-entered strings. It is the writing companion to goloop/is: where is reads (is this an email?), norm cleans a value toward the canonical form of that type and validates the result.

value, ok := norm.Email("  Example @ Gmail.com ")
// value == "Example@gmail.com", ok == true

Standard library only, plus goloop/is for validation.

Install

go get github.com/goloop/norm

Typed normalizers

Each mirrors an is predicate and returns (cleaned, valid). The cleaned value is returned even when invalid, so a UI can suggest "did you mean X?".

Function	Normalization
Email	trim, lowercase domain (local part preserved), repair internal spaces
EmailFold	Email, then lowercase the whole address (case-insensitive identity key)
URL	trim, lowercase scheme and host only (no scheme is invented)
UUID	trim, strip {} and urn:uuid:, lowercase
MAC	regroup to lowercase colon pairs (accepts :, -, ., none)
IBAN	remove spaces, uppercase, strict checksum
BankCard	remove spaces and dashes, Luhn (and optional kinds)
E164, Phone	remove spaces, parentheses, dashes, dots (no country code invented)
IMSI	remove spaces and dashes
IP, IPv4, IPv6	canonical form (compressed, lowercase IPv6)
Domain, Hostname	trim, lowercase
Base64, Base64URL	remove whitespace (unwrap)
JWT	trim, scrub
HexColor	trim, lowercase (no # invented)
RGBColor	lowercase the rgb keyword
Bin	remove whitespace, lowercase 0b prefix
Decimal	remove whitespace (ungroup digits)
Float	trim and scrub only (grammar is not repaired)
Hex, MD5, SHA1, SHA256, SHA512	trim, lowercase

Every value is first scrubbed of control and invisible characters (a zero-width space in an email is removed before validation). A value is never reported valid unless is agrees, and norm never strips arbitrary punctuation or invents data.

Character toolkit

For input that is not a fixed external format, use the character-group tools. Class is a bitmask (combine with |):

norm.Trim(s, norm.Spaces|norm.Controls) // remove groups from both ends
norm.Remove(s, norm.Invisible)          // remove a group everywhere
norm.Keep(s, norm.Letters|norm.Digits)  // keep only these groups
norm.CollapseSpace(s)                   // runs of whitespace -> one space
norm.Clean(s)                           // strip invisible/control, collapse space, trim
norm.Lower(s), norm.Upper(s)            // Unicode case

Clean is the one-call cleanup for a free-text field (a name, title or comment): it removes invisible and non-space control characters, collapses every whitespace run to a single space and trims the ends, so pasted text cannot carry zero-width bytes, hidden line breaks or ragged spacing into storage.

Groups: Spaces, Controls, Invisible (zero-width / format characters, BOM, soft hyphen, variation selectors), Punct, Symbols, Letters, Digits (decimal), Numbers (all Unicode numbers).

For the character-class predicates of is, the honest lossy counterparts are named *Only and return a plain string (not (string, bool)), so the name signals filtering rather than canonicalization:

norm.AlphaOnly("a1-b2")   // "ab"
norm.AlnumOnly("a1-b2!")  // "a1b2"
norm.DigitsOnly("+1 415") // "1415"

There is deliberately no norm.Alpha/norm.Digit (a name implying a canonical form for a lossy filter), and no mirror for grammar or policy predicates (Numeric, Title, Nickname, VariableName, SelectorName), where filtering would change meaning or identity.

Documentation

• English reference: DOC.md
• Ukrainian reference: DOC.UK.md

License

MIT - see LICENSE.

Release v1.0.2

Normalize IP addresses with a single parse.

Documentation

Overview

Package norm normalizes user-entered strings, the writing companion to the predicate module github.com/goloop/is: where is reads (is this an email?), norm cleans a value toward the canonical form of that type and validates the result.

Typed normalizers

Each typed normalizer mirrors an is predicate and returns the cleaned value together with whether it is valid:

value, ok := norm.Email("  Example @ Gmail.com ")
// value == "Example@gmail.com", ok == true

The returned string is the best-effort normalized value even when ok is false, so a UI can offer a "did you mean" suggestion. Covered types: Email, URL, UUID, MAC, IBAN, BankCard, E164 and the hex/hash family (Hex, MD5, SHA1, SHA256, SHA512).

Normalization is conservative and type-specific. Every value is first scrubbed of control and invisible characters (never valid inside these formats); safe canonical steps (trim, lowercasing a case-insensitive domain or scheme) are always applied; a more aggressive repair (removing internal whitespace) is only attempted when the value is still invalid. Every result is confirmed with an is predicate, so a value is never reported valid unless is agrees. norm never strips arbitrary punctuation and never invents data (for example a URL scheme or a phone country code).

Character toolkit

For the cases that are not a fixed external format, norm offers character group tools driven by a Class bitmask:

norm.Trim(s, norm.Spaces|norm.Controls) // remove groups from both ends
norm.Remove(s, norm.Invisible)          // remove a group everywhere
norm.Keep(s, norm.Letters|norm.Digits)  // keep only these groups
norm.CollapseSpace(s)                   // runs of whitespace -> one space
norm.Lower(s), norm.Upper(s)            // Unicode case

These deliberately replace mirrors like a hypothetical norm.Alpha or norm.Digit, which would imply a canonical form but really perform lossy filtering; Keep and Remove make that explicit.

See DOC.md (English) and DOC.UK.md (Ukrainian) for the full reference.

Index

• func AlnumOnly(s string) string
• func AlphaOnly(s string) string
• func BankCard(s string, kinds ...is.CardKind) (string, bool)
• func Base64(s string) (string, bool)
• func Base64URL(s string) (string, bool)
• func Bin(s string) (string, bool)
• func Clean(s string) string
• func CollapseSpace(s string) string
• func Decimal(s string) (string, bool)
• func DigitsOnly(s string) string
• func Domain(s string) (string, bool)
• func E164(s string) (string, bool)
• func Email(s string) (string, bool)
• func EmailFold(s string) (string, bool)
• func Float(s string) (string, bool)
• func Hex(s string) (string, bool)
• func HexColor(s string) (string, bool)
• func Hostname(s string) (string, bool)
• func IBAN(s string) (string, bool)
• func IMSI(s string) (string, bool)
• func IP(s string) (string, bool)
• func IPv4(s string) (string, bool)
• func IPv6(s string) (string, bool)
• func JWT(s string) (string, bool)
• func Keep(s string, class Class) string
• func Lower(s string) string
• func MAC(s string) (string, bool)
• func MD5(s string) (string, bool)
• func Phone(s string) (string, bool)
• func RGBColor(s string) (string, bool)
• func Remove(s string, class Class) string
• func SHA1(s string) (string, bool)
• func SHA256(s string) (string, bool)
• func SHA512(s string) (string, bool)
• func Trim(s string, class Class) string
• func TrimLeft(s string, class Class) string
• func TrimRight(s string, class Class) string
• func URL(s string) (string, bool)
• func UUID(s string) (string, bool)
• func Upper(s string) string
• type Class

Examples

• Email
• IBAN
• Keep
• MAC
• Remove

Functions

func AlnumOnly

func AlnumOnly(s string) string

AlnumOnly returns s with everything but Unicode letters and numbers removed. It is the filtering counterpart of is.Alnum, which accepts any Unicode letter or number (so, like is.Alnum, it keeps Roman numerals and other numerics, not only decimal digits).

func AlphaOnly

func AlphaOnly(s string) string

AlphaOnly returns s with everything but Unicode letters removed. It is the filtering counterpart of is.Alpha.

func BankCard

func BankCard(s string, kinds ...is.CardKind) (string, bool)

BankCard normalizes a payment card number and reports whether the result is valid. It removes spaces and dashes (the usual grouping separators), then validates the number with the Luhn check and, when kinds are given, against those card kinds. No other characters are removed.

Security: the returned value is a full card number (PAN). Do not log or store it unnecessarily, and remember that a Luhn-valid number is not proof of a real or intended card.

func Base64

func Base64(s string) (string, bool)

Base64 normalizes a standard base64 string by removing whitespace (base64 is often line-wrapped) and reports whether the result is valid. Padding and the base64 alphabet are preserved.

func Base64URL

func Base64URL(s string) (string, bool)

Base64URL normalizes a URL-safe base64 string by removing whitespace and reports whether the result is valid.

func Bin

func Bin(s string) (string, bool)

Bin normalizes a binary string by removing whitespace (binary is often space-grouped) and lowercasing, so an uppercase "0B" prefix becomes "0b", and reports whether the result is valid.

func Clean

func Clean(s string) string

Clean returns s as safe, single-line display text: it removes invisible and non-space control characters, then collapses every run of whitespace (spaces, tabs and newlines) to a single space and trims the ends. It is the common one-call cleanup for a free-text form field (a name, title or comment) before it is stored or shown, so a pasted value cannot smuggle zero-width or control bytes, ragged whitespace or hidden line breaks.

func CollapseSpace

func CollapseSpace(s string) string

CollapseSpace trims surrounding whitespace and replaces every internal run of whitespace with a single ASCII space.

func Decimal

func Decimal(s string) (string, bool)

Decimal normalizes a run of decimal digits by removing whitespace (digits are sometimes space-grouped) and reports whether the result is valid.

func DigitsOnly

func DigitsOnly(s string) string

DigitsOnly returns s with everything but decimal digits removed. It is the filtering counterpart of is.Digit.

func Domain

func Domain(s string) (string, bool)

Domain normalizes a domain name (scrub, trim and lowercase, since domains are case-insensitive) and reports whether the result is valid. It performs no IDN/punycode conversion.

func E164

func E164(s string) (string, bool)

E164 normalizes a phone number toward the E.164 format and reports whether the result is valid. It removes spaces, dashes, parentheses and dots. It is non-magical: it never guesses a country code, so a national number without a leading "+" stays invalid.

func Email

func Email(s string) (string, bool)

Email normalizes an email address and reports whether the result is valid.

It trims surrounding whitespace and lowercases the domain (domains are case-insensitive; the local part is preserved, as it is technically case-sensitive). If the address is still invalid, it removes internal whitespace as a best-effort repair. A valid input, including a quoted local part that legitimately contains spaces, is preserved because the repair runs only when the milder result does not validate.

It does not perform IDN/punycode conversion: a Unicode domain is accepted only as far as is.Email accepts it.

Example

package main

import (
	"fmt"

	"github.com/goloop/norm"
)

func main() {
	value, ok := norm.Email("  Example @ Gmail.com ")
	fmt.Println(value, ok)
}

Output:
Example@gmail.com true

func EmailFold

func EmailFold(s string) (string, bool)

EmailFold is Email with the whole address lower-cased, for using an email as a case-insensitive identity: a login key, a uniqueness constraint or a deduplication key. Email preserves the local part's case because it is technically case-sensitive; EmailFold folds it too, matching how identity systems and mail providers treat addresses in practice, so "Boss@Acme.io" and "boss@acme.io" compare and store as the same key.

Use Email when exact deliverability matters and EmailFold when the address is an identifier you look up or compare.

func Float

func Float(s string) (string, bool)

Float normalizes a floating-point literal and reports whether the result is valid. It only scrubs and trims: the sign and decimal point are grammar, so no characters are removed from the middle (which could silently change the value).

func Hex

func Hex(s string) (string, bool)

Hex normalizes a hexadecimal string (trim and lowercase) and reports whether the result is valid hex.

func HexColor

func HexColor(s string) (string, bool)

HexColor normalizes a hexadecimal color (scrub, trim and lowercase) and reports whether the result is valid. A missing "#" is not added.

func Hostname

func Hostname(s string) (string, bool)

Hostname normalizes a hostname (scrub, trim and lowercase) and reports whether the result is valid.

func IBAN

func IBAN(s string) (string, bool)

IBAN normalizes an International Bank Account Number and reports whether the result is valid. It removes spaces and uppercases the value (an IBAN is defined over uppercase letters and digits), then validates it strictly, including the checksum. No other characters are removed.

Example

package main

import (
	"fmt"

	"github.com/goloop/norm"
)

func main() {
	value, ok := norm.IBAN("de89 3704 0044 0532 0130 00")
	fmt.Println(value, ok)
}

Output:
DE89370400440532013000 true

func IMSI

func IMSI(s string) (string, bool)

IMSI normalizes an International Mobile Subscriber Identity (15 digits) by removing spaces and dashes, and reports whether the result is valid.

func IP

func IP(s string) (string, bool)

IP normalizes an IP address to its canonical form (lowercase, compressed IPv6) and reports whether the result is valid.

func IPv4

func IPv4(s string) (string, bool)

IPv4 normalizes an IPv4 address and reports whether the result is valid.

func IPv6

func IPv6(s string) (string, bool)

IPv6 normalizes an IPv6 address to its canonical, compressed lowercase form and reports whether the result is valid.

func JWT

func JWT(s string) (string, bool)

JWT normalizes a JSON Web Token (scrub and trim; a JWT contains no whitespace) and reports whether the result is well-formed.

func Keep

func Keep(s string, class Class) string

Keep deletes every character NOT in the given class from s. It is the inverse of Remove and is the honest way to filter to, say, letters: Keep(s, Letters).

Example

package main

import (
	"fmt"

	"github.com/goloop/norm"
)

func main() {
	fmt.Println(norm.Keep("a1-b2 c3!", norm.Letters|norm.Digits))
}

Output:
a1b2c3

func Lower

func Lower(s string) string

Lower returns s with all Unicode letters mapped to lower case. It is the normalizing counterpart of is.Lower and always succeeds, so it returns only a string.

func MAC

func MAC(s string) (string, bool)

MAC normalizes a MAC address to lowercase, colon-separated form and reports whether the result is valid. It accepts the common separators (":", "-", ".") and the separatorless form, and regroups an EUI-48 (12 hex digits) or EUI-64 (16 hex digits) into colon-separated pairs. No other punctuation is stripped.

Example

package main

import (
	"fmt"

	"github.com/goloop/norm"
)

func main() {
	value, ok := norm.MAC("0011.2233.4455")
	fmt.Println(value, ok)
}

Output:
00:11:22:33:44:55 true

func MD5

func MD5(s string) (string, bool)

MD5 normalizes an MD5 digest (trim and lowercase) and reports validity.

func Phone

func Phone(s string) (string, bool)

Phone normalizes a phone number by removing spaces, parentheses, dashes and dots, and reports whether the result is valid. Like E164 it is non-magical: a number without a leading "+" and country code stays invalid.

func RGBColor

func RGBColor(s string) (string, bool)

RGBColor normalizes a CSS rgb(...) color by lowercasing the "rgb" keyword (the numbers and separators are unaffected) and reports whether the result is valid.

func Remove

func Remove(s string, class Class) string

Remove deletes every character of the given class from s. A zero class leaves s unchanged.

Example

package main

import (
	"fmt"

	"github.com/goloop/norm"
)

func main() {
	// Strip zero-width and control characters that break validation.
	clean := norm.Remove("\u200bhi\ufeff", norm.Invisible)
	fmt.Printf("%q\n", clean)
}

Output:
"hi"

func SHA1

func SHA1(s string) (string, bool)

SHA1 normalizes a SHA-1 digest (trim and lowercase) and reports validity.

func SHA256

func SHA256(s string) (string, bool)

SHA256 normalizes a SHA-256 digest (trim and lowercase) and reports validity.

func SHA512

func SHA512(s string) (string, bool)

SHA512 normalizes a SHA-512 digest (trim and lowercase) and reports validity.

func Trim

func Trim(s string, class Class) string

Trim removes characters of the given class from both ends of s. A zero class leaves s unchanged.

func TrimLeft

func TrimLeft(s string, class Class) string

TrimLeft removes characters of the given class from the start of s.

func TrimRight

func TrimRight(s string, class Class) string

TrimRight removes characters of the given class from the end of s.

func URL

func URL(s string) (string, bool)

URL normalizes a URL and reports whether the result is valid.

It trims surrounding whitespace and lowercases the scheme and host only; the userinfo, path, query and fragment are left byte-for-byte untouched (no re-encoding). It never invents a missing scheme, so a schemeless input such as "example.com" stays invalid (and is returned as the best-effort value with false).

func UUID

func UUID(s string) (string, bool)

UUID normalizes a UUID and reports whether the result is valid. It trims surrounding whitespace, strips a surrounding pair of braces and a leading "urn:uuid:" prefix, and lowercases the value. It does not remove any internal characters other than the recognized wrappers.

func Upper

func Upper(s string) string

Upper returns s with all Unicode letters mapped to upper case. It is the normalizing counterpart of is.Upper and always succeeds, so it returns only a string.

Types

type Class

type Class uint

Class is a bitmask selecting one or more groups of characters. Combine groups with the bitwise OR operator, for example Spaces|Controls.

const (
	// Spaces are Unicode White_Space characters, including the regular space,
	// tab, newline and no-break space (U+00A0).
	Spaces Class = 1 << iota

	// Controls are Unicode control characters (category Cc).
	Controls

	// Invisible are zero-width and formatting characters that render to nothing
	// yet can corrupt input or hide content: Unicode format characters
	// (category Cf, such as the zero-width space, byte-order mark, soft hyphen
	// and word joiner), variation selectors and the combining grapheme joiner.
	Invisible

	// Punct are Unicode punctuation characters (category P).
	Punct

	// Symbols are Unicode symbol characters (category S), such as math and
	// currency symbols.
	Symbols

	// Letters are Unicode letters (category L).
	Letters

	// Digits are Unicode decimal digits (category Nd), matching is.Digit.
	Digits

	// Numbers are all Unicode numbers (category N: decimal digits, letter
	// numbers such as Roman numerals, and other numerics), matching the numeric
	// half of is.Alnum.
	Numbers
)