keyenc

README

keyenc

Join several untrusted strings into one key that no field's content can forge. Go and TypeScript, byte-identical.

Programs build composite keys constantly: a cache key from a path and a version, a dedupe token from a user and a stream, a map key from a kind and a host. The usual way is to concatenate with a separator, and that is correct only while no field can contain the separator.

key := kind + ":" + host          // "gitea" + "git.example.com:3000"

When a field can contain it, two different field tuples produce the same key, and every consumer of that key silently merges two things it was built to keep apart. There is no error path: the cache returns the wrong entry, the dedupe token suppresses a distinct event as already seen, the map treats two identities as one. keyenc makes the separator between fields always distinguishable from a separator inside one, so distinct tuples always produce distinct keys.

keyenc.Join("gitea", "git.example.com:3000")   // gitea:git.example.com\:3000
keyenc.Join("a:b", "c") == keyenc.Join("a", "b:c")   // false

Standard library only, zero dependencies, on both sides.

Why not just pick a separator no field can contain

That is the usual answer, and it is a bet on the whole program rather than on the key. Whether a given site is safe depends on the alphabet of every field except the last, which is not visible at the call site and not stable over time. Adding a field, reordering two, or widening one field's source from an enum to free-form upstream text each turns a correct key into a forgeable one, and none of those edits looks like it touches key encoding. \x00 narrows the bet without closing it, and it costs you a key you can put in a URL or read in a log.

Install

• Go: go get github.com/cplieger/keyenc@latest
• TS: npx jsr add @cplieger/keyenc or npm i @cplieger/keyenc

Usage

Building a key

key := keyenc.Join("streams", userID, ratingKey, audioID, subID)

const key = join("streams", userID, ratingKey, audioID, subID);

Reading one back

Some keys are parsed again — a URL path segment, a value read out of a persisted file. Split is Join's exact inverse, so the site needs one grammar rather than an encoder plus a parser that can disagree with it.

parts, err := keyenc.Split(key)
if err != nil {
    return fmt.Errorf("unrecognized key %q: %w", key, err)
}
kind, host := parts[0], parts[1]

Split accepts exactly what Join emits. A key that could not have been produced by Join is refused rather than normalized, so "Split accepted it" is a statement about where the key came from.

Nesting

A key with an inner list nests by composition. Encode the inner sequence, then pass its result as one outer component:

languages := keyenc.Join("en", "fr")
key := keyenc.Join(mediaType, mediaID, languages, videoPath)

The inner result is escaped as it becomes an outer component, so its separators cannot be read as outer ones at any depth. A second reserved character buys the same thing for exactly two levels and fails at three.

Oversized keys

A caller that folds unbounded upstream data into a key can be made to allocate without limit. Above MaxComponentBytes (8 KiB of total raw input) Join returns a fixed-size SHA-256 identity instead. Components are streamed into the hash under a length prefix, so element boundaries survive hashing exactly as escaping preserves them below the bound.

Hashing is one-way. At a site that parses its keys back, check first:

if keyenc.IsHashed(key) {
    return errNotRecoverable
}

Adopting this in an existing key

A component containing neither reserved character is emitted verbatim, so a key whose fields are already separator-free encodes byte-identically to its naive concatenation:

keyenc.Join("streams", "u-42", "1234", "3", "5")   // streams:u-42:1234:3:5

The bytes change only where the naive form was already ambiguous. So adopting keyenc at a key that is currently safe costs nothing: no re-key, no cache invalidation, and no migration for a persisted key. At a key that is currently forgeable the bytes do change for the inputs that were colliding, which is the point.

API

Symbol	Purpose
Join(parts ...string) string	Encode components as one key. Hashes above MaxComponentBytes.
Split(key string) ([]string, error)	Recover the exact components. Join's inverse.
IsHashed(key string) bool	Whether the key is a hashed identity, whose components are gone.
MaxComponentBytes	Total raw size, in bytes, above which Join hashes.
Separator, Escape	The two reserved characters, : and \.
ErrHashed, ErrMalformed	Why Split refused.

The TypeScript half exports the same surface as join, split, isHashed, MAX_COMPONENT_BYTES, SEPARATOR, ESCAPE, HashedKeyError and MalformedKeyError.

Cross-language parity

The two implementations produce identical keys for identical input, which matters wherever a key built in one language is looked up against keys built in the other. The parity is pinned by a shared golden fixture rather than by inspection: conformance/keys.json is generated from the Go implementation and asserted by the TypeScript test suite.

Two details carry that parity and are easy to get wrong in a reimplementation. The size bound is measured in UTF-8 bytes, not UTF-16 code units, so a key with multibyte text crosses the threshold at the same point in both languages. And the digest hashes UTF-8 bytes with an 8-byte big-endian length prefix per component. The fixture contains a case that fails specifically when either is wrong.

To change the grammar, regenerate and land both halves in one commit:

UPDATE_GOLDEN=1 go test ./... -run TestConformanceFixture
cd web && npm test

Unsupported by Design

• No configurable separator. Two implementations that must agree byte-for-byte cannot each carry a policy knob, and a key encoded under one separator is unreadable under another. Nesting covers what a second separator would have.
• No key derivation, no MAC, no authentication. The digest bounds a key's size. It is not a commitment, not constant-time, and not a substitute for HMAC. A key from this package tells you two field tuples differ; it tells you nothing about who produced them.
• No canonicalization of components. Join encodes what it is given. Case folding, Unicode normalization and trimming are the caller's decisions, and applying them here would make two implementations of "normalize" the thing that has to stay in sync.
• Arbitrary bytes are Go-only. A Go string is a byte sequence and a JavaScript string is a UTF-16 sequence, so the two halves agree for all valid UTF-8 and the conformance fixture covers only that. Go's own fuzz targets cover invalid UTF-8 for the Go implementation.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

GPL-3.0-or-later. See LICENSE.

Documentation

Overview

Package keyenc joins several untrusted strings into one key that no component's content can forge.

A composite key is normally built by concatenating fields with a separator:

key := userID + ":" + ratingKey + ":" + streamID

That is correct only while no field can contain the separator. When one can, two distinct field tuples produce one key, and every consumer of that key silently merges two things it was built to keep apart. The failure has no error path: a cache returns the wrong entry, a dedupe token suppresses a distinct event as already seen, a singleflight gate hands one caller another caller's result, a map treats two identities as one.

Whether a given site is exposed depends on the alphabet of every field except the last, which is a property of the whole program rather than of the key. Adding a field, reordering two, or widening one field's source from an enum to free-form upstream text can each turn a correct key into a forgeable one, and none of those edits looks like it touches key encoding.

The grammar

Join escapes each component before joining, so the separator that appears between components is always distinguishable from a separator inside one:

Join("a:b", "c") == `a\:b:c`
Join("a", "b:c") == `a:b\:c`
Join("a", "b", "c") == "a:b:c"

Two characters are reserved: ':' separates components and '\' escapes. Both are escaped element-wise, '\' first, which keeps the mapping injective. A component containing neither is emitted verbatim, so a key whose fields are already separator-free encodes byte-identically to its naive concatenation. That property is what lets an existing key move to this package without changing its bytes: the encoding changes only where the naive form was already ambiguous.

Split is the exact inverse below the size bound, so a key that must be parsed back into its fields round-trips instead of needing a second, separately-maintained parser.

Nesting

A key with an inner list nests by composition rather than by a second separator. Encode the inner sequence, then pass its result as one outer component:

inner := keyenc.Join(languages...)
key := keyenc.Join(mediaType, mediaID, inner, videoPath)

The inner result is escaped as it becomes an outer component, so its separators cannot be read as outer ones at any depth. A second reserved character would buy the same thing for exactly two levels and fail at three.

Bounding

Escaping grows a key in proportion to its input, so a caller that folds unbounded upstream data into a key can be made to allocate without limit (CWE-400). When the components' total raw size exceeds MaxComponentBytes, Join returns a fixed-size SHA-256 identity instead. Components are streamed into the hash under a length prefix, so element boundaries survive hashing exactly as escaping preserves them below the bound: ["a:b"] and ["a", "b"] hash differently.

The hashed identity carries a "sha256:" prefix and the raw encodings exclude that prefix, so the two output domains stay disjoint and no in-bound key can spell an oversized one's identity. Hashing is one-way, so Split refuses a hashed key rather than guessing; call IsHashed first at a site that must parse its keys back.

Standard library only, zero dependencies. A TypeScript twin publishes as @cplieger/keyenc and produces byte-identical keys, pinned by the conformance fixture in conformance/.

Index

• Constants
• Variables
• func IsHashed(key string) bool
• func Join(parts ...string) string
• func Split(key string) ([]string, error)

Examples

• IsHashed
• Join
• Join (Nesting)
• Join (ShiftedBoundary)
• Split
• Split (Malformed)

Constants

const Escape = '\\'

Escape is the character Join prefixes to a literal Separator or Escape inside a component.

const MaxComponentBytes = 8 << 10

MaxComponentBytes is the total raw size of a component set above which Join returns a fixed-size hashed identity instead of an escaped join.

The threshold is measured on the RAW components, not on the escaped output, so a set's representation never depends on how many separators escaping had to double. A delimiter-heavy but honest set whose escaped form is twice this bound still encodes as an escaped join; only genuinely oversized input crosses over. That keeps the shape of an honest key independent of its content, which matters wherever a key is persisted or compared across runs.

const Separator = ':'

Separator is the character Join places between components and Split reads as a component boundary. It is exported so a caller can assert its own field alphabet against the grammar, not so it can be changed.

Variables

var (
	// ErrHashed reports that a key is a hashed identity and therefore carries
	// no recoverable components. Hashing is one-way: the components are gone,
	// not merely encoded. A site that both builds oversized keys and parses
	// them back has to keep the fields it needs somewhere else.
	ErrHashed = errors.New("keyenc: key is a hashed identity and cannot be split")

	// ErrMalformed reports that a key was not produced by [Join]: it ends in a
	// dangling escape, or an escape precedes a character that never needs
	// escaping. Split's accepted language is exactly Join's image, so a
	// tampered or hand-built key is refused rather than silently normalized -
	// which matters wherever a key arrives from a persisted file, a URL
	// segment or client storage.
	ErrMalformed = errors.New("keyenc: key was not produced by Join")
)

Functions

func IsHashed

func IsHashed(key string) bool

IsHashed reports whether key is a hashed identity, i.e. whether Join reduced an oversized component set rather than encoding it. Call it before Split at a site that must parse its keys back; a key is otherwise indistinguishable from a raw one only to a caller that does not look.

Example

Above MaxComponentBytes the key becomes a fixed-size identity, so a caller folding unbounded upstream data into a key cannot be made to allocate without limit. Hashing is one-way, so a site that parses its keys back checks first.

package main

import (
	"errors"
	"fmt"
	"strings"

	"github.com/cplieger/keyenc"
)

func main() {
	oversized := keyenc.Join(strings.Repeat("x", keyenc.MaxComponentBytes+1))
	fmt.Println(keyenc.IsHashed(oversized), len(oversized))

	_, err := keyenc.Split(oversized)
	fmt.Println(errors.Is(err, keyenc.ErrHashed))
}

Output:
true 71
true

func Join

func Join(parts ...string) string

Join encodes parts as one key such that no part's content can forge a different split: each part is escaped before the separators are inserted, so Join("a:b", "c") and Join("a", "b:c") differ, where a naive concatenation makes them identical. A part containing neither reserved character is emitted verbatim, so a key built from separator-free fields is byte-identical to its naive concatenation.

Above MaxComponentBytes of total raw input it returns the set's hashed identity instead (see MaxComponentBytes and IsHashed). An in-bound set whose escaped join would itself begin with the hashed-identity prefix is also routed through the hash, which is what keeps the raw and hashed output domains disjoint and therefore keeps Join injective across the size boundary.

Two degenerate cases are worth naming because they are the only places the output is not simply an escaped join. No parts at all encodes as the empty string. A single empty part is routed through the hash, because escaping preserves element boundaries only once the join emits a byte: without the special case, Join() and Join("") would share the empty encoding and alias two distinct component sets.

Example

A key whose fields carry no separator encodes exactly as its naive concatenation, which is what lets an existing key adopt this package without changing its bytes or invalidating persisted state.

package main

import (
	"fmt"

	"github.com/cplieger/keyenc"
)

func main() {
	fmt.Println(keyenc.Join("streams", "u-42", "1234", "3", "5"))
}

Output:
streams:u-42:1234:3:5

Example (Nesting)

An inner list nests by composition: encode it, then pass the result as one outer component. The inner separators are escaped as the value becomes an outer component, so they cannot be read as outer boundaries at any depth.

package main

import (
	"fmt"

	"github.com/cplieger/keyenc"
)

func main() {
	languages := keyenc.Join("en", "fr")
	key := keyenc.Join("episode", "tvdb-1-s01e02", languages)
	fmt.Println(key)

	outer, _ := keyenc.Split(key)
	inner, _ := keyenc.Split(outer[2])
	fmt.Printf("%q\n", inner)
}

Output:
episode:tvdb-1-s01e02:en\:fr
["en" "fr"]

Example (ShiftedBoundary)

The escaping is what a naive concatenation lacks: moving the separator into a field changes the key, so two distinct field tuples can never collide.

package main

import (
	"fmt"

	"github.com/cplieger/keyenc"
)

func main() {
	fmt.Println(keyenc.Join("a:b", "c"))
	fmt.Println(keyenc.Join("a", "b:c"))
	fmt.Println(keyenc.Join("a:b", "c") == keyenc.Join("a", "b:c"))
}

Output:
a\:b:c
a:b\:c
false

func Split

func Split(key string) ([]string, error)

Split recovers the exact components Join encoded. It is Join's inverse for every in-bound key: Split(Join(parts...)) equals parts, and Join(Split(key)) equals key.

Its accepted language is exactly Join's image. A key carrying an escape before anything other than a reserved character, or a trailing dangling escape, is refused with ErrMalformed rather than normalized, so "Split accepted it" is a statement about the key's provenance. A hashed identity is refused with ErrHashed, whose components are not recoverable. The empty key splits to no components, mirroring Join's zero-part case.

Split exists so a site that parses its keys back has one grammar rather than an encoder plus a separately-maintained parser that can disagree with it.

Example

A host carrying a port is the everyday case where content meets the separator. Split recovers it exactly, so a site that parses its keys back needs no second parser.

package main

import (
	"fmt"

	"github.com/cplieger/keyenc"
)

func main() {
	key := keyenc.Join("gitea", "git.example.com:3000")
	fmt.Println(key)

	parts, err := keyenc.Split(key)
	if err != nil {
		fmt.Println("split:", err)
		return
	}
	fmt.Printf("%q\n", parts)
}

Output:
gitea:git.example.com\:3000
["gitea" "git.example.com:3000"]

Example (Malformed)

Split accepts exactly what Join emits, so a key arriving from a persisted file, a URL segment or client storage is refused rather than normalized when it could not have been produced by this package.

package main

import (
	"errors"
	"fmt"

	"github.com/cplieger/keyenc"
)

func main() {
	_, err := keyenc.Split(`a\b`)
	fmt.Println(errors.Is(err, keyenc.ErrMalformed))
}

Output:
true