textdata

README

textdata

Comment- and byte-preserving parsers and editors for human-edited config files, in Go. Every format keeps the file a person wrote: parse it, read it, change one value, save it, and every comment, blank line, key order, quoting choice and line ending you did not touch is written back exactly as it was. No third-party parser libraries: every format is parsed by code in this repository.

github.com/ubgo/textdata is a multi-module repository: a small shared base (this module), one module per format, value and provider plugins, and the CLIs. Formats today: dotenv (moved here from github.com/ubgo/dotenv) and JSONC / JSON. YAML and TOML come only when a tool needs them.

Status: nothing is released yet. No module is tagged; until the first tags, build from a clone, where go.work wires the modules together (unreleased siblings are required as v0.0.0-00010101000000-000000000000 and resolved through the workspace).

Modules

Module	Path	What
github.com/ubgo/textdata	.	the shared base: plugin, expand, atomicfile, pos, path, texttest; stdlib only
…/dotenv	dotenv/	the .env library: lossless parse, edit, Docker Compose interpolation, plugins
…/dotenv/cli	dotenv/cli/	the dotenvctl binary and its command package dotenvcmd, the operations API envkit, .env family discovery
…/jsonc	jsonc/	JSONC: lossless parse, read, edit, Disable/Restore, save, expansion; plain JSON is jsonc.Strict()
…/jsonc/cli	jsonc/cli/	the jsonc command package, mounted as textdatactl jsonc
…/clikit	clikit/	shared CLI plumbing: the {ok,data|error} output envelope and error codes, HTML reports, dry-run diff, selection grammar, provider-plugin seam
…/cmd/textdatactl	cmd/textdatactl/	the umbrella CLI: textdatactl dotenv …, textdatactl jsonc …
…/dotenv/cli/providers/github	dotenv/cli/providers/github/	provider plugin: push a selection of values to GitHub Actions secrets through gh
…/dotenv/cli/providers/vercel	dotenv/cli/providers/vercel/	provider plugin: push a selection of values to Vercel environment variables through vercel
…/plugins/gopass	plugins/gopass/	value plugin: ${gopass:path} references from a gopass store
…/plugins/file	plugins/file/	value plugin: ${file:./key.pem} references read from a file
…/plugins/extends	plugins/extends/	value plugin: loads $extends base files from somewhere other than the local disk
…/plugins/vault	plugins/vault/	value plugin: references resolved from HashiCorp Vault (KV v1 and v2)
…/plugins/aws	plugins/aws/	value plugin: references resolved from AWS Secrets Manager and SSM Parameter Store

Every module path starts with github.com/ubgo/textdata. Value plugins (plugins/) are written once against the base plugin package, require only the base, and work in every format; their cross-format tests live in the test-only integration/ module, which is never tagged. Provider plugins (dotenv/cli/providers/) push a flat key/value selection to a hosting service; each is its own module needing only clikit and cobra, so another CLI can mount one without pulling dotenv. No module requires a sibling it only uses in tests.

Guarantees every format keeps

• Parse(x).Render() == x, byte for byte, for every input, valid or not.
• An edit changes only the bytes it touches. Comments, blank lines, key order, indentation, quoting and line endings elsewhere survive.
• Never touches the process environment. Never runs a command unless the caller turns that on.
• Errors carry file:line:col and a plain message (config.jsonc:2:3: object keys must be double-quoted (unquoted keys are JSON5)).
• The base and the format modules use the Go standard library only.

Each guarantee is pinned by fuzz and property tests (through texttest), never by prose.

Dependency rules

• Formats and value plugins import the base; the base never imports a format.
• Formats never import each other or a plugin. A cross-format plugin is written once against plugin and works in every format.
• clikit imports no format. A format's CLI module imports its format, clikit and the provider plugins it mounts; provider plugins import only clikit and never parse a file.
• Format-specific rules stay in the format: dotenv's quoting (single quotes are never expanded) and its top-to-bottom self-reference rule live in dotenv, jsonc's comment attachment in jsonc.
• The base is public API with a backward-compatibility promise. Go's minimal version selection can pair jsonc vX with a newer base pulled in by another format, so base changes are additive only.

The base packages

go get github.com/ubgo/textdata

The root package holds no code; the base lives in its sub-packages, each stdlib only.

plugin

The plugin interfaces every format accepts. A plugin has a Name() and implements only the abilities it needs; a format detects them once, at install, by type assertion, and reports what it found (Info{Name, Capabilities}, rendered as vault: Lookuper, Resolver), so a method with a slightly wrong signature shows up as a missing capability instead of silently never running.

Interface	Method	Does
Plugin	Name() string	required, so an error names the plugin
Lookuper	Lookup(name) (value, ok, err)	a value for ${NAME} the file does not define; first answer wins; an error aborts
Resolver	Schemes() []string, Resolve(scheme, ref) (string, error)	${scheme:ref} references (${gopass:personal/db}); a scheme nobody owns reads as before
CommandRunner	RunCommand(cmd) (string, error)	runs $(cmd); a replacement, last installed wins; installing one turns substitution on
ValueTransformer	TransformValue(path, value) (string, error)	changes a value on read, after expansion; chained; path is the key (dotenv) or canonical path (jsonc)
ExpandObserver	ObserveExpand(path, name, resolved)	sees every reference resolution (dead-key detection)
SaveGuard	GuardSave(doc Document) error	may veto a save, never rewrite it
Validator	Validate(doc Document) error	checks a document; runs only when asked
Includer	Include(from, ref) ([]byte, string, error)	loads $extends targets from somewhere other than disk

Document is the read-only view a guard or validator receives (FormatName, Location, ValuePaths, ValueAt, Bytes); every format's document type implements it. Also here: Registry (install order, capability filters, NameFor so an error names the plugin that failed), *Error (dotenv: plugin "vault": lookup DB_PASSWORD: connection refused), and the named errors ErrUnavailable, ErrUnsupportedScheme, ErrDenied, ErrNotFound. A plugin that is down is an error, never a silent "missing": an unreachable secret store must not look like an unset variable.

expand

One expansion engine, string in, string out: $VAR, ${VAR}, the six operators (${VAR-d}, ${VAR:-d}, ${VAR+a}, ${VAR:+a}, ${VAR?e}, ${VAR:?e}), nesting (${A:-${B:-c}}), $$ as a literal dollar, $(cmd) and ${scheme:ref}. It knows no file format: an Engine{Lookup, Resolve, Run, Observe, Required, MaxDepth} takes callbacks, and the format keeps its own rules (which quoting suppresses expansion, which definition a self-reference sees, which error type ${VAR:?e} produces). Run == nil keeps $(cmd) verbatim; a failing command yields "", as a shell does. A Stack of the names being expanded cuts cycles; depth 32 is a backstop. Hand-written because os.Expand cannot express $$ or nesting.

vars := map[string]string{"HOST": "db.internal"}
eng := expand.Engine{
	Lookup: func(name string, _ *expand.Stack) (expand.Var, error) {
		v, ok := vars[name]
		return expand.Var{Value: v, Set: ok, At: expand.AtLookup}, nil
	},
}
out, err := eng.Expand("postgres://${HOST}:${PORT:-5432}/app costs 5$$", expand.NewStack())
fmt.Println(out, err) // postgres://db.internal:5432/app costs 5$ <nil>

atomicfile

Write(path, data, Options{Mode, KeepMode, TempSuffix, NoFollow, Chmod, Rename}) saves a file safely: it resolves symlinks first so the real file is replaced and the link survives (NoFollow replaces the link itself), writes a sibling temp file at 0600, fsyncs it, chmods it to the target mode explicitly (umask filters os.WriteFile's mode), renames it over the target and fsyncs the folder best-effort. KeepMode keeps an existing target's permissions. A crash leaves the old file intact; a failed write removes the temp file.

pos

Pos{Offset, Line, Col} (1-based line, byte column), an Index built once per source to map offsets to positions (CRLF files report the same lines as LF), Error{File, Pos, Msg, Err} rendering file:line:col: msg, and ErrorList (every problem in one source; Sort orders by file then offset, Err() returns nil when empty).

path

Paths for tree formats: Parse("a.\"b.c\"[2]") (dotted, with quotes for names holding ., [, ], " or \), ParsePointer("/a/b~1c/2") (RFC 6901), and ParseAny, which reads a leading / as a JSON Pointer. A Path is a list of key and index segments; String() renders the canonical dotted form and Pointer() the RFC 6901 form. dotenv keys are flat and do not use it.

texttest

The test harness that pins the guarantees: RoundTrip(t, input, render) (byte-exact round trip), Untouched / CheckUntouched (bytes outside an edited window are unchanged), DiffWindow(a, b), Seed(f, extra...) to share fuzz seeds, and Corpus, a format-agnostic set of awkward inputs (empty, BOM, CRLF, lone CR, NUL, invalid UTF-8, very long lines).

CLIs

dotenvctl is dotenv's standalone binary; textdatactl is the umbrella binary with one namespace per format. Mounting is by delegation: textdatactl dotenv <args> hands <args> to the format's command package, so flags, exit codes (0 ok, 1 failed, 2 usage), the --json envelope and exec plugins behave exactly as in the standalone binary.

go install github.com/ubgo/textdata/cmd/textdatactl@latest   # after the first release
textdatactl dotenv get API_URL --expand
textdatactl dotenv set DEBUG=true --dry-run

Releases

Each module is tagged on its own, with a path-prefixed tag for every module below the root: v0.1.1 (base), dotenv/v0.1.1, jsonc/v0.1.1, clikit/v0.1.1, plugins/vault/v0.1.1, dotenv/cli/providers/github/v0.1.1, dotenv/cli/v0.1.1, cmd/textdatactl/v0.1.1, and so on. Modules are tagged in dependency order, base first, then the formats and clikit, then the value plugins and providers, then the format CLIs, then cmd/textdatactl; at release the zero-version requires are replaced with the real versions. Version bumps are PATCH only, and releases are manual: no CI runs on push. No tag exists yet.

Development

task test       # quick: every module
task check      # the full gate
task fuzz:list  # every fuzz target
task fuzz -- ./jsonc FuzzParseRender 30s
task fuzz:all FUZZTIME=30s

Documentation

Overview

Package textdata is the shared base of the textdata formats: comment- and byte-preserving parsers and editors for human-edited config files.

The root package holds no code; the base lives in its sub-packages:

• plugin: plugin interfaces, capability detection, named plugin errors
• expand: the ${VAR} / $(cmd) / ${scheme:ref} expansion engine
• atomicfile: temp file + fsync + rename saves that keep mode and symlinks
• pos: positions and file:line:col errors
• path: a."b.c"[2] paths and JSON Pointer for tree formats
• texttest: round-trip and untouched-bytes property harness

The formats are separate modules (github.com/ubgo/textdata/dotenv, github.com/ubgo/textdata/jsonc) that import this one; the base never imports a format. The base is public API with a backward-compatibility promise: minimal version selection can pair any format version with a newer base, so changes here are additive only.