README
¶
json-transformer
Why
Go's encoding/json gives you no control over the JSON output once your struct is defined. Real-world systems frequently need to change the JSON shape without touching the data model:
- A service receives
camelCaseJSON from a frontend but must forwardsnake_caseto a downstream API. - An API gateway needs to mask sensitive fields (passwords, tokens) before logging.
- A data pipeline reads NDJSON, renames fields to match a new schema, and streams the result to another sink.
The standard workarounds all have significant costs:
| Workaround | Problem |
|---|---|
| Second struct with different tags | Doubles type definitions; breaks with dynamic schemas |
json.Unmarshal → rename → json.Marshal |
Two full encode/decode passes; entire document loaded into memory |
map[string]any manipulation |
Verbose, error-prone, loses type information |
Custom MarshalJSON per type |
Couples serialisation logic to the domain model |
json-transformer solves this in one pass: rename keys and transform values as the JSON flows through, without loading the full document into memory.
Design
One pass, no round-trips. When the input is a Go value (map, struct, slice), the library walks it directly and writes JSON with no intermediate marshal step. When the input is raw JSON ([]byte, string, io.Reader), a token-based decoder drives the output writer in lockstep — the document is never fully decoded into memory.
Zero config, zero cost. A Transformer with no options short-circuits to a direct pass-through. You pay only for what you configure.
Concurrency-safe. All per-call state is local to each invocation. One Transformer can be shared freely across goroutines.
Follows encoding/json conventions. Struct tags (json:"name,omitempty", json:"-", anonymous embedding) are respected. Map key order is non-deterministic, consistent with the standard library. Large numbers are preserved exactly via json.Number with no float64 precision loss.
Installation
go get github.com/longbridge/json-transformer
Quick start
import jsontransform "github.com/longbridge/json-transformer"
t := jsontransform.New(
jsontransform.WithRenameFunc(jsontransform.SnakeCaseRename()),
)
out, err := t.TransformBytes(map[string]any{"userName": "Alice", "userAge": 30})
// out: {"user_name":"Alice","user_age":30}
API
Creating a transformer
t := jsontransform.New(opts ...Option) *Transformer
| Option | Description |
|---|---|
WithRenameFunc(fn RenameFunc) |
Called for every object key. Return nil to keep the original name, or *string with the new name. |
WithValueTransformer(fn func(string) ValueTransformFunc) |
Called once per unique field name. Return a ValueTransformFunc to transform that field's value, or nil to leave it unchanged. |
WithPathRenameFunc(fn PathRenameFunc) / WithPathValueTransformer(fn func(string) ValueTransformFunc) |
Same, but matched by root-anchored dot path instead of bare name. See Path-aware callbacks. |
WithKeepRenamedOriginal() |
Emit renamed fields twice: the original name keeps the input value, the new name gets the transformed one. See Keeping the original field. |
Choosing a method
| Method | Use when |
|---|---|
TransformBytes(src any) ([]byte, error) |
You need the result as []byte. The most common case. |
Transform(src any, dst io.Writer) error |
You already have an io.Writer to write into (HTTP response, file, network connection). Also the right choice when src is an io.Reader. |
TransformStream(r io.Reader, w io.Writer) error |
Your input contains multiple JSON values in sequence (NDJSON). Each value is transformed and written on its own line. Use Transform for single-value input. |
out, err := t.TransformBytes(src) // → []byte
err := t.Transform(src, w) // → io.Writer
err := t.TransformStream(r, w) // NDJSON: multiple values
Input dispatch
Transform and TransformBytes accept any of the following as src:
| Type | Behaviour |
|---|---|
map[string]any |
Fast path — traverses the map directly, no marshal/unmarshal |
struct / *struct |
Fast path — reflection with cached field metadata |
slice / array |
Fast path |
[]byte / string / io.Reader |
Streaming path — parses JSON tokens without loading the full document |
Primitives (int, bool, …) |
Encoded directly with encoding/json |
Built-in rename functions
jsontransform.SnakeCaseRename() // "UserName" → "user_name"
jsontransform.CamelCaseRename() // "user_name" → "userName"
jsontransform.PascalCaseRename() // "user_name" → "UserName"
jsontransform.KebabCaseRename() // "UserName" → "user-name"
jsontransform.MapRename(map[string]string{"id": "ID"}) // exact lookup; returns nil if not found
MapRename returns nil for unknown keys, so it composes naturally with a fallback:
exact := jsontransform.MapRename(map[string]string{"id": "ID"})
fallback := jsontransform.SnakeCaseRename()
t := jsontransform.New(
jsontransform.WithRenameFunc(func(name string) *string {
if s := exact(name); s != nil {
return s
}
return fallback(name)
}),
)
Examples
Rename fields on a struct
type User struct {
UserName string `json:"userName"`
UserAge int `json:"userAge"`
}
t := jsontransform.New(jsontransform.WithRenameFunc(jsontransform.SnakeCaseRename()))
out, _ := t.TransformBytes(User{UserName: "Alice", UserAge: 30})
// {"user_name":"Alice","user_age":30}
Rename a key and mask its value
WithValueTransformer receives the original field name. Return a transform function for fields you want to handle, or nil to leave the value unchanged. Renaming and value transformation are independent and can be combined freely.
t := jsontransform.New(
jsontransform.WithRenameFunc(func(name string) *string {
if name == "pwd" {
s := "password"
return &s
}
return nil
}),
jsontransform.WithValueTransformer(func(name string) jsontransform.ValueTransformFunc {
if name == "pwd" {
return func(v any) any { return "***" }
}
return nil
}),
)
out, _ := t.TransformBytes(map[string]any{"user": "alice", "pwd": "secret"})
// {"user":"alice","password":"***"}
Transform values by pattern
Because WithValueTransformer gives you the field name, you can match by any pattern rather than a fixed string:
t := jsontransform.New(
jsontransform.WithValueTransformer(func(name string) jsontransform.ValueTransformFunc {
if strings.HasSuffix(name, "_at") {
return func(v any) any {
// reformat Unix timestamp → RFC3339
if n, ok := v.(json.Number); ok {
ts, _ := n.Int64()
return time.Unix(ts, 0).UTC().Format(time.RFC3339)
}
return v
}
}
return nil
}),
)
out, _ := t.TransformBytes(map[string]any{"created_at": 1700000000, "name": "Alice"})
// {"created_at":"2023-11-14T22:13:20Z","name":"Alice"}
Stream a large JSON file
t := jsontransform.New(jsontransform.WithRenameFunc(jsontransform.CamelCaseRename()))
f, _ := os.Open("large.json")
defer f.Close()
t.Transform(f, os.Stdout)
Process NDJSON
t := jsontransform.New(jsontransform.WithRenameFunc(jsontransform.SnakeCaseRename()))
r := strings.NewReader(`{"userId":1}` + "\n" + `{"userId":2}`)
t.TransformStream(r, os.Stdout)
// {"user_id":1}
// {"user_id":2}
Path-aware callbacks
WithPathRenameFunc / WithPathValueTransformer match fields by their
root-anchored dot-joined path instead of the bare name. Array levels are
transparent (no segment). A path callback returning nil falls back to the
name-based callback; returning "" omits the field.
tr := jsontransform.New(
jsontransform.WithPathRenameFunc(func(path string) *string {
if path == "data.list" { s := "items"; return &s }
return nil
}),
jsontransform.WithPathValueTransformer(func(path string) jsontransform.ValueTransformFunc {
if path == "data.list.ts" {
return func(v any) any { /* convert */ return v }
}
return nil
}),
)
// paths always refer to the ORIGINAL input structure: renaming data.list
// does not stop data.list.ts from matching.
Keeping the original field
WithKeepRenamedOriginal makes renaming additive: every renamed field is
emitted twice — first under its original name with the input value untouched,
then under the new name with the transform applied. Use it to add a converted
field to a response without breaking consumers that still read the old one.
tr := jsontransform.New(
jsontransform.WithRenameFunc(jsontransform.MapRename(map[string]string{"counter_id": "symbol"})),
jsontransform.WithValueTransformer(func(name string) jsontransform.ValueTransformFunc {
if name == "counter_id" {
return func(v any) any { return toSymbol(v) }
}
return nil
}),
jsontransform.WithKeepRenamedOriginal(),
)
// {"counter_id":"ST/HK/00700"} → {"counter_id":"ST/HK/00700","symbol":"700.HK"}
The retained field is verbatim: nested rename and value rules apply only to the renamed copy, so it keeps exactly the shape the input had.
Three cases are deliberately left alone:
- A field whose name does not change is emitted once — an in-place value transform cannot keep both values under one key. If you need the old value to survive, rename to a new field instead of transforming in place.
- A field omitted by a rename callback (rename to
"") stays omitted. Omission is a deletion, not a rename. - Renaming two sibling fields onto each other's names still yields duplicate keys, exactly as it does without this option. The library never deduplicates output keys.
Notes
- Field name matching (both
WithValueTransformerandRenameFunc) always uses the original field name, before any renaming is applied. The lookup function passed toWithValueTransformeris called at most once per unique field name; results are cached for the lifetime of theTransformer. - Value transformer input when processing JSON text (
[]byte/string/io.Reader): values arrive asstring,json.Number,bool,nil,map[string]any, or[]any. When processing Go values directly, the original Go type is passed. - Value transformer output is encoded as-is. It is not subject to further renaming or transformation.
- Map key order is non-deterministic, consistent with
encoding/json. - Large numbers in JSON text are preserved exactly via
json.Number; there is no float64 precision loss.
Benchmarks
Measured on Windows amd64, Intel Core i9-12900K, Go 1.24.
BenchmarkTransformMap ~487 ns/op 1 alloc/op
BenchmarkTransformStruct ~340 ns/op 2 allocs/op
BenchmarkTransformParallel ~199 ns/op 1 alloc/op (24 goroutines)
BenchmarkTransformBytes ~899 ns/op 13 allocs/op ([]byte input, fastjson path)
BenchmarkTransformNoOp ~36 ns/op 1 alloc/op (pass-through baseline, Writer)
BenchmarkTransformLargeJSON ~975 ms/op ~108 MB/s (100 MB JSON array, []byte input)
The []byte/string/io.Reader path is driven by fastjson's arena parser. The fast path (in-memory Go values) has no token-allocation overhead at all.
Run on your own machine:
go test -bench=. -benchmem ./...
License
MIT — see LICENSE.
Documentation
¶
Overview ¶
Package jsontransform provides streaming JSON transformation with field renaming and value transformation.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Option ¶
type Option func(*Transformer)
Option configures a Transformer.
func WithKeepRenamedOriginal ¶
func WithKeepRenamedOriginal() Option
WithKeepRenamedOriginal makes renaming additive instead of destructive: when a field is renamed, the original field name is emitted alongside the new one, carrying the input value untransformed. Use it to add a transformed field to a payload while keeping the field older consumers still read:
{"counter_id":"ST/HK/00700"} → {"counter_id":"ST/HK/00700","symbol":"700.HK"}
The original comes first and is written verbatim — value transformations and nested rename rules apply only to the renamed copy, so the retained field keeps exactly the shape the input had.
Three cases are deliberately left alone. A field whose name does not change is emitted once: an in-place value transformation cannot keep both values under one key. A field omitted by a rename callback (rename to "") stays omitted — omission is a deletion, not a rename. And renaming two sibling fields onto each other's names still produces duplicate keys, exactly as it does without this option; the library never deduplicates output keys.
func WithPathRenameFunc ¶
func WithPathRenameFunc(fn PathRenameFunc) Option
WithPathRenameFunc sets a path-based rename callback. It takes precedence over WithRenameFunc for fields whose path it matches (returns non-nil).
func WithPathValueTransformer ¶
func WithPathValueTransformer(fn func(path string) ValueTransformFunc) Option
WithPathValueTransformer sets a path-based value-transform callback. It takes precedence over WithValueTransformer for paths it matches.
func WithRenameFunc ¶
func WithRenameFunc(fn RenameFunc) Option
WithRenameFunc sets a function to rename object field names.
func WithValueTransformer ¶
func WithValueTransformer(fn func(fieldName string) ValueTransformFunc) Option
WithValueTransformer sets a function called once per unique field name to determine whether and how to transform its value. Return nil to leave the value unchanged for that field (fast path, no materialisation).
type PathRenameFunc ¶
PathRenameFunc is called for each object field with its root-anchored, dot-joined path (array levels are transparent). Return nil to fall back to the name-based RenameFunc, or a *string with the new leaf name ("" omits the field).
type RenameFunc ¶
RenameFunc is called for each object field name. Return nil to keep the original name, or a *string with the new name to rename it.
func CamelCaseRename ¶
func CamelCaseRename() RenameFunc
CamelCaseRename returns a RenameFunc that converts all field names to lowerCamelCase. Example: "user_name" → "userName"
func KebabCaseRename ¶
func KebabCaseRename() RenameFunc
KebabCaseRename returns a RenameFunc that converts all field names to kebab-case. Example: "UserName" → "user-name"
func MapRename ¶
func MapRename(m map[string]string) RenameFunc
MapRename returns a RenameFunc that renames fields according to the given map. Fields not found in the map are not renamed (returns nil). Can be combined with other RenameFuncs for fallback behavior.
func PascalCaseRename ¶
func PascalCaseRename() RenameFunc
PascalCaseRename returns a RenameFunc that converts all field names to PascalCase. Example: "user_name" → "UserName"
func SnakeCaseRename ¶
func SnakeCaseRename() RenameFunc
SnakeCaseRename returns a RenameFunc that converts all field names to snake_case. Example: "UserName" → "user_name"
type Transformer ¶
type Transformer struct {
// contains filtered or unexported fields
}
Transformer performs JSON transformation with optional field renaming and value transformation. A Transformer is safe for concurrent use after construction.
func New ¶
func New(opts ...Option) *Transformer
New creates a new Transformer with the given options.
Example (MapRenameWithFallback) ¶
ExampleNew_mapRenameWithFallback demonstrates combining MapRename with a fallback.
package main
import (
"fmt"
jsontransform "github.com/longbridge/json-transformer"
)
func main() {
exact := jsontransform.MapRename(map[string]string{"id": "ID"})
fallback := jsontransform.SnakeCaseRename()
t := jsontransform.New(
jsontransform.WithRenameFunc(func(name string) *string {
if s := exact(name); s != nil {
return s
}
return fallback(name)
}),
)
out, _ := t.TransformBytes(`{"id":1,"userName":"Alice"}`)
fmt.Println(string(out))
}
Output: {"ID":1,"user_name":"Alice"}
Example (MaskField) ¶
ExampleNew_maskField demonstrates renaming a field and masking its value.
package main
import (
"fmt"
jsontransform "github.com/longbridge/json-transformer"
)
func main() {
t := jsontransform.New(
jsontransform.WithRenameFunc(func(name string) *string {
if name == "pwd" {
s := "password"
return &s
}
return nil
}),
jsontransform.WithValueTransformer(func(name string) jsontransform.ValueTransformFunc {
if name == "pwd" {
return func(v any) any { return "***" }
}
return nil
}),
)
out, _ := t.TransformBytes(`{"user":"alice","pwd":"secret"}`)
fmt.Println(string(out))
}
Output: {"user":"alice","password":"***"}
Example (PatternMatch) ¶
ExampleNew_patternMatch demonstrates matching fields by suffix to reformat timestamps.
package main
import (
"encoding/json"
"fmt"
"strings"
"time"
jsontransform "github.com/longbridge/json-transformer"
)
func main() {
t := jsontransform.New(
jsontransform.WithValueTransformer(func(name string) jsontransform.ValueTransformFunc {
if strings.HasSuffix(name, "_at") {
return func(v any) any {
if n, ok := v.(json.Number); ok {
ts, _ := n.Int64()
return time.Unix(ts, 0).UTC().Format(time.RFC3339)
}
return v
}
}
return nil
}),
)
out, _ := t.TransformBytes(`{"name":"Alice","created_at":1700000000}`)
fmt.Println(string(out))
}
Output: {"name":"Alice","created_at":"2023-11-14T22:13:20Z"}
Example (RenameSnakeCase) ¶
ExampleNew_renameSnakeCase demonstrates renaming camelCase keys to snake_case.
package main
import (
"fmt"
jsontransform "github.com/longbridge/json-transformer"
)
func main() {
t := jsontransform.New(jsontransform.WithRenameFunc(jsontransform.SnakeCaseRename()))
out, _ := t.TransformBytes(map[string]any{"userName": "Alice", "userAge": 30})
fmt.Println(string(out))
// Note: map key order is non-deterministic; output may vary.
}
Output:
func (*Transformer) Transform ¶
func (t *Transformer) Transform(src any, dst io.Writer) error
Transform transforms src and writes the result to dst.
src dispatch:
- map[string]any → fast path (in-memory traversal, no marshal/unmarshal)
- struct / *struct → fast path (reflection with field-metadata cache)
- slice / array → fast path
- []byte / string / io.Reader → streaming path (token-based)
- other primitive types → encoded directly
func (*Transformer) TransformBytes ¶
func (t *Transformer) TransformBytes(src any) ([]byte, error)
TransformBytes transforms src and returns the result as []byte. It uses pooled buffers for efficiency.
func (*Transformer) TransformStream ¶
TransformStream processes multiple whitespace-separated JSON values from r and writes transformed results to w, one per line (suitable for NDJSON).
Example ¶
ExampleTransformer_TransformStream demonstrates processing NDJSON line by line.
package main
import (
"fmt"
"strings"
jsontransform "github.com/longbridge/json-transformer"
)
func main() {
t := jsontransform.New(jsontransform.WithRenameFunc(jsontransform.SnakeCaseRename()))
r := strings.NewReader(`{"userId":1}` + "\n" + `{"userId":2}`)
var buf strings.Builder
_ = t.TransformStream(r, &buf)
fmt.Println(buf.String())
}
Output: {"user_id":1} {"user_id":2}
type ValueTransformFunc ¶
ValueTransformFunc transforms a decoded Go value (string/float64/bool/nil/map/slice).
Source Files