reproto

README

reproto

A reflection-based binary serialization platform for Go object graphs — data structures where many pointers reference the same object.

Status: v1.0.0. CI is green on Linux, macOS, and Windows against the current and previous Go releases. The wire format (v2) is frozen: within v1, a payload written by any release reads on any other. Format changes now mean pkg/v2.

The problem

Go's standard serializers assume your data is a tree. Given a DAG, they lie about its shape:

type Node struct {
    Name     string
    Children []*Node
}

shared := &Node{Name: "shared"}
root := &Node{Name: "root", Children: []*Node{
    {Name: "a", Children: []*Node{shared}},
    {Name: "b", Children: []*Node{shared}},
}}

• encoding/json writes shared twice. Decode it and you get two distinct objects; mutate one and the other is stale. On a cycle it recurses forever.
• encoding/gob also duplicates, and rejects cycles outright.

Identity is part of the data. If a.Children[0] == b.Children[0] before you serialize, it must still be true after you deserialize.

Usage

import reproto "github.com/lukem570/reproto/pkg/v1"

data, err := reproto.Marshal(root)

var out *Node
err = reproto.Unmarshal(data, &out)

out.Children[0].Children[0] == out.Children[1].Children[0] // true

Cycles need no special handling:

n := &Node{Name: "self"}
n.Parent = n

data, _ := reproto.Marshal(n)
var back *Node
_ = reproto.Unmarshal(data, &back)

back.Parent == back // true

There is also a streaming Encoder/Decoder over io.Writer/io.Reader, and Inspect reports a graph's shape without encoding it:

stats, _ := reproto.Inspect(root)
// stats.Objects, stats.Shared, stats.Edges, stats.MaxDepth

No registration, no init(), no setup. Encoding never needs a registry, and decoding derives what it needs from the target: unmarshalling into a *Node already says the payload may contain Node, []*Node, and so on.

The one exception is a value behind an interface, which is named nowhere in the target's static type:

type Event struct {
    Payload any // may hold a *Click
}

reproto.Register(&Click{}) // needed: Click is named nowhere in Event

Why only interfaces? Because decoding has to go from a name in the payload back to a reflect.Type, and Go has no lookup in that direction. For an ordinary field the decoder walks there from the target's static type instead — but any has no element type and no list of implementors, so there is nothing to walk. Rebuilding the type structurally from the payload does not help either: reflect.StructOf produces a different type with no methods, so the type assertion fails and it satisfies no interface.

Get it wrong and the error lists everything missing at once, with the calls to add:

unknown type: 3 type(s) in the payload are not reachable from the decode
target and must be registered:
    reproto.Register(&Click{})     // app.Click
    reproto.Register(&Scroll{})    // app.Scroll
    reproto.Register(&KeyPress{})  // app.KeyPress

Options cover registries, limits, unknown-type handling, and slice aliasing — see options.go.

Types that encode themselves

A type can take over its own encoding by implementing MarshalReproto and UnmarshalReproto, and a type you do not own can be given a Codec with RegisterCodec. time.Time, net/netip, and math/big types work out of the box — their fields are unexported, so without a codec they would encode as empty structs.

Slice aliasing

Off by default; WithSliceAliasing() makes slices that view one backing array still share it after a round trip:

raw := []byte("HDRbody")
f := &frame{Raw: raw, Header: raw[:3], Body: raw[3:]}

data, _ := reproto.Marshal(f, reproto.WithSliceAliasing())
var out *frame
_ = reproto.Unmarshal(data, &out, reproto.WithSliceAliasing())

out.Raw[0] = 'X'
out.Header[0] == 'X' // true

A worked example

examples/depgraph models a package dependency graph, where every package is reached three ways — the root's transitive requirements, another package's dependents list, and a by-name index — and all three must be the same object.

go run ./examples/depgraph compare

A dependency DAG: 9 packages, 9 distinct objects.

  reproto  9 objects — identity preserved
  gob      23 objects — 14 copies of the shared packages
  json     23 objects — 14 copies of the shared packages

Nine objects become twenty-three, and nothing looks wrong: every field still holds the right value and reflect.DeepEqual is satisfied. What changed is that logging is now four unrelated structs. Add the Dependents back edges that make it a real graph and json refuses to encode it at all.

How it works

reproto flattens the graph into a table of uniquely-identified objects, and replaces every pointer with an integer reference into that table.

        walk                encode               decode
 object graph ──▶ object table ──▶ bytes ──▶ object table ──▶ object graph
                (identity map)              (two-pass patch)

1. Walk. A Walker traverses the graph from a root value using reflection, keeping an identity map keyed on each pointer's underlying address. The first time an object is seen it is assigned a monotonic ObjectID and queued for encoding; every later sighting emits only the ID. Cycles terminate because the second visit is just a lookup. Traversal runs off an explicit queue, so a chain of a million nodes walks in constant stack space.
2. Encode. Each queued object becomes a self-describing, length-prefixed record. Pointer fields become varint object IDs, with 0 reserved for nil.
3. Decode. Two passes. The first allocates a zero value for every object in the table, so every ID has a real address. The second fills in fields and patches pointer fields to those addresses. Forward references and cycles fall out for free — no fixup queue, no deferred callbacks.

Reflection is used only to compile a per-type plan. Plans are cached per registry, so the per-value hot path is a loop over precomputed field operations rather than repeated reflect.Type inspection.

The format

Full specification in docs/format.md.

payload     := header typetable objecttable footer
header      := magic[4]="RPRO" version:u16 flags:u16
typetable   := count:uvarint type*
objecttable := count:uvarint object*
footer      := rootid:uvarint checksum:u32?

• ObjectID 0 means nil. Real objects are 1-based indices into the object table, so a pointer field needs no separate presence flag.
• Objects are length-prefixed, so a decoder can skip a record whose type it does not know instead of failing the whole payload.
• The type table is self-describing, so a payload can be inspected — and partially recovered — without the original Go types. That is what the CLI below runs on.
• Encoding is deterministic. IDs are assigned in a fixed walk order and map entries are emitted in sorted key order, so the same graph always produces byte-identical output on every OS and architecture. A golden payload in pkg/v1/testdata is compared byte for byte on Linux, macOS, and Windows in CI.
• Named types carry a structural hash, which lets the decoder recognise an unchanged layout in one comparison and read its fields positionally. It is a fast path, not a compatibility gate — a changed layout falls back to matching fields by name.

Schema evolution

Fields are matched by name, not position. Adding a field, removing one, or reordering them are all compatible; changing a field's type is rejected. See the compatibility table in docs/format.md.

CLI

task build
dist/reproto inspect graph.rpro     # type table, object table, reference edges
dist/reproto graph   graph.rpro     # Graphviz DOT, shared objects highlighted
dist/reproto verify  graph.rpro     # structural and checksum validation

$ dist/reproto inspect pkg/v1/testdata/graph.rpro
payload  397 bytes, format version 2
root     object 1
checksum 0x99f5626d (ok)
objects  5 (2 shared by more than one reference)
types    12
...
OBJECT TABLE
     1  …Containers   66 bytes  -> #2, #3, #3
 *   2  …Node          9 bytes  -> #4, #5
 *   3  …Node          9 bytes  -> #2
     4  …Node          5 bytes  -> #3
     5  …Node          5 bytes  -> #3

* shared: #2, #3

#2 and #3 are reached by more than one path — the objects a tree-shaped format would have duplicated. Type names are elided here for width.

Layout

pkg/v1/            package reproto — public API (Marshal, Unmarshal, Encoder, Decoder)
pkg/v1/walk/       Walker, identity map, ObjectID assignment, object table
pkg/v1/schema/     type registry, type descriptors, struct tags, plan compiler
pkg/v1/codec/      encoder, decoder, payload inspection
pkg/v1/wire/       low-level binary primitives (varint, zigzag, strings, limits)
cmd/reproto/       CLI for inspecting payloads
examples/depgraph/ worked example: a package dependency graph
internal/          developer tools (coverage gate, format check) kept portable

The public API is pinned at pkg/v1 so a future incompatible format can land at pkg/v2 without breaking importers.

Struct tags

type Record struct {
    Name    string `reproto:"name"`            // rename on the wire
    Cache   []byte `reproto:"-"`               // never serialized
    Comment string `reproto:"comment,omitempty"` // dropped when zero
}

Unknown tag options are an error at registration time rather than being ignored, so a typo surfaces immediately instead of silently changing the encoding. Unexported fields are always skipped.

Performance

Identity tracking is not free. On tree-shaped data — where reproto pays for identity and buys nothing — it remains slower than encoding/gob:

BenchmarkTreeReproto-16    385000 ns/op    531600 B/op    47 allocs/op
BenchmarkTreeGob-16        124000 ns/op     33328 B/op    35 allocs/op
BenchmarkTreeJSON-16       202000 ns/op     73866 B/op     1 allocs/op

Use reproto when you need identity preserved; use gob when your data really is a tree.

The optimization pass (phase 8) cut encoding time by 43% and decoding by 37%, and allocations per marshal from 5077 to 44. Three changes did most of it: a purpose-built identity map keyed on the pointer alone rather than a Go map with an interface in its key, removing a defer from the per-value decode path, and caching each payload type's local descriptor instead of looking it up per struct.

One proposal was measured and rejected: replacing reflect.Value.Field with precomputed offsets and reflect.NewAt benchmarked three times slower, so the unsafe fast path in the plan was closed rather than implemented.

task bench:compare diffs against the committed baseline; TestAllocationBudgets fails CI on a regression.

Building

Requires Go 1.24+ and Task.

task              # list available tasks
task test
task ci           # everything CI runs
task phases       # list the per-phase tasks from todo.md
task phase:3      # check one phase in isolation
task fuzz         # FUZZTIME=5m task fuzz to go longer
task bench:compare

CI runs the full suite on Linux, macOS, and Windows, against both the current and previous Go releases, with a golden-payload determinism job on each OS, a nightly fuzz run, and a weekly pass under the address and memory sanitizers.

Testing

Beyond the usual round-trip tests:

• Identity is asserted as a partition. TestPropertyIdentityPreserved builds the full pattern of which nodes are the same object across 200 random graphs and requires it to be identical after decoding.
• A differential against the standard library. On tree-shaped input all three serializers must agree; on the diamond graph, TestGobLosesIdentity records that gob passes reflect.DeepEqual while failing pointer equality — which is why DeepEqual is not a sufficient test here.
• Fuzzing. Millions of executions against the decoder and the wire reader; no input may panic, hang, or exhaust memory.
• Sanitizers. The memory sanitizer caught a genuine bug during development: the slice-aliasing code formed a pointer one past the end of an allocation. It now keys on the last element instead.

Limitations

• Only pointers carry identity by default: two fields holding the same map decode as separate copies. Slices can opt in with WithSliceAliasing.
• Interior pointers (&s.Field) become their own object and no longer alias the field inside s.
• Map keys may not be pointers or interfaces — there is no ordering for them that is stable across runs, and determinism is a hard guarantee.
• chan and func fields must be tagged reproto:"-".
• Removing a struct field does not remove its types from the payload: every object in the table is decoded, so the reader still needs those types registered, or must use WithSkipUnknownTypes.

Licence

MIT. See LICENSE.