goserde

README

goSerde

A code-generated, zero-reflection binary serializer for Go, built for maximum encode/decode throughput when you own both ends of the wire.

goserde generates type-specific Size/Marshal/Unmarshal methods for your structs at build time. There is no reflection on the hot path, no schema language, and no runtime type registry: just straight-line Go over a single running offset, with inlinable primitives and zero-copy decoding.

Marshal    14 ns/op    0 allocs     (28× faster than encoding/json)
Unmarshal  28 ns/op    1 alloc      (81× faster than encoding/json)

Is goSerde for you?

Use it when you control both the encoder and the decoder and want the smallest, fastest possible binary representation: RPC between your own services, on-disk caches, IPC, or embedded pipelines. The default fast mode assumes trusted input on the same architecture; an opt-in safe mode adds bounds-checked decoding, copies of decoded data, and a portable little-endian format for untrusted or cross-machine input (see Modes).

Look elsewhere when you need schema evolution, a self-describing format, or forward/backward compatibility with peers on different versions. goserde's schema lives entirely in the generated code, with no field tags or type markers on the wire, so the format does not evolve on its own. For that, reach for a schema-evolving format such as Protobuf, Cap'n Proto, or FlatBuffers.

Install

# The codec support library (imported by generated code)
go get github.com/tochemey/goserde

# The code generator
go install github.com/tochemey/goserde/cmd/goserdegen@latest

The library has zero external dependencies and builds offline.

Quick start

Annotate a struct with the //goserde:generate directive:

package model

//go:generate goserdegen -dir . -out goserde_gen.go

//goserde:generate
type User struct {
	ID   uint64
	Name string
	Tags []int32
}

Generate the codec:

go generate ./...

Encode and decode with the convenience helpers:

import "github.com/tochemey/goserde/codec"

u := &User{ID: 1, Name: "ada", Tags: []int32{1, 2, 3}}

// Encode into a fresh, exactly sized buffer.
data := codec.Bytes(u)

// Decode.
var out User
if err := codec.From(&out, data); err != nil {
	log.Fatal(err)
}

On the hot path, drive the generated methods directly and reuse buffers to stay allocation-free:

buf = codec.Into(u, buf) // reuses buf when it has capacity, else allocates

var out User
out.Unmarshal(data)        // returns (bytesRead int, err error)

Supported types

Category	Types
Integers	int, int8/16/32/64, uint, uint8/16/32/64
Floats	float32, float64
Scalars	bool, string, []byte
Composites	slices []T, fixed arrays [N]T, maps map[K]V, pointers *T
Structs	nested structs (each annotated with //goserde:generate)
Time	time.Time (as int64 Unix-nanoseconds, UTC)
Unions	interface fields via //goserde:union (see Tagged unions)

Fields tagged `goserde:"-"` are excluded from the wire and decode back to their zero value, useful for fields of types goserde cannot serialize (channels, funcs).

//goserde:generate
type Session struct {
	ID    uint64
	Token string
	conn  net.Conn `goserde:"-"` // excluded
}

Tagged unions

Declare a polymorphic field by listing its concrete members above an interface. Each member must itself be a //goserde:generate struct whose pointer implements the interface:

//goserde:generate
type Circle struct{ R float64 }

func (*Circle) isShape() {}

//goserde:generate
type Square struct{ Side float64 }

func (*Square) isShape() {}

//goserde:union Circle Square
type Shape interface{ isShape() }

//goserde:generate
type Drawing struct {
	Name  string
	Shape Shape   // holds *Circle, *Square, or nil
}

On the wire a union is a varint tag (0 = nil, 1 = first member, …) followed by the member's payload. Member order is the wire contract: only ever append new members, never reorder. Decoding an unknown tag returns codec.ErrUnknownUnionTag rather than panicking.

Modes

A package is generated in exactly one mode; the method signatures are identical, so callers don't change.

Mode	Flag	Unmarshal on bad input	Decoded string/[]byte	Portable
Fast (default)	none	may panic (trusted bytes)	zero-copy (aliases input)	no (native memory)
Safe	-safe	returns codec.ErrShortBuffer	copied (owns its bytes)	yes (little-endian)

Fast mode is for trusted bytes on identical architectures: it uses unsafe zero-copy decoding and a single memmove for all-fixed-width structs. Safe mode bounds-checks every read, copies length-prefixed payloads, and encodes field-by-field in little-endian, portable across architectures and Go versions, at a modest cost.

Decoding untrusted input? Use safe mode. Fast mode trusts its bytes, and that has two consequences. First, malformed input may panic or read out of bounds, so never decode bytes you did not produce yourself (network payloads, files written by other processes, user input) in fast mode. Second, a decoded string or []byte aliases the buffer passed to Unmarshal rather than copying, so that buffer must outlive the decoded value and must not be mutated or returned to a pool while the value is in use. Safe mode removes both hazards: it returns codec.ErrShortBuffer on truncated input and copies decoded data so the result owns its bytes.

goserdegen -dir . -out goserde_gen.go        # fast (default)
goserdegen -dir . -out goserde_gen.go -safe  # safe / portable

Performance

All numbers below are from the same machine (Go 1.26, darwin/arm64, Apple M1). Reproduce with make bench (vs the standard library), make bench-shapes (across shapes), and make compare (vs mus and benc).

vs the standard library

Flat Record struct (uint64, float64, bool, string, []uint32, []byte):

Operation	goserde	encoding/json	encoding/gob¹
Marshal	14 ns, 0 alloc	399 ns, 1 alloc	191 ns
Unmarshal	28 ns, 1 alloc	2258 ns, 12 allocs	n/a

¹ gob marshal uses a reused encoder. goserde is ~28× faster than JSON on marshal and ~81× on unmarshal, with a fraction of the allocations.

vs other fast serializers

Same Record data, 3 runs each, with mus v0.10.2 and benc v1.1.8:

Serializer	Marshal	Unmarshal	Payload	Decode allocs
goserde	13.3 ns	28.0 ns	139 B	1
benc	25.3 ns	45.1 ns	143 B	1
mus (raw)	34.3 ns	68.6 ns	139 B	2
mus (varint)	34.2 ns	73.0 ns	109 B	2

goserde is ~1.9× / ~1.6× faster than benc (marshal/unmarshal) and ~2.6× faster than mus on both. The main reason is no interface dispatch in the generated code: running mus in fixed-width "raw" mode produces goserde's exact 139 B format yet barely changes its speed, so the win is the straight-line code, not the format. mus and benc are fuller-featured libraries (schema evolution, validation, versioning); goserde trades those for raw throughput.

Across struct shapes

goserde's lead is shape-dependent. Marshal is zero-alloc on every shape:

Shape	Marshal	Unmarshal	Decode allocs	Notes
Fixed-width (4 fld)	2.4 ns	2.4 ns	0	single memmove (blittable)
Flat + string/bytes	7.1 ns	6.3 ns	0	zero-copy string/[]byte
String-heavy	19.6 ns	33.0 ns	1	[]string must allocate
Nested + pointers	16.9 ns	36.1 ns	2	pointer + slice-of-struct
Map-heavy	77.5 ns	151.9 ns	4	make(map) + per-entry insert

Decode reuse: decoding repeatedly into the same value reuses its slices (by capacity) and maps (via clear). The map-heavy shape above drops from 152 ns / 4 allocs to 67 ns / 0 allocs when the destination is reused across calls, the right pattern for a read loop.

Fixed and flat structs are where goserde dominates. Map-heavy decode is its weakest point on a fresh destination, where mus and benc are competitive.

Wire format

Fixed-width little-endian numerics; LEB128 varint length prefixes for strings, slices, and maps; a 1-byte nil flag for pointers; an 8-byte Unix-nanosecond int64 for time.Time. All-fixed-width structs are encoded as a single memory copy in fast mode. There are no field names, tags, or type markers on the wire: the schema lives entirely in the generated code.

This format is not self-describing and, in fast mode, not portable across architectures or Go versions. That is the deliberate trade for speed.

Compatibility

goserde is pre-1.0: the API and wire format may change between minor versions until 1.0.0. From 1.0.0 onward the following semantic-versioning promise applies.

Covered by SemVer (stable within a major version):

• The generated method set on your types: Size() int, Marshal([]byte) int, and Unmarshal([]byte) (int, error).
• The full codec package: the convenience API (Bytes, Into, From, and the Marshaler / Unmarshaler interfaces) and the low-level encoding primitives (PutU32/U32 and friends, PutUvarint/Uvarint/UvarintSize, Zig/Zag, B2S/S2B, the float bit-casts) you can use to hand-write a codec instead of running the generator. Generated and hand-written codecs call the same primitives, so they share an identical wire format.
• The generator directives //goserde:generate and //goserde:union, and the goserde:"-" struct tag.

Wire-format stability:

• Safe mode (-safe) is the portable format: little-endian and bounds-checked, stable across architectures and Go versions. Use it for data at rest or exchanged between machines.
• Fast mode (default) is not portable. It uses native byte order and encodes all-fixed-width structs as a copy of their in-memory layout, so the bytes can differ across architectures and across Go versions whose struct layout differs. Treat fast-mode bytes as ephemeral and decode them with the same goserde version, architecture, and Go toolchain that produced them.

Regenerating codecs after upgrading goserde is always safe and recommended.

How it works

The generator (cmd/goserdegen) is standard-library only: it loads and type-checks your package with go/parser and go/types (no go/packages, no network), finds //goserde:generate structs, and emits a codec shaped exactly like the hand-tuned reference in codec/record.go. Generated code imports goserde's codec package, whose primitives (varint, zigzag, little-endian fixed-width R/W, zero-copy conversions, float bit-casts) are written to inline.

The import path of that codec package is derived automatically from the module that built goserdegen, so forking under a different module path "just works" with no flags to keep in sync.

Development

go build ./...        # build everything (zero dependencies)
go test ./...         # run the full test suite
go generate ./...     # regenerate codecs after editing annotated structs
make bench-shapes     # benchmark across struct shapes
make compare          # head-to-head vs mus and benc (needs network)