README
¶
Litz
High-performance, zero-allocation serialization library for Go.
Litz is a serialization library optimized for low-latency hot-paths. It leverages the SBC-IPR (Segmented Block Copy & In-place Pointer Resolution) paradigm combined with a HIBI (Hash Index Block Inlay) wire format to minimize parsing and allocation overhead during message processing.
For questions regarding bug reporting, pull requests, and security policies, please refer to CONTRIBUTING.md and SECURITY.md.
Key Features
- Zero-Allocation Deserialization: Resolves strings, slices, and nested structures directly pointing into the backing serialized buffer.
- In-Place Querying: The HIBI wire format indexes fields by hash, allowing O(log N) binary search lookups on the byte stream via the dynamic
litz.Dynamicreader without parsing the entire payload. - Compile-Time Layout Verification: Code generation outputs compile-time size and field-offset assertions (
unsafe.Offsetof) to guarantee structural alignment matches exactly. - Security Boundaries: Enforces memory alignment checks and math-safe bounds comparisons to prevent buffer overflow and OOM Denial of Service (DoS) vectors.
Benchmarks & Performance Metrics
Below is a performance comparison of Litz against Standard Go Protobuf (v2), Vtprotobuf (Highly optimized VT-codegen Protobuf), and MessagePack (Msgp) run on a Linux AMD64 system:
| Payload Size | Library | Marshal (ns/op) | Unmarshal (ns/op) | Heap Allocations (B/op) | Allocs (op) |
|---|---|---|---|---|---|
| Small | Litz | 2.51 ns | 0.43 ns | 0 B | 0 |
| Vtprotobuf | 14.59 ns | 5.03 ns | 3 B | 1 | |
| MessagePack | 4.50 ns | 10.25 ns | 0 B | 0 | |
| Protobuf | 64.53 ns | 58.93 ns | 0 B | 0 | |
| Medium | Litz | 5.25 ns | 2.42 ns | 0 B | 0 |
| Vtprotobuf | 38.02 ns | 41.46 ns | 32 B | 1 | |
| MessagePack | 15.30 ns | 59.52 ns | 16 B | 1 | |
| Protobuf | 127.30 ns | 124.60 ns | 16 B | 1 | |
| Large | Litz | 77.26 ns | 78.14 ns | 48 B | 1 |
| MessagePack | 90.72 ns | 337.40 ns | 136 B | 7 | |
| Vtprotobuf | 152.30 ns | 684.10 ns | 1081 B | 7 | |
| Protobuf | 378.70 ns | 752.00 ns | 536 B | 13 |
Design Trade-offs & Limitations
While Litz is optimized for speed, it makes several major trade-offs:
- Architecture Portability (Little-Endian only): The wire format uses little-endian byte ordering directly matching host hardware registers (amd64, arm64, wasm). Paying no cost for byte-swapping means Litz is not portable to big-endian architectures.
- Fixed-Size Fields (Larger Wire Footprint): Litz does not use variable-length integer encoding (varints). Numerical types are serialized as fixed 8-byte, 4-byte, or 2-byte values, leading to larger wire payloads compared to Protobuf or Msgpack.
- Buffer Lifetime Coupling (Zero-Copy): Deserialized string and slice fields point directly into the source byte buffer. Reusing or discarding the source buffer will lead to use-after-free corruption unless the deserialized struct is explicitly duplicated via
Clone(). - Homogeneous Slices Only: Collection elements must share a single consistent data type. Heterogeneous (mixed-type) slices are unsupported and will reject marshaling.
Quick Start
1. Install the Generator
go install github.com/cuprite-io/litz/cmd/litz-gen@latest
2. Annotate Struct Definitions
Annotate target structs in your package with the //litz:generate comment directive:
package user
//litz:generate
type Profile struct {
ID uint64
Name string
Active bool
}
3. Generate Code
Run the code generator to produce layout mirrors and marshaling helpers (outputs to *_test.go files if generated solely for testing packages):
litz-gen -dir . -out profile_gen.go
4. Serialize & Deserialize
package main
import (
"fmt"
"log"
"github.com/cuprite-io/litz"
"mypackage/user" // Import generated package
)
func main() {
input := &user.Profile{ID: 101, Name: "Jane Doe", Active: true}
// 1. Marshal struct to buffer
buf, err := user.MarshalProfile(input, nil)
if err != nil {
log.Fatalf("failed to marshal: %v", err)
}
// 2. Unmarshal buffer back to struct
var output user.Profile
if err := user.UnmarshalProfile(buf, &output); err != nil {
log.Fatalf("failed to unmarshal: %v", err)
}
fmt.Printf("Deserialized User: %s (ID: %d)\n", output.Name, output.ID)
// 3. Optional: Dynamic lookup of fields without full unmarshaling
dyn := litz.NewDynamic(buf, litz.TypeMap)
if activeVal := dyn.Get("Active"); activeVal != nil {
fmt.Printf("Dynamic Active check: %t\n", activeVal.Bool())
}
}
Documentation
¶
Index ¶
- Constants
- Variables
- func AlignedBuffer(size int) []byte
- func CloneAny(v any) any
- func HashKey(key string) uint32
- func MarshalAny(v any) ([]byte, uint8, error)
- func SliceSwizzle[T any](buf []byte, offset uintptr, length int) []T
- func StringSwizzle(buf []byte, offset uintptr, length int) string
- func StringSwizzleUnchecked(buf []byte, offset uintptr, length int) string
- type Dynamic
- func (d *Dynamic) Bool() bool
- func (d *Dynamic) Bytes() []byte
- func (d *Dynamic) Float() float64
- func (d *Dynamic) Get(key string) *Dynamic
- func (d *Dynamic) GetOK(key string) (*Dynamic, bool)
- func (d *Dynamic) Int() int64
- func (d *Dynamic) Interface() any
- func (d *Dynamic) IsNil() bool
- func (d *Dynamic) Keys() []string
- func (d *Dynamic) Len() int
- func (d *Dynamic) Map(keys []string) map[string]*Dynamic
- func (d *Dynamic) Raw() []byte
- func (d *Dynamic) Slice() []*Dynamic
- func (d *Dynamic) String() string
- func (d *Dynamic) ToMap() (map[string]any, error)
- func (d *Dynamic) ToSlice() ([]any, error)
- func (d *Dynamic) Type() uint8
- func (d *Dynamic) Uint() uint64
- type Pool
Constants ¶
const ( TypeNull uint8 = iota TypeInt TypeFloat TypeBool TypeString TypeBytes TypeMap TypeSlice TypeUint )
HIBI Type Constants
const Version = "v0.1.5"
Version is the current version of the Litz serialization library.
Variables ¶
var ( ErrBufferTooShort = errors.New("litz.Unmarshal: buffer too short for fixed part") ErrStringOutOfBounds = errors.New("litz.Unmarshal: string out of bounds") ErrSliceOutOfBounds = errors.New("litz.Unmarshal: slice out of bounds") ErrPointerOutOfBounds = errors.New("litz.Unmarshal: nested pointer out of bounds") ErrSizeOverflow = errors.New("litz.Marshal: size integer overflow") ErrInvalidHeader = errors.New("litz.Unmarshal: invalid format signature or version") ErrInvalidHIBIType = errors.New("litz.Dynamic: invalid type for this operation") )
Sentinel Errors to avoid heap allocations on error paths
Functions ¶
func AlignedBuffer ¶
AlignedBuffer allocates a byte slice. Go's runtime allocator aligns heap allocations to 8-byte boundaries automatically for sizes >= 8 bytes.
func CloneAny ¶
CloneAny performs a deep copy of common interface{} types to prevent use-after-free.
func MarshalAny ¶
MarshalAny serializes any Go value into the HIBI format. Returns the serialized bytes, type identifier, and error.
func SliceSwizzle ¶
SliceSwizzle converts an offset in buf to a valid Go slice. WARNING: The returned slice points directly into the buffer memory.
func StringSwizzle ¶
StringSwizzle converts an offset in buf to a valid Go string. WARNING: The returned string points directly into the buffer memory. The buffer MUST outlive the returned string to avoid use-after-free corruption.
func StringSwizzleUnchecked ¶
StringSwizzleUnchecked is an unchecked variant of StringSwizzle. Re-uses direct string backing arrays without runtime bounds checking. WARNING: Calling this with a corrupted or malicious offset will trigger a segmentation fault or memory read violation.
Types ¶
type Dynamic ¶
type Dynamic struct {
// contains filtered or unexported fields
}
Dynamic represents unstructured schema-less data backed by raw bytes. It implements the Hash-Indexed Binary Index (HIBI) protocol.
func NewDynamic ¶
NewDynamic creates a new Dynamic reader wrapping the given HIBI buffer and type code.
func (*Dynamic) Get ¶
Get performs an O(log N) binary search lookup for a key inside a HIBI map. Crucially, it resolves hash collisions by verifying the full key string.
func (*Dynamic) GetOK ¶ added in v0.1.4
GetOK is like Get, but also returns a boolean indicating whether the key was found.
func (*Dynamic) Interface ¶ added in v0.1.4
Interface converts the Dynamic value back to a standard Go interface representation.
func (*Dynamic) Map ¶
Map converts the HIBI payload into a standard Go map[string]*Dynamic. Validates that this Dynamic object is actually a Map.
func (*Dynamic) Slice ¶
Slice returns the dynamic elements if this Dynamic object is a slice. Validates that this Dynamic object is actually a Slice and resolves the dynamic element type from the slice header.
func (*Dynamic) ToMap ¶ added in v0.1.4
ToMap converts the Dynamic object back to a standard Go map[string]any. Returns an error if the underlying value is not a HIBI map.
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool is a wrapper around sync.Pool for reusing serialization buffers.
func (*Pool) Put ¶
Put returns a buffer back to the pool. To prevent memory bloat during massive payload spikes, buffers with capacity larger than 16MB are discarded rather than returned to the pool. Note: We accept and return *[]byte (pointer to slice header) rather than []byte to prevent the Go runtime from allocating interface boxing containers on sync.Pool.Put, maintaining true zero-allocation execution on recycled paths.
Source Files
Directories