qdf

package module
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2026 License: MIT Imports: 24 Imported by: 0

README

qdf

ci codeql codecov benchmarks Go Reference Go Report Card Go License Status

Quantum-inspired data serialization format designed for ultra-fast and lightweight data exchange.

qdf gives you encoding/json-style ergonomics (Marshal / Unmarshal / NewEncoder / NewDecoder) and a compact tagged binary wire format with an optional inline string-interning table. The reflect path ships with specialised encoders for the common slice / map shapes. A go:generate tool emits reflection-free MarshalQDF / UnmarshalQDF methods when you need the last 10–15 %.

Where the name comes from

Quantum Density Format is a borrowed metaphor, not physics. A classical CPU cannot store qubits; what qdf borrows from quantum information theory is a way of thinking about repetitive structured data.

The starting observation: in a payload like

{ "users": [
    {"country": "LT"},
    {"country": "LT"},
    {"country": "LT"} ] }

the byte "LT" is written three times even though, from an information-theoretic point of view, it is a single state with high probability. Classical formats (JSON, MessagePack, CBOR, Protobuf) treat data as tree → bytes. qdf treats it as

data = (state set, references, entropy weights)

— a finite set of distinct values plus a stream that "collapses" each position onto one of them. The wire dialect that ships today realises the practical subset of that idea:

Quantum-inspired concept What it maps to in qdf today
State set — the discrete values a position can take. The Dense-mode intern table. The first occurrence of a string or []byte value writes it in full and assigns it a stable ID.
Wavefunction collapse — a single observation picks one state. Every subsequent occurrence emits a state_ref tag plus a varint ID. Decoding "collapses" the reference back to its stored value.
Density — keep only what carries information; minimise entropy H(D | C) against the existing context. Repeated keys / values across a message (and across a Dense stream) cost 1–3 bytes rather than their full length. Numeric and bool slices use QPack codecs (FOR, Delta+FOR, RLE, dictionary, Gorilla XOR, ALP decimal, raw-LE bulk) that auto-select the smallest predicted form per slice; slices of homogeneous structs transpose to per-column codecs, and an enum-like string column is dictionary-coded (distinct table + bit-packed index per row) when that beats per-value interning. A mixed struct — mostly scalar/string columns with one or two map / slice / nested fields (the common log / AD / span / RTB record shape) — transposes its eligible columns and keeps the rest as a per-row residual tail (hybrid columnar, tagHybridColStruct, 0xF7), under OptCompression, instead of falling back to fully row-major. High-cardinality string columns (URLs, log lines, paths) where the whole-string dictionary can't help are eligible for FSST (tagColStrFSST, 0xF6), which learns a symbol table of up to 255 substrings and compresses at the byte level — emitted only when strictly smaller. Field-name headers in generated code are pre-encoded once per type and concatenated without further work.
Probabilistic / residual coding — predict, then store only the deviation. QPack's Delta+FOR codec is exactly this for monotonic integer sequences: encode the first value plus the bit-packed residual against a aᵢ = aᵢ₋₁ + minΔ predictor. Gorilla does the same for floats by XOR-ing each sample against the previous one and storing the differing bits only.
Entanglement — correlated values that constrain each other (e.g. city = "Vilnius" ⇒ country = "Lithuania"). Three stacked predictors on the Dense state stream. Markov-0 (tagStateRepeat, 0xE8) collapses an immediate repeat of the previously emitted state-ref to a single byte. MTF rank (tagStateMTF, 0xE9) encodes the LRU rank of the touched ID when that varuint is shorter than the raw id. Markov-1 pair (tagStatePair, 0xEA) stores the most-recent successor observed after each prev ID (top-1, K=1) and encodes a hit as 0xEA + 0x00 (rank always 0) — wins when the prev → curr transition is predictable and the raw id needs a multi-byte varuint. The encoder picks the shortest of the four variants per emission, so the wire is never larger than the plain tagStateRef encoding. A full conditional-probability table beyond order-1 stays in the reserved 0xEB / 0xED..0xEF block.
Shape interning — repeated structure means the layout itself is information. Dense mode emits structs (and map[string]any of stable shape via the reflect-struct path) through tagMapShape (0xEC). First occurrence declares the shape inline: 0xEC, 0, varuint(N), N × key. Subsequent occurrences of the same shape emit 0xEC + varuint(shapeID) + N × value — keys are not re-emitted. Per-record saving on an array of identical-shape structs is N × 2 bytes for the elided state-refs plus the map header. The shape table is per-stream, addressed by *typeDesc on the encoder and by sequential ID on the wire. Shapes never collide across types because the encoder keys the binding on the descriptor pointer.
Arithmetic / range coding (rANS) — push the encoded stream to its Shannon limit. Shipped behind OptRANS (in OptCompression): a static order-0 rANS pass over the whole body, applied only when it shrinks (never larger). Squeezes the residual byte-entropy the structural codecs leave — e.g. −37 % on trace batches — at ~4–6× CPU, so it stays in the opt-in compression tier.

The conceptual roadmap (state table → entanglement graph → predictive encoder) is preserved in the codebase: tag space and the encoder interface are designed to let those layers slot in above the current one without a format break. The MVP that ships now is the part of the idea that has a positive CPU / size tradeoff on day-one workloads — logs, traces, columnar rows, embedding vectors.

The realistic conclusion: the real shift is not the "quantum" framing but the move from "data as bytes" to "data as a model of the world that produced it".

Why another format

encoding/json is verbose and slow. vmihailenco/msgpack/v5 is fast but allocates a lot on decode and has no built-in deduplication for repeated keys / values — which is the dominant cost on real telemetry, log, and trace payloads where the same handful of strings (service names, log levels, region codes, span attributes) appears thousands of times per batch.

qdf attacks two costs at once:

  1. Wire size on repetitive data. A Dense-mode encoder maintains an inline interning table: the first time "region":"eu-west-1" appears it is written in full; subsequent occurrences emit a 2-byte back-reference. Streams share that table across messages, so a 1000-event log batch with eight distinct region codes carries each code once. The decoder follows the same protocol — no out-of-band dictionary, no two-pass encode.

  2. CPU and allocations on decode. The reflect path uses a cached per-type descriptor with unsafe.Pointer + field offset access instead of reflect.Value.Field(i), plus specialised type-specific decoders for []string, []int*, []float*, map[string]string, map[string]int, map[string]any, etc. A direct-mapped key interner deduplicates map keys across decode calls without a pool or chained hash table. On a map[string]any decode with repeated keys the allocator runs 3 times per call versus json's 37.

The format is single-pass and streaming-safe in both directions. There is no schema language, no IDL, no compile-time registration — types are picked up via Go struct tags (qdf:"name", falling back to json:"name").

Measured against the field

On realistic batch payloads (i7-9750H · Go 1.26), qdf's wire is smaller than protobuf on every fixture — because it dedups and columnar-compresses across records, while json/msgpack/protobuf/flatbuffers encode each record independently:

wire size vs protobuf OTLP traces logs RTB events IoT floats
qdf_balanced −57 % −43 % −21 % −39 % −24 %
qdf_compression −68 % −60 % −38 % −39 % −29 %

It also allocates far less under load (IoT encode: qdf_balanced 3 allocs/op vs protobuf 385). Honest trade-offs: qdf_speed wire ≈ msgpack; qdf_compression encode trades CPU for bytes; protobuf and flatbuffers win raw decode throughput and single-tiny-message size. Full tables (json / msgpack / protobuf / flatbuffers × five fixtures, wire + memory) and the reproduction recipe are in docs/COMPETITIVE.md.

A concrete head-to-head on a realistic Active-Directory export (5 000 mixed ADUser records — unique GUID/UPN/email, occasionally-repeating names, moderately-repeating groups/departments; i7-9750H, same data through each library):

5 000 AD users wire encode decode
encoding/json 3 833 KB 14.1 ms 57.8 ms
msgpack 3 370 KB 8.7 ms 16.7 ms
qdf — OptBalanced (default) 1 830 KB 6.6 ms 7.9 ms
qdf — OptCompression 616 KB 32.1 ms 13.2 ms

OptBalanced is a clean sweep — smaller and faster than both json and msgpack on every axis (decode ≈ 7× faster than json, ≈ 2× faster than msgpack; ≈ half the allocations). OptCompression is 6.2× smaller than json / 5.5× smaller than msgpack and still decodes faster than both, paying encode CPU for it. The mixed-struct shape (a map + a slice field amid the scalars) is exactly what hybrid columnar (OptCompression) unlocks — without it those records would never have been transposed at all.

Wire layout
+-----+-----+-----+-----+-----+
| 'Q' | 'D' | 'F' | ver |flags|
+-----+-----+-----+-----+-----+
| body (tagged values)        |
+-----------------------------+

A 5-byte header (3-byte magic + 1-byte version + 1-byte flags) and a tagged body. The flag bit distinguishes Fast from Dense; the decoder auto-detects.

Tag space is msgpack-inspired with a few additions:

Range Purpose
0x00..0x7F positive fixint (0..127)
0x80..0x9F fixstr (len 0..31)
0xA0..0xBF fixarr (len 0..31)
0xC0..0xCF nil, bool, int / uint / float of width 8–64
0xCD..0xCF str8 / str16 / str32
0xD0..0xD2 bin8 / bin16 / bin32
0xD3..0xD4 arr16 / arr32
0xD5..0xD7 map8 / map16 / map32
0xD8..0xDF negfixint (-1..-8)
0xE0..0xE2 intern_str / state_ref / intern_bin (Dense)
0xE3 QPack: bit-packed []bool
0xE4 QPack: raw-LE numeric slice
0xE5 QPack: Frame-of-Reference bit-packed ints
0xE6 QPack: Delta + zigzag + FOR (monotonic ints)
0xE7 QPack: Gorilla XOR-coded float slice
0xE8 Dense: state-ref repeat (Markov-0 predictor)
0xE9 Dense: state-ref MTF rank (Move-To-Front)
0xEA Dense: state-ref pair rank (Markov-1)
0xEB QPack: run-length encoded integer slice
0xEC Dense: struct/map shape declare / reuse
0xED QPack: dictionary-coded integer slice
0xEE QPack: Patched FOR integer slice (outliers)
0xEF QPack: columnar []struct container
0xF0..0xF3 ext / timestamp
0xF4 QPack: ALP decimal-coded []float64 slice
0xF5 QPack: dictionary-coded string column
0xF6 QPack: FSST-coded string column (inside columnar)
0xF7..0xFF reserved (rANS, n-gram graph, future)

The 5th header byte holds two flag bits: FlagDense (0x01) for the intern dialect, and FlagQPack (0x02) as an early hint that the body may carry codec tags from the QPack codec range (0xE3..0xEF, 0xF4..0xF6). A reader that does not implement the QPack tags fails with ErrBadTag on first contact; it never decodes a packed payload as scalar by accident.

Alpha note: qdf is pre-1.0. The wire format is still being shaped — tags may shift, version may bump, and "older" decoders here mean "earlier in alpha", not stable releases. No backwards-compat promise yet.

All multi-byte integers and floats are little-endian. amd64 and arm64 are the supported targets.

Quick start

go get github.com/alex60217101990/qdf

Requires Go 1.26.

package main

import (
    "fmt"

    "github.com/alex60217101990/qdf"
)

type Event struct {
    ID      int               `qdf:"id"`
    Source  string            `qdf:"source"`
    Payload []byte            `qdf:"payload"`
    Attrs   map[string]string `qdf:"attrs"`
}

func main() {
    in := Event{
        ID:     42,
        Source: "ingest",
        Attrs:  map[string]string{"region": "eu-west-1", "version": "v3"},
    }

    b, err := qdf.Marshal(in, qdf.OptSpeed)
    if err != nil {
        panic(err)
    }
    fmt.Printf("wire: %d bytes\n", len(b))

    var out Event
    if err := qdf.Unmarshal(b, &out); err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", out)
}
Encode profile is per-call

There is one encode entry point — Marshal(v, opts) — and the opts bit-mask picks which codecs run for that specific call. The convenience bundles cover the common tradeoffs:

Bundle When to use
qdf.OptSpeed Fast path. Tightest CPU cost; size comparable to msgpack. Drop-in for encoding/json behaviour.
qdf.OptBalanced Repetitive payloads — logs, telemetry, columnar rows. Strings intern once; numeric and bool slices use QPack codecs; struct shapes intern; Markov-1 + MTF run on state-refs.
qdf.OptCompression OptBalanced plus the heavier wire-size codecs: Gorilla XOR for smooth float series, ALP for quantized/decimal []float64, FSST substring-level compression for high-cardinality string columns (URLs, log lines, paths), hybrid columnar (transposes the eligible columns of a mixed struct — one with a map / slice / nested field — keeping the rest row-major, so log / AD / span records get columnar treatment they otherwise couldn't), and a final order-0 rANS entropy pass over the whole body (never larger). Trades encode CPU for smaller wire — pick it for backup / cold storage.
custom mix Or-combine individual bits (OptDense | OptQPack | OptShapeIntern …) when one of the bundles is one click off the desired tradeoff. 
b, _ := qdf.Marshal(event,    qdf.OptSpeed)        // hot path
b, _ := qdf.Marshal(batch,    qdf.OptBalanced)     // telemetry / logs
b, _ := qdf.Marshal(snapshot, qdf.OptCompression)  // backup / archive
b, _ := qdf.Marshal(payload,                       // tuned mix
    qdf.OptDense|qdf.OptQPack|qdf.OptShapeIntern)

A single qdf.Unmarshal reads everything. The wire header is self- describing; the receiver never has to know which option mix the sender picked.

Picking the right combo for a concrete workload — hot path, telemetry, metric series, embeddings, backup — is covered in docs/CHOOSING.md, with head-to-head numbers vs json and msgpack per scenario.

Dense mode internals — what each bit buys you

OptDense activates the inline intern table. The four codec bits that compose OptBalanced layer on top of it. They share one rule: the encoder picks the shortest variant per emission, so Dense wire is never larger than the same payload at OptSpeed. If the predictors do not pay, the byte cost is identical to a plain state-ref.

Tag Predictor Wins when Doesn't help when
0xE8 Markov-0: same ID as the previous emission Runs of the same value (region, region, region). Distinct values in a row.
0xE9 MTF: encode LRU rank instead of raw id Heavy reuse of a small hot subset of strings that were interned early. Random access pattern with no recency.
0xEA Markov-1: top-1 predictor of the previous emission's successor Correlated pairs — country → city, service → region. Only emits when the raw id needs ≥ 2 varuint bytes, so the wire never grows on small state tables. Unique transitions, low-cardinality state.
0xEC Shape interning: declare struct shape once, reuse by id Arrays / streams of the same struct type. Per-record saving is ≈ N × 2 bytes (elided key state-refs + map header). One-shot encodes of a unique struct type.

The state table is shared per stream in StreamEncoder / StreamDecoder, so a batch of 1 000 log events with eight distinct region codes ships each region once. Marshal* calls do not share — each call starts with an empty table.

Where Dense is worth it: logs, traces, telemetry, columnar rows, event batches, audit streams, snapshot dumps — anything where a small vocabulary of strings and a stable struct shape repeat thousands of times. Expected envelope: 25 – 55 % smaller than JSON at 2–6× the encode speed.

Where Dense is not worth it: single-shot encodes of small unique payloads (Markov / shape predictors never reach their amortisation point, you pay ≤ 2 bytes of prelude for nothing), or strict-throughput paths where decode CPU is the bottleneck — Dense decode is ~15 % slower than Fast because of the state-table lookups. Reach for OptSpeed or OptQPack there.

Where it is wrong: cryptographic / forensic contexts that need byte-stable wire across implementations and versions. Dense embeds predictor state, intern-ID ordering, and shape IDs that depend on emission order. Use OptSpeed if you hash or sign the wire.

Bit reference
Bit What it gates
OptDense Inline intern table; required by everything below.
OptQPack Numeric / bool slice codecs (FOR, Delta+FOR, Gorilla, bit-pack).
OptShapeIntern tagMapShape for struct emissions (declare once, reuse by id).
OptPairPred Markov-1 successor predictor (tagStatePair, 0xEA).
OptMTF Move-to-Front rank coding on state-refs (tagStateMTF, 0xE9).
OptColumnIndex Column-length index on columnar []struct for selective decode (opt-in, ~4 B/column; default wire unchanged when off).
OptMapShape Key-set interning for map[string]V fields (declare keys once, reuse by shape id). ~−24 % encode CPU and ~−26 % wire on recurring-key tag-maps (telemetry/logs); opt-in, default wire unchanged when off.
Bundle Composition
OptSpeed 0 — Fast mode, no codecs.
OptBalanced OptDense | OptQPack | OptShapeIntern | OptPairPred | OptMTF.
OptCompression OptBalanced + Gorilla XOR + ALP decimal floats + FSST for high-cardinality string columns + an order-0 rANS entropy pass (never larger). Trades CPU for wire size; for backup / cold storage.

Options is a uint32 carried by value, so Marshal and AppendMarshal add zero per-call allocations over the pool / output-clone overhead. Encoders are pooled in a single shared pool (encPool) keyed only by buffer; the configuration is applied on each acquire via applyOpts. Markov-0 (tagStateRepeat, 0xE8) is always on inside OptDense because it costs nothing on the wire when the predictor misses.

A note on tuning knobs that do not live on the bit-mask: the intern threshold (SetIntern) and the cycle-depth ceiling (SetMaxDepth) stay on the *Encoder itself. Use the low-level encoder API for those.

Generic API (Go 1.18+ generics)

Marshal(v any, opts) boxes the argument through interface{} and then makes one reflect copy for value-typed inputs. The generic helpers skip both: T is fixed at the call site, reflect.TypeFor[T]() resolves at compile time, and unsafe.Pointer(&v) points directly at the caller's stack.

Generic Equivalent to
MarshalT[T any] Marshal (typed)
AppendMarshalT[T any] AppendMarshal (typed)
UnmarshalT[T any] Unmarshal (typed)

Wire output is byte-identical. The win is on the encode side: one fewer allocation per call (-80 B) and 25–40 % faster on small/medium payloads.

buf, _ := qdf.MarshalT(event, qdf.OptSpeed)
var out Event
_ = qdf.UnmarshalT(buf, &out)
Direct entry points (skip reflection entirely)

MarshalT still goes through descOf to walk the type. Types that already implement MarshalQDF / UnmarshalQDF (either hand-written or generated by cmd/qdfgen) can bypass the descriptor cache and the runtime type assertion via the Direct generics. The type parameter is constrained to the Marshaler / Unmarshaler interface, so the method call is resolved at compile time and the compiler inlines it.

Direct When to use
MarshalDirect[T Marshaler](v T) ([]byte, error) Hot paths where the type is known to have generated codecs.
AppendMarshalDirect[T Marshaler](dst []byte, v T) Same, when the caller owns the destination buffer.
UnmarshalDirect[T Unmarshaler](data []byte, out T) Inverse of MarshalDirect. Falls back to Unmarshal on FlagDense because generated code does not maintain the intern table.

Wire is identical to Marshal / Unmarshal; the path is Fast-mode only (Dense or QPack stay on the reflect path so the intern table is correctly resolved). On the cmd/qdfgen Sample fixture (11-field struct):

Encode ns/op B/op allocs
json 2760 576 8
qdf_reflect 910 480 3
qdf_codegen 800 504 6
qdf_direct 545 160 1

(i7-9750H · Go 1.26 · -count=6 median. Absolute ns are machine-specific; the ratios — qdf_direct ≈ 5× faster than json, ≈ 1.6× faster than the reflect path, at 1 alloc vs json's 8 — are the stable signal.)

Decode-side performance is bounded by the receiver's UnmarshalQDF. The reflect path is heavily pooled (Decoder pool + per-decoder key intern cache); generated code from qdfgen uses Decoder.InternKey and matches it. Hand-rolled receivers that allocate a new Decoder per call will not.

buf, _ := qdf.MarshalDirect(&event)
var out Event
_ = qdf.UnmarshalDirect(buf, &out)
QPack: math-driven slice codecs

QPack auto-selects, per slice, the codec with the smallest predicted wire form. Round-trip is lossless for every type (NaN/±Inf survive on floats).

Codec Trigger Math
Bitpack []bool 1 bit per element, LSB-first per byte. 8× smaller for free.
Raw-LE numeric slice, large delta range Unsafe-slice cast → single memmove of LE bytes. 10-50× faster.
FOR numeric slice, clustered values Frame-of-Reference: store min and ceil(log₂(max-min+1))-bit deltas.
Delta+FOR monotonic / near-monotonic integers Δᵢ = aᵢ - aᵢ₋₁, zigzag bias, FOR over the deltas.
Patched FOR integer slice with rare outliers (latency spikes) FOR body at a reduced width b + an exception list for the few values that don't fit. ~50% smaller than FOR on spiky columns.
Gorilla float slices (explicit opt-in via low-level API) XOR with previous, run-length leading/meaningful-bit window. (Facebook VLDB 2015.)
FSST high-cardinality columnar string column (URLs, log lines, paths) under OptCompression Learns a symbol table of up to 255 substrings (1–8 bytes each); replaces frequent substrings with 1-byte codes. Complements the dictionary codec: dict deduplicates whole strings; FSST compresses at the substring level where cardinality is too high for a dictionary. ~76–79 % wire reduction on URL/log-line corpora. (Boncz/Neumann/Leis, VLDB 2020.)

Head-to-head on a mixed 256-bool / 512-monotonic-u64 / 512-i64 / 256-f64 payload (Intel i7-9750H):

Format Bytes Encode (ns/op) Decode (ns/op)
json 10,739 48,000 200,000
msgpack 11,808 64,000 80,000
qdf_fast 6,694 6,500 14,000
qdf_qpack 2,132 2,300 2,600
qdf_dense 2,134 2,500 2,500

For sequences where the deltas alone collapse, the gain is much larger — e.g. 1024 consecutive Unix-second timestamps shrink from 8201 bytes (raw) to 16 bytes (Delta+FOR), a 512× reduction.

Streaming
var w bytes.Buffer
enc := qdf.NewStreamEncoder(&w, qdf.Dense)
for _, ev := range events {
    if err := enc.Encode(ev); err != nil {
        return err
    }
}
enc.Close()

dec := qdf.NewStreamDecoder(&w)
defer dec.Close()
for {
    var ev Event
    if err := dec.Decode(&ev); err == io.EOF {
        break
    } else if err != nil {
        return err
    }
    // ...
}

Dense streams preserve the intern table across messages — the second occurrence of "region":"eu-west-1" in the batch is a 2-byte reference rather than a 13-byte string.

Each message is length-framed (a uvarint byte-count precedes its body; the 5-byte header is written once up front), so a message of any size round-trips — even across a reader that delivers one byte at a time — and io.EOF cleanly marks the end. Struct tags and all Dense/codec options work exactly as in one-shot Marshal, and dec.SetNoCopy(true) gives zero-copy decode whose aliases stay valid for the stream's lifetime. The encoder reuses one pooled buffer (allocation-free steady state); target a *bytes.Buffer to collect into a []byte. The three whole-batch features — OptColumnIndex, Where/Select pushdown, and OptRANS — are not part of streaming (they act on a single complete batch; use one-shot Marshal/Unmarshal for those).

Selective columnar decode (OptColumnIndex)

qdf transposes a slice of flat structs into a columnar container and compresses it column-by-column. With OptColumnIndex the producer writes a fixed-width column-length index (one uint32 per column, ~4 B each) after the shape declaration. A consumer that only needs some columns then advances past the rest using the index instead of decoding them — cost becomes O(columns you read), not O(all columns).

Two ways to read a subset:

// Producer opts in to the index.
buf, _ := qdf.Marshal(events, qdf.OptBalanced|qdf.OptColumnIndex)

// 1) Typed subset — no new API. Decode into a struct whose fields are a
//    subset of the wire (matched by qdf tag / field name). Wire columns
//    absent from the target are skipped; target fields absent from the
//    wire are left zero.
type Subset struct {
    Level   string `qdf:"level"`
    Service string `qdf:"service"`
}
var sub []Subset
_ = qdf.Unmarshal(buf, &sub)

// 2) Explicit column names — drives a typed subset or the dynamic
//    *[]map[string]any form. With no fields it behaves like Unmarshal.
var rows []map[string]any
_ = qdf.UnmarshalColumns(buf, &rows, "level", "service")

Decoding a 3-field subset of a 16-field columnar batch (i7-9750H):

ns/op B/op allocs/op
full (16 fields) 94,000 283,100 66
subset (3 fields) 16,440 53,610 32

≈ 5.7× faster, ≈5.3× fewer bytes — it decodes 3 columns and skips 13.

Notes: the option only affects payloads the encoder transposes into the columnar container; the flag is backpatched only when the index is actually emitted, so the default columnar wire stays byte-identical when the option is off (and the default non-indexed decode path has no regression). Without the index a subset decode still returns correct results by decoding and discarding unwanted columns — correct, just not fast. The index is a single-message feature and is not emitted in streaming mode.

Predicate pushdown (Where / Select) — the headline feature

This is what no other Go serializer does. On a columnar []struct batch you can filter rows and project columns inside Unmarshal — the decoder tests typed predicates against whole columns, builds a row bitmask, and materializes only the surviving rows of the columns you keep. The rows you filter out are never reconstructed; the columns you don't select are never touched. It is "SELECT ts, code WHERE level='ERROR' AND code>=500" over a qdf blob, with no database, no second pass, and zero per-value boxing.

buf, _ := qdf.Marshal(events, qdf.OptBalanced|qdf.OptColumnIndex)

type Hot struct {
    TS   int64 `qdf:"ts"`
    Code int32 `qdf:"code"`
}
var hot []Hot // filter on level (absent from Hot), keep ts+code of matches
_ = qdf.Unmarshal(buf, &hot,
    qdf.Where("level", func(s string) bool { return s == "ERROR" }),
    qdf.Where("code", func(c int32) bool { return c >= 500 }))

On a wide 9-column, 2000-row batch at 1% selectivity, pushdown moves ≈4.4× fewer bytes and runs ≈2.3× faster than a full decode + manual filter loop — because it never materializes the rows or columns it discards.

This is the unique selling point of qdf versus encoding/json, msgpack, protobuf, gob, and friends — none of them can read back a filtered, projected subset of a batch without first decoding the whole thing. The full rationale, API reference, nullable/error semantics, internals, and a step-by-step tutorial live in docs/PREDICATE-PUSHDOWN.md.

Zero-extra-copy encode

AppendMarshal lets callers own the destination buffer:

out, err := qdf.AppendMarshal(out[:0], v)

Pair with a goroutine-local buffer to drop the per-call allocation entirely.

Zero-copy decode (WithNoCopy)

Decode is allocation-bound: copying each string/[]byte body out of the input buffer dominates. WithNoCopy() makes the decoded values alias the input buffer instead of copying — near-zero allocations, ~1.7× faster on string-heavy batches.

// data must outlive `out` and must NOT be mutated/reused while `out` is in use.
var out LogBatch
_ = qdf.Unmarshal(data, &out, qdf.WithNoCopy())

⚠️ Lifetime contract. The decoded strings/bytes point INTO data. If data is recycled, mutated, or freed before you finish with out, the values silently corrupt — a use-after-free the race detector cannot catch. Never use it on a pooled/reused buffer (e.g. an HTTP request body). Safe for caller-owned, long-lived, immutable input (mmap, a file read into memory, batch analytics). It is opt-in for this reason; the default copies.

Measured on a 1000-row string-heavy batch (i7-9750H): 7002 → 3 allocs/op, −38 % B/op, ~1.7× faster. It works on both the reflect path and codegen types (generated UnmarshalQDFOpts threads the flag through nested decodes). For a Decoder/StreamDecoder you drive yourself, use SetNoCopy(true).

Arena decode (WithArena)

When the wire buffer is recycled (a pooled HTTP request body) WithNoCopy is unsafe — its aliases dangle the moment the buffer returns to the pool. A decode Arena is the safe alternative: it copies the strings out, but into one dense bump-allocated block per epoch instead of one heap allocation per string. The wire buffer is free to go back to its pool immediately, and across a batch/stream the block allocation amortizes to ~zero.

a := qdf.NewArena()                 // one arena per epoch (batch / request)
for i, msg := range messages {
    _ = qdf.Unmarshal(msg, &out[i], qdf.WithArena(a))
}
// out's strings live in `a`; the GC frees it when `out` is dropped — no Reset,
// no lifetime bookkeeping. (Call a.Reset() to reuse it across epochs for zero
// steady-state alloc; see the lifetime contract in docs/ARENA.md.)

Safe by default (strings are owned copies, the GC keeps the block alive via an interior pointer), opt-in, and adaptive — the win scales with string density and is never negative:

Corpus time allocs/op (off → on)
Telemetry log batch (1 000) −35 % 3 502 → 3
AD / directory export (200) −26 % 4 856 → 605
Event batch ([]byte + string) −13 % 1 003 → 504
IoT sensor batch (numeric-heavy) −4 % 291 → 164

Works on the reflect path and codegen types (generated UnmarshalQDFArena threads the arena through nested decodes); drive a Decoder/StreamDecoder directly with SetArena(a). Full lifetime contract, WithNoCopy comparison, and internals in docs/ARENA.md.

Decode into a pooled slice (backing reuse)

The decoded result slice backing is the dominant decode allocation. When you pass a *[]T whose slice already has enough capacity for the row count, the decoder reuses that backing instead of allocating a fresh one — ideal for a server that decodes many messages into a pooled slice:

out := pool.Get().([]Row)[:0] // pre-sized; cap >= incoming row count
_ = qdf.Unmarshal(data, &out)  // reuses out's backing, no result allocation

The reused backing is overwritten in place (elements are zeroed first, so a shorter/older wire schema can't leak stale data), so don't keep another live slice or map reference from the target aliasing it across the call. A nil or too-small destination allocates fresh — default var out []T usage is unchanged, no opt-in flag needed. Works on the row-major and columnar decode paths, for every element type (pointer-free elements take a raw clear, pointer-containing ones a barrier-correct typed clear).

Map recycling. The per-element maps in a reused []struct{…map…} target (or a []map) are the dominant allocation for map-bearing payloads — telemetry tag maps, label sets. Decoding into a reused target now recycles those maps (clears and refills them in place) instead of re-allocating, since make(map) + bucket growth is ~83 % of map-heavy decode bytes. This is automatic and data-adaptive (no flag): it kicks in only when the destination already holds maps, covers maps at any struct-nesting depth, the generated fast paths, the reflect and schemaless (map[string]any) paths, and successive StreamDecoder frames into the same target. Wire is unchanged; nil-vs-empty and stale-key semantics are preserved (recycled maps are cleared before refill). Same contract caveat: a map value you retained from a previous decode of the target is cleared and reused on the next decode into it.

Measured, decode into a pre-sized target (i7-9750H):

element shape B/op (fresh → reuse) ns/op
all-numeric []struct (columnar) 65,815 → 68 (−99.9 %) ~−30 %
all-numeric []struct (row-major) 131,000 → 162 (−99.9 %) ~−9 %
string-bearing telemetry []struct 393,924 → 229,948 (−42 %) ~−9 %
[]struct{map[string]string} (256×16) 341,000 → 25,000 (−92 %) ~−17 %

Code generation

For fixed-schema types where every nanosecond matters, generate reflection-free methods:

go install github.com/alex60217101990/qdf/cmd/qdfgen@latest

//go:generate qdfgen -type Event,User .
type Event struct {
    ID     int    `qdf:"id"`
    Source string `qdf:"source"`
    // ...
}

qdfgen writes <package>_qdf.go with concrete MarshalQDF / UnmarshalQDF methods using only the public qdf API. No reflect.* at runtime. See cmd/qdfgen/README.md for flags and supported types.

On the test fixture (11-field struct with nested struct, slice, map, pointer, fixed array, []byte, time.Time) the generated code is 2.65× faster than encoding/json on encode and 8.49× faster on decode.

Benchmarks

📈 Live trend dashboard: https://alex60217101990.github.io/qdf/dev/bench/

Every push to main re-runs the benchmark suite in CI and appends the result to that dashboard — one chart per benchmark, ns/op (and where shown B/op / allocs/op) over the commit timeline. A curve trending down is a speed/size win; a sustained step up is a regression (a lone spike is shared-runner noise — read the trend, not single points). See docs/BENCHMARKS.md for how to read the graphs. Wire-size numbers (TestCorpusCodec_Sizes) are deterministic, so changes there are real.

The tables below are a point-in-time snapshot — Darwin amd64 / Intel i7-9750H @ 2.6 GHz, Go 1.26.0, -benchtime=2s, -race cross-check. Full numbers, realistic / unique-data scenarios, memory tables and reproduction commands are in docs/BENCH.md; the live dashboard above has the current numbers.

May 2026 perf series (commits ada9fd7, 2ea3b48, 02d6aac, 7090e25, c0517e8, 95d3c21, 001864b) rebuilt the encode + decode hot paths: MRU ring side-cache for MTF rank, flat open-addressed intern table replacing map[string]uint32, packed lruLink, large-payload buffer probe-and-grow, cached decode interning (decState.stringValues), 4-way mruRank unroll, opt-in Encoder.PreIntern API, and the qdfgen codegen path wired into the bench matrix. qdf is now strictly faster than msgpack on encode AND decode for every workload in the matrix (-5 % HotPath, -40 % TelemetryBatch, -96 % MetricSeries on encode; -55…-98 % on decode). See docs/GUIDE.md for the up-to-date Profile_* matrix.

Encode — ns/op
Payload json msgpack qdf_fast qdf_dense vs json vs msgpack
Tiny ({id,name}) 192 286 227 398 0.85× 1.26×
Flat (20 primitive fields) 1182 1262 480 948 2.46× 2.63×
Nested (4 deep) 446 793 331 786 1.35× 2.40×
Deep linked-list (16) 1157 3060 477 814 2.43× 6.42×
Wide × 1000 991 k 991 k 213 k 289 k 4.66× 4.66×
Log batch × 1000 998 k 624 k 186 k 365 k 5.36× 3.35×
Map-heavy (40 entries) 8446 5282 1391 2019 6.07× 3.80×
[]float32 × 512 30 054 18 270 1821 – 16.5× 10.0×
[]float64 × 512 39 819 23 856 2450 – 16.3× 9.74×
UniqueLog (fresh per iter) 2419 2080 1510 – 1.60× 1.38×
Decode — ns/op
Payload json msgpack qdf_fast vs json vs msgpack
Tiny 729 383 170 4.29× 2.25×
Flat 4411 1862 1122 3.93× 1.66×
Nested 2483 1206 425 5.84× 2.84×
Deep16 7363 4170 1972 3.73× 2.11×
Wide × 1000 4.40 M 1.83 M 990 k 4.44× 1.85×
Log batch × 1000 3.29 M 1.32 M 449 k 7.31× 2.93×
Map-heavy (40 entries) 16 621 8096 3108 5.35× 2.61×
[]float32 × 512 72 928 28 521 3960 18.4× 7.20×
UniqueLog (fresh bytes) 3569 1296 509 7.01× 2.55×
UniqueLog (RunParallel) 667 682 487 1.37× 1.40×
Memory — bytes per decode
Payload json msgpack qdf_fast vs json vs msgpack
Tiny 248 77 29 0.12× 0.38×
Nested 664 160 112 0.17× 0.70×
Log batch × 1000 442 536 407 698 251 838 0.57× 0.62×
Map-heavy 4912 3089 2359 0.48× 0.76×
[]float32 × 512 4384 4282 2113 0.48× 0.49×
MapStringAny (rep. keys) 790 – 345 0.44× –
Allocations per decode
Payload json msgpack qdf_fast
Nested 15 6 5
Map-heavy (40 entries) 124 112 32
MapStringAny (rep. keys) 37 – 3
[]float32 × 512 16 8 3
Decode arena (WithArena)

Opt-in qdf_fast decode with a per-epoch Arena, measured off → on over an epoch loop (i7-9750H, reflect path, Reset per message). The arena copies string bodies into one dense block instead of one allocation each; the win scales with string density and never regresses.

Corpus ns/op (off → on) allocs/op (off → on)
Telemetry log batch × 1000 258 k → 167 k −35 % 3 502 → 3
AD / directory export × 200 450 k → 335 k −26 % 4 856 → 605
Event batch × 500 ([]byte + str) 116 k → 101 k −13 % 1 003 → 504
IoT sensor batch (numeric-heavy) 102 k → 98 k −4 % 291 → 164
LogEntry single, RunParallel 248 → 205 −17 % 8 → 2

[]byte fields stay copy-only (caller-mutable); numeric-heavy payloads see a small win and no regression. Lifetime contract in docs/ARENA.md.

Encoded size
Payload json msgpack qdf_fast qdf_dense dense vs json
Flat 210 134 132 138 0.66×
Deep16 239 139 166 63 0.26×
Wide × 1000 212 901 135 626 128 632 66 702 0.31×
Log batch × 1000 251 902 185 639 185 649 85 440 0.34×

On a 1000-entry Dense stream the same log batch encodes to 85 bytes per entry, against 251 for json and 186 for msgpack — shared intern table across messages plus per-record struct headers collapsed via shape interning (tagMapShape).

Realistic corpus

Numbers from realistic_corpus_test.go builders that mirror real telemetry workloads. Full breakdown plus encode latency in docs/BENCH.md.

TelemetryBatch (1 000 log events, repeating service / region / level)
Format bytes vs json encode ns/op
json 252 497 1.00× —
qdf_fast 186 674 0.74× 272 k
qdf_qpack 186 674 0.74× 261 k
qdf_dense 50 129 0.20× 1.0 M

Dense pays ~4× CPU for 5.0× smaller wire vs JSON — string-intern

  • Markov-0 + MTF + Markov-1 pair + shape interning crush the repeating service / region / level / host fields AND elide per-record struct headers. QPack does not help much here (per-event numeric fields are scalar, not slice-shaped).
MetricSeries (1 024 numeric timestamps + values)
Format bytes vs json
json 30 043 1.00×
qdf_fast 14 442 0.48×
qdf_qpack 8 307 0.28×
qdf_dense 8 315 0.28×

Here QPack pulls its weight: []int64 timestamp column is monotonic and Delta+FOR compresses it to near-zero bytes per element; the []float64 value column rides raw-LE bulk. Dense and QPack converge because string overhead is tiny.

Large payload (~150 MiB, 200 000 records, every supported type)

bench/largepayload_test.go builds a 200 000-record corpus that exercises every qdf-supported field type (scalars, hot/cold strings, nested map, []int32 / []float64, []byte, UUIDs) and measures size + encode/decode latency + working-set memory delta across json, msgpack, qdf_fast, qdf_qpack, qdf_dense.

Sizes (200 000 records):

Format bytes (MiB) vs json
json 142.10 1.00×
msgpack 92.99 0.65×
qdf_fast 91.99 0.65×
qdf_qpack 89.65 0.63×
qdf_dense 71.19 0.50×

Latency + memory (100 000 records):

Format encode (ms) decode (ms) encode heap delta
json 1 070 1 744 199 MiB
msgpack 296 597 64 MiB
qdf_fast 142 300 94 MiB
qdf_qpack 147 231 93 MiB
qdf_dense 169 216 9.7 MiB

Reproduce:

go test -C bench -run TestSizes_LargePayload -count=1 -timeout=10m
go test -C bench -run TestMem_LargePayload   -count=1 -timeout=10m

Both helpers skip under -short; the size test allocates ~600 MiB during the run, the mem test ~400 MiB.

Reproduce
# Default build
cd bench
go test -bench=. -benchmem -benchtime=2s -timeout=10m

# QPack head-to-head: Marshal(_, OptSpeed) vs Marshal(_, OptQPack)
# vs Marshal(_, OptBalanced) vs json / msgpack on the same payload
go test -bench='BenchmarkQPack_' -benchmem

# QPack micro-benchmarks (codec internals, in the root module)
cd ..
go test -bench='BenchmarkQPack' -benchmem

# AVX2 bit-unpack (asm, requires CPUID AVX2 at run time)
go test -tags qdf_simd -bench='BenchmarkBitUnpackFast' -benchmem

# Realistic unique-data scenarios
cd bench
go test -bench='BenchmarkEncode_UniqueLog|BenchmarkDecode_UniqueLog|BenchmarkEncode_MixedTypes|BenchmarkEncode_RandomSize' -benchmem

# Encoded sizes
go test -run TestSizes -v

# Codegen vs reflect on the Sample fixture
cd ../internal/codegen_test
go test -run TestGenerate .
go test -bench=. -benchmem -benchtime=2s
Profile-guided optimisation (downstream)

Go's PGO applies to the main package being built, so a library like qdf cannot ship its own profile. If you build a service that imports qdf, collect a representative profile of your workload and drop it next to your main:

# Run your service / load test under cpuprofile.
go test -bench=. -cpuprofile=cpu.pprof -benchtime=10s ./...
mv cpu.pprof default.pgo            # same dir as your main package
go build .                          # auto-picks up default.pgo

The Go toolchain will then recompile the qdf functions on the hot path with PGO-driven inlining and devirtualisation. Typical gain is 5–15 % across the encode/decode pipeline on top of the numbers above.

Build tags

Opt-in fast paths. None are required for the headline numbers; the defaults already include the specialised slice / map encoders, the QPack codecs, and a 128-bit sliding-window bit-unpacker for FOR / Delta+FOR.

Tag Effect Build prerequisite
qdf_reflect2 Swap reflect.MakeSlice / MakeMapWithSize for modern-go/reflect2 unsafe equivalents. Smaller decode allocations on map / slice heavy workloads. none — pure Go
qdf_simd SIMD fast paths for the QPack integer/bool codecs. amd64 (AVX2): decode bits ∈ {8,16,32} (VPMOVZX) and every width 1–28 (VPBROADCASTQ+VPSRLVQ), encode {8,16,32} (VPSHUFB) and {10,12,14,20} (VPSLLVQ+lane-OR), []bool pack; runtime CPUID gate, non-AVX2 falls back. arm64 (NEON): decode 1–28 plus 32 (encode/bool scalar for now), baseline. ~3-11× over pure-Go on accelerated widths. Output byte-identical to scalar; other arches compile a stub. See docs/USAGE.md. amd64 (AVX2) / arm64 (NEON) 
go build -tags qdf_reflect2 ./...
go build -tags qdf_simd ./...
go build -tags "qdf_simd qdf_reflect2" ./...   # combined

The qdf_simd path only changes the speed of QPack-encoded numeric and bool slices at the accelerated widths; string / map / struct paths, the float codecs, and other bit widths run the pure-Go path unchanged. It is a pure speed switch — the wire format and output are identical.

Correctness

The test suite runs under -race and covers:

  • Primitive round-trip across every wire-tag boundary.
  • Boundary integers from math.MinInt64 to math.MaxUint64.
  • Strings at the fixstr / str8 / str16 / str32 boundaries (0, 1, 31, 32, 255, 256, 65535, 65536, 1 MiB) plus Unicode and invalid UTF-8.
  • Every specialised fast path against the generic reflect path.
  • Truncated input — every prefix from 0 to N-1 decodes to an error, never a panic.
  • Bad magic / bad version — errors, not panics.
  • Cross-mode interop (Fast bytes decoded via Dense decoder and vice versa).
  • Streaming with mixed message types under Dense intern.
  • Concurrency: 32 × 500 marshal+unmarshal goroutines under -race.
  • QPack codecs: each codec (bitpack-bool, raw-LE, FOR, Delta+FOR, Gorilla) has its own round-trip suite covering edge cases — empty slices, single elements, near-MaxUint64, MinInt64, NaN / ±Inf / ±0 / denormals, monotonic / constant / mixed-direction sequences.
  • TestCompleteness_AllModes runs one big payload (every QPack- eligible type, every edge case, nested structs, maps, string interning) through OptSpeed, OptQPack, and OptBalanced, then asserts bit-for-bit IEEE-754 equality for floats and reflect.DeepEqual elsewhere.
  • TestCompleteness_StreamingDense exercises three messages through NewStreamEncoder in Dense mode (carries QPack + shared intern table); TestCompleteness_FuzzRandomStructsQPack generates 50 random struct shapes and round-trips them through all three encoders.
  • AVX2 bit-unpack (under qdf_simd) has parity tests against scalar zero-extend for n in {0,1,2,3,4,5,7,8,9,15,16,17,1000,4096} at bits ∈ {8, 16, 32}, plus a bitPackU64LE → bitUnpackU64LE round- trip.
  • tagStateRepeat (Markov-0 state-ref predictor): tests that the predictor fires on runs of identical interned values, does not fire on alternation, invalidates correctly across an inline-string emission, stays in sync with Decoder.Skip, and round-trips through the public Marshal(v, OptBalanced) / Unmarshal boundary.
  • Property-based round-trip fuzzers (fuzz_property_test.go) drive a deterministic value generator from the fuzz bytes, encode through every Marshal entry point, and assert Unmarshal(Marshal(v)) == v. Targets cover Int64Slice, Uint64Slice, Float64Slice, BoolSlice, MapStringInt, StructTriad, and an AllModesAgree fuzzer that asserts the three encoder dialects decode to the same Go value. Caught one Markov-0 predictor bug in encodeStruct (commit dfc30ae).
  • Golden-file wire pinning (testdata/golden/*.bin, golden_test.go): every representative payload has its OptSpeed / OptQPack / OptBalanced bytes committed to disk. A later wire-format change either matches the bytes byte-for-byte or fails the test. Map-shaped cases skip the byte-pin half (Go map iteration is randomised) but keep the decode-and-compare half. Regenerate intentionally via go test -run TestGolden -update.
  • Deterministic truncation matrix (truncation_test.go, ~2700 sub-tests): for every representative payload, every prefix payload[:i] is fed through Unmarshal into six destination types, every 4th byte is mutated to one of nine "bad" alternatives (00, FF, 80, 55, AA, and every QPack tag), and malformed 5-byte headers are exhaustively rejected. The decoder must return a typed error in every case; panic budget is zero.
  • Unknown-field skip (unknown_field_test.go): encoder writes a 10-field struct, decoder declares only 3; subsequent declared fields still decode correctly. Each of the 10 fields is also pulled out alone via its own single-field destination type.
  • Pathological-shape stress (stress_nesting_test.go): 256-deep linked list, 512-key map, balanced binary tree (depth 12), 1 MiB string + 1 MiB []byte + 100k-element u64/f64 vectors, and a 2000-deep chain under debug.SetMaxStack(64 MiB).
  • Realistic payload corpus (realistic_corpus_test.go): telemetry batches with repeating service/region/level fields, metric time-series, wide column-store rows with recursive Children — round-tripped through every Marshal entry point and cross-checked for three-way agreement.
  • Allocation-budget tests (alloc_budget_test.go): each hot entry point has an upper-bound allocation budget enforced via testing.AllocsPerRun. A regression in alloc count fails CI rather than silently bloating downstream callers.
  • Tag and named-type matrix (tag_matrix_test.go): every primitive type is exercised through a named alias (type MyInt int64), the tag fallback chain qdf → json → field name is verified, the qdf:"-" skip directive is checked via a map[string]any decode, and embedded-struct (non-)flattening is pinned.
  • Differential testing (bench/diff_test.go): qdf round-trips agree semantically with encoding/json and github.com/vmihailenco/msgpack/v5 on a shared payload, including NaN / ±Inf / ±0 preservation against msgpack.
  • Pointer-cycle detection via depth counter: the encoder increments Encoder.depth on every pointer dereference and returns ErrCycleDetected once the count exceeds DefaultMaxDepth (10000). Cheaper than a per-pointer set (no allocation per call) and catches both genuine *T → *T cycles and pathologically deep payloads. Override the cap via Encoder.SetMaxDepth.
  • OOM protection on every length-prefixed read path: a hostile varuint encoding a value > 2^62 cannot drive the decoder cursor negative because the byte-count check happens in uint64 against (len(d.buf) - d.i) * 8 / bitsPer before any signed cast. Applies to tagPackBool/Raw/For/DeltaFor/Gorilla in both the slice-read and Skip paths.
  • Stream Close idempotency: both StreamEncoder.Close and StreamDecoder.Close are safe to call multiple times.
  • Fuzz: FuzzDecoder_NeverPanics, FuzzRoundTrip_StringSlice, FuzzQPackBool, FuzzQPackRawUint64, FuzzRoundTrip_Int64Slice, FuzzRoundTrip_Uint64Slice, FuzzRoundTrip_Float64Slice, FuzzRoundTrip_BoolSlice, FuzzRoundTrip_MapStringInt, FuzzRoundTrip_StructTriad, FuzzRoundTrip_AllModesAgree — persistent corpus under testdata/fuzz/. 10 M+ executions clean across the suite after the Skip-overflow and encodeStruct-predictor fixes; both repros are saved under testdata/fuzz/.

Length prefixes are validated against the remaining buffer (Decoder.CheckLength) before any make, so a hostile payload claiming a multi-billion-element map cannot OOM the process. The QPack-tag Skip paths perform the same overflow-safe check on uint64 element counts before any signed cast, so a 10-byte varuint encoding a value > 2^62 cannot drive the decoder cursor negative.

go test -race -count=1 ./...

# Property-based round-trip fuzz (each fuzzer asserts
# Unmarshal(Marshal(v)) == v on randomly generated values)
go test -run=^$ -fuzz=FuzzRoundTrip_StructTriad     -fuzztime=60s
go test -run=^$ -fuzz=FuzzRoundTrip_AllModesAgree   -fuzztime=60s
go test -run=^$ -fuzz=FuzzRoundTrip_Int64Slice      -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_Uint64Slice     -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_Float64Slice    -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_BoolSlice       -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_MapStringInt    -fuzztime=30s

# Decoder safety fuzz (asserts never-panic on hostile input)
go test -run=^$ -fuzz=FuzzDecoder_NeverPanics       -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_StringSlice     -fuzztime=30s
go test -run=^$ -fuzz=FuzzQPackBool                 -fuzztime=30s
go test -run=^$ -fuzz=FuzzQPackRawUint64            -fuzztime=30s

# Differential vs msgpack and encoding/json
go test -C bench -run=TestDiff -count=1

# Regenerate wire-format golden fixtures (intentional after a wire
# bump only)
go test -run=TestGolden -update

# Build-tag combinations
go test -tags qdf_reflect2          -race ./...
go test -tags qdf_simd              -race ./...
go test -tags "qdf_simd qdf_reflect2" -race ./...

Architecture diagrams

Mermaid diagrams (GitHub renders them natively) covering every layer of the format — concept overview, wire layout, codec picker, Dense interning, columnar/selective-decode, and the algorithmic performance wins.

diagrams/README.md — start here: concept overview flowchart and index of all diagrams.

Diagram Topic
architecture.md Marshal/Unmarshal end-to-end: pool, typeDesc cache, mode dispatch, rANS pass
wire-format.md 5-byte header, flags bit map, full tag space
options-and-modes.md Options bits, Speed/Balanced/Compression bundles, Fast vs Dense decision
qpack-codecs.md Codec picker (FOR/Delta/RLE/dict/PFOR/Gorilla/ALP), never-larger floor, SIMD
dense-interning.md Intern table, state-ref predictors (Markov-0/MTF/Markov-1/shape)
columnar-and-selective-decode.md []struct transpose, Time split, Nullable slab, colIndex, 3VL pushdown
performance.md 10 algorithmic wins grouped by CPU / wire size / memory

Status

Alpha. The wire format is stable for the 0x01 version byte; future versions will bump it. The public API may change before the first tagged release — pin a commit if you depend on it.

License

MIT. See LICENSE.

Documentation

Overview

Package qdf is a compact, streaming-friendly binary serialization format.

Quick start

One encode entry point: Marshal(v, opts). The Options bit-mask picks which codecs run for that call. Convenience bundles cover the common tradeoffs: 

OptSpeed       — Fast mode, no codecs (matches encoding/json shape)
OptBalanced    — Dense + QPack + shape interning + Markov-1 + MTF
OptCompression — OptBalanced + heavier float codecs: Gorilla XOR for
                 smooth time-series and ALP for quantized/decimal
                 float64 (large wire reduction at higher encode CPU)

A single decoder handles every variant; the wire header self- describes the dialect. 

b, err := qdf.Marshal(v, qdf.OptSpeed)        // hot path
b, err := qdf.Marshal(v, qdf.OptBalanced)     // telemetry / logs
b, err := qdf.Marshal(v, qdf.OptCompression)  // backup / archive

err := qdf.Unmarshal(b, &v)

enc := qdf.NewStreamEncoder(w, qdf.OptBalanced)
dec := qdf.NewStreamDecoder(r)

Marshal returns a freshly-allocated slice owned by the caller. AppendMarshal is the zero-extra-copy variant.

Data shapes and codecs

Under OptBalanced the encoder picks a codec per slice from the data itself: integer slices use Frame-of-Reference, delta, run-length or dictionary coding; bool slices bit-pack; repeated strings are interned and back-referenced. A slice of homogeneous flat structs is transposed and compressed column-by-column automatically — numeric columns get the integer codecs, an enum-like string column is dictionary-coded (distinct table + a bit-packed index per row) when that beats per-value interning, an optional (*T, T a scalar or string) field becomes a presence bitmap plus a dense present-only column instead of forcing the struct back to row-major, other repeated string columns collapse — with no flag, and a per-array probe falls back to the plain row encoding when columnar would not help. The float codecs that trade encode CPU for size live behind OptCompression: Gorilla XOR on smooth series, ALP on quantized/decimal float64 grids (quantized telemetry, prices, latencies).

Concurrency

Marshal, Unmarshal and AppendMarshal are safe for concurrent use: each call leases its own encoder/decoder from a pool. A single Encoder, Decoder or StreamEncoder value is not safe to share across goroutines — use one per goroutine. A Dense document is a sequential stream, so one document cannot be encoded or decoded in parallel; to parallelize a large dataset, split it into independent shards and Marshal each separately.

Selective decode

Under OptColumnIndex a columnar []struct payload carries a fixed-width uint32 column-length index, written after the shape declaration and before the column bodies (~4 bytes per column). A reader wanting only some columns then advances past the rest using the index instead of decoding them, so the cost becomes O(columns you read), not O(all columns). Two entry points: decode into a subset struct whose fields are a subset of the wire (matched by qdf tag / field name) with plain Unmarshal — wire columns absent from the target are skipped, target fields absent from the wire are left zero — or name the columns explicitly with UnmarshalColumns (also drives the dynamic *[]map[string]any form). The option is a true no-op on non-columnar payloads: the flag is backpatched only when the index is actually emitted, so the default columnar wire stays byte-identical when it is off. Without the index a subset decode still returns correct results by decoding and discarding unwanted columns. The index is a single-message feature and is not emitted in streaming mode.

Predicate pushdown

Unmarshal accepts trailing QueryOption arguments that filter and project a columnar []struct payload before it is materialized. Where(field, pred) keeps only the rows for which a typed predicate holds — func(T) bool over the column's element type, AND-ed across multiple Where clauses, with zero per-value boxing — and Select(fields...) restricts the output to a named subset of columns. The filter field need not appear in the output type, and the result can be either *[]Struct or *[]map[string]any. The decoder reads each predicate column whole, evaluates it into a row bitmask, ANDs the masks, then compacts and materializes only the surviving rows of the projected columns; OptColumnIndex lets it seek past skipped columns instead of decode-and-discard. A nullable column's nil rows never match. Pushdown fires only on columnar payloads — a non-columnar payload (a single struct, or a batch the columnar probe declined) returns an error wrapping ErrUnsupported; a missing field yields ErrFieldNotFound and a predicate whose argument type mismatches the column yields ErrTypeMismatch, both as a *QueryError (errors.Is / errors.As). Versus a full decode followed by a manual loop, pushdown moves materially fewer bytes (about 4x less on a wide batch with 1% selectivity) and runs roughly 2x faster. No other Go serializer reads back a filtered, projected subset of a batch without decoding the whole thing — see docs/PREDICATE-PUSHDOWN.md for the full guide, rationale, and tutorial.

See docs/USAGE.md in the repository for a fuller guide.

Public API surface

The package contract is split across four layers; everything else under internal/ is implementation detail and may change between releases without notice.

Top-level entry points (file: qdf.go): 

func Marshal(v any, opts Options) ([]byte, error)
func AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)
func Unmarshal(data []byte, out any) error
type Options uint32   // bit-mask of OptDense, OptQPack, OptMTF, …

Selective columnar decode (file: columnar_select.go): 

func UnmarshalColumns(data []byte, out any, fields ...string) error

Predicate pushdown — filter rows and project columns on a columnar []struct, AND-ed typed predicates with zero per-value boxing (file: query.go): 

func Unmarshal(data []byte, out any, opts ...QueryOption) error
func Where[T Queryable](field string, pred func(T) bool) QueryOption
func Select(fields ...string) QueryOption

Typed convenience wrappers — generic, zero-extra-reflection (file: qdf_generic.go): 

func MarshalT[T any](v T, opts Options) ([]byte, error)
func AppendMarshalT[T any](dst []byte, v T, opts Options) ([]byte, error)
func UnmarshalT[T any](data []byte) (T, error)

Low-level encoder / decoder for callers driving the wire directly or interoperating with the qdfgen code generator (files: encoder.go, decoder.go, stream.go): 

type Encoder struct{ … }
    NewEncoder(mode Mode) *Encoder
    NewEncoderWith(opts Options) *Encoder
    NewEncoderOnBuf(buf []byte, mode Mode) *Encoder
    (e *Encoder) WriteString / WriteBytes / WriteInt / WriteUint /
                WriteFloat32 / WriteFloat64 / WriteBool / WriteNil /
                WriteArrayHeader / WriteMapHeader /
                WriteTimestamp / WriteStringInline
    (e *Encoder) AppendBytes / EnsureHeader / Bytes / Take /
                Reset / SetBuffer / AdoptBuffer / SetIntern /
                SetMaxDepth / SetQPack / QPack / ApplyOpts /
                EncodeValue / PreIntern
type Decoder struct{ … }
    NewDecoder() *Decoder
    NewDecoderOnBuf(buf []byte) *Decoder
    (d *Decoder) ReadString / ReadStringBytes / ReadBytes /
                ReadBool / ReadInt / ReadUint / ReadFloat32 /
                ReadFloat64 / ReadNil / ReadArrayHeader /
                ReadMapHeader / ReadTimestamp / Skip /
                PeekTag / IsNil / Pos / Remaining / RemainingBytes /
                Advance / SetInput / SetNoCopy / MarkHeaderRead /
                CheckLength / InternKey
type StreamEncoder, type StreamDecoder         // io.Writer / io.Reader

User-side hook points (file: marshaler.go): 

type Marshaler interface   { MarshalQDF(dst []byte) ([]byte, error) }
type Unmarshaler interface { UnmarshalQDF(src []byte) (int, error) }

The qdfgen code generator (cmd/qdfgen) emits MarshalQDF / UnmarshalQDF methods for user struct types; the codegen path uses the same Encoder / Decoder primitives listed above and requires no reflection at runtime.

Example

Example demonstrates the simplest possible round-trip: Marshal a value into a freshly-allocated byte slice and Unmarshal it back into a typed receiver. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		ID     int    `qdf:"id"`
		Source string `qdf:"source"`
	}

	in := Event{ID: 42, Source: "ingest"}
	buf, err := qdf.Marshal(in, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var out Event
	if err := qdf.Unmarshal(buf, &out); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", out)
}

Output:
{ID:42 Source:ingest}
Example (RoundTripSlice)

Example_roundTripSlice demonstrates encoding a slice of structs. QDF's reflect path resolves the element type once per typeDesc; every element after the first emits its tag stream straight from the cached descriptor. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	rows := []Row{
		{"auth", 200},
		{"billing", 503},
		{"auth", 200},
	}

	buf, _ := qdf.Marshal(rows, qdf.OptBalanced)

	var got []Row
	_ = qdf.Unmarshal(buf, &got)
	for _, r := range got {
		fmt.Printf("%s=%d\n", r.Service, r.Status)
	}
}

Output:
auth=200
billing=503
auth=200

Index

Examples

Constants

View Source 
const (
	Magic0   byte = 'Q'
	Magic1   byte = 'D'
	Magic2   byte = 'F'
	Version1 byte = 0x01
)
View Source 
const (
	FlagDense byte = 1 << 0
	// FlagQPack signals that the encoder may emit QPack codec tags
	// (0xE3..0xEF). A decoder that does not recognize the tags will fail
	// with ErrBadTag on the first packed slice; the flag is an early hint
	// so callers can refuse the buffer up front.
	FlagQPack byte = 1 << 1

	// FlagRANS signals that the body (everything after the 5-byte header) is
	// rANS-compressed: varuint(origLen) + 256-entry frequency table +
	// rANS stream. The decoder reverses it before reading tags. Set only
	// when the rANS form is strictly smaller than the plain body.
	FlagRANS byte = 1 << 2

	// FlagColIndex marks a tagColStruct (columnar []struct) payload that
	// carries a fixed-width column-length table right after the shape
	// declaration and before the column bodies. The table is K little-endian
	// uint32 entries (one per column, K = number of columns), each the byte
	// length of the corresponding column body. It lets a reader skip columns
	// it does not need without decoding them. Opt-in (~4 bytes per column);
	// only meaningful when the value is columnar — for any other top-level
	// shape the bit is set but no index block exists.
	FlagColIndex byte = 1 << 3
)

Flag bits in the 5th header byte.

View Source 
const DefaultMaxDepth = 10_000

DefaultMaxDepth caps reflect-path pointer/struct recursion. Set large enough for any legitimate payload (10 000) while still rejecting genuine cycles before the goroutine stack overflows.

View Source 
const TagNil = tagNil

TagNil exposes the nil-value tag for comparison with PeekTag.

Variables

View Source 
var (
	ErrShortBuffer    = errors.New("qdf: short buffer")
	ErrBadMagic       = errors.New("qdf: bad magic / not a qdf stream")
	ErrBadVersion     = errors.New("qdf: unsupported wire version")
	ErrBadTag         = errors.New("qdf: unknown tag")
	ErrTypeMismatch   = errors.New("qdf: type mismatch on decode")
	ErrInvalidLength  = errors.New("qdf: invalid length prefix")
	ErrUnknownStateID = errors.New("qdf: unknown state-table id")
	ErrUnsupported    = errors.New("qdf: unsupported type")
	ErrCycleDetected  = errors.New("qdf: pointer cycle detected (max depth exceeded)")
	ErrFieldNotFound  = errors.New("qdf: query predicate field not found")
	// ErrStreamBadFlags is returned by StreamDecoder.Decode when the stream
	// header carries whole-payload flags that do not apply to a frame stream
	// (FlagRANS / FlagColIndex). A conforming StreamEncoder never sets them; a
	// hostile stream claiming them would, if honored, swap the shared window
	// buffer mid-stream and desync framing. The decoder is latched broken.
	ErrStreamBadFlags = errors.New("qdf: stream header has unsupported whole-payload flags")
)
View Source 
var ErrStreamBroken = errors.New("qdf: stream encoder broken by a prior mid-message error")

ErrStreamBroken is returned by StreamEncoder.Encode after a previous Encode failed mid-message: the cross-message encoder state can no longer be trusted, so the stream must be abandoned (the valid prefix may still be flushed).

View Source 
var ErrStreamDecoderBroken = errors.New("qdf: stream decoder broken by a prior mid-message error")

ErrStreamDecoderBroken is returned by StreamDecoder.Decode after a previous Decode failed mid-message: the read cursor and the shared dense state can no longer be trusted (subsequent frames would misparse), so the stream must be abandoned.

Functions

func AppendMarshal 

func AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)

AppendMarshal encodes v and appends the result to dst. Reuse the returned slice as dst on the next call to avoid per-message allocations.

Example

ExampleAppendMarshal shows the zero-extra-copy form. The caller keeps a reusable byte buffer and asks Marshal to append into it, returning the extended slice. Combined with a sync.Pool of buffers this drives encode allocations to zero on the hot path. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Tick struct {
		Ts    int64   `qdf:"ts"`
		Price float64 `qdf:"price"`
	}

	buf := make([]byte, 0, 64) // caller-owned, reusable
	t := Tick{Ts: 1_700_000_000, Price: 99.5}

	var err error
	buf, err = qdf.AppendMarshal(buf, t, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var out Tick
	if err := qdf.Unmarshal(buf, &out); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", out)
}

Output:
{Ts:1700000000 Price:99.5}

func AppendMarshalDirect 

func AppendMarshalDirect[T Marshaler](dst []byte, v T) ([]byte, error)

AppendMarshalDirect appends the serialisation of v to dst.

func AppendMarshalT 

func AppendMarshalT[T any](dst []byte, v T, opts Options) ([]byte, error)

AppendMarshalT is the generic equivalent of AppendMarshal.

func Marshal 

func Marshal(v any, opts Options) ([]byte, error)

Marshal encodes v with the given option bit-mask. Combine bits with | to opt into individual codecs, or use one of the OptSpeed / OptBalanced / OptCompression bundles. 

b, _ := qdf.Marshal(event,    qdf.OptSpeed)         // hot path
b, _ := qdf.Marshal(batch,    qdf.OptBalanced)      // telemetry
b, _ := qdf.Marshal(snapshot, qdf.OptCompression)   // backup
b, _ := qdf.Marshal(payload,                        // tuned
    qdf.OptDense|qdf.OptQPack|qdf.OptShapeIntern)

Big-output detach (in marshalDict): cloning a multi-megabyte buffer to hand the caller their own copy used to dominate Large-payload profiles (slices.Clone + runtime.memmove). Above marshalDetachThreshold the pool would drop the buffer in putEnc anyway (cap exceeds maxPooledBuf), so handing the original to the caller and leaving the pool encoder with a nil buf is strictly cheaper. Small payloads stay on the clone path so the warm 4 KiB pool buffer survives.

func MarshalDirect 

func MarshalDirect[T Marshaler](v T) ([]byte, error)

MarshalDirect serialises a value through its MarshalQDF method. Skips the public Marshal entry point's any-boxing, reflect.New, and descriptor lookup. Roughly 2-4× faster than Marshal for generated types on small payloads, with one fewer allocation per call.

v must be non-nil: MarshalDirect calls v.MarshalQDF directly, so a nil pointer-receiver value panics. Use Marshal, which emits an explicit nil, if the value may be nil.

func MarshalT 

func MarshalT[T any](v T, opts Options) ([]byte, error)

MarshalT is the generic equivalent of Marshal. T is fixed at the call site; opts copies by value, so the call adds zero heap allocations over MarshalT itself.

Example

ExampleMarshalT shows the generic typed wrapper. T is fixed at the call site, so the compiler skips the reflect.TypeOf step that Marshal(v any, opts) pays inside; encode/decode of value types stays on the stack. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Point struct {
		X int32 `qdf:"x"`
		Y int32 `qdf:"y"`
	}

	buf, err := qdf.MarshalT(Point{X: 3, Y: 4}, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var p Point
	if err := qdf.UnmarshalT(buf, &p); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", p)
}

Output:
{X:3 Y:4}

func Unmarshal 

func Unmarshal(data []byte, out any, opts ...QueryOption) error

Unmarshal decodes data into out, which must be a non-nil pointer. The wire dialect is detected from the header — the same Unmarshal reads any output produced by Marshal regardless of the encode-side option bits.

Slice-backing reuse: when out is a *[]T whose slice already has enough capacity for the decoded element count, the decoder reuses that backing array instead of allocating a new one — eliminating the result allocation (the dominant decode cost) on a decode into a pooled / pre-sized slice. The reused backing is overwritten in place (its elements are zeroed first, so no stale data leaks), so do not share it with other live slices across the call. A nil or too-small destination allocates fresh, so default usage is unaffected.

The optional QueryOptions (Where / Select) turn the call into a filtering/projecting columnar decode: predicates are AND-ed and the named columns projected. They apply only to a columnar []struct, []map[string]any or []any payload; on any other shape a query call returns a *QueryError wrapping ErrUnsupported. With no options the behavior is exactly the plain decode above.

Round-trip fidelity: decoding into the same concrete Go type reproduces the value exactly, including the nil-vs-empty distinction for slices, maps and pointers (a nil []T decodes as nil, an empty []T{} as empty — like encoding/json's null vs []). Two deliberate normalizations are NOT bit-exact: decoding into an interface (any / map[string]any) canonicalizes integers to int64 / uint64 and structs to map[string]any (the schemaless type contract), and a time.Time loses any monotonic-clock reading on encode (as the standard library strips it for transport). Values nested deeper than the configured max depth, and maps with both int and uint keys of the same small value, are the documented exceptions.

Example (PredicatePushdown)

ExampleUnmarshal_predicatePushdown keeps only the rows matching typed predicates (AND-ed), decoding just the projected columns of those rows.

Predicate pushdown only fires when the payload is columnar, which the encoder picks for a sufficiently large []struct batch, so this example builds a 20-row batch rather than a handful of literals. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		TS    int64  `qdf:"ts"`
		Level string `qdf:"level"`
		Code  int32  `qdf:"code"`
	}
	in := make([]Event, 0, 20)
	for i := range 20 {
		lvl, code := "INFO", int32(200)
		switch i % 5 {
		case 0:
			lvl, code = "ERROR", 500
		case 1:
			lvl, code = "ERROR", 404
		case 2:
			lvl, code = "WARN", 503
		}
		in = append(in, Event{TS: int64(i + 1), Level: lvl, Code: code})
	}
	buf, _ := qdf.Marshal(in, qdf.OptBalanced|qdf.OptColumnIndex)

	type Out struct {
		TS   int64 `qdf:"ts"`
		Code int32 `qdf:"code"`
	}
	var hot []Out // filter on level (not in Out), return ts+code
	_ = qdf.Unmarshal(buf, &hot,
		qdf.Where("level", func(s string) bool { return s == "ERROR" }),
		qdf.Where("code", func(c int32) bool { return c >= 500 }))

	for _, e := range hot {
		fmt.Printf("ts=%d code=%d\n", e.TS, e.Code)
	}
}

Output:
ts=1 code=500
ts=6 code=500
ts=11 code=500
ts=16 code=500

func UnmarshalColumns added in v0.0.2 

func UnmarshalColumns(data []byte, out any, fields ...string) error

UnmarshalColumns decodes only the named columns of a columnar []struct payload into out, leaving the rest undecoded. out may point at a typed subset slice (e.g. *[]Subset whose fields are the wanted columns, matched by name like Unmarshal) or at the dynamic *[]map[string]any form, where only the named columns are stored. When the payload carries the column-length index (OptColumnIndex), unrequested columns are skipped without decoding; otherwise they are decoded but dropped.

fields names the wire columns to keep. With no fields it behaves like Unmarshal.

Example

ExampleUnmarshalColumns decodes only a named subset of columns from a columnar []struct payload. The producer opts in to OptColumnIndex so the payload carries a column-length index; the decoder then advances past the columns it was not asked for instead of decoding them. Here a 4-field struct batch is decoded into a 2-field typed subset, naming the wanted columns explicitly. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		TS      int64  `qdf:"ts"`
		Level   string `qdf:"level"`
		Service string `qdf:"service"`
		Code    int32  `qdf:"code"`
	}
	in := make([]Event, 4)
	levels := []string{"INFO", "WARN", "ERROR", "INFO"}
	for i := range in {
		in[i] = Event{
			TS:      int64(1700000000 + i),
			Level:   levels[i],
			Service: "billing",
			Code:    int32(200 + i),
		}
	}

	// OptColumnIndex writes the column-length index; the default columnar
	// wire is byte-identical without it.
	buf, _ := qdf.Marshal(in, qdf.OptBalanced|qdf.OptColumnIndex)

	// Decode only the "level" and "service" columns into a typed subset.
	// The "ts" and "code" columns are skipped via the index, not decoded.
	type Subset struct {
		Level   string `qdf:"level"`
		Service string `qdf:"service"`
	}
	var out []Subset
	if err := qdf.UnmarshalColumns(buf, &out, "level", "service"); err != nil {
		fmt.Println("error:", err)
		return
	}
	for _, e := range out {
		fmt.Printf("level=%-5s service=%s\n", e.Level, e.Service)
	}
}

Output:
level=INFO  service=billing
level=WARN  service=billing
level=ERROR service=billing
level=INFO  service=billing

func UnmarshalDirect 

func UnmarshalDirect[T Unmarshaler](data []byte, out T) error

UnmarshalDirect dispatches to out.UnmarshalQDF after validating the 5-byte header. It is the inverse of MarshalDirect; the same wire constraints apply (Fast-mode only).

If the input carries the FlagDense bit, UnmarshalDirect falls back to the full Unmarshal path because generated code cannot resolve state-ref tags without the decoder's intern table.

Decode-side performance is bounded by the user's UnmarshalQDF implementation: the reflect path is heavily pooled (Decoder pool + per-decoder key intern cache) and tends to win against naive hand-rolled receivers. Generated code from cmd/qdfgen uses Decoder.InternKey for map / struct keys and matches or beats the reflect path; ad-hoc UnmarshalQDF methods that call NewDecoderOnBuf + ReadString in a tight loop will not.

func UnmarshalNested added in v0.0.2 

func UnmarshalNested(u Unmarshaler, src []byte, noCopy bool) (int, error)

UnmarshalNested decodes one nested Unmarshaler value from src, honoring noCopy when u also implements UnmarshalerOpts. External Unmarshalers without the Opts method fall back to a copying decode. Used by decodeUnmarshaler and by cmd/qdfgen-generated code; exported for the latter.

func UnmarshalNestedArena added in v0.0.6 

func UnmarshalNestedArena(u Unmarshaler, src []byte, noCopy bool, a *Arena) (int, error)

UnmarshalNestedArena is UnmarshalNested threading a decode arena: when a is non-nil and u implements UnmarshalerArena, the nested decode packs its strings into a. Otherwise it falls back to the plain (copying) nested decode. Exported for cmd/qdfgen-generated code.

func UnmarshalT 

func UnmarshalT[T any](data []byte, out *T) error

UnmarshalT is the generic equivalent of Unmarshal. The destination pointer must not be nil.

Types

type Arena added in v0.0.6 

type Arena struct {
	// contains filtered or unexported fields
}

Arena is a bump allocator for decoded string (and []byte-as-string) bodies. Pass it to a decode via WithArena (or Decoder.SetArena) and every copied string is packed into the arena's dense, contiguous blocks instead of one heap allocation per string. Across an epoch of many decodes this amortizes the block allocation to near zero and keeps the strings cache-local; the GC scans each block as a single object instead of one object per string.

Lifetime / safety: strings returned by an arena decode ALIAS the arena's memory. They stay valid as long as the Arena (or any string from it) is reachable — the GC keeps a block alive from an interior pointer, so you need not track the buffer manually. The SAFE pattern is one Arena per epoch (request / batch / stream window), then drop it: 

a := qdf.NewArena()
for _, msg := range batch {
    var v Event
    _ = qdf.Unmarshal(msg, &v, qdf.WithArena(a))
    use(v)          // v's strings live in a
}
// drop a; the GC frees every block once the decoded values die.

Reset reuses the arena across epochs for zero allocation, but is UNSAFE while any string from the previous epoch is still live (see Reset).

An Arena is NOT safe for concurrent use; give each goroutine its own.

Example

ExampleArena shows the safe default pattern: one arena per epoch (here, one batch of messages). Every decoded string is bump-packed into the arena instead of a separate heap allocation. No Reset and no manual lifetime management are needed — the arena and its blocks are reclaimed by the GC once the decoded values are dropped. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Level   string `qdf:"level"`
		Msg     string `qdf:"msg"`
	}

	// Pretend these arrived over the wire.
	msgs := make([][]byte, 3)
	for i, m := range []Event{
		{"api", "info", "request handled"},
		{"worker", "warn", "queue backlog"},
		{"edge", "error", "upstream timeout"},
	} {
		b, err := qdf.Marshal(m, qdf.OptSpeed)
		if err != nil {
			panic(err)
		}
		msgs[i] = b
	}

	a := qdf.NewArena() // one arena for this batch
	out := make([]Event, len(msgs))
	for i, msg := range msgs {
		if err := qdf.Unmarshal(msg, &out[i], qdf.WithArena(a)); err != nil {
			panic(err)
		}
	}
	// out's strings live in `a`; when `out` is dropped the GC frees the arena.

	for _, e := range out {
		fmt.Printf("%s/%s: %s\n", e.Service, e.Level, e.Msg)
	}
}

Output:
api/info: request handled
worker/warn: queue backlog
edge/error: upstream timeout
Example (Reset)

ExampleArena_reset shows the max-performance pattern: reuse one arena across an unbounded stream, calling Reset at each epoch boundary for zero steady-state allocation.

SAFETY: Reset rewinds the arena and the next decode overwrites its memory, so every value decoded before a Reset must be done being used before that Reset runs. Here each event is processed inside the loop body, before the next iteration's Reset. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Msg     string `qdf:"msg"`
	}

	stream := []Event{
		{"api", "ok"},
		{"worker", "retry"},
	}
	encoded := make([][]byte, len(stream))
	for i, e := range stream {
		b, _ := qdf.Marshal(e, qdf.OptSpeed)
		encoded[i] = b
	}

	a := qdf.NewArena()
	for _, msg := range encoded {
		a.Reset() // safe: the previous iteration's value is no longer used

		var ev Event
		if err := qdf.Unmarshal(msg, &ev, qdf.WithArena(a)); err != nil {
			panic(err)
		}
		fmt.Printf("%s: %s\n", ev.Service, ev.Msg)
	}
}

Output:
api: ok
worker: retry

func NewArena added in v0.0.6 

func NewArena() *Arena

NewArena returns an empty arena. The first decode allocates the first block.

func (*Arena) Reset added in v0.0.6 

func (a *Arena) Reset()

Reset rewinds the arena to reuse its current block, so the next epoch of decodes allocates nothing.

UNSAFE: Reset invalidates every string previously returned from this arena — the next decode overwrites the same memory. Call it ONLY once every value decoded into the arena since the last Reset (or since NewArena) is dead. This is a manual use-after-free contract the race detector cannot catch; if unsure, drop the Arena and make a new one instead (the GC then frees it safely).

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads QDF wire data from a single input buffer. Call SetInput to bind a buffer and the typed Read* methods to walk it. Behaviour is undefined if the input is mutated while the decoder holds it.

func NewDecoder 

func NewDecoder() *Decoder

NewDecoder constructs a Decoder. Bind input via SetInput.

func NewDecoderOnBuf 

func NewDecoderOnBuf(buf []byte) *Decoder

NewDecoderOnBuf binds a Decoder to buf. Equivalent to NewDecoder followed by SetInput(buf), without the SetInput branch checks.

func (*Decoder) Advance 

func (d *Decoder) Advance(n int)

Advance moves the read cursor forward by n bytes.

func (*Decoder) CheckLength 

func (d *Decoder) CheckLength(n int, perElem int) error

CheckLength returns ErrShortBuffer when n claimed elements cannot fit in the remaining input. Use before allocating per-element storage.

func (*Decoder) InternKey 

func (d *Decoder) InternKey(b []byte) string

InternKey returns a string equal to b, sharing storage with prior identical keys when the cache has them.

func (*Decoder) IsNil 

func (d *Decoder) IsNil() (bool, error)

IsNil reports whether the next value is the nil tag, consuming it on true. Returns (false, nil) for any other tag.

Example

ExampleDecoder_IsNil consumes a nil tag if present, returning false (without advancing) for any other tag. Useful when a field is optional on the wire and the caller wants to keep the read cursor stable on a non-nil branch. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	buf, _ := qdf.Marshal([]any{nil, "hello"}, qdf.OptSpeed)
	dec := qdf.NewDecoderOnBuf(buf)

	n, _ := dec.ReadArrayHeader()
	for range n {
		isNil, _ := dec.IsNil()
		if isNil {
			fmt.Println("<nil>")
			continue
		}
		s, _ := dec.ReadString()
		fmt.Println(s)
	}
}

Output:
<nil>
hello

func (*Decoder) MarkHeaderRead 

func (d *Decoder) MarkHeaderRead()

MarkHeaderRead tells the decoder the magic+version header is already consumed (e.g. the buffer is a tail slice from a parent decoder). The next read will skip the header check.

func (*Decoder) PeekTag 

func (d *Decoder) PeekTag() (byte, error)

PeekTag returns the next tag byte without advancing the cursor.

Example

ExampleDecoder_PeekTag inspects the next wire tag without consuming it. Handy when implementing a dispatch loop over a schemaless payload, or when an Unmarshaler needs to branch on the upcoming type. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	buf, _ := qdf.Marshal(int64(-42), qdf.OptSpeed)
	dec := qdf.NewDecoderOnBuf(buf)
	tag, _ := dec.PeekTag()
	if tag == qdf.TagNil {
		fmt.Println("nil value")
		return
	}
	v, _ := dec.ReadInt()
	fmt.Printf("int=%d (tag=0x%02x)\n", v, tag)
}

Output:
int=-42 (tag=0xc7)

func (*Decoder) Pos 

func (d *Decoder) Pos() int

Pos returns the current read offset.

func (*Decoder) ReadArrayHeader 

func (d *Decoder) ReadArrayHeader() (int, error)

ReadArrayHeader returns the element count.

func (*Decoder) ReadBool 

func (d *Decoder) ReadBool() (bool, error)

func (*Decoder) ReadBytes 

func (d *Decoder) ReadBytes() ([]byte, error)

ReadBytes returns a []byte value. With noCopy = true the result aliases the input.

func (*Decoder) ReadFloat32 

func (d *Decoder) ReadFloat32() (float32, error)

func (*Decoder) ReadFloat64 

func (d *Decoder) ReadFloat64() (float64, error)

func (*Decoder) ReadInt 

func (d *Decoder) ReadInt() (int64, error)

func (*Decoder) ReadMapHeader 

func (d *Decoder) ReadMapHeader() (int, error)

ReadMapHeader returns the key/value pair count.

func (*Decoder) ReadNil 

func (d *Decoder) ReadNil() error

func (*Decoder) ReadString 

func (d *Decoder) ReadString() (string, error)

ReadString returns the next string. If the decoder is in noCopy mode the returned string aliases the input buffer; otherwise a copy is made.

When the tag at the cursor resolves through the intern table — tagInternStr, tagStateRef, tagStateMTF, tagStatePair, tagStateRepeat — readStringBytes leaves d.state.lastID set to the entry it just touched. We return the pre-materialised d.state.stringValues[id] from that path instead of allocating a fresh `string(b)` copy on every state-ref hit. Inline reads (fixstr, str8/16/32) set lastID = lruInvalidID, fall through, and pay the copy as before.

func (*Decoder) ReadStringBytes 

func (d *Decoder) ReadStringBytes() ([]byte, error)

ReadStringBytes returns the next string or []byte value without copying. The returned slice aliases either the input buffer or the decoder's state-table storage. Callers that retain it beyond the input's lifetime must copy.

func (*Decoder) ReadTimestamp added in v0.0.2 

func (d *Decoder) ReadTimestamp() (sec int64, nsec uint32, err error)

ReadTimestamp reads a full-range timestamp and returns (sec, nsec, err). sec is seconds since Unix epoch (may be negative for pre-1970 instants). nsec is nanoseconds in [0, 999_999_999]. This replaces the old fixed-8-byte UnixNano encoding (clean break).

func (*Decoder) ReadUint 

func (d *Decoder) ReadUint() (uint64, error)

ReadUint reads a value that was encoded with WriteUint or WriteInt with a non-negative argument. Returns an error if the next value is negative.

func (*Decoder) Remaining 

func (d *Decoder) Remaining() int

Remaining returns the number of unread bytes.

func (*Decoder) RemainingBytes 

func (d *Decoder) RemainingBytes() []byte

RemainingBytes returns the unread portion of the input buffer. The result aliases the buffer and is invalidated by the next SetInput.

func (*Decoder) SetArena added in v0.0.6 

func (d *Decoder) SetArena(a *Arena)

SetArena directs the decoder to pack copied inline string bodies into a, bump-packed, instead of allocating one string per field. Pass nil to disable. The decoded strings alias a's memory; see Arena for the lifetime contract. Ignored while noCopy is set. Not concurrent-safe: one Arena per goroutine.

func (*Decoder) SetInput 

func (d *Decoder) SetInput(buf []byte)

SetInput rebinds the decoder to buf, dropping any prior state table.

func (*Decoder) SetNoCopy 

func (d *Decoder) SetNoCopy(v bool)

SetNoCopy switches the decoder into aliasing mode: string and []byte reads return slices that share storage with the input buffer. Faster (~2x on string-heavy payloads) with near-zero allocations.

The aliases are valid only while the input is alive and unmodified. This is safe — and free — when the input is caller-owned and not reused: an mmap, a file fully read into memory, or a buffer allocated fresh per message and never pooled (the aliasing headers keep it alive via the GC, so you need not track its lifetime). The hazard is buffer REUSE or mutation (a sync.Pool buffer, an overwritten scratch slice), which silently corrupts already-decoded values — a manual use-after-free the race detector will not catch. See WithNoCopy for the full contract.

Example

ExampleDecoder_SetNoCopy returns string / []byte values that alias the input buffer instead of allocating per-value copies. Use this for read-heavy hot paths where the input buffer outlives every value the decoder hands back; do not retain the values past the buffer's lifetime. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	in := []string{"alpha", "beta", "gamma"}
	buf, _ := qdf.Marshal(in, qdf.OptSpeed)

	dec := qdf.NewDecoderOnBuf(buf)
	dec.SetNoCopy(true)

	n, _ := dec.ReadArrayHeader()
	for range n {
		s, _ := dec.ReadString() // alias into buf
		fmt.Println(s)
	}
}

Output:
alpha
beta
gamma

func (*Decoder) Skip 

func (d *Decoder) Skip() error

Skip advances past one value without materializing it.

Skip recurses through nested arrays and maps, so it bounds nesting depth via descend/ascend exactly like the reflect decode path: an unknown struct field whose value is a deeply-nested array is skipped here, and without this guard a hostile payload of N nested arrays would overflow the goroutine stack (an unrecoverable fatal error). The recursive d.Skip() calls below re-enter this guard, so every level is counted.

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes a single QDF value into a growing internal buffer. Reset drops the buffer contents and (in Dense mode) the intern table so the encoder can be reused.

Example

ExampleEncoder drives the encoder by hand instead of via the top-level Marshal pool. Use this when you need to:

  • keep a reusable buffer alive across many encodes;
  • call PreIntern with a known hot string pool;
  • mix WriteX primitives with reflect-driven EncodeValue.
package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	enc := qdf.NewEncoderWith(qdf.OptSpeed)

	// Primitives + struct fields can be mixed by hand.
	enc.WriteMapHeader(2)
	enc.WriteString("name")
	enc.WriteString("alice")
	enc.WriteString("age")
	enc.WriteInt(33)

	dec := qdf.NewDecoderOnBuf(enc.Bytes())
	n, _ := dec.ReadMapHeader()
	for range n {
		k, _ := dec.ReadString()
		switch k {
		case "name":
			v, _ := dec.ReadString()
			fmt.Printf("%s=%s\n", k, v)
		case "age":
			v, _ := dec.ReadInt()
			fmt.Printf("%s=%d\n", k, v)
		}
	}
}

Output:
name=alice
age=33

func NewEncoder 

func NewEncoder(mode Mode) *Encoder

NewEncoder returns an Encoder. The internal buffer is allocated lazily on first write.

func NewEncoderOnBuf 

func NewEncoderOnBuf(buf []byte, mode Mode) *Encoder

NewEncoderOnBuf returns an Encoder that appends to buf at its current length. The buffer is NOT truncated; pass an empty slice for a fresh encoding. Call SetBuffer afterwards to truncate.

func NewEncoderWith 

func NewEncoderWith(opts Options) *Encoder

NewEncoderWith returns an Encoder configured by the option bit-mask directly. Dense state machinery is allocated only when OptDense is set. Defaults for intern threshold / depth match NewEncoder.

func (*Encoder) AdoptBuffer 

func (e *Encoder) AdoptBuffer(b []byte)

AdoptBuffer installs b as the working buffer, preserving its current length. Used to continue writing into a buffer that already carries data — for example, after a nested type returned its extended slice. headerOut is left unchanged.

func (*Encoder) AppendBytes 

func (e *Encoder) AppendBytes(p []byte)

AppendBytes appends raw, already-valid wire bytes. Bypasses tag dispatch; used by generated code to emit pre-encoded field-name prefixes.

func (*Encoder) ApplyOpts 

func (e *Encoder) ApplyOpts(opts Options)

ApplyOpts reconfigures the encoder via the options bit-mask. Equivalent to recreating the encoder with NewEncoderWith but preserves the existing pool buffer and (Dense-mode) intern state. The pool-backed Marshal* entry points call this on every acquire; callers driving an Encoder directly use it to switch between OptSpeed and OptBalanced without re-allocating.

func (*Encoder) Bytes 

func (e *Encoder) Bytes() []byte

Bytes returns the encoded payload. It aliases the encoder's buffer and is only valid until the next write or Reset.

func (*Encoder) EncodeValue 

func (e *Encoder) EncodeValue(v any) error

EncodeValue runs the reflect-driven Marshal pipeline on v against this Encoder. Convenience for callers driving an Encoder directly (e.g. with PreIntern) — equivalent to what the pool-backed Marshal does internally.

Example

ExampleEncoder_EncodeValue drives the reflect-based encode path against any Go value directly through the encoder, bypassing the pool used by qdf.Marshal. Combine with PreIntern, custom SetBuffer, or SetIntern tuning for a long-lived encoder that shapes its state for a specific workload. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Config struct {
		Name  string `qdf:"name"`
		Limit int    `qdf:"limit"`
	}

	enc := qdf.NewEncoderWith(qdf.OptBalanced)
	if err := enc.EncodeValue(Config{Name: "service.cpu", Limit: 1000}); err != nil {
		panic(err)
	}

	var out Config
	_ = qdf.Unmarshal(enc.Bytes(), &out)
	fmt.Printf("%+v\n", out)
}

Output:
{Name:service.cpu Limit:1000}

func (*Encoder) EnsureHeader 

func (e *Encoder) EnsureHeader()

EnsureHeader forces a header write if one has not been emitted yet.

func (*Encoder) MarkHeaderWritten 

func (e *Encoder) MarkHeaderWritten()

MarkHeaderWritten tells the encoder the QDF header is already present in its buffer (e.g. left there by a parent encoder). The next write will skip the header emission.

func (*Encoder) PreIntern 

func (e *Encoder) PreIntern(strs ...string)

PreIntern registers the given strings against the encoder's intern table up front. Subsequent WriteString / WriteBytes calls that pass the SAME backing string header (i.e. the same underlying byte pointer and length) skip the hash + slot probe and emit a state-ref directly against the cached intern id.

Intended for power users who know their hot string pool ahead of time — service names, region codes, enum-like values drawn from a fixed slice. Real telemetry payloads draw 90 %+ of their dense intern hits from such pools. Skipping the hash/probe on those calls is the main per-emit cost left on the encode hot path.

Requires OptDense to be applied first (otherwise the call is a no-op). The registered identities are dropped on Reset so a pooled encoder does not carry caller-supplied pointers across recycles.

Safety: the caller must keep the backing memory of every PreIntern'd string alive for the lifetime of the next encode call. For literals embedded in a slice / global / struct field this is automatic; for short-lived stack strings the caller is responsible.

Example

ExampleEncoder_PreIntern registers a known string pool against the encoder so subsequent WriteString calls with the same backing pointer skip the intern table's hash + slot probe and emit a state-ref directly. Use this when the caller knows the hot vocabulary up front (service names, region codes, enum- like fields).

The caller must keep the registered strings' backing memory alive for the lifetime of the next encode call. String literals embedded in a slice, global, or struct field stay alive automatically; short-lived stack strings do not. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Region  string `qdf:"region"`
		Status  int    `qdf:"status"`
	}

	// Stable hot pool — backing slices live for the program's
	// lifetime, so the registered pointers stay valid across
	// many encode calls on the same encoder.
	services := []string{"billing", "auth", "ingest"}
	regions := []string{"eu-west-1", "us-east-1"}

	enc := qdf.NewEncoderWith(qdf.OptBalanced)
	enc.PreIntern(services...)
	enc.PreIntern(regions...)

	rows := []Row{
		{services[0], regions[0], 200},
		{services[1], regions[1], 500},
		{services[2], regions[0], 200},
	}
	if err := enc.EncodeValue(rows); err != nil {
		panic(err)
	}

	var out []Row
	_ = qdf.Unmarshal(enc.Bytes(), &out)
	for _, r := range out {
		fmt.Printf("%s/%s=%d\n", r.Service, r.Region, r.Status)
	}
}

Output:
billing/eu-west-1=200
auth/us-east-1=500
ingest/eu-west-1=200

func (*Encoder) QPack 

func (e *Encoder) QPack() bool

QPack reports whether QPack codec emission is enabled.

func (*Encoder) Reset 

func (e *Encoder) Reset()

Reset truncates the buffer and resets the intern table. Capacities are preserved. opts / mode / qpack are also reset to OptSpeed so a pooled encoder does not leak its previous configuration into the next caller — apply the desired options explicitly via applyOpts / SetQPack after Reset.

func (*Encoder) SetBuffer 

func (e *Encoder) SetBuffer(b []byte)

SetBuffer installs a caller-owned buffer (truncated to length 0).

func (*Encoder) SetIntern 

func (e *Encoder) SetIntern(min int, cap int)

SetIntern overrides the Dense-mode tuning knobs. Zero values keep the current setting.

func (*Encoder) SetMaxDepth 

func (e *Encoder) SetMaxDepth(d int)

SetMaxDepth caps reflect-path pointer/struct recursion. The default (DefaultMaxDepth = 10000) is sufficient for any normal payload and rejects pointer cycles before they stack-overflow the goroutine. Set to 0 to disable the check entirely — only safe when the caller can prove the input graph is acyclic.

func (*Encoder) SetQPack 

func (e *Encoder) SetQPack(v bool)

SetQPack toggles QPack codec emission. When true, slice fast paths produce packed/encoded forms (bitpacked bools, FOR-packed integers, Gorilla-encoded floats, raw-LE bulk) instead of per-element tag streams. Setting must happen before the first write of the value (the header is emitted lazily and carries the FlagQPack hint when this is on).

func (*Encoder) Take 

func (e *Encoder) Take() []byte

Take returns the encoded payload and detaches it from the encoder. The caller takes ownership.

func (*Encoder) WriteArrayHeader 

func (e *Encoder) WriteArrayHeader(n int)

WriteArrayHeader writes the header for an array of n elements. The caller must follow with exactly n element writes.

func (*Encoder) WriteBool 

func (e *Encoder) WriteBool(v bool)

func (*Encoder) WriteBytes 

func (e *Encoder) WriteBytes(b []byte)

WriteBytes writes a []byte. In Dense mode, eligible payloads are intern-encoded. The intern table is keyed on the byte sequence, so a string and a []byte with identical content share an ID.

func (*Encoder) WriteFloat32 

func (e *Encoder) WriteFloat32(v float32)

func (*Encoder) WriteFloat64 

func (e *Encoder) WriteFloat64(v float64)

func (*Encoder) WriteInt 

func (e *Encoder) WriteInt(v int64)

func (*Encoder) WriteMapHeader 

func (e *Encoder) WriteMapHeader(n int)

WriteMapHeader writes the header for a map of n entries. The caller must follow with exactly n key/value pairs.

func (*Encoder) WriteNil 

func (e *Encoder) WriteNil()

func (*Encoder) WriteString 

func (e *Encoder) WriteString(s string)

WriteString writes s. In Dense mode (OptDense set), eligible strings are intern-encoded.

func (*Encoder) WriteStringInline 

func (e *Encoder) WriteStringInline(s string)

WriteStringInline forces an in-line encoding even when Dense intern would be eligible. Use for fields known to be unique per message.

func (*Encoder) WriteTimestamp added in v0.0.2 

func (e *Encoder) WriteTimestamp(sec int64, nsec uint32)

WriteTimestamp writes a full-range timestamp as two uvarints: sec (zigzag-encoded signed int64 seconds since Unix epoch) and nsec (unsigned uint32 nanoseconds in [0, 999_999_999]). This replaces the old fixed-8-byte UnixNano encoding (clean break).

func (*Encoder) WriteUint 

func (e *Encoder) WriteUint(v uint64)

type FSSTDict added in v0.0.2 

type FSSTDict struct {
	// contains filtered or unexported fields
}

FSSTDict is a pre-trained, immutable FSST symbol table.

Training the per-column symbol table is the dominant cost of FSST encoding. Train one FSSTDict over representative samples and reuse it across many FSSTDict.Marshal calls: each encode then skips training and pays only compression — the train-once, encode-many pattern (the "Static" in FSST).

An FSSTDict is bounded (≤255 symbols, a few KB) and never mutated after training, so it is safe for concurrent use and holds a fixed, small amount of memory. It cannot grow, so reusing one across the life of a process does not leak. The wire stays self-describing: each FSST column still carries its table, so output produced with a dictionary decodes with a plain Unmarshal.

func TrainFSSTDict added in v0.0.2 

func TrainFSSTDict(samples [][]byte) *FSSTDict

TrainFSSTDict learns an FSST symbol table from representative byte samples (typically a slice of the string column you will encode). Deterministic: the same samples always yield the same dictionary.

func TrainFSSTDictStrings added in v0.0.2 

func TrainFSSTDictStrings(samples []string) *FSSTDict

TrainFSSTDictStrings is TrainFSSTDict over []string, viewing each string's bytes without copying (the strings are only read during training).

func (*FSSTDict) AppendMarshal added in v0.0.2 

func (d *FSSTDict) AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)

AppendMarshal encodes v with this dictionary and appends to dst (see AppendMarshal). Enables FSST and its columnar prerequisites.

func (*FSSTDict) Marshal added in v0.0.2 

func (d *FSSTDict) Marshal(v any, opts Options) ([]byte, error)

Marshal encodes v using this pre-trained dictionary for FSST string columns. It enables the FSST codec and its columnar prerequisites (Dense, QPack, ShapeIntern), so even FSSTDict.Marshal(v, OptSpeed) compresses; combine with OptCompression to add the float codecs and rANS.

type Marshaler 

type Marshaler interface {
	MarshalQDF(dst []byte) ([]byte, error)
}

Marshaler is implemented by types that know how to serialize themselves into the QDF wire format. Implementations should append exactly one value to dst and return the extended slice.

Example

ExampleMarshaler shows a custom MarshalQDF / UnmarshalQDF pair. Use the interface when you want a compact wire form for a type — e.g. packing a small struct into a single primitive, versioning a payload, or pre-validating bytes before they hit the wire. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

// Color is a 24-bit RGB value stored as a single uint32 on the
// wire. It implements Marshaler / Unmarshaler so qdf encodes it
// as a primitive integer instead of recursing into reflect.
type Color struct{ R, G, B uint8 }

// MarshalQDF packs the three bytes into a uint32 and writes it
// using the encoder primitives from a temporary qdf.Encoder
// over the caller's buffer. This shape — temporary encoder over
// the caller's dst slice — is what qdfgen produces by default.
func (c Color) MarshalQDF(dst []byte) ([]byte, error) {
	enc := qdf.NewEncoderOnBuf(dst, qdf.Fast)
	enc.MarkHeaderWritten() // dst already contains the parent header
	enc.WriteUint(uint64(c.R)<<16 | uint64(c.G)<<8 | uint64(c.B))
	return enc.Bytes(), nil
}

// UnmarshalQDF reverses the pack and returns the number of bytes
// consumed from src.
func (c *Color) UnmarshalQDF(src []byte) (int, error) {
	dec := qdf.NewDecoderOnBuf(src)
	dec.MarkHeaderRead()
	v, err := dec.ReadUint()
	if err != nil {
		return 0, err
	}
	c.R = uint8(v >> 16)
	c.G = uint8(v >> 8)
	c.B = uint8(v)
	return dec.Pos(), nil
}

// ExampleMarshaler shows a custom MarshalQDF / UnmarshalQDF pair.
// Use the interface when you want a compact wire form for a
// type — e.g. packing a small struct into a single primitive,
// versioning a payload, or pre-validating bytes before they hit
// the wire.
func main() {
	in := Color{R: 0x12, G: 0x34, B: 0x56}
	buf, _ := qdf.MarshalDirect(&in)

	var out Color
	_ = qdf.UnmarshalDirect(buf, &out)
	fmt.Printf("R=%02x G=%02x B=%02x\n", out.R, out.G, out.B)
}

Output:
R=12 G=34 B=56

type Mode 

type Mode uint8

Mode selects the wire dialect. 

const (
	// Fast writes strings and []byte in line. No intern bookkeeping.
	Fast Mode = 0

	// Dense writes repeated strings/bytes once and references them by ID
	// thereafter. Smaller output on repetitive payloads; slightly slower
	// per call.
	Dense Mode = 1
)

type Options 

type Options uint32

Options is a bit-mask of per-call encoder feature toggles. A zero value (OptSpeed) gives the fastest path: Fast mode, no codecs, no predictors. Or-combine bits with | to opt in to individual codecs. Pre-built bundles cover the common "max speed", "balanced", and "max compression" use cases without having to remember the layout.

Options carry by value (uint32) so Marshal / AppendMarshal add zero allocations over the pool / output-clone overhead. The encoder checks each bit with a single AND on the hot path.

Example

ExampleOptions shows how the Options bit-mask picks which codec layers run for a Marshal call. The same input encodes to a progressively smaller wire as more codecs opt in, with a small CPU penalty per added codec. The decoder reads any dialect off the wire header so changing Options is a producer-side decision. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Region  string `qdf:"region"`
		Status  int    `qdf:"status"`
	}
	// Repetitive batch — every row shares the same service and region.
	in := make([]Row, 50)
	for i := range in {
		in[i] = Row{Service: "billing", Region: "eu-west-1", Status: 200}
	}

	for _, p := range []struct {
		name string
		opts qdf.Options
	}{
		{"OptSpeed", qdf.OptSpeed},
		{"OptBalanced", qdf.OptBalanced},
	} {
		buf, _ := qdf.Marshal(in, p.opts)
		fmt.Printf("%-12s wire=%d bytes\n", p.name, len(buf))
	}
}

Output:
OptSpeed     wire=2208 bytes
OptBalanced  wire=161 bytes
Example (ColumnarStringDict)

ExampleOptions_columnarStringDict shows the per-column string dictionary that fires automatically under OptBalanced for a slice of structs whose string fields are enum-like — a small set of distinct values scattered across rows. The distinct strings are stored once and each row keeps a few-bit index, so the wire is far smaller than one interned reference per value. It is gated never-worse: run-heavy or high-cardinality columns keep the per-value path, and the choice is invisible to the decoder. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type LogRow struct {
		TS      int64  `qdf:"ts"`      // sequential — Delta+FOR makes columnar win
		Level   string `qdf:"level"`   // enum — string dictionary
		Service string `qdf:"service"` // enum — string dictionary
	}
	levels := []string{"INFO", "WARN", "ERROR"}
	services := []string{"api", "auth", "billing"}
	in := make([]LogRow, 300)
	for i := range in {
		in[i] = LogRow{
			TS:      int64(1700000000 + i),
			Level:   levels[i%len(levels)],
			Service: services[(i*7)%len(services)],
		}
	}

	buf, _ := qdf.Marshal(in, qdf.OptBalanced)

	var out []LogRow
	_ = qdf.Unmarshal(buf, &out)
	fmt.Printf("rows=%d wire=%d bytes roundtrip=%v\n", len(in), len(buf), out[0] == in[0] && out[299] == in[299])

}

Output:
rows=300 wire=234 bytes roundtrip=true
Example (StreamingDictShared)

ExampleOptions_streamingDictShared shows the headline Dense-mode win: across many calls of a StreamEncoder the intern table / shape table / predictors all survive, so the second record onwards trades long strings for 1-3 byte state-refs. Per-message Marshal does not share the dictionary across calls; only the stream API does. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	// 100 rows, all the same Service literal. Marshal-per-call
	// reuses the qdf encoder pool but resets state every call.
	one, _ := qdf.Marshal(Row{Service: "billing", Status: 200}, qdf.OptBalanced)
	fmt.Printf("per-message wire = %d bytes\n", len(one))

}

Output:
per-message wire = 36 bytes

const (
	// OptDense activates the inline intern table. First occurrences of
	// strings and []byte payloads are stored once and back-referenced
	// by ID (1-3 bytes per reuse). Required for any of the state-ref
	// codecs (Repeat / MTF / Pair) and for tagMapShape.
	OptDense Options = 1 << iota

	// OptQPack enables the numeric / boolean slice codecs. Bools
	// bit-pack; 32- and 64-bit integer slices ([]int/int32/int64,
	// []uint/uint32/uint64) run the codec picker — Frame-of-Reference,
	// Delta+FOR, RLE, dictionary, and Patched FOR — widening 32-bit
	// values to 64-bit for selection and narrowing back on decode;
	// float slices stay on raw-LE. Auto-selected per slice; the
	// encoder falls back to raw-LE when nothing wins. Gorilla XOR
	// for floats lives behind OptGorillaFloat (bit 5) — see
	// OptCompression for the bundle that turns it on.
	OptQPack

	// OptShapeIntern routes struct emissions through tagMapShape: each
	// distinct struct type declares its field ordering on first emit;
	// subsequent emits write only the shape ID + values. Requires
	// OptDense (the shape table lives on the intern table side).
	OptShapeIntern

	// OptPairPred enables the Markov-1 successor predictor
	// (tagStatePair, 0xEA). Per previous state-ref ID the encoder stores
	// its most-recent successor (top-1, K=1); a hit emits the transition
	// rank in a single byte (always rank 0). Requires OptDense.
	OptPairPred

	// OptMTF enables Move-to-Front rank coding (tagStateMTF, 0xE9).
	// When the LRU rank of a state-ref ID needs fewer varuint bytes
	// than the raw id, the rank is emitted instead. Requires OptDense.
	OptMTF

	// OptGorillaFloat opts in to the Gorilla XOR codec for []float64
	// (and []float32) slices. Gorilla collapses smooth time-series
	// data dramatically (~75 % wire reduction on quantised metric
	// streams) but trades that for ~10× more CPU on the encode/decode
	// path because the body is bit-level. Off by default in
	// OptBalanced for that reason; included in OptCompression for
	// archive-style workloads where wire size dominates. Requires
	// OptQPack.
	OptGorillaFloat

	// OptRANS adds a final static order-0 rANS entropy pass over the whole
	// encoded body. Opt-in (bundled into OptCompression) because it trades
	// encode/decode CPU for wire size. The encoder applies it only when the
	// rANS form is strictly smaller than the plain body, so it never makes a
	// buffer larger. Whole-buffer, so the streaming encoder ignores it.
	OptRANS

	// OptColumnIndex makes a columnar []struct payload carry a column-length
	// index so readers can decode a subset of columns without decoding the
	// rest. Opt-in, ~4 bytes per column. Only affects payloads the encoder
	// transposes into the columnar container (tagColStruct); has no effect on
	// other shapes. Sets FlagColIndex on the header.
	OptColumnIndex

	// OptFSST opts in to the FSST string codec for columnar string columns
	// (high-cardinality, substring-sharing text: log lines, URLs, paths). A
	// CPU-for-size trade (trains a per-column symbol table), so it is excluded
	// from OptBalanced and bundled into OptCompression. Never-larger: emitted
	// only when strictly smaller than the dictionary / per-value forms.
	OptFSST

	// OptMapShape interns map key-sets the way OptShapeIntern interns struct
	// field-names: the first map with a given set of (string) keys declares the
	// key ordering via tagMapShape; subsequent maps with the same key-set emit
	// only the shape ID + values in canonical (sorted) key order. Cuts encode
	// CPU and wire on maps with recurring keys (telemetry tags, log labels).
	// Requires OptDense (the shape table lives on the intern side). Opt-in in
	// v1; never-worse fallback to a plain map header when a key-set does not
	// recur.
	OptMapShape

	// OptSpeed is the zero-bit preset: Fast mode, no codecs, no
	// predictors. Maximum throughput, smallest CPU footprint.
	OptSpeed Options = 0

	// OptBalanced bundles every codec that does not trade CPU for
	// compression beyond its sweet spot. The right default for
	// telemetry, log batches, and any payload with repetitive
	// strings or numeric slices. Notably excludes OptGorillaFloat —
	// reach for OptCompression when the float slices in the payload
	// are smooth time-series and wire size matters more than encode
	// latency.
	OptBalanced Options = OptDense | OptQPack | OptShapeIntern | OptPairPred | OptMTF

	// OptCompression bundles every codec that the encoder will spend
	// CPU on for wire-size gains. On top of OptBalanced it adds
	// OptGorillaFloat (Gorilla XOR for float slices), which trades
	// encode/decode CPU for wire size, so it stays out of the
	// OptBalanced default; future heavy codecs (rANS, dictionary
	// preloading) will land in this bundle without breaking the name.
	OptCompression Options = OptBalanced | OptGorillaFloat | OptRANS | OptFSST
)

func (Options) Has 

func (o Options) Has(bit Options) bool

Has reports whether the named bit is set. Compiles to a single AND + compare. Use on the encoder hot path to gate codec emission.

type QueryError added in v0.0.2 

type QueryError struct {
	Op    string  // e.g. "predicate pushdown"
	Field string  // filter/projection field involved, if any
	Want  colKind // kind implied by the predicate's T (kind mismatches)
	Got   colKind // kind found on the wire (kind mismatches)
	Err   error   // wrapped sentinel
}

QueryError describes why a filtering/projecting decode (Unmarshal with QueryOptions) could not proceed. It wraps one of ErrUnsupported, ErrTypeMismatch, or ErrFieldNotFound, so callers can categorise the failure with errors.Is and read the specifics with errors.As.

func (*QueryError) Error added in v0.0.2 

func (e *QueryError) Error() string

func (*QueryError) Unwrap added in v0.0.2 

func (e *QueryError) Unwrap() error

type QueryOption added in v0.0.2 

type QueryOption struct {
	// contains filtered or unexported fields
}

QueryOption configures a filtering/projecting Unmarshal. Construct with Where, And, Or, Not (predicate tree) or Select (projection). Exactly one of node / selectFields is set.

func And added in v0.0.2 

func And(opts ...QueryOption) QueryOption

And keeps rows where every sub-predicate is true. Multiple top-level Where options are an implicit And; And is the explicit, nestable form.

func Not added in v0.0.2 

func Not(opt QueryOption) QueryOption

Not keeps rows where the sub-predicate is not true. Under three-valued NULL semantics a nil (absent) nullable row is UNKNOWN, so Not(pred) excludes nil rows just as the bare predicate does.

func Or added in v0.0.2 

func Or(opts ...QueryOption) QueryOption

Or keeps rows where at least one sub-predicate is true.

Example

ExampleOr demonstrates OR/NOT predicate combinators: keep events that are either level=="ERROR" or code>=500, while excluding level=="DEBUG" rows. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Level string `qdf:"level"`
		Code  int32  `qdf:"code"`
	}
	events := make([]Event, 30)
	for i := range events {
		if i%2 == 0 {
			events[i] = Event{Level: "ERROR", Code: int32(500 + i)}
		} else {
			events[i] = Event{Level: "INFO", Code: int32(i)}
		}
	}
	buf, _ := qdf.Marshal(events, qdf.OptBalanced|qdf.OptColumnIndex)

	var hot []Event
	_ = qdf.Unmarshal(buf, &hot,
		qdf.Or(
			qdf.Where("level", func(s string) bool { return s == "ERROR" }),
			qdf.Where("code", func(c int32) bool { return c >= 500 }),
		),
		qdf.Not(qdf.Where("level", func(s string) bool { return s == "DEBUG" })),
	)
	fmt.Println(len(hot))
}

Output:
15

func Select added in v0.0.2 

func Select(fields ...string) QueryOption

Select restricts decoding to the named columns. Top-level only; nesting a Select inside And/Or/Not is reported as ErrUnsupported.

func Where added in v0.0.2 

func Where[T Queryable](field string, pred func(T) bool) QueryOption

Where keeps only rows whose field column satisfies pred. T must match the column's native kind (any integer width, float32/64, string, or bool); a mismatch is reported as ErrTypeMismatch at decode time. If field is absent from the wire the call returns ErrFieldNotFound. Multiple Where options are AND-ed. The predicate is called once per row with the native value — no interface boxing per value.

func WithArena added in v0.0.6 

func WithArena(a *Arena) QueryOption

WithArena makes the decode pack copied inline string values into a, instead of allocating one string per field — near-zero allocations across an epoch of decodes (see Arena). The decoded strings alias a's memory and follow Arena's lifetime contract. Ignored together with WithNoCopy (aliasing already skips the copy). Composes with predicates / Select / WithNoCopy.

func WithNoCopy added in v0.0.2 

func WithNoCopy() QueryOption

WithNoCopy makes the decode return string and []byte values that ALIAS the input buffer instead of copying them. On string-heavy payloads this is ~2x faster with near-zero allocations.

DANGER — lifetime contract: the returned values are valid ONLY while the input buffer passed to Unmarshal stays alive and is never modified or reused. Do NOT use WithNoCopy when the input may be recycled, mutated, or freed before you finish reading the decoded values — e.g. a pooled HTTP request body. The decoded strings would silently become garbage. This is a manual-memory use-after-free, not a data race: the race detector will NOT catch it.

Safe for caller-owned, immutable input such as an mmap or a file read fully into memory.

Also safe — and the common zero-copy high-throughput pattern — when the input buffer is allocated FRESH per message and never pooled or overwritten: e.g. io.ReadAll(body), or a make([]byte, n) you read one message into and do not reuse. You do NOT have to track the buffer's lifetime manually: the aliasing string/[]byte headers in the decoded value keep the backing buffer alive through the garbage collector (Go marks the whole allocation from an interior pointer), so the buffer lives exactly as long as the values that point into it. The hazard is exclusively buffer REUSE (a sync.Pool buffer returned after the handler, a scratch slice you overwrite on the next read) or mutation — not lifetime. If every decode reads into its own fresh slice, WithNoCopy is safe and gives the full ~2x / near-zero-alloc win for free.

Scope: WithNoCopy affects the reflect decode path. A type that implements Unmarshaler (including codegen-generated UnmarshalQDF) decodes through its own decoder via the byte-only UnmarshalQDF(data) interface, which cannot inherit this flag, so such types still copy. Use SetNoCopy on a Decoder/StreamDecoder you drive directly if you need zero-copy there.

Example

ExampleWithNoCopy decodes with zero-copy: the returned string / []byte fields alias the input buffer instead of being copied, cutting allocations to near zero (~1.7x faster on string-heavy batches).

Lifetime contract: the decoded values are valid only while data stays alive and is never modified or reused. Never use WithNoCopy when data is a pooled or recycled buffer (e.g. an HTTP request body) — the values would silently corrupt. It is opt-in for this reason; the default Unmarshal copies. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Message string `qdf:"message"`
	}
	// data is owned here and outlives `out`, so noCopy is safe.
	data, _ := qdf.Marshal(&Event{Service: "api", Message: "ok"}, qdf.OptSpeed)

	var out Event
	if err := qdf.Unmarshal(data, &out, qdf.WithNoCopy()); err != nil {
		panic(err)
	}
	fmt.Println(out.Service, out.Message)
}

Output:
api ok

type Queryable added in v0.0.2 

type Queryable interface {
	int | int8 | int16 | int32 | int64 |
		uint | uint8 | uint16 | uint32 | uint64 | uintptr |
		float32 | float64 | string | bool
}

Queryable is the set of column element types a predicate can match. Exact base types only: Where dispatches on the concrete func(T) bool type once at construction, which matches base types but not user-defined named types.

type StreamDecoder 

type StreamDecoder struct {
	// contains filtered or unexported fields
}

StreamDecoder reads a sequence of values from an io.Reader. The intern table is preserved across Decode calls to match StreamEncoder.

The decoder grows its window buffer as needed. State-table entries alias the buffer, so the window is never compacted during a stream; total memory tracks the stream length. For unbounded streams partition into envelopes and create a fresh StreamDecoder per envelope.

func NewStreamDecoder 

func NewStreamDecoder(r io.Reader) *StreamDecoder

NewStreamDecoder returns a stream decoder reading from r.

func (*StreamDecoder) Close 

func (s *StreamDecoder) Close() error

Close releases the scratch buffer to the pool. The underlying reader is not closed. Idempotent: subsequent calls are a safe no-op.

func (*StreamDecoder) Decode 

func (s *StreamDecoder) Decode(out any) error

Decode reads the next value into out. out must be a pointer. Returns io.EOF at a clean end of stream. Each message is length-framed, so a message of any size is buffered in full before decoding — no 4 KiB window limit.

func (*StreamDecoder) SetNoCopy 

func (s *StreamDecoder) SetNoCopy(v bool)

SetNoCopy makes Decode return string / []byte values that alias the decoder's internal window buffer instead of copying them — eliminating the per-value copy that dominates decode allocation.

Unlike the one-shot Decoder.SetNoCopy (which aliases the caller's input buffer and is unsafe the moment that buffer is reused — the typical server recycles its read buffer right after the handler returns), the stream owns its window buffer, so aliasing is safe for the lifetime of the stream:

  • A decoded value stays valid until Close. Close returns the window to a pool, after which any retained aliased value is undefined — copy (strings.Clone / append) anything you need past Close.
  • The window grows but is never compacted mid-stream (the Dense intern table already aliases it for cross-message back-references), so a growth does not invalidate earlier values, and noCopy adds no extra memory: the bytes are retained either way.

Because the window is retained for the whole stream regardless, this is the safe home for zero-copy decode — process or copy each value before Close and you pay zero per-value allocation across every message. No-op after Close.

Example

ExampleStreamDecoder_SetNoCopy decodes a multi-message stream without copying the decoded strings out of the input buffer. The decoded values alias the io.Reader's payload; the caller must not retain the values past the next Decode call. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
	}

	// Build a 2-message stream.
	var sink bytes.Buffer
	enc := qdf.NewStreamEncoderWith(&sink, qdf.OptBalanced)
	_ = enc.Encode(Event{Service: "auth"})
	_ = enc.Encode(Event{Service: "auth"})
	_ = enc.Flush()
	_ = enc.Close()

	dec := qdf.NewStreamDecoder(&sink)
	dec.SetNoCopy(true) // aliased reads — zero allocations
	defer dec.Close()   //nolint:errcheck // example
	var ev Event
	for dec.Decode(&ev) == nil {
		fmt.Println(ev.Service)
	}
}

Output:
auth
auth

type StreamEncoder 

type StreamEncoder struct {
	// contains filtered or unexported fields
}

StreamEncoder writes a sequence of values into an io.Writer. The header is emitted once before the first value; in Dense mode the intern table survives across Encode calls so back-references span the whole stream.

Example

ExampleStreamEncoder writes a batch of messages into a single io.Writer where the Dense-mode intern table, shape table, and predictors survive across calls. The first message of every shape pays for the key intern records; every subsequent message of the same shape emits values only.

This is the form to reach for when piping events to a file or a network socket — per-message Marshal would reset the state table each call and lose the cross-message compression. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	var sink bytes.Buffer
	enc := qdf.NewStreamEncoderWith(&sink, qdf.OptBalanced)
	defer enc.Close() //nolint:errcheck // example

	events := []Event{
		{"billing", 200},
		{"billing", 200},
		{"billing", 500},
	}
	for _, e := range events {
		if err := enc.Encode(e); err != nil {
			panic(err)
		}
	}
	if err := enc.Flush(); err != nil {
		panic(err)
	}

	dec := qdf.NewStreamDecoder(&sink)
	for {
		var out Event
		if err := dec.Decode(&out); err != nil {
			break
		}
		fmt.Printf("%s=%d\n", out.Service, out.Status)
	}
}

Output:
billing=200
billing=200
billing=500

func NewStreamEncoder 

func NewStreamEncoder(w io.Writer, mode Mode) *StreamEncoder

NewStreamEncoder returns a stream encoder backed by w. Dense mode activates the full balanced codec set (OptBalanced); Fast mode emits raw tagged bytes (OptSpeed). For finer per-stream tuning, build the StreamEncoder via NewStreamEncoderWith.

func NewStreamEncoderWith 

func NewStreamEncoderWith(w io.Writer, opts Options) *StreamEncoder

NewStreamEncoderWith returns a stream encoder with the given Options bit-mask. The intern table (when OptDense is set) survives across Encode calls so back-references span the whole stream.

func (*StreamEncoder) Close 

func (s *StreamEncoder) Close() error

Close flushes pending data and releases the scratch buffer to the pool. The underlying writer is not closed. Idempotent: subsequent calls are a safe no-op.

func (*StreamEncoder) Encode 

func (s *StreamEncoder) Encode(v any) error

Encode writes v as the next value in the stream. Each message is framed with a uvarint byte-length prefix so the decoder can buffer a whole message — of any size — before decoding it. The 5-byte stream header is written once, before the first frame, and is not itself framed. The encoder flushes internally when its buffer crosses 16 KiB; call Flush to push earlier.

func (*StreamEncoder) Flush 

func (s *StreamEncoder) Flush() error

Flush writes any buffered bytes to the underlying writer.

type Unmarshaler 

type Unmarshaler interface {
	UnmarshalQDF(src []byte) (n int, err error)
}

Unmarshaler is implemented by types that know how to deserialize themselves from a QDF wire-format slice. Implementations should consume exactly one value from src and return the number of bytes consumed.

A custom UnmarshalQDF reads the Fast wire format. Marshal honours this: a type implementing Marshaler always emits its Fast-format body and is framed as Fast regardless of the Options passed, so Marshaler+Unmarshaler types round-trip under any Options. A type that implements Unmarshaler WITHOUT also implementing Marshaler is encoded structurally; under a Dense/QPack tier that produces a Dense body its Fast-only UnmarshalQDF cannot read. Implement both interfaces (or neither) to avoid this — generated code from cmd/qdfgen always implements both.

type UnmarshalerArena added in v0.0.6 

type UnmarshalerArena interface {
	UnmarshalerOpts
	UnmarshalQDFArena(src []byte, noCopy bool, a *Arena) (n int, err error)
}

UnmarshalerArena is an optional extension that also accepts a decode Arena, so the implementation packs copied string/[]byte fields into it (see WithArena) instead of one allocation per field. Generated code from cmd/qdfgen implements this; UnmarshalQDFOpts delegates to it with a nil arena. Decoders honor it only when the caller passed an arena.

type UnmarshalerOpts added in v0.0.2 

type UnmarshalerOpts interface {
	Unmarshaler
	UnmarshalQDFOpts(src []byte, noCopy bool) (n int, err error)
}

UnmarshalerOpts is an optional extension of Unmarshaler that accepts the noCopy flag. When noCopy is true, the implementation should decode string and []byte fields as aliases of src (see WithNoCopy) instead of copying. Generated code from cmd/qdfgen implements this; the plain UnmarshalQDF delegates to it with noCopy=false. Decoders honor it only when the caller opted into noCopy.

Source Files

View all Source files

Directories

Path Synopsis
internal
bitpack
Package bitpack implements the pure bit-packing and SIMD kernel layer shared by qdf's integer codecs (FOR, Delta+FOR, dict, PFOR, ALP).
 Package bitpack implements the pure bit-packing and SIMD kernel layer shared by qdf's integer codecs (FOR, Delta+FOR, dict, PFOR, ALP).
bufpool
Package bufpool implements size-classed, sharded byte-slice pools.
 Package bufpool implements size-classed, sharded byte-slice pools.
bumparena
Package bumparena is a monotonic bump allocator for decoded string bodies.
 Package bumparena is a monotonic bump allocator for decoded string bodies.
endian
Package endian exposes a build-time constant for native byte order.
 Package endian exposes a build-time constant for native byte order.
fsst
Package fsst implements FSST (Fast Static Symbol Table) string compression: a learned table of up to 255 substrings (each 1..8 bytes) replaced by 1-byte codes, with code 255 reserved as an escape for a following literal byte.
 Package fsst implements FSST (Fast Static Symbol Table) string compression: a learned table of up to 255 substrings (each 1..8 bytes) replaced by 1-byte codes, with code 255 reserved as an escape for a following literal byte.
intern
Package intern provides a small fixed-size string cache used by the decoder to deduplicate repeated map keys.
 Package intern provides a small fixed-size string cache used by the decoder to deduplicate repeated map keys.
internarena
Package internarena is a bytes-only bump-pointer allocator for the Dense-mode intern table.
 Package internarena is a bytes-only bump-pointer allocator for the Dense-mode intern table.
mapsgen command
internal/mapsgen — code generator for typed map encode/decode fast paths.
 internal/mapsgen — code generator for typed map encode/decode fast paths.
rans
Package rans is a static order-0 byte range Asymmetric Numeral System coder (the canonical rans_byte: 32-bit state, byte renormalization, a 12-bit normalized frequency table).
 Package rans is a static order-0 byte range Asymmetric Numeral System coder (the canonical rans_byte: 32-bit state, byte renormalization, a 12-bit normalized frequency table).
reflectutil
Package reflectutil holds the reflect-based helpers the qdf codec uses to allocate slices and maps without going through reflect.Value.
 Package reflectutil holds the reflect-based helpers the qdf codec uses to allocate slices and maps without going through reflect.Value.
unsafestr
Package unsafestr provides a zero-copy conversion from []byte to string.
 Package unsafestr provides a zero-copy conversion from []byte to string.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.