qdf

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: MIT Imports: 30 Imported by: 0

README

qdf

ci codeql codecov benchmarks Go Reference Go Report Card Go License Status

Fast, compact binary serialization for Go.
encoding/json-style API · columnar per-column codecs · up to 68% smaller than Protobuf, 6× smaller than JSON

qdf is a single-pass, streaming binary format with encoding/json ergonomics (Marshal / Unmarshal / NewEncoder / NewDecoder) — no IDL, no schema registry, types picked up from struct tags. It dedups repeated strings and keys inline and columnar-compresses slices of structs with per-column codecs (frame-of-reference, delta, RLE, dictionary, Gorilla XOR, FSST, rANS), so it shrinks across records where per-record formats can't. A go:generate tool emits reflection-free MarshalQDF / UnmarshalQDF for the last 10–15 %.

Wire size on real telemetry batches · GitHub Actions ubuntu-latest · Go 1.26
batch encoding/json protobuf qdf balanced qdf compression
OTLP traces 1 027 KB 562 KB 138 KB 130 KB
logs 245 KB 156 KB 44 KB 44 KB
RTB bids 559 KB 328 KB 244 KB 201 KB
events 123 KB 65 KB 40 KB 40 KB
IoT floats 469 KB 208 KB 158 KB 148 KB

qdf's wire is smaller than Protobuf on every fixture (−24 … −77 %) and 2.3–7.5× smaller than JSON — it interns repeated strings and columnar-compresses across records, where per-record formats can't. On a real Active-Directory host dump it also decodes ≈6× faster than JSON and ~1.5× faster than msgpack at ~40 % of the allocations, and encodes faster than both on telemetry-shaped batches (msgpack edges encode on some high-cardinality sets). Honest trade-offs: OptCompression spends encode CPU for the smallest wire, and protobuf/flatbuffers still win raw single-tiny-message decode. Full tables + reproduction recipe: docs/COMPETITIVE.md.

Deep dives: qdf — decodes less, packs harder, and lets you query the bytes · Shrinking AI embeddings on the wire — a lossy vector codec that beats Google's TurboQuant at equal recall

When to use qdf

Reach for qdf when… Consider something else when…
You batch or stream structured records — logs, traces, spans, metrics, events, rows — where strings and keys repeat across the batch. You send one tiny message at a time with no repetition: protobuf / flatbuffers carry less fixed per-message overhead.
You want encoding/json ergonomics (struct tags, no IDL, no codegen step) at a fraction of the size and decode cost. You need a cross-language schema contract with generated stubs in many languages: reach for protobuf / an IDL.
Your workload is decode-heavy and dominated by materialising strings — pair it with a decode arena / no-copy. You need random access into a huge mmap'd file without decoding: flatbuffers / capnproto are built for that.
You want to filter columnar batches without a full decode (predicate pushdown, selective columns). You need maximum encode throughput on high-cardinality data with no repetition: msgpack can edge qdf on encode there.
You store AI embeddings and can trade bit-exactness for size (lossy vector codec at a fidelity budget). You require a frozen, versioned wire spec today: qdf is alpha and the format still moves (see Status).

qdf is Go-only and pure-Go (zero dependencies). One Unmarshal reads any output Marshal produces regardless of the encode-side option bits, so you can turn codecs on later without a migration.

Design philosophy — from "data as bytes" to "data as a model"

qdf treats a message not as a tree to serialize byte-for-byte, but as a small set of distinct values plus a stream of references and residuals against the context already seen. (The original working name, Quantum Density Format, was a borrowed metaphor for this state/collapse view — not physics; the engineering below is what actually ships.)

The starting observation: in a payload like

{ "users": [
    {"country": "LT"},
    {"country": "LT"},
    {"country": "LT"} ] }

the byte "LT" is written three times even though, from an information-theoretic point of view, it is a single state with high probability. Classical formats (JSON, MessagePack, CBOR, Protobuf) treat data as tree → bytes. qdf treats it as

data = (state set, references, entropy weights)

— a finite set of distinct values plus a stream that "collapses" each position onto one of them. The wire dialect that ships today realises the practical subset of that idea:

Concept What it maps to in qdf today
State set — the discrete values a position can take. The Dense-mode intern table. The first occurrence of a string or []byte value writes it in full and assigns it a stable ID.
Wavefunction collapse — a single observation picks one state. Every subsequent occurrence emits a state_ref tag plus a varint ID. Decoding "collapses" the reference back to its stored value.
Density — keep only what carries information; minimise entropy H(D | C) against the existing context. Repeated keys / values across a message (and across a Dense stream) cost 1–3 bytes rather than their full length. Numeric and bool slices use QPack codecs (FOR, Delta+FOR, RLE, dictionary, Gorilla XOR, ALP decimal, raw-LE bulk) that auto-select the smallest predicted form per slice; slices of homogeneous structs transpose to per-column codecs, and an enum-like string column is dictionary-coded (distinct table + bit-packed index per row) when that beats the per-value floor of one ref byte per row — including a skewed low-cardinality column whose dominant value clusters into few runs (the gate compares the index against n, not the run count, so clustered enums are no longer wrongly sent to the per-value path). When the index itself is skewed, it is QPack-coded (tagColStrDictQ, 0xFC) — RLE / dictionary / FOR over the per-row indices — so the index costs its entropy, not ceil(log2 d) bits, emitted only when strictly smaller than the flat pack (the Balanced-tier counterpart to the whole-body rANS pass). A dictionary whose distinct values share prefixes (SID / path / DN / URL columns) is front-coded (tagColStrDictFC, 0xFA): the table is sorted and each entry stored as sharedPrefixLen + suffix, emitted only when strictly smaller (−36.5 % on real AD dictionary tables, up to −92.6 % on SID columns). A mixed struct — mostly scalar/string columns with one or two map / slice / nested fields (the common log / AD / span / RTB record shape) — transposes its eligible columns and keeps the rest as a per-row residual tail (hybrid columnar, tagHybridColStruct, 0xF7), under OptCompression, instead of falling back to fully row-major. High-cardinality string columns (URLs, log lines, paths) where the whole-string dictionary can't help are eligible for FSST (tagColStrFSST, 0xF6), which learns a symbol table of up to 255 substrings and compresses at the byte level — emitted only when strictly smaller. A high-cardinality column whose values are drawn from a small alphabet (hex / base32 / base64 / decimal IDs — trace, span, request IDs, hashes, GUIDs) is alphabet-packed (tagColStrAlpha, 0xFB): each character is stored in ceil(log2 |A|) bits instead of 8 (hex halves the body), the one class the dictionary, front-coding and FSST all miss — captured without the rANS CPU cost, so it lands on the Balanced tier, emitted only when strictly smaller. Field-name headers in generated code are pre-encoded once per type and concatenated without further work.
Probabilistic / residual coding — predict, then store only the deviation. QPack's Delta+FOR codec is exactly this for monotonic integer sequences: encode the first value plus the bit-packed residual against a aᵢ = aᵢ₋₁ + minΔ predictor. Gorilla does the same for floats by XOR-ing each sample against the previous one and storing the differing bits only.
Entanglement — correlated values that constrain each other (e.g. city = "Vilnius" ⇒ country = "Lithuania"). Three stacked predictors on the Dense state stream. Markov-0 (tagStateRepeat, 0xE8) collapses an immediate repeat of the previously emitted state-ref to a single byte. MTF rank (tagStateMTF, 0xE9) encodes the LRU rank of the touched ID when that varuint is shorter than the raw id. Markov-1 pair (tagStatePair, 0xEA) stores the most-recent successor observed after each prev ID (top-1, K=1) and encodes a hit as 0xEA + 0x00 (rank always 0) — wins when the prev → curr transition is predictable and the raw id needs a multi-byte varuint. The encoder picks the shortest of the four variants per emission, so the wire is never larger than the plain tagStateRef encoding. A full conditional-probability table beyond order-1 stays in the reserved 0xEB / 0xED..0xEF block.
Shape interning — repeated structure means the layout itself is information. Dense mode emits structs (and map[string]any of stable shape via the reflect-struct path) through tagMapShape (0xEC). First occurrence declares the shape inline: 0xEC, 0, varuint(N), N × key. Subsequent occurrences of the same shape emit 0xEC + varuint(shapeID) + N × value — keys are not re-emitted. Per-record saving on an array of identical-shape structs is N × 2 bytes for the elided state-refs plus the map header. The shape table is per-stream, addressed by *typeDesc on the encoder and by sequential ID on the wire. Shapes never collide across types because the encoder keys the binding on the descriptor pointer.
Arithmetic / range coding (rANS) — push the encoded stream to its Shannon limit. Shipped behind OptRANS (in OptCompression): a static order-0 rANS pass over the whole body, applied only when it shrinks (never larger). Squeezes the residual byte-entropy the structural codecs leave — e.g. −37 % on trace batches — at ~4–6× CPU, so it stays in the opt-in compression tier.

The conceptual roadmap (state table → entanglement graph → predictive encoder) is preserved in the codebase: tag space and the encoder interface are designed to let those layers slot in above the current one without a format break. The MVP that ships now is the part of the idea that has a positive CPU / size tradeoff on day-one workloads — logs, traces, columnar rows, embedding vectors.

The realistic conclusion: the real shift is not the "quantum" framing but the move from "data as bytes" to "data as a model of the world that produced it".

Why another format

encoding/json is verbose and slow. vmihailenco/msgpack/v5 is fast but allocates a lot on decode and has no built-in deduplication for repeated keys / values — which is the dominant cost on real telemetry, log, and trace payloads where the same handful of strings (service names, log levels, region codes, span attributes) appears thousands of times per batch.

qdf attacks two costs at once:

  1. Wire size on repetitive data. A Dense-mode encoder maintains an inline interning table: the first time "region":"eu-west-1" appears it is written in full; subsequent occurrences emit a 2-byte back-reference. Streams share that table across messages, so a 1000-event log batch with eight distinct region codes carries each code once. The decoder follows the same protocol — no out-of-band dictionary, no two-pass encode.

  2. CPU and allocations on decode. The reflect path uses a cached per-type descriptor with unsafe.Pointer + field offset access instead of reflect.Value.Field(i), plus specialised type-specific decoders for []string, []int*, []float*, map[string]string, map[string]int, map[string]any, etc. A direct-mapped key interner deduplicates map keys across decode calls without a pool or chained hash table. On a map[string]any decode with repeated keys the allocator runs 3 times per call versus json's 37.

The format is single-pass and streaming-safe in both directions. There is no schema language, no IDL, no compile-time registration — types are picked up via Go struct tags (qdf:"name", falling back to json:"name").

Measured against the field

On realistic batch payloads (GitHub Actions ubuntu-latest · Go 1.26), qdf's wire is smaller than protobuf on every fixture — because it dedups and columnar-compresses across records, while json/msgpack/protobuf/flatbuffers encode each record independently:

wire size vs protobuf OTLP traces logs RTB events IoT floats
qdf_balanced −75 % −72 % −25 % −39 % −24 %
qdf_compression −77 % −72 % −39 % −39 % −29 %

It also allocates far less under load (IoT encode: qdf_balanced 3 allocs/op vs protobuf 385). Honest trade-offs: qdf_speed wire ≈ msgpack; qdf_compression encode trades CPU for bytes; protobuf and flatbuffers win raw single-tiny-message decode. Full tables (json / msgpack / protobuf / flatbuffers × five fixtures, wire + memory) and the reproduction recipe are in docs/COMPETITIVE.md.

A concrete head-to-head on a real Active-Directory host dump (adalanche localmachine data — unique GUID/SID identifiers alongside repeating group, department, and object-class strings; same data through each library, typed structs):

AD host dump wire encode decode decode allocs
encoding/json 532 KB 1.5 ms 4.9 ms 15 851
vmihailenco/msgpack 500 KB 0.65 ms 1.2 ms 13 261
qdf — OptBalanced (default) 209 KB 0.77 ms 0.79 ms 6 413
qdf — OptCompression 127 KB 5.7 ms 1.6 ms 5 597

OptBalanced is 2.5× smaller than json / 2.4× smaller than msgpack, decodes ≈6× faster than json and ~1.5× faster than msgpack, at ~40 % of json's allocations. msgpack edges it on encode wall-time for this high-cardinality set (qdf wins encode on the telemetry fixtures above); OptCompression trades encode CPU for the smallest wire. Protobuf/flatbuffers still win raw single-message decode.

Wire layout
+-----+-----+-----+-----+-----+
| 'Q' | 'D' | 'F' | ver |flags|
+-----+-----+-----+-----+-----+
| body (tagged values)        |
+-----------------------------+

A 5-byte header (3-byte magic + 1-byte version + 1-byte flags) and a tagged body. The flag bit distinguishes Fast from Dense; the decoder auto-detects.

Tag space is msgpack-inspired with a few additions:

Range Purpose
0x00..0x7F positive fixint (0..127)
0x80..0x9F fixstr (len 0..31)
0xA0..0xBF fixarr (len 0..31)
0xC0..0xCF nil, bool, int / uint / float of width 8–64
0xCD..0xCF str8 / str16 / str32
0xD0..0xD2 bin8 / bin16 / bin32
0xD3..0xD4 arr16 / arr32
0xD5..0xD7 map8 / map16 / map32
0xD8..0xDF negfixint (-1..-8)
0xE0..0xE2 intern_str / state_ref / intern_bin (Dense)
0xE3 QPack: bit-packed []bool
0xE4 QPack: raw-LE numeric slice
0xE5 QPack: Frame-of-Reference bit-packed ints
0xE6 QPack: Delta + zigzag + FOR (monotonic ints)
0xE7 QPack: Gorilla XOR-coded float slice
0xE8 Dense: state-ref repeat (Markov-0 predictor)
0xE9 Dense: state-ref MTF rank (Move-To-Front)
0xEA Dense: state-ref pair rank (Markov-1)
0xEB QPack: run-length encoded integer slice
0xEC Dense: struct/map shape declare / reuse
0xED QPack: dictionary-coded integer slice
0xEE QPack: Patched FOR integer slice (outliers)
0xEF QPack: columnar []struct container
0xF0..0xF3 ext / timestamp
0xF4 QPack: ALP decimal-coded []float64 / []float32 slice (kind byte selects width)
0xF5 QPack: dictionary-coded string column
0xF6 QPack: FSST-coded string column (inside columnar)
0xF7 QPack: hybrid columnar container (mixed []struct)
0xF8 QPack: bulk-slab string column (inside columnar)
0xF9 QPack: constant string column (inside columnar)
0xFA QPack: front-coded dictionary string column (inside columnar)
0xFB QPack: alphabet-packed string column (inside columnar)
0xFC QPack: dictionary string column with QPack-coded index (inside columnar)
0xFD..0xFF reserved (n-gram graph, future)

The 5th header byte holds two flag bits: FlagDense (0x01) for the intern dialect, and FlagQPack (0x02) as an early hint that the body may carry codec tags from the QPack codec range (0xE3..0xEF, 0xF4..0xFC). A reader that does not implement the QPack tags fails with ErrBadTag on first contact; it never decodes a packed payload as scalar by accident.

Alpha note: qdf is pre-1.0. The wire format is still being shaped — tags may shift, version may bump, and "older" decoders here mean "earlier in alpha", not stable releases. No backwards-compat promise yet.

All multi-byte integers and floats are little-endian. amd64 and arm64 are the supported targets.

Quick start

go get github.com/alex60217101990/qdf

Requires Go 1.26.

Runnable examples (go run ./examples/<name>): telemetry (dedup vs json) · query (predicate pushdown) · embeddings (lossy vector codec) · streaming (shared-dict stream) · decodealloc (arena zero-alloc decode). More runnable snippets — arena, no-copy, options, delta, columns — render on pkg.go.dev.

package main

import (
    "fmt"

    "github.com/alex60217101990/qdf"
)

type Event struct {
    ID      int               `qdf:"id"`
    Source  string            `qdf:"source"`
    Payload []byte            `qdf:"payload"`
    Attrs   map[string]string `qdf:"attrs"`
}

func main() {
    in := Event{
        ID:     42,
        Source: "ingest",
        Attrs:  map[string]string{"region": "eu-west-1", "version": "v3"},
    }

    b, err := qdf.Marshal(in, qdf.OptSpeed)
    if err != nil {
        panic(err)
    }
    fmt.Printf("wire: %d bytes\n", len(b))

    var out Event
    if err := qdf.Unmarshal(b, &out); err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", out)
}
Encode profile is per-call

There is one encode entry point — Marshal(v, opts) — and the opts bit-mask picks which codecs run for that specific call. The convenience bundles cover the common tradeoffs:

Bundle When to use
qdf.OptSpeed Fast path. Tightest CPU cost; size comparable to msgpack. Drop-in for encoding/json behaviour.
qdf.OptBalanced Repetitive payloads — logs, telemetry, columnar rows. Strings intern once; numeric and bool slices use QPack codecs; struct shapes intern; Markov-1 + MTF run on state-refs.
qdf.OptCompression OptBalanced plus the heavier wire-size codecs: Gorilla XOR for smooth float series, ALP for quantized/decimal []float64 and []float32, FSST substring-level compression for high-cardinality string columns (URLs, log lines, paths), hybrid columnar (transposes the eligible columns of a mixed struct — one with a map / slice / nested field — keeping the rest row-major, so log / AD / span records get columnar treatment they otherwise couldn't), and a final order-0 rANS entropy pass over the whole body (never larger). Trades encode CPU for smaller wire — pick it for backup / cold storage.
custom mix Or-combine individual bits (OptDense | OptQPack | OptShapeIntern …) when one of the bundles is one click off the desired tradeoff. 
b, _ := qdf.Marshal(event,    qdf.OptSpeed)        // hot path
b, _ := qdf.Marshal(batch,    qdf.OptBalanced)     // telemetry / logs
b, _ := qdf.Marshal(snapshot, qdf.OptCompression)  // backup / archive
b, _ := qdf.Marshal(payload,                       // tuned mix
    qdf.OptDense|qdf.OptQPack|qdf.OptShapeIntern)

A single qdf.Unmarshal reads everything. The wire header is self- describing; the receiver never has to know which option mix the sender picked.

Picking the right combo for a concrete workload — hot path, telemetry, metric series, embeddings, backup — is covered in docs/CHOOSING.md, with head-to-head numbers vs json and msgpack per scenario.

Dense mode internals — what each bit buys you

OptDense activates the inline intern table. The four codec bits that compose OptBalanced layer on top of it. They share one rule: the encoder picks the shortest variant per emission, so Dense wire is never larger than the same payload at OptSpeed. If the predictors do not pay, the byte cost is identical to a plain state-ref.

Tag Predictor Wins when Doesn't help when
0xE8 Markov-0: same ID as the previous emission Runs of the same value (region, region, region). Distinct values in a row.
0xE9 MTF: encode LRU rank instead of raw id Heavy reuse of a small hot subset of strings that were interned early. Random access pattern with no recency.
0xEA Markov-1: top-1 predictor of the previous emission's successor Correlated pairs — country → city, service → region. Only emits when the raw id needs ≥ 2 varuint bytes, so the wire never grows on small state tables. Unique transitions, low-cardinality state.
0xEC Shape interning: declare struct shape once, reuse by id Arrays / streams of the same struct type. Per-record saving is ≈ N × 2 bytes (elided key state-refs + map header). One-shot encodes of a unique struct type.

The state table is shared per stream in StreamEncoder / StreamDecoder, so a batch of 1 000 log events with eight distinct region codes ships each region once. Marshal* calls do not share — each call starts with an empty table.

Where Dense is worth it: logs, traces, telemetry, columnar rows, event batches, audit streams, snapshot dumps — anything where a small vocabulary of strings and a stable struct shape repeat thousands of times. Expected envelope: 25 – 55 % smaller than JSON at 2–6× the encode speed.

Where Dense is not worth it: single-shot encodes of small unique payloads (Markov / shape predictors never reach their amortisation point, you pay ≤ 2 bytes of prelude for nothing), or strict-throughput paths where decode CPU is the bottleneck — Dense decode is ~15 % slower than Fast because of the state-table lookups. Reach for OptSpeed or OptQPack there.

Where it is wrong: cryptographic / forensic contexts that need byte-stable wire across implementations and versions. Dense embeds predictor state, intern-ID ordering, and shape IDs that depend on emission order. Use OptSpeed if you hash or sign the wire.

Bit reference
Bit What it gates
OptDense Inline intern table; required by everything below.
OptQPack Numeric / bool slice codecs (FOR, Delta+FOR, Gorilla, bit-pack).
OptShapeIntern tagMapShape for struct emissions (declare once, reuse by id).
OptPairPred Markov-1 successor predictor (tagStatePair, 0xEA).
OptMTF Move-to-Front rank coding on state-refs (tagStateMTF, 0xE9).
OptColumnIndex Column-length index on columnar []struct for selective decode (opt-in, ~4 B/column; default wire unchanged when off).
OptMapShape Key-set interning for map[string]V fields (declare keys once, reuse by shape id). ~−24 % encode CPU and ~−26 % wire on recurring-key tag-maps (telemetry/logs); opt-in, default wire unchanged when off.
Bundle Composition
OptSpeed 0 — Fast mode, no codecs.
OptBalanced OptDense | OptQPack | OptShapeIntern | OptPairPred | OptMTF.
OptCompression OptBalanced + Gorilla XOR + ALP decimal floats ([]float64 / []float32) + FSST for high-cardinality string columns + an order-0 rANS entropy pass (never larger). Trades CPU for wire size; for backup / cold storage.

Options is a uint32 carried by value, so Marshal and AppendMarshal add zero per-call allocations over the pool / output-clone overhead. Encoders are pooled in a single shared pool (encPool) keyed only by buffer; the configuration is applied on each acquire via applyOpts. Markov-0 (tagStateRepeat, 0xE8) is always on inside OptDense because it costs nothing on the wire when the predictor misses.

A note on tuning knobs that do not live on the bit-mask: the intern threshold (SetIntern) and the cycle-depth ceiling (SetMaxDepth) stay on the *Encoder itself. Use the low-level encoder API for those.

Generic API (Go 1.18+ generics)

Marshal(v any, opts) boxes the argument through interface{} and then makes one reflect copy for value-typed inputs. The generic helpers skip both: T is fixed at the call site, reflect.TypeFor[T]() resolves at compile time, and unsafe.Pointer(&v) points directly at the caller's stack.

Generic Equivalent to
MarshalT[T any] Marshal (typed)
AppendMarshalT[T any] AppendMarshal (typed)
UnmarshalT[T any] Unmarshal (typed)

Wire output is byte-identical. The win is on the encode side: one fewer allocation per call (-80 B) and 25–40 % faster on small/medium payloads.

buf, _ := qdf.MarshalT(event, qdf.OptSpeed)
var out Event
_ = qdf.UnmarshalT(buf, &out)
Direct entry points (skip reflection entirely)

MarshalT still goes through descOf to walk the type. Types that already implement MarshalQDF / UnmarshalQDF (either hand-written or generated by cmd/qdfgen) can bypass the descriptor cache and the runtime type assertion via the Direct generics. The type parameter is constrained to the Marshaler / Unmarshaler interface, so the method call is resolved at compile time and the compiler inlines it.

Direct When to use
MarshalDirect[T Marshaler](v T) ([]byte, error) Hot paths where the type is known to have generated codecs.
AppendMarshalDirect[T Marshaler](dst []byte, v T) Same, when the caller owns the destination buffer.
UnmarshalDirect[T Unmarshaler](data []byte, out T) Inverse of MarshalDirect. Falls back to Unmarshal on FlagDense because generated code does not maintain the intern table.

Wire is identical to Marshal / Unmarshal; the path is Fast-mode only (Dense or QPack stay on the reflect path so the intern table is correctly resolved). On the cmd/qdfgen Sample fixture (11-field struct):

Encode ns/op B/op allocs
json 2760 576 8
qdf_reflect 910 480 3
qdf_codegen 800 504 6
qdf_direct 545 160 1

(i7-9750H · Go 1.26 · -count=6 median. Absolute ns are machine-specific; the ratios — qdf_direct ≈ 5× faster than json, ≈ 1.6× faster than the reflect path, at 1 alloc vs json's 8 — are the stable signal.)

Decode-side performance is bounded by the receiver's UnmarshalQDF. The reflect path is heavily pooled (Decoder pool + per-decoder key intern cache); generated code from qdfgen uses Decoder.InternKey and matches it. Hand-rolled receivers that allocate a new Decoder per call will not.

buf, _ := qdf.MarshalDirect(&event)
var out Event
_ = qdf.UnmarshalDirect(buf, &out)
QPack: math-driven slice codecs

QPack auto-selects, per slice, the codec with the smallest predicted wire form. Round-trip is lossless for every type (NaN/±Inf survive on floats).

Codec Trigger Math
Bitpack []bool 1 bit per element, LSB-first per byte. 8× smaller for free.
Raw-LE numeric slice, large delta range Unsafe-slice cast → single memmove of LE bytes. 10-50× faster.
FOR numeric slice, clustered values Frame-of-Reference: store min and ceil(log₂(max-min+1))-bit deltas.
Delta+FOR monotonic / near-monotonic integers Δᵢ = aᵢ - aᵢ₋₁, zigzag bias, FOR over the deltas.
Patched FOR integer slice with rare outliers (latency spikes) FOR body at a reduced width b + an exception list for the few values that don't fit. ~50% smaller than FOR on spiky columns.
Gorilla float slices (explicit opt-in via low-level API) XOR with previous, run-length leading/meaningful-bit window. (Facebook VLDB 2015.)
FSST high-cardinality columnar string column (URLs, log lines, paths) under OptCompression Learns a symbol table of up to 255 substrings (1–8 bytes each); replaces frequent substrings with 1-byte codes. Complements the dictionary codec: dict deduplicates whole strings; FSST compresses at the substring level where cardinality is too high for a dictionary. ~76–79 % wire reduction on URL/log-line corpora. (Boncz/Neumann/Leis, VLDB 2020.)

Head-to-head on a mixed 256-bool / 512-monotonic-u64 / 512-i64 / 256-f64 payload (Intel i7-9750H):

Format Bytes Encode (ns/op) Decode (ns/op)
json 10,739 48,000 200,000
msgpack 11,808 64,000 80,000
qdf_fast 6,694 6,500 14,000
qdf_qpack 2,132 2,300 2,600
qdf_dense 2,134 2,500 2,500

For sequences where the deltas alone collapse, the gain is much larger — e.g. 1024 consecutive Unix-second timestamps shrink from 8201 bytes (raw) to 16 bytes (Delta+FOR), a 512× reduction.

Streaming
var w bytes.Buffer
enc := qdf.NewStreamEncoder(&w, qdf.Dense)
for _, ev := range events {
    if err := enc.Encode(ev); err != nil {
        return err
    }
}
enc.Close()

dec := qdf.NewStreamDecoder(&w)
defer dec.Close()
for {
    var ev Event
    if err := dec.Decode(&ev); err == io.EOF {
        break
    } else if err != nil {
        return err
    }
    // ...
}

Dense streams preserve the intern table across messages — the second occurrence of "region":"eu-west-1" in the batch is a 2-byte reference rather than a 13-byte string.

Each message is length-framed (a uvarint byte-count precedes its body; the 5-byte header is written once up front), so a message of any size round-trips — even across a reader that delivers one byte at a time — and io.EOF cleanly marks the end. Struct tags and all Dense/codec options work exactly as in one-shot Marshal. Decode-side allocation has two levers: dec.SetNoCopy(true) aliases the window (zero copies, values valid for the stream's lifetime), and dec.SetArena(a) bump-packs string bodies into a caller-owned arena (one amortized copy, values valid as long as the arena — Reset it per batch). To reuse one encoder/decoder across independent streams (pay the intern-table construction once), Reset them between streams; pair with the arena to bound a long stream's memory. The encoder reuses one pooled buffer (allocation-free steady state); target a *bytes.Buffer to collect into a []byte. The three whole-batch features — OptColumnIndex, Where/Select pushdown, and OptRANS — are not part of streaming (they act on a single complete batch; use one-shot Marshal/Unmarshal for those).

Selective columnar decode (OptColumnIndex)

qdf transposes a slice of flat structs into a columnar container and compresses it column-by-column. With OptColumnIndex the producer writes a fixed-width column-length index (one uint32 per column, ~4 B each) after the shape declaration. A consumer that only needs some columns then advances past the rest using the index instead of decoding them — cost becomes O(columns you read), not O(all columns).

Two ways to read a subset:

// Producer opts in to the index.
buf, _ := qdf.Marshal(events, qdf.OptBalanced|qdf.OptColumnIndex)

// 1) Typed subset — no new API. Decode into a struct whose fields are a
//    subset of the wire (matched by qdf tag / field name). Wire columns
//    absent from the target are skipped; target fields absent from the
//    wire are left zero.
type Subset struct {
    Level   string `qdf:"level"`
    Service string `qdf:"service"`
}
var sub []Subset
_ = qdf.Unmarshal(buf, &sub)

// 2) Explicit column names — drives a typed subset or the dynamic
//    *[]map[string]any form. With no fields it behaves like Unmarshal.
var rows []map[string]any
_ = qdf.UnmarshalColumns(buf, &rows, "level", "service")

Decoding a 3-field subset of a 16-field columnar batch (i7-9750H):

ns/op B/op allocs/op
full (16 fields) 94,000 283,100 66
subset (3 fields) 16,440 53,610 32

≈ 5.7× faster, ≈5.3× fewer bytes — it decodes 3 columns and skips 13.

Notes: the option only affects payloads the encoder transposes into the columnar container; the flag is backpatched only when the index is actually emitted, so the default columnar wire stays byte-identical when the option is off (and the default non-indexed decode path has no regression). Without the index a subset decode still returns correct results by decoding and discarding unwanted columns — correct, just not fast. The index is a single-message feature and is not emitted in streaming mode.

Predicate pushdown (Where / Select) — the headline feature

This is what no other Go serializer does. On a columnar []struct batch you can filter rows and project columns inside Unmarshal — the decoder tests typed predicates against whole columns, builds a row bitmask, and materializes only the surviving rows of the columns you keep. The rows you filter out are never reconstructed; the columns you don't select are never touched. It is "SELECT ts, code WHERE level='ERROR' AND code>=500" over a qdf blob, with no database, no second pass, and zero per-value boxing.

buf, _ := qdf.Marshal(events, qdf.OptBalanced|qdf.OptColumnIndex)

type Hot struct {
    TS   int64 `qdf:"ts"`
    Code int32 `qdf:"code"`
}
var hot []Hot // filter on level (absent from Hot), keep ts+code of matches
_ = qdf.Unmarshal(buf, &hot,
    qdf.Where("level", func(s string) bool { return s == "ERROR" }),
    qdf.Where("code", func(c int32) bool { return c >= 500 }))

On a wide 9-column, 2000-row batch at 1% selectivity, pushdown moves ≈4.4× fewer bytes and runs ≈2.3× faster than a full decode + manual filter loop — because it never materializes the rows or columns it discards.

This is the unique selling point of qdf versus encoding/json, msgpack, protobuf, gob, and friends — none of them can read back a filtered, projected subset of a batch without first decoding the whole thing. The full rationale, API reference, nullable/error semantics, internals, and a step-by-step tutorial live in docs/PREDICATE-PUSHDOWN.md.

Go one level deeper with zone maps (OptZoneMap): an ordered int/uint/float64 column is chunked into 256-row zones with a [min,max] summary, so a range filter (WhereRange / WhereCmp) skips whole zones without decoding them — 87–97 % of an ordered column. Guide: docs/ZONEMAP.md.

Structural delta (Diff / Apply)

qdf can compute a structural delta between two values and ship only what changed. Diff(old, new, opts) walks both values field/element/key-wise and emits a self-describing patch carrying only the locations that differ; Apply(&base, patch) merges it back in place to reconstruct new. This is the state-sync primitive no mainstream Go format ships natively — k8s reconciliation, config hot-reload, CDC / event sourcing, realtime netcode.

patch, _ := qdf.Diff(old, updated, qdf.OptBalanced) // only the changes
// ... ship `patch` (often a few dozen bytes) ...

base := old
_ = qdf.Apply(&base, patch) // base now == updated, in place

Structs (incl. nested), scalars / string / []byte / [N]byte, positional slices/arrays, and per-key maps (with tombstone deletes) are all diffed. schemaFP + baseFP fingerprints reject a patch applied to the wrong type or a divergent base (ErrPatchSchemaMismatch / ErrPatchBaseMismatch). On a 1000-record batch with one changed field the patch is 42 B vs a 26 677 B full Marshal — ~635× smaller. Full semantics, wire format, and the deletion / nil-vs-empty rules are in docs/DELTA.md.

Canonical encoding

OptCanonical makes logically-equal values serialize to byte-identical output — map keys are emitted in sorted order (all key kinds) and floats are normalized (-0.0 → +0.0, any NaN → a canonical quiet NaN). The bytes are ordinary qdf (decode unaffected) and safe to hash, sign, content-address, or dedup.

b, _ := qdf.Marshal(v, qdf.OptBalanced|qdf.OptCanonical)
sum := sha256.Sum256(b) // stable across runs, machines, and map rebuilds

Zero overhead for map-free / normal-float values, composes with Diff, and is lossy only for the sign of zero and NaN payloads (use the default mode for a bit-exact float round-trip). Full guarantee, scope, and caveats are in docs/CANONICAL.md.

Zero-extra-copy encode

AppendMarshal lets callers own the destination buffer:

out, err := qdf.AppendMarshal(out[:0], v)

Pair with a goroutine-local buffer to drop the per-call allocation entirely.

Decode performance levers — the four ways to cut decode allocations (decode fewer fields, [N]byte ids, WithNoCopy, WithArena), with measured numbers and when to use each, are collected in docs/DECODE-PERF.md.

Zero-copy decode (WithNoCopy)

Decode is allocation-bound: copying each string/[]byte body out of the input buffer dominates. WithNoCopy() makes the decoded values alias the input buffer instead of copying — near-zero allocations, ~1.7× faster on string-heavy batches.

// data must outlive `out` and must NOT be mutated/reused while `out` is in use.
var out LogBatch
_ = qdf.Unmarshal(data, &out, qdf.WithNoCopy())

⚠️ Lifetime contract. The decoded strings/bytes point INTO data. If data is recycled, mutated, or freed before you finish with out, the values silently corrupt — a use-after-free the race detector cannot catch. Never use it on a pooled/reused buffer (e.g. an HTTP request body). Safe for caller-owned, long-lived, immutable input (mmap, a file read into memory, batch analytics). It is opt-in for this reason; the default copies.

Measured on a 1000-row string-heavy batch (i7-9750H): 7002 → 3 allocs/op, −38 % B/op, ~1.7× faster. It works on both the reflect path and codegen types (generated UnmarshalQDFOpts threads the flag through nested decodes). For a Decoder/StreamDecoder you drive yourself, use SetNoCopy(true).

Arena decode (WithArena)

When the wire buffer is recycled (a pooled HTTP request body) WithNoCopy is unsafe — its aliases dangle the moment the buffer returns to the pool. A decode Arena is the safe alternative: it copies the strings out, but into one dense bump-allocated block per epoch instead of one heap allocation per string. The wire buffer is free to go back to its pool immediately, and across a batch/stream the block allocation amortizes to ~zero.

a := qdf.NewArena()                 // one arena per epoch (batch / request)
for i, msg := range messages {
    _ = qdf.Unmarshal(msg, &out[i], qdf.WithArena(a))
}
// out's strings live in `a`; the GC frees it when `out` is dropped — no Reset,
// no lifetime bookkeeping. (Call a.Reset() to reuse it across epochs for zero
// steady-state alloc; see the lifetime contract in docs/ARENA.md.)

Safe by default (strings are owned copies, the GC keeps the block alive via an interior pointer), opt-in, and adaptive — the win scales with string density and is never negative:

Corpus time allocs/op (off → on)
Telemetry log batch (1 000) −35 % 3 502 → 3
AD / directory export (200) −26 % 4 856 → 605
Event batch ([]byte + string) −13 % 1 003 → 504
IoT sensor batch (numeric-heavy) −4 % 291 → 164

Works on the reflect path and codegen types (generated UnmarshalQDFArena threads the arena through nested decodes); drive a Decoder/StreamDecoder directly with SetArena(a). Full lifetime contract, WithNoCopy comparison, and internals in docs/ARENA.md.

Decode into a pooled slice (backing reuse)

The decoded result slice backing is the dominant decode allocation. When you pass a *[]T whose slice already has enough capacity for the row count, the decoder reuses that backing instead of allocating a fresh one — ideal for a server that decodes many messages into a pooled slice:

out := pool.Get().([]Row)[:0] // pre-sized; cap >= incoming row count
_ = qdf.Unmarshal(data, &out)  // reuses out's backing, no result allocation

The reused backing is overwritten in place (elements are zeroed first, so a shorter/older wire schema can't leak stale data), so don't keep another live slice or map reference from the target aliasing it across the call. A nil or too-small destination allocates fresh — default var out []T usage is unchanged, no opt-in flag needed. Works on the row-major and columnar decode paths, for every element type (pointer-free elements take a raw clear, pointer-containing ones a barrier-correct typed clear).

Map recycling. The per-element maps in a reused []struct{…map…} target (or a []map) are the dominant allocation for map-bearing payloads — telemetry tag maps, label sets. Decoding into a reused target now recycles those maps (clears and refills them in place) instead of re-allocating, since make(map) + bucket growth is ~83 % of map-heavy decode bytes. This is automatic and data-adaptive (no flag): it kicks in only when the destination already holds maps, covers maps at any struct-nesting depth, the generated fast paths, the reflect and schemaless (map[string]any) paths, and successive StreamDecoder frames into the same target. Wire is unchanged; nil-vs-empty and stale-key semantics are preserved (recycled maps are cleared before refill). Same contract caveat: a map value you retained from a previous decode of the target is cleared and reused on the next decode into it.

Measured, decode into a pre-sized target (i7-9750H):

element shape B/op (fresh → reuse) ns/op
all-numeric []struct (columnar) 65,815 → 68 (−99.9 %) ~−30 %
all-numeric []struct (row-major) 131,000 → 162 (−99.9 %) ~−9 %
string-bearing telemetry []struct 393,924 → 229,948 (−42 %) ~−9 %
[]struct{map[string]string} (256×16) 341,000 → 25,000 (−92 %) ~−17 %
Lossy vector codec (OptLossyVec)

[]float32 and []float64 embedding fields can be compressed with a configurable fidelity budget — cosine similarity, relative L2 error, or SNR. The codec is opt-in and lossy; the default mode is always bit-exact. It rotates (Hadamard), quantizes (scalar or E8 lattice — whichever is smaller), and entropy-codes the result, so at equal reconstruction quality it is ~17–22 % smaller than scalar / TurboQuant-style quantization.

enc := qdf.NewEncoderWith(qdf.OptBalanced | qdf.OptLossyVec)
enc.SetVectorBudget(qdf.MinCosine(0.999)) // or MaxRelError / TargetSNR
_ = enc.EncodeValue(rows)
data := enc.Bytes()

Built for embedding stores. The natural shape of a RAG index or vector database is a []struct — {ID, Title, Emb []float32} — i.e. id + metadata + vector per record. qdf serializes the whole thing as one self-describing blob, and under OptLossyVec it batches the vector field across all rows into a single column block, so the per-vector header and entropy-coding overhead are amortized once over the batch instead of paid per row (a per-row encoding is ~70 % larger). A 256-dim float32 vector lands at roughly a third of its 1 KiB raw size while cosine recall is preserved — no separate vector store, no second metadata store, no custom format.

For columns whose vectors have widely varying magnitudes (model-weight rows, un-normalized vectors) the codec automatically switches to a polar split — storing each vector's norm separately and quantizing the unit direction — so one tight step serves every vector and the column keeps its fidelity budget. Unit- norm embeddings, where it cannot help, pay nothing for it.

NaN and Inf values are preserved exactly via an exception list in the wire format — they are never corrupted or silently dropped. The encode never exceeds the lossless size (never-worse), and reuses pooled scratch for near-zero warm allocations. See the runnable Example_aiEmbeddingStore.

Full details — pipeline, budget knobs, 0xFD wire format, the diagram, benchmark numbers, and runnable examples — are in docs/LOSSY-VECTOR.md.

Code generation

For fixed-schema types where every nanosecond matters, generate reflection-free methods:

go install github.com/alex60217101990/qdf/cmd/qdfgen@latest

//go:generate qdfgen -type Event,User .
type Event struct {
    ID     int    `qdf:"id"`
    Source string `qdf:"source"`
    // ...
}

qdfgen writes <package>_qdf.go with concrete MarshalQDF / UnmarshalQDF methods using only the public qdf API. No reflect.* at runtime. See cmd/qdfgen/README.md for flags and supported types.

On the test fixture (11-field struct with nested struct, slice, map, pointer, fixed array, []byte, time.Time) the generated code is 2.65× faster than encoding/json on encode and 8.49× faster on decode.

Benchmarks

📈 Live trend dashboard: https://alex60217101990.github.io/qdf/dev/bench/

Every push to main re-runs the benchmark suite in CI and appends the result to that dashboard — one chart per benchmark, ns/op (and where shown B/op / allocs/op) over the commit timeline. A curve trending down is a speed/size win; a sustained step up is a regression (a lone spike is shared-runner noise — read the trend, not single points). See docs/BENCHMARKS.md for how to read the graphs. Wire-size numbers (TestCorpusCodec_Sizes) are deterministic, so changes there are real.

The tables below are a point-in-time snapshot — Darwin amd64 / Intel i7-9750H @ 2.6 GHz, Go 1.26.0, -benchtime=2s, -race cross-check. Full numbers, realistic / unique-data scenarios, memory tables and reproduction commands are in docs/BENCH.md; the live dashboard above has the current numbers.

May 2026 perf series (commits ada9fd7, 2ea3b48, 02d6aac, 7090e25, c0517e8, 95d3c21, 001864b) rebuilt the encode + decode hot paths: MRU ring side-cache for MTF rank, flat open-addressed intern table replacing map[string]uint32, packed lruLink, large-payload buffer probe-and-grow, cached decode interning (decState.stringValues), 4-way mruRank unroll, opt-in Encoder.PreIntern API, and the qdfgen codegen path wired into the bench matrix. qdf is now strictly faster than msgpack on encode AND decode for every workload in the matrix (-5 % HotPath, -40 % TelemetryBatch, -96 % MetricSeries on encode; -55…-98 % on decode). See docs/GUIDE.md for the up-to-date Profile_* matrix.

Encode — ns/op
Payload json msgpack qdf_fast qdf_dense vs json vs msgpack
Tiny ({id,name}) 192 286 227 398 0.85× 1.26×
Flat (20 primitive fields) 1182 1262 480 948 2.46× 2.63×
Nested (4 deep) 446 793 331 786 1.35× 2.40×
Deep linked-list (16) 1157 3060 477 814 2.43× 6.42×
Wide × 1000 991 k 991 k 213 k 289 k 4.66× 4.66×
Log batch × 1000 998 k 624 k 186 k 365 k 5.36× 3.35×
Map-heavy (40 entries) 8446 5282 1391 2019 6.07× 3.80×
[]float32 × 512 30 054 18 270 1821 – 16.5× 10.0×
[]float64 × 512 39 819 23 856 2450 – 16.3× 9.74×
UniqueLog (fresh per iter) 2419 2080 1510 – 1.60× 1.38×
Decode — ns/op
Payload json msgpack qdf_fast vs json vs msgpack
Tiny 729 383 170 4.29× 2.25×
Flat 4411 1862 1122 3.93× 1.66×
Nested 2483 1206 425 5.84× 2.84×
Deep16 7363 4170 1972 3.73× 2.11×
Wide × 1000 4.40 M 1.83 M 990 k 4.44× 1.85×
Log batch × 1000 3.29 M 1.32 M 449 k 7.31× 2.93×
Map-heavy (40 entries) 16 621 8096 3108 5.35× 2.61×
[]float32 × 512 72 928 28 521 3960 18.4× 7.20×
UniqueLog (fresh bytes) 3569 1296 509 7.01× 2.55×
UniqueLog (RunParallel) 667 682 487 1.37× 1.40×
Memory — bytes per decode
Payload json msgpack qdf_fast vs json vs msgpack
Tiny 248 77 29 0.12× 0.38×
Nested 664 160 112 0.17× 0.70×
Log batch × 1000 442 536 407 698 251 838 0.57× 0.62×
Map-heavy 4912 3089 2359 0.48× 0.76×
[]float32 × 512 4384 4282 2113 0.48× 0.49×
MapStringAny (rep. keys) 790 – 345 0.44× –
Allocations per decode
Payload json msgpack qdf_fast
Nested 15 6 5
Map-heavy (40 entries) 124 112 32
MapStringAny (rep. keys) 37 – 3
[]float32 × 512 16 8 3
Decode arena (WithArena)

Opt-in decode with a per-epoch Arena, measured off → on over an epoch loop (i7-9750H, reflect path, Reset per message). The arena copies string bodies into one dense block instead of one allocation each; the win scales with string density and never regresses. It works across all tiers — OptSpeed inline strings and the OptDense/OptCompression interned strings are both arena-backed (numbers below are Speed wire; Dense/Compression reach the same allocation floor — see docs/ARENA.md).

Corpus ns/op (off → on) allocs/op (off → on)
Telemetry log batch × 1000 258 k → 167 k −35 % 3 502 → 3
AD / directory export × 200 450 k → 335 k −26 % 4 856 → 605
Event batch × 500 ([]byte + str) 116 k → 101 k −13 % 1 003 → 504
IoT sensor batch (numeric-heavy) 102 k → 98 k −4 % 291 → 164
LogEntry single, RunParallel 248 → 205 −17 % 8 → 2

[]byte fields stay copy-only (caller-mutable); numeric-heavy payloads see a small win and no regression. Lifetime contract in docs/ARENA.md.

Encoded size
Payload json msgpack qdf_fast qdf_dense dense vs json
Flat 210 134 132 138 0.66×
Deep16 239 139 166 63 0.26×
Wide × 1000 212 901 135 626 128 632 66 702 0.31×
Log batch × 1000 251 902 185 639 185 649 85 440 0.34×

On a 1000-entry Dense stream the same log batch encodes to 85 bytes per entry, against 251 for json and 186 for msgpack — shared intern table across messages plus per-record struct headers collapsed via shape interning (tagMapShape).

Realistic corpus

Numbers from realistic_corpus_test.go builders that mirror real telemetry workloads. Full breakdown plus encode latency in docs/BENCH.md.

TelemetryBatch (1 000 log events, repeating service / region / level)
Format bytes vs json encode ns/op
json 252 497 1.00× —
qdf_fast 186 674 0.74× 272 k
qdf_qpack 186 674 0.74× 261 k
qdf_dense 50 129 0.20× 1.0 M

Dense pays ~4× CPU for 5.0× smaller wire vs JSON — string-intern

  • Markov-0 + MTF + Markov-1 pair + shape interning crush the repeating service / region / level / host fields AND elide per-record struct headers. QPack does not help much here (per-event numeric fields are scalar, not slice-shaped).
MetricSeries (1 024 numeric timestamps + values)
Format bytes vs json
json 30 043 1.00×
qdf_fast 14 442 0.48×
qdf_qpack 8 307 0.28×
qdf_dense 8 315 0.28×

Here QPack pulls its weight: []int64 timestamp column is monotonic and Delta+FOR compresses it to near-zero bytes per element; the []float64 value column rides raw-LE bulk. Dense and QPack converge because string overhead is tiny.

Large payload (~150 MiB, 200 000 records, every supported type)

bench/largepayload_test.go builds a 200 000-record corpus that exercises every qdf-supported field type (scalars, hot/cold strings, nested map, []int32 / []float64, []byte, UUIDs) and measures size + encode/decode latency + working-set memory delta across json, msgpack, qdf_fast, qdf_qpack, qdf_dense.

Sizes (200 000 records):

Format bytes (MiB) vs json
json 142.10 1.00×
msgpack 92.99 0.65×
qdf_fast 91.99 0.65×
qdf_qpack 89.65 0.63×
qdf_dense 71.19 0.50×

Latency + memory (100 000 records):

Format encode (ms) decode (ms) encode heap delta
json 1 070 1 744 199 MiB
msgpack 296 597 64 MiB
qdf_fast 142 300 94 MiB
qdf_qpack 147 231 93 MiB
qdf_dense 169 216 9.7 MiB

Reproduce:

go test -C bench -run TestSizes_LargePayload -count=1 -timeout=10m
go test -C bench -run TestMem_LargePayload   -count=1 -timeout=10m

Both helpers skip under -short; the size test allocates ~600 MiB during the run, the mem test ~400 MiB.

Reproduce
# Default build
cd bench
go test -bench=. -benchmem -benchtime=2s -timeout=10m

# QPack head-to-head: Marshal(_, OptSpeed) vs Marshal(_, OptQPack)
# vs Marshal(_, OptBalanced) vs json / msgpack on the same payload
go test -bench='BenchmarkQPack_' -benchmem

# QPack micro-benchmarks (codec internals, in the root module)
cd ..
go test -bench='BenchmarkQPack' -benchmem

# AVX2 bit-unpack (asm, requires CPUID AVX2 at run time)
go test -tags qdf_simd -bench='BenchmarkBitUnpackFast' -benchmem

# Realistic unique-data scenarios
cd bench
go test -bench='BenchmarkEncode_UniqueLog|BenchmarkDecode_UniqueLog|BenchmarkEncode_MixedTypes|BenchmarkEncode_RandomSize' -benchmem

# Encoded sizes
go test -run TestSizes -v

# Codegen vs reflect on the Sample fixture
cd ../internal/codegen_test
go test -run TestGenerate .
go test -bench=. -benchmem -benchtime=2s
Profile-guided optimisation (downstream)

Go's PGO applies to the main package being built, so a library like qdf cannot ship its own profile. If you build a service that imports qdf, collect a representative profile of your workload and drop it next to your main:

# Run your service / load test under cpuprofile.
go test -bench=. -cpuprofile=cpu.pprof -benchtime=10s ./...
mv cpu.pprof default.pgo            # same dir as your main package
go build .                          # auto-picks up default.pgo

The Go toolchain will then recompile the qdf functions on the hot path with PGO-driven inlining and devirtualisation. Typical gain is 5–15 % across the encode/decode pipeline on top of the numbers above.

Build tags

Opt-in fast paths. None are required for the headline numbers; the defaults already include the specialised slice / map encoders, the QPack codecs, and a 128-bit sliding-window bit-unpacker for FOR / Delta+FOR.

Tag Effect Build prerequisite
qdf_reflect2 Swap reflect.MakeSlice / MakeMapWithSize for modern-go/reflect2 unsafe equivalents. Smaller decode allocations on map / slice heavy workloads. none — pure Go
qdf_simd SIMD fast paths for the QPack integer/bool codecs. amd64 (AVX2): decode bits ∈ {8,16,32} (VPMOVZX) and every width 1–28 (VPBROADCASTQ+VPSRLVQ), encode {8,16,32} (VPSHUFB) and {10,12,14,20} (VPSLLVQ+lane-OR), []bool pack; runtime CPUID gate, non-AVX2 falls back. arm64 (NEON): decode 1–28 plus 32 (encode/bool scalar for now), baseline. ~3-11× over pure-Go on accelerated widths. Output byte-identical to scalar; other arches compile a stub. See docs/USAGE.md. amd64 (AVX2) / arm64 (NEON) 
go build -tags qdf_reflect2 ./...
go build -tags qdf_simd ./...
go build -tags "qdf_simd qdf_reflect2" ./...   # combined

The qdf_simd path only changes the speed of QPack-encoded numeric and bool slices at the accelerated widths; string / map / struct paths, the float codecs, and other bit widths run the pure-Go path unchanged. It is a pure speed switch — the wire format and output are identical.

Correctness

The test suite runs under -race and covers:

  • Primitive round-trip across every wire-tag boundary.
  • Boundary integers from math.MinInt64 to math.MaxUint64.
  • Strings at the fixstr / str8 / str16 / str32 boundaries (0, 1, 31, 32, 255, 256, 65535, 65536, 1 MiB) plus Unicode and invalid UTF-8.
  • Every specialised fast path against the generic reflect path.
  • Truncated input — every prefix from 0 to N-1 decodes to an error, never a panic.
  • Bad magic / bad version — errors, not panics.
  • Cross-mode interop (Fast bytes decoded via Dense decoder and vice versa).
  • Streaming with mixed message types under Dense intern.
  • Concurrency: 32 × 500 marshal+unmarshal goroutines under -race.
  • QPack codecs: each codec (bitpack-bool, raw-LE, FOR, Delta+FOR, Gorilla) has its own round-trip suite covering edge cases — empty slices, single elements, near-MaxUint64, MinInt64, NaN / ±Inf / ±0 / denormals, monotonic / constant / mixed-direction sequences.
  • TestCompleteness_AllModes runs one big payload (every QPack- eligible type, every edge case, nested structs, maps, string interning) through OptSpeed, OptQPack, and OptBalanced, then asserts bit-for-bit IEEE-754 equality for floats and reflect.DeepEqual elsewhere.
  • TestCompleteness_StreamingDense exercises three messages through NewStreamEncoder in Dense mode (carries QPack + shared intern table); TestCompleteness_FuzzRandomStructsQPack generates 50 random struct shapes and round-trips them through all three encoders.
  • AVX2 bit-unpack (under qdf_simd) has parity tests against scalar zero-extend for n in {0,1,2,3,4,5,7,8,9,15,16,17,1000,4096} at bits ∈ {8, 16, 32}, plus a bitPackU64LE → bitUnpackU64LE round- trip.
  • tagStateRepeat (Markov-0 state-ref predictor): tests that the predictor fires on runs of identical interned values, does not fire on alternation, invalidates correctly across an inline-string emission, stays in sync with Decoder.Skip, and round-trips through the public Marshal(v, OptBalanced) / Unmarshal boundary.
  • Property-based round-trip fuzzers (fuzz_property_test.go) drive a deterministic value generator from the fuzz bytes, encode through every Marshal entry point, and assert Unmarshal(Marshal(v)) == v. Targets cover Int64Slice, Uint64Slice, Float64Slice, BoolSlice, MapStringInt, StructTriad, and an AllModesAgree fuzzer that asserts the three encoder dialects decode to the same Go value. Caught one Markov-0 predictor bug in encodeStruct (commit dfc30ae).
  • Golden-file wire pinning (testdata/golden/*.bin, golden_test.go): every representative payload has its OptSpeed / OptQPack / OptBalanced bytes committed to disk. A later wire-format change either matches the bytes byte-for-byte or fails the test. Map-shaped cases skip the byte-pin half (Go map iteration is randomised) but keep the decode-and-compare half. Regenerate intentionally via go test -run TestGolden -update.
  • Deterministic truncation matrix (truncation_test.go, ~2700 sub-tests): for every representative payload, every prefix payload[:i] is fed through Unmarshal into six destination types, every 4th byte is mutated to one of nine "bad" alternatives (00, FF, 80, 55, AA, and every QPack tag), and malformed 5-byte headers are exhaustively rejected. The decoder must return a typed error in every case; panic budget is zero.
  • Unknown-field skip (unknown_field_test.go): encoder writes a 10-field struct, decoder declares only 3; subsequent declared fields still decode correctly. Each of the 10 fields is also pulled out alone via its own single-field destination type.
  • Pathological-shape stress (stress_nesting_test.go): 256-deep linked list, 512-key map, balanced binary tree (depth 12), 1 MiB string + 1 MiB []byte + 100k-element u64/f64 vectors, and a 2000-deep chain under debug.SetMaxStack(64 MiB).
  • Realistic payload corpus (realistic_corpus_test.go): telemetry batches with repeating service/region/level fields, metric time-series, wide column-store rows with recursive Children — round-tripped through every Marshal entry point and cross-checked for three-way agreement.
  • Allocation-budget tests (alloc_budget_test.go): each hot entry point has an upper-bound allocation budget enforced via testing.AllocsPerRun. A regression in alloc count fails CI rather than silently bloating downstream callers.
  • Tag and named-type matrix (tag_matrix_test.go): every primitive type is exercised through a named alias (type MyInt int64), the tag fallback chain qdf → json → field name is verified, the qdf:"-" skip directive is checked via a map[string]any decode, and embedded-struct (non-)flattening is pinned.
  • Differential testing (bench/diff_test.go): qdf round-trips agree semantically with encoding/json and github.com/vmihailenco/msgpack/v5 on a shared payload, including NaN / ±Inf / ±0 preservation against msgpack.
  • Pointer-cycle detection via depth counter: the encoder increments Encoder.depth on every pointer dereference and returns ErrCycleDetected once the count exceeds DefaultMaxDepth (10000). Cheaper than a per-pointer set (no allocation per call) and catches both genuine *T → *T cycles and pathologically deep payloads. Override the cap via Encoder.SetMaxDepth.
  • OOM protection on every length-prefixed read path: a hostile varuint encoding a value > 2^62 cannot drive the decoder cursor negative because the byte-count check happens in uint64 against (len(d.buf) - d.i) * 8 / bitsPer before any signed cast. Applies to tagPackBool/Raw/For/DeltaFor/Gorilla in both the slice-read and Skip paths.
  • Stream Close idempotency: both StreamEncoder.Close and StreamDecoder.Close are safe to call multiple times.
  • Fuzz: FuzzDecoder_NeverPanics, FuzzRoundTrip_StringSlice, FuzzQPackBool, FuzzQPackRawUint64, FuzzRoundTrip_Int64Slice, FuzzRoundTrip_Uint64Slice, FuzzRoundTrip_Float64Slice, FuzzRoundTrip_BoolSlice, FuzzRoundTrip_MapStringInt, FuzzRoundTrip_StructTriad, FuzzRoundTrip_AllModesAgree — persistent corpus under testdata/fuzz/. 10 M+ executions clean across the suite after the Skip-overflow and encodeStruct-predictor fixes; both repros are saved under testdata/fuzz/.

Length prefixes are validated against the remaining buffer (Decoder.CheckLength) before any make, so a hostile payload claiming a multi-billion-element map cannot OOM the process. The QPack-tag Skip paths perform the same overflow-safe check on uint64 element counts before any signed cast, so a 10-byte varuint encoding a value > 2^62 cannot drive the decoder cursor negative.

go test -race -count=1 ./...

# Property-based round-trip fuzz (each fuzzer asserts
# Unmarshal(Marshal(v)) == v on randomly generated values)
go test -run=^$ -fuzz=FuzzRoundTrip_StructTriad     -fuzztime=60s
go test -run=^$ -fuzz=FuzzRoundTrip_AllModesAgree   -fuzztime=60s
go test -run=^$ -fuzz=FuzzRoundTrip_Int64Slice      -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_Uint64Slice     -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_Float64Slice    -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_BoolSlice       -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_MapStringInt    -fuzztime=30s

# Decoder safety fuzz (asserts never-panic on hostile input)
go test -run=^$ -fuzz=FuzzDecoder_NeverPanics       -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_StringSlice     -fuzztime=30s
go test -run=^$ -fuzz=FuzzQPackBool                 -fuzztime=30s
go test -run=^$ -fuzz=FuzzQPackRawUint64            -fuzztime=30s

# Differential vs msgpack and encoding/json
go test -C bench -run=TestDiff -count=1

# Regenerate wire-format golden fixtures (intentional after a wire
# bump only)
go test -run=TestGolden -update

# Build-tag combinations
go test -tags qdf_reflect2          -race ./...
go test -tags qdf_simd              -race ./...
go test -tags "qdf_simd qdf_reflect2" -race ./...

Architecture diagrams

Mermaid diagrams (GitHub renders them natively) covering every layer of the format — concept overview, wire layout, codec picker, Dense interning, columnar/selective-decode, and the algorithmic performance wins.

diagrams/README.md — start here: concept overview flowchart and index of all diagrams.

Diagram Topic
architecture.md Marshal/Unmarshal end-to-end: pool, typeDesc cache, mode dispatch, rANS pass
wire-format.md 5-byte header, flags bit map, full tag space
options-and-modes.md Options bits, Speed/Balanced/Compression bundles, Fast vs Dense decision
qpack-codecs.md Codec picker (FOR/Delta/RLE/dict/PFOR/Gorilla/ALP), never-larger floor, SIMD
dense-interning.md Intern table, state-ref predictors (Markov-0/MTF/Markov-1/shape)
columnar-and-selective-decode.md []struct transpose, Time split, Nullable slab, colIndex, 3VL pushdown
delta.md Diff/Apply pipeline, 'Q','D','P' patch wire format, keyed slices, columnar column-diff, baseline registry
canonical.md OptCanonical: sorted map keys (every kind), -0.0/NaN float normalization, byte-identical-for-equal-values
performance.md 10 algorithmic wins grouped by CPU / wire size / memory

Status

Alpha. The wire format is stable for the 0x01 version byte; future versions will bump it. The public API may change before the first tagged release — pin a commit if you depend on it.

License

MIT. See LICENSE.

Documentation

Overview

Package qdf is a compact, streaming-friendly binary serialization format.

Quick start

One encode entry point: Marshal(v, opts). The Options bit-mask picks which codecs run for that call. Convenience bundles cover the common tradeoffs: 

OptSpeed       — Fast mode, no codecs (matches encoding/json shape)
OptBalanced    — Dense + QPack + shape interning + Markov-1 + MTF
OptCompression — OptBalanced + heavier codecs: Gorilla XOR and ALP for
                 float series, a final order-0 rANS entropy pass, and
                 FSST for columnar string columns (large wire reduction
                 at higher encode CPU)

A single decoder handles every variant; the wire header self- describes the dialect. 

b, err := qdf.Marshal(v, qdf.OptSpeed)        // hot path
b, err := qdf.Marshal(v, qdf.OptBalanced)     // telemetry / logs
b, err := qdf.Marshal(v, qdf.OptCompression)  // backup / archive

err := qdf.Unmarshal(b, &v)

enc := qdf.NewStreamEncoder(w, qdf.OptBalanced)
dec := qdf.NewStreamDecoder(r)

Marshal returns a freshly-allocated slice owned by the caller. AppendMarshal is the zero-extra-copy variant.

Data shapes and codecs

Under OptBalanced the encoder picks a codec per slice from the data itself: integer slices use Frame-of-Reference, delta, run-length or dictionary coding; bool slices bit-pack; repeated strings are interned and back-referenced. A slice of homogeneous flat structs is transposed and compressed column-by-column automatically — numeric columns get the integer codecs, an enum-like string column is dictionary-coded (distinct table + a bit-packed index per row) when that beats per-value interning, an optional (*T, T a scalar or string) field becomes a presence bitmap plus a dense present-only column instead of forcing the struct back to row-major, other repeated string columns collapse — with no flag, and a per-array probe falls back to the plain row encoding when columnar would not help. The float codecs that trade encode CPU for size live behind OptCompression: Gorilla XOR on smooth series, ALP on quantized/decimal float64 grids (quantized telemetry, prices, latencies).

Concurrency

Marshal, Unmarshal and AppendMarshal are safe for concurrent use: each call leases its own encoder/decoder from a pool. A single Encoder, Decoder or StreamEncoder value is not safe to share across goroutines — use one per goroutine. A Dense document is a sequential stream, so one document cannot be encoded or decoded in parallel; to parallelize a large dataset, split it into independent shards and Marshal each separately.

Selective decode

Under OptColumnIndex a columnar []struct payload carries a fixed-width uint32 column-length index, written after the shape declaration and before the column bodies (~4 bytes per column). A reader wanting only some columns then advances past the rest using the index instead of decoding them, so the cost becomes O(columns you read), not O(all columns). Two entry points: decode into a subset struct whose fields are a subset of the wire (matched by qdf tag / field name) with plain Unmarshal — wire columns absent from the target are skipped, target fields absent from the wire are left zero — or name the columns explicitly with UnmarshalColumns (also drives the dynamic *[]map[string]any form). The option is a true no-op on non-columnar payloads: the flag is backpatched only when the index is actually emitted, so the default columnar wire stays byte-identical when it is off. Without the index a subset decode still returns correct results by decoding and discarding unwanted columns. The index is a single-message feature and is not emitted in streaming mode.

Predicate pushdown

Unmarshal accepts trailing QueryOption arguments that filter and project a columnar []struct payload before it is materialized. Where(field, pred) keeps only the rows for which a typed predicate holds — func(T) bool over the column's element type, AND-ed across multiple Where clauses, with zero per-value boxing — and Select(fields...) restricts the output to a named subset of columns. The filter field need not appear in the output type, and the result can be either *[]Struct or *[]map[string]any. The decoder reads each predicate column whole, evaluates it into a row bitmask, ANDs the masks, then compacts and materializes only the surviving rows of the projected columns; OptColumnIndex lets it seek past skipped columns instead of decode-and-discard. A nullable column's nil rows never match. Pushdown fires only on columnar payloads — a non-columnar payload (a single struct, or a batch the columnar probe declined) returns an error wrapping ErrUnsupported; a missing field yields ErrFieldNotFound and a predicate whose argument type mismatches the column yields ErrTypeMismatch, both as a *QueryError (errors.Is / errors.As). Versus a full decode followed by a manual loop, pushdown moves materially fewer bytes (about 4x less on a wide batch with 1% selectivity) and runs roughly 2x faster. No other Go serializer reads back a filtered, projected subset of a batch without decoding the whole thing — see docs/PREDICATE-PUSHDOWN.md for the full guide, rationale, and tutorial.

Structural delta

Diff(old, new, opts) computes a patch carrying only the locations that changed; Apply(&base, patch) merges it back in place. The patch is self-describing — the receiver never has to know which Options produced it — and far smaller than a re-encode because unchanged fields, elements, and keys cost no bytes. Slices whose elements have a stable identity field tagged ",key" match by key (cheap reorder / middle edit) instead of by position, and an equal-length columnar batch is diffed column-by-column when that wins. Two fingerprints guard Apply against the wrong type or the wrong base. BaselineRegistry[T] applies a chain of patches in a state-sync stream without threading the previous value by hand. See docs/DELTA.md.

Canonical encoding

OptCanonical makes the same logical value serialize to byte-identical output: map keys are emitted in sorted order (every key kind) and floats are normalized (-0.0 → +0.0, any NaN → one quiet NaN). The bytes are then safe to hash, sign, content-address, or deduplicate. It is encode-side only and decodes like any other qdf output; it is lossy for the sign of zero and the NaN payload, so use the default mode for a bit-exact float round-trip. See docs/CANONICAL.md.

See docs/USAGE.md in the repository for a fuller guide.

Public API surface

The package contract is split across four layers; everything else under internal/ is implementation detail and may change between releases without notice.

Top-level entry points (file: qdf.go): 

func Marshal(v any, opts Options) ([]byte, error)
func AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)
func Unmarshal(data []byte, out any) error
type Options uint32   // bit-mask of OptDense, OptQPack, OptMTF, …

Selective columnar decode (file: columnar_select.go): 

func UnmarshalColumns(data []byte, out any, fields ...string) error

Predicate pushdown — filter rows and project columns on a columnar []struct, AND-ed typed predicates with zero per-value boxing (file: query.go): 

func Unmarshal(data []byte, out any, opts ...QueryOption) error
func Where[T Queryable](field string, pred func(T) bool) QueryOption
func Select(fields ...string) QueryOption

Typed convenience wrappers — generic, zero-extra-reflection (file: qdf_generic.go): 

func MarshalT[T any](v T, opts Options) ([]byte, error)
func AppendMarshalT[T any](dst []byte, v T, opts Options) ([]byte, error)
func UnmarshalT[T any](data []byte) (T, error)

Structural delta — patch / merge two values, key-matched slices, and content-addressed baselines for state-sync streams (files: delta.go, delta_baseline.go): 

func Diff[T any](old, new T, opts Options) ([]byte, error)
func AppendDiff[T any](dst []byte, old, new T, opts Options) ([]byte, error)
func Apply[T any](base *T, patch []byte) error
func ApplyArena[T any](base *T, patch []byte, arena *Arena) error
type BaselineRegistry[T any]   // NewBaselineRegistry / Register / Apply / Len

Low-level encoder / decoder for callers driving the wire directly or interoperating with the qdfgen code generator (files: encoder.go, decoder.go, stream.go): 

type Encoder struct{ … }
    NewEncoder(mode Mode) *Encoder
    NewEncoderWith(opts Options) *Encoder
    NewEncoderOnBuf(buf []byte, mode Mode) *Encoder
    (e *Encoder) WriteString / WriteBytes / WriteInt / WriteUint /
                WriteFloat32 / WriteFloat64 / WriteBool / WriteNil /
                WriteArrayHeader / WriteMapHeader /
                WriteTimestamp / WriteStringInline
    (e *Encoder) AppendBytes / EnsureHeader / Bytes / Take /
                Reset / SetBuffer / AdoptBuffer / SetIntern /
                SetMaxDepth / SetQPack / QPack / ApplyOpts /
                EncodeValue / PreIntern
type Decoder struct{ … }
    NewDecoder() *Decoder
    NewDecoderOnBuf(buf []byte) *Decoder
    (d *Decoder) ReadString / ReadStringBytes / ReadBytes /
                ReadBool / ReadInt / ReadUint / ReadFloat32 /
                ReadFloat64 / ReadNil / ReadArrayHeader /
                ReadMapHeader / ReadTimestamp / Skip /
                PeekTag / IsNil / Pos / Remaining / RemainingBytes /
                Advance / SetInput / SetNoCopy / MarkHeaderRead /
                CheckLength / InternKey
type StreamEncoder, type StreamDecoder         // io.Writer / io.Reader

User-side hook points (file: marshaler.go): 

type Marshaler interface   { MarshalQDF(dst []byte) ([]byte, error) }
type Unmarshaler interface { UnmarshalQDF(src []byte) (int, error) }

The qdfgen code generator (cmd/qdfgen) emits MarshalQDF / UnmarshalQDF methods for user struct types; the codegen path uses the same Encoder / Decoder primitives listed above and requires no reflection at runtime.

Example

Example demonstrates the simplest possible round-trip: Marshal a value into a freshly-allocated byte slice and Unmarshal it back into a typed receiver. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		ID     int    `qdf:"id"`
		Source string `qdf:"source"`
	}

	in := Event{ID: 42, Source: "ingest"}
	buf, err := qdf.Marshal(in, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var out Event
	if err := qdf.Unmarshal(buf, &out); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", out)
}

Output:
{ID:42 Source:ingest}
Example (AiEmbeddingStore)

Example_aiEmbeddingStore is the headline use case: a RAG index / vector database stored as ONE self-describing qdf blob — id + metadata + the embedding vector per record — instead of a separate vector store plus a metadata store.

Why this is a good fit:

  • The records are a []struct with a []float32 vector field. Under OptLossyVec qdf batches that field across ALL rows into a single column block, so the per-vector header and entropy-coding overhead are amortized once over the whole batch (not paid per row) — a 256-dim float32 vector lands at roughly a third of its 1 KiB raw size.
  • The fidelity budget is the metric the index relies on (cosine here), so nearest-neighbour results are preserved while the bytes shrink.
  • It is never-worse (a column that would not shrink stays lossless) and the default path is bit-exact, so turning the flag off restores exact vectors.
package main

import (
	"fmt"
	"math"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Doc struct {
		ID    string
		Title string
		Emb   []float32 // the embedding — batched + lossily compressed
	}

	// A small deterministic "corpus" of 128 documents, 256-dim embeddings.
	const n, dim = 128, 256
	docs := make([]Doc, n)
	for i := range docs {
		v := make([]float32, dim)
		for j := range v {
			v[j] = float32(math.Sin(float64(i)*0.3 + float64(j)*0.02))
		}
		docs[i] = Doc{ID: fmt.Sprintf("doc-%04d", i), Title: "title", Emb: v}
	}

	// Store the whole index in one blob; cosine recall kept >= 0.999.
	data, err := qdf.Marshal(docs, qdf.OptBalanced|qdf.OptLossyVec)
	if err != nil {
		panic(err)
	}

	var out []Doc
	if err := qdf.Unmarshal(data, &out); err != nil {
		panic(err)
	}

	cos := func(a, b []float32) float64 {
		var dot, na, nb float64
		for j := range a {
			x, y := float64(a[j]), float64(b[j])
			dot, na, nb = dot+x*y, na+x*x, nb+y*y
		}
		return dot / (math.Sqrt(na)*math.Sqrt(nb) + 1e-30)
	}
	// Top-1 retrieval: for a handful of original query vectors, the nearest
	// DECODED vector must still be the same document (recall survives the loss).
	hits := 0
	for _, q := range []int{0, 17, 63, 100} {
		best, bestC := -1, -1.0
		for i := range out {
			if c := cos(docs[q].Emb, out[i].Emb); c > bestC {
				bestC, best = c, i
			}
		}
		if best == q {
			hits++
		}
	}

	rawF32 := n * dim * 4
	fmt.Println("metadata intact:", out[0].ID == "doc-0000" && out[0].Title == "title")
	fmt.Println("smaller than raw f32:", len(data) < rawF32/2) // well under half
	fmt.Println("top-1 recall preserved:", hits == 4)

}

Output:
metadata intact: true
smaller than raw f32: true
top-1 recall preserved: true
Example (RoundTripSlice)

Example_roundTripSlice demonstrates encoding a slice of structs. QDF's reflect path resolves the element type once per typeDesc; every element after the first emits its tag stream straight from the cached descriptor. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	rows := []Row{
		{"auth", 200},
		{"billing", 503},
		{"auth", 200},
	}

	buf, _ := qdf.Marshal(rows, qdf.OptBalanced)

	var got []Row
	_ = qdf.Unmarshal(buf, &got)
	for _, r := range got {
		fmt.Printf("%s=%d\n", r.Service, r.Status)
	}
}

Output:
auth=200
billing=503
auth=200

Index

Examples

Constants

View Source 
const (
	Magic0   byte = 'Q'
	Magic1   byte = 'D'
	Magic2   byte = 'F'
	Version1 byte = 0x01
)
View Source 
const (
	FlagDense byte = 1 << 0
	// FlagQPack signals that the encoder may emit QPack codec tags
	// (0xE3..0xEF). A decoder that does not recognize the tags will fail
	// with ErrBadTag on the first packed slice; the flag is an early hint
	// so callers can refuse the buffer up front.
	FlagQPack byte = 1 << 1

	// FlagRANS signals that the body (everything after the 5-byte header) is
	// rANS-compressed: varuint(origLen) + 256-entry frequency table +
	// rANS stream. The decoder reverses it before reading tags. Set only
	// when the rANS form is strictly smaller than the plain body.
	FlagRANS byte = 1 << 2

	// FlagColIndex marks a tagColStruct (columnar []struct) payload that
	// carries a fixed-width column-length table right after the shape
	// declaration and before the column bodies. The table is K little-endian
	// uint32 entries (one per column, K = number of columns), each the byte
	// length of the corresponding column body. It lets a reader skip columns
	// it does not need without decoding them. Opt-in (~4 bytes per column);
	// only meaningful when the value is columnar — for any other top-level
	// shape the bit is set but no index block exists.
	FlagColIndex byte = 1 << 3
)

Flag bits in the 5th header byte.

View Source 
const DefaultMaxDepth = 10_000

DefaultMaxDepth caps reflect-path pointer/struct recursion. Set large enough for any legitimate payload (10 000) while still rejecting genuine cycles before the goroutine stack overflows.

View Source 
const TagNil = tagNil

TagNil exposes the nil-value tag for comparison with PeekTag.

Variables

View Source 
var (
	// ErrInvalidPatch is returned when a patch blob is truncated, has a bad
	// magic/version, or is otherwise malformed.
	ErrInvalidPatch = errors.New("qdf: invalid or truncated patch")
	// ErrPatchSchemaMismatch is returned by Apply when the patch was built for
	// a different type than the supplied base.
	ErrPatchSchemaMismatch = errors.New("qdf: patch schema fingerprint mismatch")
	// ErrPatchBaseMismatch is returned by Apply when the patch carries a base
	// fingerprint that does not match the supplied base value.
	ErrPatchBaseMismatch = errors.New("qdf: patch base fingerprint mismatch")
)
View Source 
var (
	// ErrBaselineRequired is returned by (*BaselineRegistry).Apply when the
	// patch carries no base fingerprint (the producer used
	// OptDeltaNoBaseFingerprint), so its baseline cannot be content-addressed.
	ErrBaselineRequired = errors.New("qdf: patch has no base fingerprint; baseline registry requires one")
	// ErrBaselineEvicted is returned by (*BaselineRegistry).Apply when the
	// patch's baseline id is not resolvable — it was never registered, or the
	// caller dropped its strong reference and the GC reclaimed it.
	ErrBaselineEvicted = errors.New("qdf: patch baseline not found in registry (never registered or GC-reclaimed)")
)
View Source 
var (
	ErrShortBuffer    = errors.New("qdf: short buffer")
	ErrBadMagic       = errors.New("qdf: bad magic / not a qdf stream")
	ErrBadVersion     = errors.New("qdf: unsupported wire version")
	ErrBadTag         = errors.New("qdf: unknown tag")
	ErrTypeMismatch   = errors.New("qdf: type mismatch on decode")
	ErrInvalidLength  = errors.New("qdf: invalid length prefix")
	ErrUnknownStateID = errors.New("qdf: unknown state-table id")
	ErrUnsupported    = errors.New("qdf: unsupported type")
	ErrCycleDetected  = errors.New("qdf: pointer cycle detected (max depth exceeded)")
	ErrFieldNotFound  = errors.New("qdf: query predicate field not found")
	// ErrStreamBadFlags is returned by StreamDecoder.Decode when the stream
	// header carries whole-payload flags that do not apply to a frame stream
	// (FlagRANS / FlagColIndex). A conforming StreamEncoder never sets them; a
	// hostile stream claiming them would, if honored, swap the shared window
	// buffer mid-stream and desync framing. The decoder is latched broken.
	ErrStreamBadFlags = errors.New("qdf: stream header has unsupported whole-payload flags")
)
View Source 
var ErrStreamBroken = errors.New("qdf: stream encoder broken by a prior mid-message error")

ErrStreamBroken is returned by StreamEncoder.Encode after a previous Encode failed mid-message: the cross-message encoder state can no longer be trusted, so the stream must be abandoned (the valid prefix may still be flushed).

View Source 
var ErrStreamDecoderBroken = errors.New("qdf: stream decoder broken by a prior mid-message error")

ErrStreamDecoderBroken is returned by StreamDecoder.Decode after a previous Decode failed mid-message: the read cursor and the shared dense state can no longer be trusted (subsequent frames would misparse), so the stream must be abandoned.

Functions

func AppendDiff added in v0.0.8 

func AppendDiff[T any](dst []byte, old, new T, opts Options) ([]byte, error)

AppendDiff appends the patch to dst and returns the extended slice.

func AppendMarshal 

func AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)

AppendMarshal encodes v and appends the result to dst. Reuse the returned slice as dst on the next call to avoid per-message allocations.

Example

ExampleAppendMarshal shows the zero-extra-copy form. The caller keeps a reusable byte buffer and asks Marshal to append into it, returning the extended slice. Combined with a sync.Pool of buffers this drives encode allocations to zero on the hot path. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Tick struct {
		Ts    int64   `qdf:"ts"`
		Price float64 `qdf:"price"`
	}

	buf := make([]byte, 0, 64) // caller-owned, reusable
	t := Tick{Ts: 1_700_000_000, Price: 99.5}

	var err error
	buf, err = qdf.AppendMarshal(buf, t, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var out Tick
	if err := qdf.Unmarshal(buf, &out); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", out)
}

Output:
{Ts:1700000000 Price:99.5}

func AppendMarshalDirect 

func AppendMarshalDirect[T Marshaler](dst []byte, v T) ([]byte, error)

AppendMarshalDirect appends the serialisation of v to dst.

func AppendMarshalT 

func AppendMarshalT[T any](dst []byte, v T, opts Options) ([]byte, error)

AppendMarshalT is the generic equivalent of AppendMarshal.

func Apply added in v0.0.8 

func Apply[T any](base *T, patch []byte) error

Apply merges patch onto base in place, reconstructing new. Unchanged fields are untouched.

func ApplyArena added in v0.0.8 

func ApplyArena[T any](base *T, patch []byte, arena *Arena) error

ApplyArena is Apply with a caller-provided decode arena: every string (and []byte-as-string) value the patch REPLACES is copied into arena's contiguous bump blocks instead of one heap allocation per string. It is otherwise byte-for-byte identical to Apply — same result, same errors, same wire — and only differs in where replaced string bodies live.

This helps only when a patch replaces MANY strings (a large changed []string or []struct{...string} field, a map with many changed string values). A typical small patch changes few strings and the arena has little to batch; in that case Apply is the simpler choice.

Lifetime / safety (identical to the decode arena, see Arena): strings written into base by ApplyArena ALIAS arena's memory and stay valid only as long as arena (or any string from it) is reachable. The safe pattern is one Arena per epoch, then drop it; Reset reuses it but invalidates every string from the prior epoch. Unchanged string fields already present in base are NOT touched and do not alias arena.

func CheckColumnarBytes added in v0.1.1 

func CheckColumnarBytes(n int, elemSize uintptr) error

CheckColumnarBytes is the exported guard cmd/qdfgen-generated columnar decoders call after ReadColStructHeader / ReadHybridColStructHeader, before allocating the row slice, to reject the same memory-amplification a hostile row count would cause on the reflect path. elemSize is unsafe.Sizeof(row).

func DecodeNested added in v0.0.8 

func DecodeNested(d *Decoder, u Unmarshaler) error

DecodeNested decodes one nested value from the shared decoder d. When u implements DecoderUnmarshaler it reads directly from d (no new decoder, and it inherits d's noCopy / arena). Otherwise it falls back to the buffer-based UnmarshalQDF over d's remaining bytes — honoring d's noCopy / arena via the Opts / Arena extensions — and advances d by the bytes consumed. Exported for cmd/qdfgen-generated code.

func Diff added in v0.0.8 

func Diff[T any](old, new T, opts Options) ([]byte, error)

Diff computes a patch carrying only the structural difference (new − old).

Example

ExampleDiff shows the structural delta: Diff computes a patch carrying only the locations that changed, and Apply merges it back onto a base in place. The patch is far smaller than a full re-encode because unchanged fields cost no bytes, and it is self-describing — the receiver hands the bytes straight to Apply and never has to know which Options produced them. 

package main

import (
	"fmt"
	"reflect"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Config struct {
		Replicas int               `qdf:"replicas"`
		Image    string            `qdf:"image"`
		Env      map[string]string `qdf:"env"`
	}

	old := Config{Replicas: 3, Image: "app:v1", Env: map[string]string{"LOG": "info", "REGION": "eu"}}
	// Only Replicas and one Env key change.
	updated := Config{Replicas: 5, Image: "app:v1", Env: map[string]string{"LOG": "debug", "REGION": "eu"}}

	patch, _ := qdf.Diff(old, updated, qdf.OptBalanced)
	full, _ := qdf.Marshal(updated, qdf.OptBalanced)

	base := old
	_ = qdf.Apply(&base, patch)

	fmt.Println("applied == updated:", reflect.DeepEqual(base, updated))
	fmt.Println("patch smaller than full encode:", len(patch) < len(full))

}

Output:
applied == updated: true
patch smaller than full encode: true
Example (KeyedSlice)

ExampleDiff_keyedSlice shows keyed-slice matching: tag an element's stable identity field with ",key" and a []struct patch matches elements by that key instead of by position. Reordering the slice then ships only the new key order (no element values), where a positional diff would reship the whole shifted tail. 

package main

import (
	"fmt"
	"reflect"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Entity struct {
		ID string  `qdf:"id,key"`
		X  float64 `qdf:"x"`
	}

	old := []Entity{{"a", 1}, {"b", 2}, {"c", 3}}
	// Same entities, reversed order, with one value edit.
	updated := []Entity{{"c", 3}, {"b", 20}, {"a", 1}}

	patch, _ := qdf.Diff(old, updated, qdf.OptBalanced)

	base := append([]Entity(nil), old...)
	_ = qdf.Apply(&base, patch)

	fmt.Println("reordered roundtrip:", reflect.DeepEqual(base, updated))

}

Output:
reordered roundtrip: true

func EncodeNested added in v0.0.7 

func EncodeNested(e *Encoder, m Marshaler) error

EncodeNested encodes a nested Marshaler m into the shared encoder e. When m implements EncoderMarshaler it writes directly (no new Encoder); otherwise it falls back to the buffer-based MarshalQDF + AdoptBuffer. Exported for cmd/qdfgen-generated code.

func Marshal 

func Marshal(v any, opts Options) ([]byte, error)

Marshal encodes v with the given option bit-mask. Combine bits with | to opt into individual codecs, or use one of the OptSpeed / OptBalanced / OptCompression bundles. 

b, _ := qdf.Marshal(event,    qdf.OptSpeed)         // hot path
b, _ := qdf.Marshal(batch,    qdf.OptBalanced)      // telemetry
b, _ := qdf.Marshal(snapshot, qdf.OptCompression)   // backup
b, _ := qdf.Marshal(payload,                        // tuned
    qdf.OptDense|qdf.OptQPack|qdf.OptShapeIntern)

Big-output detach (in marshalDict): cloning a multi-megabyte buffer to hand the caller their own copy used to dominate Large-payload profiles (slices.Clone + runtime.memmove). Above marshalDetachThreshold the pool would drop the buffer in putEnc anyway (cap exceeds maxPooledBuf), so handing the original to the caller and leaving the pool encoder with a nil buf is strictly cheaper. Small payloads stay on the clone path so the warm 4 KiB pool buffer survives.

func MarshalDirect 

func MarshalDirect[T Marshaler](v T) ([]byte, error)

MarshalDirect serialises a value through its MarshalQDF method. Skips the public Marshal entry point's any-boxing, reflect.New, and descriptor lookup. Roughly 2-4× faster than Marshal for generated types on small payloads, with one fewer allocation per call.

v must be non-nil: MarshalDirect calls v.MarshalQDF directly, so a nil pointer-receiver value panics. Use Marshal, which emits an explicit nil, if the value may be nil.

func MarshalT 

func MarshalT[T any](v T, opts Options) ([]byte, error)

MarshalT is the generic equivalent of Marshal. T is fixed at the call site; opts copies by value, so the call adds zero heap allocations over MarshalT itself.

Example

ExampleMarshalT shows the generic typed wrapper. T is fixed at the call site, so the compiler skips the reflect.TypeOf step that Marshal(v any, opts) pays inside; encode/decode of value types stays on the stack. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Point struct {
		X int32 `qdf:"x"`
		Y int32 `qdf:"y"`
	}

	buf, err := qdf.MarshalT(Point{X: 3, Y: 4}, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var p Point
	if err := qdf.UnmarshalT(buf, &p); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", p)
}

Output:
{X:3 Y:4}

func StringColumnsBeneficial added in v0.0.8 

func StringColumnsBeneficial(cols ...[]string) bool

StringColumnsBeneficial gates a PURE string-only element's columnar encode in generated code (mirrors columnarProbe with internAware=false).

func StringColumnsBeneficialHybrid added in v0.1.1 

func StringColumnsBeneficialHybrid(cols ...[]string) bool

StringColumnsBeneficialHybrid gates a HYBRID (residual-bearing) string-only element's columnar encode in generated code (mirrors columnarProbe with internAware=true: intern-credited baseline + alpha-packing estimate + the wider gain), so codegen flips such an element into the columnar form exactly when reflect does.

func Unmarshal 

func Unmarshal(data []byte, out any, opts ...QueryOption) error

Unmarshal decodes data into out, which must be a non-nil pointer. The wire dialect is detected from the header — the same Unmarshal reads any output produced by Marshal regardless of the encode-side option bits.

Slice-backing reuse: when out is a *[]T whose slice already has enough capacity for the decoded element count, the decoder reuses that backing array instead of allocating a new one — eliminating the result allocation (the dominant decode cost) on a decode into a pooled / pre-sized slice. The reused backing is overwritten in place (its elements are zeroed first, so no stale data leaks), so do not share it with other live slices across the call. A nil or too-small destination allocates fresh, so default usage is unaffected.

The optional QueryOptions (Where / Select) turn the call into a filtering/projecting columnar decode: predicates are AND-ed and the named columns projected. They apply only to a columnar []struct, []map[string]any or []any payload; on any other shape a query call returns a *QueryError wrapping ErrUnsupported. With no options the behavior is exactly the plain decode above.

Round-trip fidelity: decoding into the same concrete Go type reproduces the value exactly, including the nil-vs-empty distinction for slices, maps and pointers (a nil []T decodes as nil, an empty []T{} as empty — like encoding/json's null vs []). Two deliberate normalizations are NOT bit-exact: decoding into an interface (any / map[string]any) canonicalizes integers to int64 / uint64 and structs to map[string]any (the schemaless type contract), and a time.Time loses any monotonic-clock reading on encode (as the standard library strips it for transport). Values nested deeper than the configured max depth, and maps with both int and uint keys of the same small value, are the documented exceptions.

Example (PredicatePushdown)

ExampleUnmarshal_predicatePushdown keeps only the rows matching typed predicates (AND-ed), decoding just the projected columns of those rows.

Predicate pushdown only fires when the payload is columnar, which the encoder picks for a sufficiently large []struct batch, so this example builds a 20-row batch rather than a handful of literals. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		TS    int64  `qdf:"ts"`
		Level string `qdf:"level"`
		Code  int32  `qdf:"code"`
	}
	in := make([]Event, 0, 20)
	for i := range 20 {
		lvl, code := "INFO", int32(200)
		switch i % 5 {
		case 0:
			lvl, code = "ERROR", 500
		case 1:
			lvl, code = "ERROR", 404
		case 2:
			lvl, code = "WARN", 503
		}
		in = append(in, Event{TS: int64(i + 1), Level: lvl, Code: code})
	}
	buf, _ := qdf.Marshal(in, qdf.OptBalanced|qdf.OptColumnIndex)

	type Out struct {
		TS   int64 `qdf:"ts"`
		Code int32 `qdf:"code"`
	}
	var hot []Out // filter on level (not in Out), return ts+code
	_ = qdf.Unmarshal(buf, &hot,
		qdf.Where("level", func(s string) bool { return s == "ERROR" }),
		qdf.Where("code", func(c int32) bool { return c >= 500 }))

	for _, e := range hot {
		fmt.Printf("ts=%d code=%d\n", e.TS, e.Code)
	}
}

Output:
ts=1 code=500
ts=6 code=500
ts=11 code=500
ts=16 code=500

func UnmarshalColumns added in v0.0.2 

func UnmarshalColumns(data []byte, out any, fields ...string) error

UnmarshalColumns decodes only the named columns of a columnar []struct payload into out, leaving the rest undecoded. out may point at a typed subset slice (e.g. *[]Subset whose fields are the wanted columns, matched by name like Unmarshal) or at the dynamic *[]map[string]any form, where only the named columns are stored. When the payload carries the column-length index (OptColumnIndex), unrequested columns are skipped without decoding; otherwise they are decoded but dropped.

fields names the wire columns to keep. With no fields it behaves like Unmarshal.

Example

ExampleUnmarshalColumns decodes only a named subset of columns from a columnar []struct payload. The producer opts in to OptColumnIndex so the payload carries a column-length index; the decoder then advances past the columns it was not asked for instead of decoding them. Here a 4-field struct batch is decoded into a 2-field typed subset, naming the wanted columns explicitly. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		TS      int64  `qdf:"ts"`
		Level   string `qdf:"level"`
		Service string `qdf:"service"`
		Code    int32  `qdf:"code"`
	}
	in := make([]Event, 4)
	levels := []string{"INFO", "WARN", "ERROR", "INFO"}
	for i := range in {
		in[i] = Event{
			TS:      int64(1700000000 + i),
			Level:   levels[i],
			Service: "billing",
			Code:    int32(200 + i),
		}
	}

	// OptColumnIndex writes the column-length index; the default columnar
	// wire is byte-identical without it.
	buf, _ := qdf.Marshal(in, qdf.OptBalanced|qdf.OptColumnIndex)

	// Decode only the "level" and "service" columns into a typed subset.
	// The "ts" and "code" columns are skipped via the index, not decoded.
	type Subset struct {
		Level   string `qdf:"level"`
		Service string `qdf:"service"`
	}
	var out []Subset
	if err := qdf.UnmarshalColumns(buf, &out, "level", "service"); err != nil {
		fmt.Println("error:", err)
		return
	}
	for _, e := range out {
		fmt.Printf("level=%-5s service=%s\n", e.Level, e.Service)
	}
}

Output:
level=INFO  service=billing
level=WARN  service=billing
level=ERROR service=billing
level=INFO  service=billing

func UnmarshalDirect 

func UnmarshalDirect[T Unmarshaler](data []byte, out T) error

UnmarshalDirect dispatches to out.UnmarshalQDF after validating the 5-byte header. It is the inverse of MarshalDirect; the same wire constraints apply (Fast-mode only).

If the input carries the FlagDense bit, UnmarshalDirect falls back to the full Unmarshal path because generated code cannot resolve state-ref tags without the decoder's intern table.

Decode-side performance is bounded by the user's UnmarshalQDF implementation: the reflect path is heavily pooled (Decoder pool + per-decoder key intern cache) and tends to win against naive hand-rolled receivers. Generated code from cmd/qdfgen uses Decoder.InternKey for map / struct keys and matches or beats the reflect path; ad-hoc UnmarshalQDF methods that call NewDecoderOnBuf + ReadString in a tight loop will not.

func UnmarshalNested added in v0.0.2 

func UnmarshalNested(u Unmarshaler, src []byte, noCopy bool) (int, error)

UnmarshalNested decodes one nested Unmarshaler value from src, honoring noCopy when u also implements UnmarshalerOpts. External Unmarshalers without the Opts method fall back to a copying decode. Used by decodeUnmarshaler and by cmd/qdfgen-generated code; exported for the latter.

func UnmarshalNestedArena added in v0.0.6 

func UnmarshalNestedArena(u Unmarshaler, src []byte, noCopy bool, a *Arena) (int, error)

UnmarshalNestedArena is UnmarshalNested threading a decode arena: when a is non-nil and u implements UnmarshalerArena, the nested decode packs its strings into a. Otherwise it falls back to the plain (copying) nested decode. Exported for cmd/qdfgen-generated code.

func UnmarshalT 

func UnmarshalT[T any](data []byte, out *T) error

UnmarshalT is the generic equivalent of Unmarshal. The destination pointer must not be nil.

Types

type Arena added in v0.0.6 

type Arena struct {
	// contains filtered or unexported fields
}

Arena is a bump allocator for decoded string (and []byte-as-string) bodies. Pass it to a decode via WithArena (or Decoder.SetArena) and every copied string is packed into the arena's dense, contiguous blocks instead of one heap allocation per string. Across an epoch of many decodes this amortizes the block allocation to near zero and keeps the strings cache-local; the GC scans each block as a single object instead of one object per string.

Lifetime / safety: strings returned by an arena decode ALIAS the arena's memory. They stay valid as long as the Arena (or any string from it) is reachable — the GC keeps a block alive from an interior pointer, so you need not track the buffer manually. The SAFE pattern is one Arena per epoch (request / batch / stream window), then drop it: 

a := qdf.NewArena()
for _, msg := range batch {
    var v Event
    _ = qdf.Unmarshal(msg, &v, qdf.WithArena(a))
    use(v)          // v's strings live in a
}
// drop a; the GC frees every block once the decoded values die.

Reset reuses the arena across epochs for zero allocation, but is UNSAFE while any string from the previous epoch is still live (see Reset).

An Arena is NOT safe for concurrent use; give each goroutine its own.

Example

ExampleArena shows the safe default pattern: one arena per epoch (here, one batch of messages). Every decoded string is bump-packed into the arena instead of a separate heap allocation. No Reset and no manual lifetime management are needed — the arena and its blocks are reclaimed by the GC once the decoded values are dropped. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Level   string `qdf:"level"`
		Msg     string `qdf:"msg"`
	}

	// Pretend these arrived over the wire.
	msgs := make([][]byte, 3)
	for i, m := range []Event{
		{"api", "info", "request handled"},
		{"worker", "warn", "queue backlog"},
		{"edge", "error", "upstream timeout"},
	} {
		b, err := qdf.Marshal(m, qdf.OptSpeed)
		if err != nil {
			panic(err)
		}
		msgs[i] = b
	}

	a := qdf.NewArena() // one arena for this batch
	out := make([]Event, len(msgs))
	for i, msg := range msgs {
		if err := qdf.Unmarshal(msg, &out[i], qdf.WithArena(a)); err != nil {
			panic(err)
		}
	}
	// out's strings live in `a`; when `out` is dropped the GC frees the arena.

	for _, e := range out {
		fmt.Printf("%s/%s: %s\n", e.Service, e.Level, e.Msg)
	}
}

Output:
api/info: request handled
worker/warn: queue backlog
edge/error: upstream timeout
Example (Reset)

ExampleArena_reset shows the max-performance pattern: reuse one arena across an unbounded stream, calling Reset at each epoch boundary for zero steady-state allocation.

SAFETY: Reset rewinds the arena and the next decode overwrites its memory, so every value decoded before a Reset must be done being used before that Reset runs. Here each event is processed inside the loop body, before the next iteration's Reset. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Msg     string `qdf:"msg"`
	}

	stream := []Event{
		{"api", "ok"},
		{"worker", "retry"},
	}
	encoded := make([][]byte, len(stream))
	for i, e := range stream {
		b, _ := qdf.Marshal(e, qdf.OptSpeed)
		encoded[i] = b
	}

	a := qdf.NewArena()
	for _, msg := range encoded {
		a.Reset() // safe: the previous iteration's value is no longer used

		var ev Event
		if err := qdf.Unmarshal(msg, &ev, qdf.WithArena(a)); err != nil {
			panic(err)
		}
		fmt.Printf("%s: %s\n", ev.Service, ev.Msg)
	}
}

Output:
api: ok
worker: retry

func NewArena added in v0.0.6 

func NewArena() *Arena

NewArena returns an empty arena. The first decode allocates the first block.

func (*Arena) Reset added in v0.0.6 

func (a *Arena) Reset()

Reset rewinds the arena to reuse its current block, so the next epoch of decodes allocates nothing.

UNSAFE: Reset invalidates every string previously returned from this arena — the next decode overwrites the same memory. Call it ONLY once every value decoded into the arena since the last Reset (or since NewArena) is dead. This is a manual use-after-free contract the race detector cannot catch; if unsure, drop the Arena and make a new one instead (the GC then frees it safely).

type BaselineRegistry added in v0.0.8 

type BaselineRegistry[T any] struct {
	// contains filtered or unexported fields
}

BaselineRegistry is a consumer-side, content-addressed cache of recent baselines of type T, used to apply a chain of patches in a state-sync stream without threading the previous value by hand. It maps each baseline's content fingerprint (the same value Diff embeds as baseFP) to a weak.Pointer, so the GC reclaims any baseline the caller no longer references. The registry never pins a baseline alive.

A BaselineRegistry is safe for concurrent use.

Example

ExampleBaselineRegistry shows applying a chain of patches in a state-sync stream without threading the previous value by hand. The registry resolves each patch's baseline from the baseFP that Diff already embeds (no wire change), holds baselines through weak pointers so the GC reclaims anything the caller drops, and auto-registers each result so it can base the next patch. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type State struct {
		Seq  int    `qdf:"seq"`
		Note string `qdf:"note"`
	}

	s0 := &State{Seq: 1, Note: "init"}
	s1 := State{Seq: 2, Note: "init"}
	s2 := State{Seq: 3, Note: "done"}

	// Producer ships ordinary diffs along the chain.
	patch1, _ := qdf.Diff(*s0, s1, qdf.OptBalanced)
	patch2, _ := qdf.Diff(s1, s2, qdf.OptBalanced)

	// Consumer: bootstrap from a full value, then apply the chain.
	reg := qdf.NewBaselineRegistry[State]()
	reg.Register(s0)

	got1, _ := reg.Apply(patch1) // resolves s0 by baseFP → s1
	got2, _ := reg.Apply(patch2) // resolves got1 (== s1) → s2
	_ = got1                     // keep the previous result reachable for the next step

	fmt.Printf("got2 = {Seq:%d Note:%s}\n", got2.Seq, got2.Note)

}

Output:
got2 = {Seq:3 Note:done}

func NewBaselineRegistry added in v0.0.8 

func NewBaselineRegistry[T any]() *BaselineRegistry[T]

NewBaselineRegistry returns an empty registry for baselines of type T.

func (*BaselineRegistry[T]) Apply added in v0.0.8 

func (r *BaselineRegistry[T]) Apply(patch []byte) (*T, error)

Apply resolves the patch's baseline (by its embedded baseFP) to a *T, applies the patch onto a fresh deep copy of it, registers the result as a new baseline (so it can base the next patch in the stream), and returns the new *T. The caller should keep the returned pointer to chain further patches off it; dropping it lets the GC reclaim it.

Errors: ErrBaselineRequired (the patch carries no baseFP — produced with OptDeltaNoBaseFingerprint), ErrBaselineEvicted (the baseline id is not resolvable), plus any error from the underlying Apply.

Because baselines are content-addressed by a 64-bit fingerprint, two distinct values that happen to share a fingerprint (probability ~N²/2⁶⁴, negligible for thousands of live baselines) collide on the same map key, and the later registration overwrites the earlier one; the earlier baseline then resolves as ErrBaselineEvicted even while the caller still holds it.

func (*BaselineRegistry[T]) Len added in v0.0.8 

func (r *BaselineRegistry[T]) Len() int

Len reports the number of live (non-evicted) baselines, pruning dead weak entries as a side effect.

func (*BaselineRegistry[T]) Register added in v0.0.8 

func (r *BaselineRegistry[T]) Register(v *T) uint64

Register stores v as a resolvable baseline and returns its content id — the same fingerprint Diff embeds as baseFP. The registry holds v through a weak.Pointer; the CALLER must keep v reachable for it to remain resolvable.

type CmpOp added in v0.1.1 

type CmpOp uint8

CmpOp is a comparison operator for WhereCmp, built from three primitive bits — "accept less-than", "accept equal", "accept greater-than" — so the named operators compose and the per-row test and zone bounds derive directly from the bits (no per-operator switch). Combine bits for custom relations, e.g. NE. 

const (
	LT CmpOp = cmpLT         // field <  val
	LE CmpOp = cmpLT | cmpEQ // field <= val
	EQ CmpOp = cmpEQ         // field == val
	GE CmpOp = cmpGT | cmpEQ // field >= val
	GT CmpOp = cmpGT         // field >  val
	NE CmpOp = cmpLT | cmpGT // field != val
)

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads QDF wire data from a single input buffer. Call SetInput to bind a buffer and the typed Read* methods to walk it. Behaviour is undefined if the input is mutated while the decoder holds it. Field order groups the pointer-bearing and 8-byte fields first, then the int counters, then the 1-byte flags last so the interspersed bools do not each force their own padding word (176 bytes vs 208 for the source order).

func NewDecoder 

func NewDecoder() *Decoder

NewDecoder constructs a Decoder. Bind input via SetInput.

func NewDecoderOnBuf 

func NewDecoderOnBuf(buf []byte) *Decoder

NewDecoderOnBuf binds a Decoder to buf. Equivalent to NewDecoder followed by SetInput(buf), without the SetInput branch checks.

func (*Decoder) Advance 

func (d *Decoder) Advance(n int)

Advance moves the read cursor forward by n bytes.

func (*Decoder) CheckLength 

func (d *Decoder) CheckLength(n int, perElem int) error

CheckLength returns ErrShortBuffer when n claimed elements cannot fit in the remaining input. Use before allocating per-element storage.

func (*Decoder) ClearColMaxLen added in v0.0.8 

func (d *Decoder) ClearColMaxLen()

ClearColMaxLen resets the per-column length bound. Generated hybrid decode calls it between the eligible columns (bounded to n) and the residual block (whose nested slices/maps may legitimately exceed n), mirroring decodeHybridColumnar.

func (*Decoder) DecodeValue added in v0.0.8 

func (d *Decoder) DecodeValue(out *any) error

DecodeValue decodes the next value as a dynamic any — the schemaless form (string, bool, int64/uint64/float64, []any, map[string]any, …) that decoding into an interface{} produces, mirroring encoding/json. It is the decode counterpart of Encoder.EncodeValue and is what qdfgen-generated code calls for an interface{} (any) struct field, so a code-generated type can still carry fully dynamic data on that field.

func (*Decoder) InternKey 

func (d *Decoder) InternKey(b []byte) string

InternKey returns a string equal to b, sharing storage with prior identical keys when the cache has them.

func (*Decoder) IsNil 

func (d *Decoder) IsNil() (bool, error)

IsNil reports whether the next value is the nil tag, consuming it on true. Returns (false, nil) for any other tag.

Example

ExampleDecoder_IsNil consumes a nil tag if present, returning false (without advancing) for any other tag. Useful when a field is optional on the wire and the caller wants to keep the read cursor stable on a non-nil branch. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	buf, _ := qdf.Marshal([]any{nil, "hello"}, qdf.OptSpeed)
	dec := qdf.NewDecoderOnBuf(buf)

	n, _ := dec.ReadArrayHeader()
	for range n {
		isNil, _ := dec.IsNil()
		if isNil {
			fmt.Println("<nil>")
			continue
		}
		s, _ := dec.ReadString()
		fmt.Println(s)
	}
}

Output:
<nil>
hello

func (*Decoder) MarkHeaderRead 

func (d *Decoder) MarkHeaderRead()

MarkHeaderRead tells the decoder the magic+version header is already consumed (e.g. the buffer is a tail slice from a parent decoder). The next read will skip the header check.

func (*Decoder) PeekColStruct added in v0.0.8 

func (d *Decoder) PeekColStruct() bool

PeekColStruct reports whether the next byte is a columnar container frame, without consuming it. Generated decode uses this to pick the columnar path vs the row-major fallback (a tiny slice the encoder kept row-major, or a reflect-produced row-major encoding of the same field).

func (*Decoder) PeekHybridColStruct added in v0.0.8 

func (d *Decoder) PeekHybridColStruct() bool

PeekHybridColStruct reports whether the next byte is a hybrid columnar frame (some fields columnar, some residual row-major), without consuming it.

func (*Decoder) PeekTag 

func (d *Decoder) PeekTag() (byte, error)

PeekTag returns the next tag byte without advancing the cursor.

Example

ExampleDecoder_PeekTag inspects the next wire tag without consuming it. Handy when implementing a dispatch loop over a schemaless payload, or when an Unmarshaler needs to branch on the upcoming type. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	buf, _ := qdf.Marshal(int64(-42), qdf.OptSpeed)
	dec := qdf.NewDecoderOnBuf(buf)
	tag, _ := dec.PeekTag()
	if tag == qdf.TagNil {
		fmt.Println("nil value")
		return
	}
	v, _ := dec.ReadInt()
	fmt.Printf("int=%d (tag=0x%02x)\n", v, tag)
}

Output:
int=-42 (tag=0xc7)

func (*Decoder) Pos 

func (d *Decoder) Pos() int

Pos returns the current read offset.

func (*Decoder) ReadArrayHeader 

func (d *Decoder) ReadArrayHeader() (int, error)

ReadArrayHeader returns the element count.

func (*Decoder) ReadBool 

func (d *Decoder) ReadBool() (bool, error)

func (*Decoder) ReadBoolColumn added in v0.0.8 

func (d *Decoder) ReadBoolColumn(n int) ([]bool, error)

ReadBoolColumn decodes one bool column of n values.

func (*Decoder) ReadBytes 

func (d *Decoder) ReadBytes() ([]byte, error)

ReadBytes returns a []byte value. With noCopy = true the result aliases the input.

func (*Decoder) ReadColNullMask added in v0.0.8 

func (d *Decoder) ReadColNullMask(n int) ([]byte, int, error)

ReadColNullMask reads a nullable column's presence bitmap for n rows and returns it (aliased into the input buffer, read-only) plus the present (set-bit) count. The dense column that follows has exactly that many values.

func (*Decoder) ReadColStructHeader added in v0.0.8 

func (d *Decoder) ReadColStructHeader() (int, []string, []byte, error)

ReadColStructHeader consumes the columnar container header and returns the row count and column shape (names + kind bytes). It mirrors readColShape's non-index path. n is bounded by maxColumnarElems.

func (*Decoder) ReadFloat32 

func (d *Decoder) ReadFloat32() (float32, error)

func (*Decoder) ReadFloat32Column added in v0.0.8 

func (d *Decoder) ReadFloat32Column(n int) ([]float32, error)

ReadFloat32Column decodes one float32 column (32-bit patterns via the unsigned codec, mirroring WriteFloat32Column). The bits land in colScratchU64 and are widened into colScratchF32.

func (*Decoder) ReadFloat64 

func (d *Decoder) ReadFloat64() (float64, error)

func (*Decoder) ReadFloat64Column added in v0.0.8 

func (d *Decoder) ReadFloat64Column(n int) ([]float64, error)

ReadFloat64Column decodes one float64 column of n values.

func (*Decoder) ReadHybridColStructHeader added in v0.0.8 

func (d *Decoder) ReadHybridColStructHeader() (int, []string, []byte, error)

ReadHybridColStructHeader consumes the hybrid columnar header and returns the row count and full shape (every field in declaration order; residual fields carry kind byte 0xFF). n is bounded by maxColumnarElems.

func (*Decoder) ReadInt 

func (d *Decoder) ReadInt() (int64, error)

func (*Decoder) ReadIntColumn added in v0.0.8 

func (d *Decoder) ReadIntColumn(n int) ([]int64, error)

ReadIntColumn decodes one signed-integer column of n values.

func (*Decoder) ReadMapHeader 

func (d *Decoder) ReadMapHeader() (int, error)

ReadMapHeader returns the key/value pair count.

func (*Decoder) ReadNil 

func (d *Decoder) ReadNil() error

func (*Decoder) ReadString 

func (d *Decoder) ReadString() (string, error)

ReadString returns the next string. If the decoder is in noCopy mode the returned string aliases the input buffer; otherwise a copy is made.

When the tag at the cursor resolves through the intern table — tagInternStr, tagStateRef, tagStateMTF, tagStatePair, tagStateRepeat — readStringBytes leaves d.state.lastID set to the entry it just touched. We return the pre-materialised d.state.stringValues[id] from that path instead of allocating a fresh `string(b)` copy on every state-ref hit. Inline reads (fixstr, str8/16/32) set lastID = lruInvalidID, fall through, and pay the copy as before.

func (*Decoder) ReadStringBytes 

func (d *Decoder) ReadStringBytes() ([]byte, error)

ReadStringBytes returns the next string or []byte value without copying. The returned slice aliases either the input buffer or the decoder's state-table storage. Callers that retain it beyond the input's lifetime must copy.

func (*Decoder) ReadStringColumn added in v0.0.8 

func (d *Decoder) ReadStringColumn(n int) ([]string, error)

ReadStringColumn decodes one string column of n values, mirroring the reflect colKindString decode: a dictionary / FSST / raw-slab block (tag present) goes through the shared block reader; otherwise the plain per-value path reads via ReadString so state-ref / MTF / repeat encodings written by the picker's plain fallback resolve correctly (and share the decode intern cache).

The returned slice is decode-state scratch reused across the columns of one struct decode (every path now pools it — see colStrScratch); it is valid only until the next string-column read, so scatter it into the destination before reading the next column. Every internal columnar decoder (codegen + reflect + delta + nullable) drains each column this way.

func (*Decoder) ReadStructHeader added in v0.0.8 

func (d *Decoder) ReadStructHeader() (names []string, plainN int, shaped bool, err error)

ReadStructHeader reads a struct/map header for code-generated DecodeQDF, transparently handling both forms a struct takes on the wire:

  • a shape-interned header (tagMapShape, see Encoder.StructShape) → returns the field names in encoded order with shaped=true; the caller reads len(names) values in that order.
  • a plain map header (tagMap8/16/32) → returns the entry count as plainN with shaped=false; the caller reads plainN (name, value) pairs inline.

This lets a generated decoder consume both qdfgen shape output and a plain qdf map (e.g. from a non-shaped encoder, an older generated type, or the reflect path under OptSpeed) without a wire-format negotiation. Exported for cmd/qdfgen-generated code.

func (*Decoder) ReadTimeColumn added in v0.0.8 

func (d *Decoder) ReadTimeColumn(n int) ([]int64, []uint64, error)

ReadTimeColumn decodes a time.Time column's two sub-columns and returns the sec / nsec slices; the caller reconstructs each value as time.Unix(sec[i], int64(nsec[i])).UTC(). nsec is validated in range.

func (*Decoder) ReadTimestamp added in v0.0.2 

func (d *Decoder) ReadTimestamp() (sec int64, nsec uint32, err error)

ReadTimestamp reads a full-range timestamp and returns (sec, nsec, err). sec is seconds since Unix epoch (may be negative for pre-1970 instants). nsec is nanoseconds in [0, 999_999_999]. This replaces the old fixed-8-byte UnixNano encoding (clean break).

func (*Decoder) ReadUint 

func (d *Decoder) ReadUint() (uint64, error)

ReadUint reads a value that was encoded with WriteUint or WriteInt with a non-negative argument. Returns an error if the next value is negative.

func (*Decoder) ReadUintColumn added in v0.0.8 

func (d *Decoder) ReadUintColumn(n int) ([]uint64, error)

ReadUintColumn decodes one unsigned-integer column of n values.

func (*Decoder) Remaining 

func (d *Decoder) Remaining() int

Remaining returns the number of unread bytes.

func (*Decoder) RemainingBytes 

func (d *Decoder) RemainingBytes() []byte

RemainingBytes returns the unread portion of the input buffer. The result aliases the buffer and is invalidated by the next SetInput.

func (*Decoder) SetArena added in v0.0.6 

func (d *Decoder) SetArena(a *Arena)

SetArena directs the decoder to pack copied inline string bodies into a, bump-packed, instead of allocating one string per field. Pass nil to disable. The decoded strings alias a's memory; see Arena for the lifetime contract. Ignored while noCopy is set. Not concurrent-safe: one Arena per goroutine.

func (*Decoder) SetInput 

func (d *Decoder) SetInput(buf []byte)

SetInput rebinds the decoder to buf, dropping any prior state table.

func (*Decoder) SetNoCopy 

func (d *Decoder) SetNoCopy(v bool)

SetNoCopy switches the decoder into aliasing mode: string and []byte reads return slices that share storage with the input buffer. Faster (~2x on string-heavy payloads) with near-zero allocations.

The aliases are valid only while the input is alive and unmodified. This is safe — and free — when the input is caller-owned and not reused: an mmap, a file fully read into memory, or a buffer allocated fresh per message and never pooled (the aliasing headers keep it alive via the GC, so you need not track its lifetime). The hazard is buffer REUSE or mutation (a sync.Pool buffer, an overwritten scratch slice), which silently corrupts already-decoded values — a manual use-after-free the race detector will not catch. See WithNoCopy for the full contract.

Example

ExampleDecoder_SetNoCopy returns string / []byte values that alias the input buffer instead of allocating per-value copies. Use this for read-heavy hot paths where the input buffer outlives every value the decoder hands back; do not retain the values past the buffer's lifetime. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	in := []string{"alpha", "beta", "gamma"}
	buf, _ := qdf.Marshal(in, qdf.OptSpeed)

	dec := qdf.NewDecoderOnBuf(buf)
	dec.SetNoCopy(true)

	n, _ := dec.ReadArrayHeader()
	for range n {
		s, _ := dec.ReadString() // alias into buf
		fmt.Println(s)
	}
}

Output:
alpha
beta
gamma

func (*Decoder) Skip 

func (d *Decoder) Skip() error

Skip advances past one value without materializing it.

Skip recurses through nested arrays and maps, so it bounds nesting depth via descend/ascend exactly like the reflect decode path: an unknown struct field whose value is a deeply-nested array is skipped here, and without this guard a hostile payload of N nested arrays would overflow the goroutine stack (an unrecoverable fatal error). The recursive d.Skip() calls below re-enter this guard, so every level is counted.

type DecoderUnmarshaler added in v0.0.8 

type DecoderUnmarshaler interface {
	Unmarshaler
	DecodeQDF(d *Decoder) error
}

DecoderUnmarshaler is the decode counterpart of EncoderMarshaler: it reads its value from a SHARED Decoder, advancing it, so a parent can thread one decoder through a whole value graph instead of opening one per nested value (one *Decoder plus its scratch / intern state per nested value otherwise). Generated code (cmd/qdfgen) implements it; DecodeNested prefers it. noCopy and arena live on the shared decoder, so a threaded nested decode inherits them with no extra arguments.

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes a single QDF value into a growing internal buffer. Reset drops the buffer contents and (in Dense mode) the intern table so the encoder can be reused. Field order groups the pointer-bearing / slice / map fields and the 8-byte counters first, then the 1-byte flags (mode + the many bools) last so the interspersed bools do not each force a padding word (200 bytes vs 232 for the source order).

Example

ExampleEncoder drives the encoder by hand instead of via the top-level Marshal pool. Use this when you need to:

  • keep a reusable buffer alive across many encodes;
  • call PreIntern with a known hot string pool;
  • mix WriteX primitives with reflect-driven EncodeValue.
package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	enc := qdf.NewEncoderWith(qdf.OptSpeed)

	// Primitives + struct fields can be mixed by hand.
	enc.WriteMapHeader(2)
	enc.WriteString("name")
	enc.WriteString("alice")
	enc.WriteString("age")
	enc.WriteInt(33)

	dec := qdf.NewDecoderOnBuf(enc.Bytes())
	n, _ := dec.ReadMapHeader()
	for range n {
		k, _ := dec.ReadString()
		switch k {
		case "name":
			v, _ := dec.ReadString()
			fmt.Printf("%s=%s\n", k, v)
		case "age":
			v, _ := dec.ReadInt()
			fmt.Printf("%s=%d\n", k, v)
		}
	}
}

Output:
name=alice
age=33
Example (LossyVector)

ExampleEncoder_lossyVector shows the opt-in lossy vector codec. With OptLossyVec and a fidelity budget, []float32/[]float64 embedding fields are rotated, quantized, and entropy-coded to a fraction of their raw size; the budget (here cosine similarity >= 0.999) bounds the approximation. Decoding needs no special flag — the 0xFD tag is recognized automatically. 

package main

import (
	"fmt"
	"math"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Doc struct {
		ID  string
		Emb []float32
	}

	// 64 deterministic embedding vectors of dimension 256.
	docs := make([]Doc, 64)
	for i := range docs {
		v := make([]float32, 256)
		for j := range v {
			v[j] = float32(math.Sin(float64(i*256+j) * 0.01))
		}
		docs[i] = Doc{ID: fmt.Sprintf("doc-%d", i), Emb: v}
	}

	enc := qdf.NewEncoderWith(qdf.OptBalanced | qdf.OptLossyVec)
	enc.SetVectorBudget(qdf.MinCosine(0.999))
	if err := enc.EncodeValue(docs); err != nil {
		panic(err)
	}
	data := enc.Bytes()

	var out []Doc
	if err := qdf.Unmarshal(data, &out); err != nil {
		panic(err)
	}

	// Cosine similarity of the first reconstructed vector vs the original.
	var dot, na, nb float64
	for j := range docs[0].Emb {
		a, b := float64(docs[0].Emb[j]), float64(out[0].Emb[j])
		dot += a * b
		na += a * a
		nb += b * b
	}
	cosine := dot / (math.Sqrt(na) * math.Sqrt(nb))

	rawF32 := len(docs) * 256 * 4
	fmt.Println("decoded count:", len(out))
	fmt.Println("smaller than raw f32:", len(data) < rawF32)
	fmt.Println("cosine >= 0.999:", cosine >= 0.999)

}

Output:
decoded count: 64
smaller than raw f32: true
cosine >= 0.999: true

func NewEncoder 

func NewEncoder(mode Mode) *Encoder

NewEncoder returns an Encoder. The internal buffer is allocated lazily on first write.

func NewEncoderOnBuf 

func NewEncoderOnBuf(buf []byte, mode Mode) *Encoder

NewEncoderOnBuf returns an Encoder that appends to buf at its current length. The buffer is NOT truncated; pass an empty slice for a fresh encoding. Call SetBuffer afterwards to truncate.

func NewEncoderWith 

func NewEncoderWith(opts Options) *Encoder

NewEncoderWith returns an Encoder configured by the option bit-mask directly. Dense state machinery is allocated only when OptDense is set. Defaults for intern threshold / depth match NewEncoder.

func (*Encoder) AdoptBuffer 

func (e *Encoder) AdoptBuffer(b []byte)

AdoptBuffer installs b as the working buffer, preserving its current length. Used to continue writing into a buffer that already carries data — for example, after a nested type returned its extended slice. headerOut is left unchanged.

func (*Encoder) AppendBytes 

func (e *Encoder) AppendBytes(p []byte)

AppendBytes appends raw, already-valid wire bytes. Bypasses tag dispatch; used by generated code to emit pre-encoded field-name prefixes.

func (*Encoder) ApplyOpts 

func (e *Encoder) ApplyOpts(opts Options)

ApplyOpts reconfigures the encoder via the options bit-mask. Equivalent to recreating the encoder with NewEncoderWith but preserves the existing pool buffer and (Dense-mode) intern state. The pool-backed Marshal* entry points call this on every acquire; callers driving an Encoder directly use it to switch between OptSpeed and OptBalanced without re-allocating.

func (*Encoder) Bytes 

func (e *Encoder) Bytes() []byte

Bytes returns the encoded payload. It aliases the encoder's buffer and is only valid until the next write or Reset.

func (*Encoder) Canonical added in v0.1.1 

func (e *Encoder) Canonical() bool

Canonical reports whether OptCanonical is set on this encoder. Generated EncodeQDF code calls it to decide whether to emit map entries in sorted (deterministic) key order, matching the reflect encoder's canonical path.

func (*Encoder) EncodeValue 

func (e *Encoder) EncodeValue(v any) error

EncodeValue runs the reflect-driven Marshal pipeline on v against this Encoder. Convenience for callers driving an Encoder directly (e.g. with PreIntern) — equivalent to what the pool-backed Marshal does internally.

Example

ExampleEncoder_EncodeValue drives the reflect-based encode path against any Go value directly through the encoder, bypassing the pool used by qdf.Marshal. Combine with PreIntern, custom SetBuffer, or SetIntern tuning for a long-lived encoder that shapes its state for a specific workload. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Config struct {
		Name  string `qdf:"name"`
		Limit int    `qdf:"limit"`
	}

	enc := qdf.NewEncoderWith(qdf.OptBalanced)
	if err := enc.EncodeValue(Config{Name: "service.cpu", Limit: 1000}); err != nil {
		panic(err)
	}

	var out Config
	_ = qdf.Unmarshal(enc.Bytes(), &out)
	fmt.Printf("%+v\n", out)
}

Output:
{Name:service.cpu Limit:1000}

func (*Encoder) EnsureHeader 

func (e *Encoder) EnsureHeader()

EnsureHeader forces a header write if one has not been emitted yet.

func (*Encoder) MarkHeaderWritten 

func (e *Encoder) MarkHeaderWritten()

MarkHeaderWritten tells the encoder the QDF header is already present in its buffer (e.g. left there by a parent encoder). The next write will skip the header emission.

func (*Encoder) PreIntern 

func (e *Encoder) PreIntern(strs ...string)

PreIntern registers the given strings against the encoder's intern table up front. Subsequent WriteString / WriteBytes calls that pass the SAME backing string header (i.e. the same underlying byte pointer and length) skip the hash + slot probe and emit a state-ref directly against the cached intern id.

Intended for power users who know their hot string pool ahead of time — service names, region codes, enum-like values drawn from a fixed slice. Real telemetry payloads draw 90 %+ of their dense intern hits from such pools. Skipping the hash/probe on those calls is the main per-emit cost left on the encode hot path.

Requires OptDense to be applied first (otherwise the call is a no-op). The registered identities are dropped on Reset so a pooled encoder does not carry caller-supplied pointers across recycles.

Safety: the caller must keep the backing memory of every PreIntern'd string alive for the lifetime of the next encode call. For literals embedded in a slice / global / struct field this is automatic; for short-lived stack strings the caller is responsible.

Example

ExampleEncoder_PreIntern registers a known string pool against the encoder so subsequent WriteString calls with the same backing pointer skip the intern table's hash + slot probe and emit a state-ref directly. Use this when the caller knows the hot vocabulary up front (service names, region codes, enum- like fields).

The caller must keep the registered strings' backing memory alive for the lifetime of the next encode call. String literals embedded in a slice, global, or struct field stay alive automatically; short-lived stack strings do not. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Region  string `qdf:"region"`
		Status  int    `qdf:"status"`
	}

	// Stable hot pool — backing slices live for the program's
	// lifetime, so the registered pointers stay valid across
	// many encode calls on the same encoder.
	services := []string{"billing", "auth", "ingest"}
	regions := []string{"eu-west-1", "us-east-1"}

	enc := qdf.NewEncoderWith(qdf.OptBalanced)
	enc.PreIntern(services...)
	enc.PreIntern(regions...)

	rows := []Row{
		{services[0], regions[0], 200},
		{services[1], regions[1], 500},
		{services[2], regions[0], 200},
	}
	if err := enc.EncodeValue(rows); err != nil {
		panic(err)
	}

	var out []Row
	_ = qdf.Unmarshal(enc.Bytes(), &out)
	for _, r := range out {
		fmt.Printf("%s/%s=%d\n", r.Service, r.Region, r.Status)
	}
}

Output:
billing/eu-west-1=200
auth/us-east-1=500
ingest/eu-west-1=200

func (*Encoder) QPack 

func (e *Encoder) QPack() bool

QPack reports whether QPack codec emission is enabled.

func (*Encoder) Reset 

func (e *Encoder) Reset()

Reset truncates the buffer and resets the intern table. Capacities are preserved. opts / mode / qpack are also reset to OptSpeed so a pooled encoder does not leak its previous configuration into the next caller — apply the desired options explicitly via applyOpts / SetQPack after Reset.

func (*Encoder) ScratchBool added in v0.0.8 

func (e *Encoder) ScratchBool(n int) []bool

ScratchBool returns a reusable []bool of length n for a bool column.

func (*Encoder) ScratchFloat32 added in v0.0.8 

func (e *Encoder) ScratchFloat32(n int) []float32

ScratchFloat32 returns a reusable []float32 of length n for a float32 column.

func (*Encoder) ScratchFloat64 added in v0.0.8 

func (e *Encoder) ScratchFloat64(n int) []float64

ScratchFloat64 returns a reusable []float64 of length n for a float64 column.

func (*Encoder) ScratchInt added in v0.0.8 

func (e *Encoder) ScratchInt(n int) []int64

ScratchInt returns a reusable []int64 of length n for a signed column.

func (*Encoder) ScratchMask added in v0.0.8 

func (e *Encoder) ScratchMask(n int) []byte

ScratchMask returns a zeroed reusable presence bitmap covering n rows ((n+7)/8 bytes). Generated code sets bit i for a present (non-nil) nullable column row, then passes it to WriteColNullMask.

func (*Encoder) ScratchString added in v0.0.8 

func (e *Encoder) ScratchString(n int) []string

ScratchString returns a reusable []string of length n for a string column.

func (*Encoder) ScratchUint added in v0.0.8 

func (e *Encoder) ScratchUint(n int) []uint64

ScratchUint returns a reusable []uint64 of length n for an unsigned column.

func (*Encoder) SetBuffer 

func (e *Encoder) SetBuffer(b []byte)

SetBuffer installs a caller-owned buffer (truncated to length 0).

func (*Encoder) SetIntern 

func (e *Encoder) SetIntern(min int, cap int)

SetIntern overrides the Dense-mode tuning knobs. Zero values keep the current setting.

func (*Encoder) SetMaxDepth 

func (e *Encoder) SetMaxDepth(d int)

SetMaxDepth caps reflect-path pointer/struct recursion. The default (DefaultMaxDepth = 10000) is sufficient for any normal payload and rejects pointer cycles before they stack-overflow the goroutine. Set to 0 to disable the check entirely — only safe when the caller can prove the input graph is acyclic.

func (*Encoder) SetQPack 

func (e *Encoder) SetQPack(v bool)

SetQPack toggles QPack codec emission. When true, slice fast paths produce packed/encoded forms (bitpacked bools, FOR-packed integers, Gorilla-encoded floats, raw-LE bulk) instead of per-element tag streams. Setting must happen before the first write of the value (the header is emitted lazily and carries the FlagQPack hint when this is on).

func (*Encoder) SetVectorBudget added in v0.1.1 

func (e *Encoder) SetVectorBudget(b VectorBudget)

SetVectorBudget sets the fidelity target used when OptLossyVec is active. No effect unless OptLossyVec is in the encoder's options.

func (*Encoder) StructShape added in v0.0.8 

func (e *Encoder) StructShape(token *byte, fieldHdrs [][]byte)

StructShape begins a shape-interned struct emission for a code-generated type (the decode-time counterpart is Decoder.ReadStructHeader). token is a stable per-type address — a package-level var the generated EncodeQDF passes; fieldHdrs are the pre-encoded fixstr/strN field-name headers in field order.

On the first emit for token on this encoder it DECLARES the shape — tagMapShape, id 0, the field count, then the names — so the decoder registers it; every later emit writes only tagMapShape + the shape ID. The caller then writes the field VALUES in field order (no names). Across a slice of the same struct threaded through one encoder, the field names are written once instead of per record. Reuses the shared shape-ID space, so a generated buffer stays decodable by the reflection path (tagMapShape is standard wire).

func (*Encoder) Take 

func (e *Encoder) Take() []byte

Take returns the encoded payload and detaches it from the encoder. The caller takes ownership.

func (*Encoder) WriteArrayHeader 

func (e *Encoder) WriteArrayHeader(n int)

WriteArrayHeader writes the header for an array of n elements. The caller must follow with exactly n element writes. n must not exceed math.MaxUint32 — the wire array count is a uint32, so a larger n is unrepresentable and panics rather than silently truncating the count (which would desync the decoder against the n bodies that follow).

func (*Encoder) WriteBool 

func (e *Encoder) WriteBool(v bool)

func (*Encoder) WriteBoolColumn added in v0.0.8 

func (e *Encoder) WriteBoolColumn(s []bool) error

WriteBoolColumn encodes a bool column.

func (*Encoder) WriteBytes 

func (e *Encoder) WriteBytes(b []byte)

WriteBytes writes a []byte. In Dense mode, eligible payloads are intern-encoded. The intern table is keyed on the byte sequence, so a string and a []byte with identical content share an ID.

func (*Encoder) WriteColNullMask added in v0.0.8 

func (e *Encoder) WriteColNullMask(mask []byte)

WriteColNullMask appends a nullable column's presence bitmap (raw bytes, bit i set ⇒ row i present), written before the dense column of present values. Layout matches encodeNullableColumn so the reflect path can cross-decode.

func (*Encoder) WriteColStructHeader added in v0.0.8 

func (e *Encoder) WriteColStructHeader(n int, names []string, kinds []byte)

WriteColStructHeader writes the columnar container header: tagColStruct, the row count n, and the column shape (field names + kind bytes). The shape is declared inline the first time this encoder sees it and reused by id afterwards, sharing encState's colStruct shape-id space with the reflect path. kinds[i] is the colKind byte for column i (see classifyColKind: int*->0, uint*->1, float64->2, bool->3, float32->6); it must match what the reflect encoder would emit for the same field so cross-decoding works.

names and kinds back the encoder's shape-id cache by reference on the declaring call; the caller must NOT mutate or recycle them for a different shape afterward (generated code passes immutable package-level vars).

func (*Encoder) WriteFloat32 

func (e *Encoder) WriteFloat32(v float32)

func (*Encoder) WriteFloat32Column added in v0.0.8 

func (e *Encoder) WriteFloat32Column(s []float32) error

WriteFloat32Column encodes a float32 column as its 32-bit patterns through the unsigned codec, bit-exact and matching the reflect colKindFloat32 path. The bit-conversion temp reuses colScratchU64 (free once any uint column it shares the buffer with has already been encoded), so no allocation.

func (*Encoder) WriteFloat64 

func (e *Encoder) WriteFloat64(v float64)

func (*Encoder) WriteFloat64Column added in v0.0.8 

func (e *Encoder) WriteFloat64Column(s []float64) error

WriteFloat64Column encodes a SCALAR float64 column losslessly. OptLossyVec targets genuine []float64/[]float32 VECTOR fields, not scalar columns, so this path never emits a lossy 0xFD block.

func (*Encoder) WriteHybridColStructHeader added in v0.0.8 

func (e *Encoder) WriteHybridColStructHeader(n int, names []string, kinds []byte)

WriteHybridColStructHeader writes the hybrid columnar container header (tagHybridColStruct, row count, shape). The shape lists EVERY field in declaration order; residual (non-columnar) fields carry kind byte 0xFF (residualKind). Shares the hybrid shape-id space with the reflect path.

func (*Encoder) WriteInt 

func (e *Encoder) WriteInt(v int64)

func (*Encoder) WriteIntColumn added in v0.0.8 

func (e *Encoder) WriteIntColumn(s []int64) error

WriteIntColumn encodes a column gathered from a signed-integer field. The adaptive picker chooses raw/FOR/Delta/RLE/Dict/PFOR per value range.

func (*Encoder) WriteMapHeader 

func (e *Encoder) WriteMapHeader(n int)

WriteMapHeader writes the header for a map of n entries. The caller must follow with exactly n key/value pairs. n must not exceed math.MaxUint32 — the wire map count is a uint32, so a larger n is unrepresentable and panics rather than silently truncating the count.

func (*Encoder) WriteNil 

func (e *Encoder) WriteNil()

func (*Encoder) WriteString 

func (e *Encoder) WriteString(s string)

WriteString writes s. In Dense mode (OptDense set), eligible strings are intern-encoded.

func (*Encoder) WriteStringColumn added in v0.0.8 

func (e *Encoder) WriteStringColumn(s []string)

WriteStringColumn encodes a string column, reusing the Balanced string-column picker (dictionary / FSST / raw-slab / plain), which is never larger than the per-value row-major encoding. Same wire as the reflect colKindString path.

func (*Encoder) WriteStringInline 

func (e *Encoder) WriteStringInline(s string)

WriteStringInline forces an in-line encoding even when Dense intern would be eligible. Use for fields known to be unique per message.

func (*Encoder) WriteTimeColumn added in v0.0.8 

func (e *Encoder) WriteTimeColumn(secs []int64, nsec []uint64) error

WriteTimeColumn encodes a time.Time column as two sub-columns — sec ([]int64, Delta+FOR compresses monotonic series) then nsec ([]uint64) — matching the reflect colKindTime path. The caller gathers secs/nsec from the time fields (t.UTC().Unix() / t.Nanosecond()); reuse ScratchInt/ScratchUint for them.

func (*Encoder) WriteTimestamp added in v0.0.2 

func (e *Encoder) WriteTimestamp(sec int64, nsec uint32)

WriteTimestamp writes a full-range timestamp as two uvarints: sec (zigzag-encoded signed int64 seconds since Unix epoch) and nsec (unsigned uint32 nanoseconds in [0, 999_999_999]). This replaces the old fixed-8-byte UnixNano encoding (clean break).

func (*Encoder) WriteUint 

func (e *Encoder) WriteUint(v uint64)

func (*Encoder) WriteUintColumn added in v0.0.8 

func (e *Encoder) WriteUintColumn(s []uint64) error

WriteUintColumn encodes an unsigned-integer column.

type EncoderMarshaler added in v0.0.7 

type EncoderMarshaler interface {
	Marshaler
	EncodeQDF(e *Encoder) error
}

EncoderMarshaler is an optional extension of Marshaler that writes directly into a shared Encoder. The buffer-based MarshalQDF forces a nested value to build its own Encoder on the parent's bytes (one *Encoder heap allocation per nested value); EncodeQDF lets a parent thread one Encoder through the whole graph. Generated code (cmd/qdfgen) implements it; EncodeNested prefers it.

type FSSTDict added in v0.0.2 

type FSSTDict struct {
	// contains filtered or unexported fields
}

FSSTDict is a pre-trained, immutable FSST symbol table.

Training the per-column symbol table is the dominant cost of FSST encoding. Train one FSSTDict over representative samples and reuse it across many FSSTDict.Marshal calls: each encode then skips training and pays only compression — the train-once, encode-many pattern (the "Static" in FSST).

An FSSTDict is bounded (≤255 symbols, a few KB) and never mutated after training, so it is safe for concurrent use and holds a fixed, small amount of memory. It cannot grow, so reusing one across the life of a process does not leak. The wire stays self-describing: each FSST column still carries its table, so output produced with a dictionary decodes with a plain Unmarshal.

func TrainFSSTDict added in v0.0.2 

func TrainFSSTDict(samples [][]byte) *FSSTDict

TrainFSSTDict learns an FSST symbol table from representative byte samples (typically a slice of the string column you will encode). Deterministic: the same samples always yield the same dictionary.

func TrainFSSTDictStrings added in v0.0.2 

func TrainFSSTDictStrings(samples []string) *FSSTDict

TrainFSSTDictStrings is TrainFSSTDict over []string, viewing each string's bytes without copying (the strings are only read during training).

func (*FSSTDict) AppendMarshal added in v0.0.2 

func (d *FSSTDict) AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)

AppendMarshal encodes v with this dictionary and appends to dst (see AppendMarshal). Enables FSST and its columnar prerequisites.

func (*FSSTDict) Marshal added in v0.0.2 

func (d *FSSTDict) Marshal(v any, opts Options) ([]byte, error)

Marshal encodes v using this pre-trained dictionary for FSST string columns. It enables the FSST codec and its columnar prerequisites (Dense, QPack, ShapeIntern), so even FSSTDict.Marshal(v, OptSpeed) compresses; combine with OptCompression to add the float codecs and rANS.

type Marshaler 

type Marshaler interface {
	MarshalQDF(dst []byte) ([]byte, error)
}

Marshaler is implemented by types that know how to serialize themselves into the QDF wire format. Implementations should append exactly one value to dst and return the extended slice.

Example

ExampleMarshaler shows a custom MarshalQDF / UnmarshalQDF pair. Use the interface when you want a compact wire form for a type — e.g. packing a small struct into a single primitive, versioning a payload, or pre-validating bytes before they hit the wire. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

// Color is a 24-bit RGB value stored as a single uint32 on the
// wire. It implements Marshaler / Unmarshaler so qdf encodes it
// as a primitive integer instead of recursing into reflect.
type Color struct{ R, G, B uint8 }

// MarshalQDF packs the three bytes into a uint32 and writes it
// using the encoder primitives from a temporary qdf.Encoder
// over the caller's buffer. This shape — temporary encoder over
// the caller's dst slice — is what qdfgen produces by default.
func (c Color) MarshalQDF(dst []byte) ([]byte, error) {
	enc := qdf.NewEncoderOnBuf(dst, qdf.Fast)
	enc.MarkHeaderWritten() // dst already contains the parent header
	enc.WriteUint(uint64(c.R)<<16 | uint64(c.G)<<8 | uint64(c.B))
	return enc.Bytes(), nil
}

// UnmarshalQDF reverses the pack and returns the number of bytes
// consumed from src.
func (c *Color) UnmarshalQDF(src []byte) (int, error) {
	dec := qdf.NewDecoderOnBuf(src)
	dec.MarkHeaderRead()
	v, err := dec.ReadUint()
	if err != nil {
		return 0, err
	}
	c.R = uint8(v >> 16)
	c.G = uint8(v >> 8)
	c.B = uint8(v)
	return dec.Pos(), nil
}

// ExampleMarshaler shows a custom MarshalQDF / UnmarshalQDF pair.
// Use the interface when you want a compact wire form for a
// type — e.g. packing a small struct into a single primitive,
// versioning a payload, or pre-validating bytes before they hit
// the wire.
func main() {
	in := Color{R: 0x12, G: 0x34, B: 0x56}
	buf, _ := qdf.MarshalDirect(&in)

	var out Color
	_ = qdf.UnmarshalDirect(buf, &out)
	fmt.Printf("R=%02x G=%02x B=%02x\n", out.R, out.G, out.B)
}

Output:
R=12 G=34 B=56

type Mode 

type Mode uint8

Mode selects the wire dialect. 

const (
	// Fast writes strings and []byte in line. No intern bookkeeping.
	Fast Mode = 0

	// Dense writes repeated strings/bytes once and references them by ID
	// thereafter. Smaller output on repetitive payloads; slightly slower
	// per call.
	Dense Mode = 1
)

type Options 

type Options uint32

Options is a bit-mask of per-call encoder feature toggles. A zero value (OptSpeed) gives the fastest path: Fast mode, no codecs, no predictors. Or-combine bits with | to opt in to individual codecs. Pre-built bundles cover the common "max speed", "balanced", and "max compression" use cases without having to remember the layout.

Options carry by value (uint32) so Marshal / AppendMarshal add zero allocations over the pool / output-clone overhead. The encoder checks each bit with a single AND on the hot path.

Example

ExampleOptions shows how the Options bit-mask picks which codec layers run for a Marshal call. The same input encodes to a progressively smaller wire as more codecs opt in, with a small CPU penalty per added codec. The decoder reads any dialect off the wire header so changing Options is a producer-side decision. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Region  string `qdf:"region"`
		Status  int    `qdf:"status"`
	}
	// Repetitive batch — every row shares the same service and region.
	in := make([]Row, 50)
	for i := range in {
		in[i] = Row{Service: "billing", Region: "eu-west-1", Status: 200}
	}

	for _, p := range []struct {
		name string
		opts qdf.Options
	}{
		{"OptSpeed", qdf.OptSpeed},
		{"OptBalanced", qdf.OptBalanced},
	} {
		buf, _ := qdf.Marshal(in, p.opts)
		fmt.Printf("%-12s wire=%d bytes\n", p.name, len(buf))
	}
}

Output:
OptSpeed     wire=2208 bytes
OptBalanced  wire=161 bytes
Example (Canonical)

ExampleOptions_canonical shows OptCanonical: two values that are logically equal but built differently — maps inserted in a different order, and one cost carrying -0.0 instead of +0.0 — serialize to byte-identical output, so a hash of the bytes is stable. That makes a canonical encoding safe to hash, sign, content-address, or deduplicate.

Without OptCanonical the two encodings may differ: Go randomizes map iteration order per range, and -0.0 carries a distinct sign bit. OptCanonical sorts map keys (every key kind) and normalizes floats (-0.0 → +0.0, any NaN → one quiet NaN), collapsing both to a single stable encoding. It is encode-side only — the bytes decode like any other qdf output. 

package main

import (
	"crypto/sha256"
	"fmt"
	"math"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Record struct {
		ID   string         `qdf:"id"`
		Tags map[string]int `qdf:"tags"`
		Cost float64        `qdf:"cost"`
	}

	a := Record{ID: "x", Tags: map[string]int{"a": 1, "b": 2}, Cost: 0.0}
	b := Record{ID: "x", Tags: map[string]int{"b": 2, "a": 1}, Cost: math.Copysign(0, -1)}

	ba, _ := qdf.Marshal(a, qdf.OptBalanced|qdf.OptCanonical)
	bb, _ := qdf.Marshal(b, qdf.OptBalanced|qdf.OptCanonical)

	fmt.Println("hashes equal:", sha256.Sum256(ba) == sha256.Sum256(bb))

}

Output:
hashes equal: true
Example (ColumnarStringDict)

ExampleOptions_columnarStringDict shows the per-column string dictionary that fires automatically under OptBalanced for a slice of structs whose string fields are enum-like — a small set of distinct values scattered across rows. The distinct strings are stored once and each row keeps a few-bit index, so the wire is far smaller than one interned reference per value. It is gated never-worse: run-heavy or high-cardinality columns keep the per-value path, and the choice is invisible to the decoder. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type LogRow struct {
		TS      int64  `qdf:"ts"`      // sequential — Delta+FOR makes columnar win
		Level   string `qdf:"level"`   // enum — string dictionary
		Service string `qdf:"service"` // enum — string dictionary
	}
	levels := []string{"INFO", "WARN", "ERROR"}
	services := []string{"api", "auth", "billing"}
	in := make([]LogRow, 300)
	for i := range in {
		in[i] = LogRow{
			TS:      int64(1700000000 + i),
			Level:   levels[i%len(levels)],
			Service: services[(i*7)%len(services)],
		}
	}

	buf, _ := qdf.Marshal(in, qdf.OptBalanced)

	var out []LogRow
	_ = qdf.Unmarshal(buf, &out)
	fmt.Printf("rows=%d wire=%d bytes roundtrip=%v\n", len(in), len(buf), out[0] == in[0] && out[299] == in[299])

}

Output:
rows=300 wire=234 bytes roundtrip=true
Example (StreamingDictShared)

ExampleOptions_streamingDictShared shows the headline Dense-mode win: across many calls of a StreamEncoder the intern table / shape table / predictors all survive, so the second record onwards trades long strings for 1-3 byte state-refs. Per-message Marshal does not share the dictionary across calls; only the stream API does. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	// 100 rows, all the same Service literal. Marshal-per-call
	// reuses the qdf encoder pool but resets state every call.
	one, _ := qdf.Marshal(Row{Service: "billing", Status: 200}, qdf.OptBalanced)
	fmt.Printf("per-message wire = %d bytes\n", len(one))

}

Output:
per-message wire = 36 bytes

const (
	// OptDense activates the inline intern table. First occurrences of
	// strings and []byte payloads are stored once and back-referenced
	// by ID (1-3 bytes per reuse). Required for any of the state-ref
	// codecs (Repeat / MTF / Pair) and for tagMapShape.
	OptDense Options = 1 << iota

	// OptQPack enables the numeric / boolean slice codecs. Bools
	// bit-pack; 32- and 64-bit integer slices ([]int/int32/int64,
	// []uint/uint32/uint64) run the codec picker — Frame-of-Reference,
	// Delta+FOR, RLE, dictionary, and Patched FOR — widening 32-bit
	// values to 64-bit for selection and narrowing back on decode;
	// float slices stay on raw-LE. Auto-selected per slice; the
	// encoder falls back to raw-LE when nothing wins. Gorilla XOR
	// for floats lives behind OptGorillaFloat (bit 5) — see
	// OptCompression for the bundle that turns it on.
	OptQPack

	// OptShapeIntern routes struct emissions through tagMapShape: each
	// distinct struct type declares its field ordering on first emit;
	// subsequent emits write only the shape ID + values. Requires
	// OptDense (the shape table lives on the intern table side).
	OptShapeIntern

	// OptPairPred enables the Markov-1 successor predictor
	// (tagStatePair, 0xEA). Per previous state-ref ID the encoder stores
	// its most-recent successor (top-1, K=1); a hit emits the transition
	// rank in a single byte (always rank 0). Requires OptDense.
	OptPairPred

	// OptMTF enables Move-to-Front rank coding (tagStateMTF, 0xE9).
	// When the LRU rank of a state-ref ID needs fewer varuint bytes
	// than the raw id, the rank is emitted instead. Requires OptDense.
	OptMTF

	// OptGorillaFloat opts in to the Gorilla XOR codec for []float64
	// (and []float32) slices. Gorilla collapses smooth time-series
	// data dramatically (~75 % wire reduction on quantised metric
	// streams) but trades that for ~10× more CPU on the encode/decode
	// path because the body is bit-level. Off by default in
	// OptBalanced for that reason; included in OptCompression for
	// archive-style workloads where wire size dominates. Requires
	// OptQPack.
	OptGorillaFloat

	// OptRANS adds a final static order-0 rANS entropy pass over the whole
	// encoded body. Opt-in (bundled into OptCompression) because it trades
	// encode/decode CPU for wire size. The encoder applies it only when the
	// rANS form is strictly smaller than the plain body, so it never makes a
	// buffer larger. Whole-buffer, so the streaming encoder ignores it.
	OptRANS

	// OptColumnIndex makes a columnar []struct payload carry a column-length
	// index so readers can decode a subset of columns without decoding the
	// rest. Opt-in, ~4 bytes per column. Only affects payloads the encoder
	// transposes into the columnar container (tagColStruct); has no effect on
	// other shapes. Sets FlagColIndex on the header.
	OptColumnIndex

	// OptFSST opts in to the FSST string codec for columnar string columns
	// (high-cardinality, substring-sharing text: log lines, URLs, paths). A
	// CPU-for-size trade (trains a per-column symbol table), so it is excluded
	// from OptBalanced and bundled into OptCompression. Never-larger: emitted
	// only when strictly smaller than the dictionary / per-value forms.
	OptFSST

	// OptMapShape interns map key-sets the way OptShapeIntern interns struct
	// field-names: the first map with a given set of (string) keys declares the
	// key ordering via tagMapShape; subsequent maps with the same key-set emit
	// only the shape ID + values in canonical (sorted) key order. Cuts encode
	// CPU and wire on maps with recurring keys (telemetry tags, log labels).
	// Requires OptDense (the shape table lives on the intern side). Opt-in in
	// v1; never-worse fallback to a plain map header when a key-set does not
	// recur.
	OptMapShape

	// OptDeltaNoBaseFingerprint disables the patch base fingerprint that Diff/Apply
	// use to detect a mismatched base. With it set, Diff omits the fingerprint and
	// Apply performs no base check — skipping a full reflect walk of the value on
	// BOTH sides (a large speedup for a big base with a tiny patch), at the cost of
	// the wrong-base safety guard. Use only when the caller guarantees Apply's base
	// is exactly the value Diff was computed against. No effect outside Diff/Apply.
	OptDeltaNoBaseFingerprint // bit 10

	// OptCanonical guarantees byte-identical output for logically-equal values:
	// map keys are emitted in sorted order (all key kinds) and floats are
	// normalized (-0.0 → +0.0, any NaN → a canonical quiet NaN). Encode-side
	// only; the bytes are ordinary qdf and decode normally. Intended for hashing
	// / signing / dedup of serialized output. Lossy for the sign of zero and NaN
	// payloads (use the default mode for bit-exact float round-trip).
	OptCanonical // bit 11

	// OptLossyVec enables the opt-in lossy float-vector codec (tag 0xFD) for
	// []float32/[]float64 columns. LOSSY: reconstruction is approximate within
	// the VectorBudget set on the Encoder (default MinCosine(0.999)). Never
	// fires unless this bit is set; never larger than the raw/lossless form.
	OptLossyVec // bit 12

	// OptZoneMap stores eligible columnar integer columns as zone-chunked
	// containers (tag 0xF1): the column is split into 256-row zones, each zone
	// independently codec-picked, prefixed by a per-zone [min,max] zonemap. A
	// bound-carrying predicate (WhereRange / WhereGE / WhereLE / WhereEq) then
	// skips zones that provably cannot match WITHOUT decoding them — a large
	// speedup for range/equality queries over positionally-ordered columns
	// (timestamps, sorted IDs). An explicit size-for-query-speed trade (chunking
	// + zonemap cost wire), so it is opt-in; without the bit the wire is
	// unchanged. v1 covers int/uint columns; string/float are a follow-up.
	OptZoneMap // bit 13

	// OptSpeed is the zero-bit preset: Fast mode, no codecs, no
	// predictors. Maximum throughput, smallest CPU footprint.
	OptSpeed Options = 0

	// OptBalanced bundles every codec that does not trade CPU for
	// compression beyond its sweet spot. The right default for
	// telemetry, log batches, and any payload with repetitive
	// strings or numeric slices. Notably excludes OptGorillaFloat —
	// reach for OptCompression when the float slices in the payload
	// are smooth time-series and wire size matters more than encode
	// latency.
	OptBalanced Options = OptDense | OptQPack | OptShapeIntern | OptPairPred | OptMTF

	// OptCompression bundles every codec that the encoder will spend
	// CPU on for wire-size gains. On top of OptBalanced it adds
	// OptGorillaFloat (Gorilla XOR for float slices), OptRANS (a final
	// order-0 rANS entropy pass over the whole body), and OptFSST (the
	// FSST string codec for columnar string columns). Each trades
	// encode/decode CPU for wire size, which is why they stay out of the
	// OptBalanced default; future heavy codecs land in this bundle without
	// breaking the name.
	OptCompression Options = OptBalanced | OptGorillaFloat | OptRANS | OptFSST
)

func (Options) Has 

func (o Options) Has(bit Options) bool

Has reports whether the named bit is set. Compiles to a single AND + compare. Use on the encoder hot path to gate codec emission.

type Ordered added in v0.1.1 

type Ordered interface {
	~int | ~int8 | ~int16 | ~int32 | ~int64 |
		~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~uintptr |
		~float32 | ~float64 | ~string
}

Ordered is Queryable minus bool — the kinds that support <,<=,>,>=.

type QueryError added in v0.0.2 

type QueryError struct {
	Err   error   // wrapped sentinel
	Op    string  // e.g. "predicate pushdown"
	Field string  // filter/projection field involved, if any
	Want  colKind // kind implied by the predicate's T (kind mismatches)
	Got   colKind // kind found on the wire (kind mismatches)
}

QueryError describes why a filtering/projecting decode (Unmarshal with QueryOptions) could not proceed. It wraps one of ErrUnsupported, ErrTypeMismatch, or ErrFieldNotFound, so callers can categorise the failure with errors.Is and read the specifics with errors.As.

func (*QueryError) Error added in v0.0.2 

func (e *QueryError) Error() string

func (*QueryError) Unwrap added in v0.0.2 

func (e *QueryError) Unwrap() error

type QueryOption added in v0.0.2 

type QueryOption struct {
	// contains filtered or unexported fields
}

QueryOption configures a filtering/projecting Unmarshal. Construct with Where, And, Or, Not (predicate tree) or Select (projection). Exactly one of node / selectFields is set.

func And added in v0.0.2 

func And(opts ...QueryOption) QueryOption

And keeps rows where every sub-predicate is true. Multiple top-level Where options are an implicit And; And is the explicit, nestable form.

func Not added in v0.0.2 

func Not(opt QueryOption) QueryOption

Not keeps rows where the sub-predicate is not true. Under three-valued NULL semantics a nil (absent) nullable row is UNKNOWN, so Not(pred) excludes nil rows just as the bare predicate does.

func Or added in v0.0.2 

func Or(opts ...QueryOption) QueryOption

Or keeps rows where at least one sub-predicate is true.

Example

ExampleOr demonstrates OR/NOT predicate combinators: keep events that are either level=="ERROR" or code>=500, while excluding level=="DEBUG" rows. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Level string `qdf:"level"`
		Code  int32  `qdf:"code"`
	}
	events := make([]Event, 30)
	for i := range events {
		if i%2 == 0 {
			events[i] = Event{Level: "ERROR", Code: int32(500 + i)}
		} else {
			events[i] = Event{Level: "INFO", Code: int32(i)}
		}
	}
	buf, _ := qdf.Marshal(events, qdf.OptBalanced|qdf.OptColumnIndex)

	var hot []Event
	_ = qdf.Unmarshal(buf, &hot,
		qdf.Or(
			qdf.Where("level", func(s string) bool { return s == "ERROR" }),
			qdf.Where("code", func(c int32) bool { return c >= 500 }),
		),
		qdf.Not(qdf.Where("level", func(s string) bool { return s == "DEBUG" })),
	)
	fmt.Println(len(hot))
}

Output:
15

func Select added in v0.0.2 

func Select(fields ...string) QueryOption

Select restricts decoding to the named columns. Top-level only; nesting a Select inside And/Or/Not is reported as ErrUnsupported.

func Where added in v0.0.2 

func Where[T Queryable](field string, pred func(T) bool) QueryOption

Where keeps only rows whose field column satisfies pred. T must match the column's native kind (any integer width, float32/64, string, or bool); a mismatch is reported as ErrTypeMismatch at decode time. If field is absent from the wire the call returns ErrFieldNotFound. Multiple Where options are AND-ed. The predicate is called once per row with the native value — no interface boxing per value.

func WhereCmp added in v0.1.1 

func WhereCmp[T Ordered](field string, op CmpOp, val T) QueryOption

WhereCmp keeps rows where (field op val): one of GE / GT / LE / LT / EQ. It is the bound-carrying form of a single comparison, enabling zone-skip on OptZoneMap int/uint columns. For a two-sided range use WhereRange.

func WhereRange added in v0.1.1 

func WhereRange[T Ordered](field string, lo, hi T) QueryOption

WhereRange keeps rows where lo <= field <= hi (inclusive). Bound-carrying: enables zone-skip on OptZoneMap int/uint columns.

func WithArena added in v0.0.6 

func WithArena(a *Arena) QueryOption

WithArena makes the decode pack copied inline string values into a, instead of allocating one string per field — near-zero allocations across an epoch of decodes (see Arena). The decoded strings alias a's memory and follow Arena's lifetime contract. Ignored together with WithNoCopy (aliasing already skips the copy). Composes with predicates / Select / WithNoCopy.

func WithNoCopy added in v0.0.2 

func WithNoCopy() QueryOption

WithNoCopy makes the decode return string and []byte values that ALIAS the input buffer instead of copying them. On string-heavy payloads this is ~2x faster with near-zero allocations.

DANGER — lifetime contract: the returned values are valid ONLY while the input buffer passed to Unmarshal stays alive and is never modified or reused. Do NOT use WithNoCopy when the input may be recycled, mutated, or freed before you finish reading the decoded values — e.g. a pooled HTTP request body. The decoded strings would silently become garbage. This is a manual-memory use-after-free, not a data race: the race detector will NOT catch it.

Safe for caller-owned, immutable input such as an mmap or a file read fully into memory.

Also safe — and the common zero-copy high-throughput pattern — when the input buffer is allocated FRESH per message and never pooled or overwritten: e.g. io.ReadAll(body), or a make([]byte, n) you read one message into and do not reuse. You do NOT have to track the buffer's lifetime manually: the aliasing string/[]byte headers in the decoded value keep the backing buffer alive through the garbage collector (Go marks the whole allocation from an interior pointer), so the buffer lives exactly as long as the values that point into it. The hazard is exclusively buffer REUSE (a sync.Pool buffer returned after the handler, a scratch slice you overwrite on the next read) or mutation — not lifetime. If every decode reads into its own fresh slice, WithNoCopy is safe and gives the full ~2x / near-zero-alloc win for free.

Scope: WithNoCopy affects the reflect decode path. A type that implements Unmarshaler (including codegen-generated UnmarshalQDF) decodes through its own decoder via the byte-only UnmarshalQDF(data) interface, which cannot inherit this flag, so such types still copy. Use SetNoCopy on a Decoder/StreamDecoder you drive directly if you need zero-copy there.

Example

ExampleWithNoCopy decodes with zero-copy: the returned string / []byte fields alias the input buffer instead of being copied, cutting allocations to near zero (~1.7x faster on string-heavy batches).

Lifetime contract: the decoded values are valid only while data stays alive and is never modified or reused. Never use WithNoCopy when data is a pooled or recycled buffer (e.g. an HTTP request body) — the values would silently corrupt. It is opt-in for this reason; the default Unmarshal copies. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Message string `qdf:"message"`
	}
	// data is owned here and outlives `out`, so noCopy is safe.
	data, _ := qdf.Marshal(&Event{Service: "api", Message: "ok"}, qdf.OptSpeed)

	var out Event
	if err := qdf.Unmarshal(data, &out, qdf.WithNoCopy()); err != nil {
		panic(err)
	}
	fmt.Println(out.Service, out.Message)
}

Output:
api ok

type Queryable added in v0.0.2 

type Queryable interface {
	int | int8 | int16 | int32 | int64 |
		uint | uint8 | uint16 | uint32 | uint64 | uintptr |
		float32 | float64 | string | bool
}

Queryable is the set of column element types a predicate can match. Exact base types only: Where dispatches on the concrete func(T) bool type once at construction, which matches base types but not user-defined named types.

type StreamDecoder 

type StreamDecoder struct {
	// contains filtered or unexported fields
}

StreamDecoder reads a sequence of values from an io.Reader. The intern table is preserved across Decode calls to match StreamEncoder.

The decoder grows its window buffer as needed. State-table entries alias the buffer, so the window is never compacted during a stream; total memory tracks the stream length. For unbounded streams partition into envelopes: decode one envelope, then Reset this decoder for the next (Reset reuses the decoder, its dense state, and the window backing — only the contents are cleared, so the window's high-water memory is reused, not re-grown, and the per-envelope footprint stays bounded). Pair Reset with SetArena to also bound the decoded-value memory across envelopes.

func NewStreamDecoder 

func NewStreamDecoder(r io.Reader) *StreamDecoder

NewStreamDecoder returns a stream decoder reading from r.

func (*StreamDecoder) Close 

func (s *StreamDecoder) Close() error

Close releases the scratch buffer to the pool. The underlying reader is not closed. Idempotent: subsequent calls are a safe no-op.

func (*StreamDecoder) Decode 

func (s *StreamDecoder) Decode(out any) error

Decode reads the next value into out. out must be a pointer. Returns io.EOF at a clean end of stream. Each message is length-framed, so a message of any size is buffered in full before decoding — no 4 KiB window limit.

func (*StreamDecoder) Reset added in v0.0.8 

func (s *StreamDecoder) Reset(r io.Reader)

Reset prepares the decoder to read a NEW, independent stream from r, reusing the decoder, its dense state, and the window buffer instead of allocating a fresh one. The noCopy setting is preserved; the cross-message state is cleared so the new stream is decoded from scratch. A no-op after Close.

func (*StreamDecoder) SetArena added in v0.0.8 

func (s *StreamDecoder) SetArena(a *Arena)

SetArena directs Decode to pack copied string / []byte-as-string bodies into the caller-owned Arena (bump-allocated, amortized to ~zero alloc) instead of one heap allocation per value. It is the streaming counterpart of Unmarshal(..., WithArena(a)).

Why both this AND SetNoCopy: they trade off lifetime vs copies.

  • SetNoCopy aliases the window: zero copies, but a decoded value lives only as long as the window — i.e. until Close (or until a Reset for a new stream). The window only ever grows (it is never compacted mid-stream), so for a long stream its memory tracks the whole stream length.
  • SetArena copies once into the arena: the decoded values then live as long as the Arena (independent of the window), and you control that — keep the arena across batches, Reset it once a batch's values are dead to reuse its blocks for the next batch at zero allocation, or drop it and let the GC free it. This is the right choice when you process a stream in batches and want each batch's decoded memory released without holding the growing window, or when values must outlive a Reset/new stream.

SetArena is ignored while SetNoCopy(true) is set (no-copy already avoids the copy the arena would pack). The setting is preserved across Reset; pass nil to clear it. Arena strings ALIAS the arena — see Arena for the use-after-free contract around Arena.Reset. No-op after Close. One Arena per goroutine.

Memory note: SetArena bounds the DECODED-VALUE memory (via Arena.Reset), not the read WINDOW. To bound a long stream's total footprint, partition it into envelopes and reuse this decoder across them with Reset (see Reset) — that is what caps the window; the arena caps the values.

Example

ExampleStreamDecoder_SetArena decodes a stream with string bodies bump-packed into a caller-owned Arena instead of one heap allocation per value. The arena is Reset per envelope so its blocks are reused; the decoder is Reset alongside to cap the read window. Together they bound a long stream's memory. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
	}

	var sink bytes.Buffer
	enc := qdf.NewStreamEncoderWith(&sink, qdf.OptBalanced)
	_ = enc.Encode(Event{Service: "billing"})
	_ = enc.Encode(Event{Service: "auth"})
	_ = enc.Flush()
	_ = enc.Close()
	stream := sink.Bytes()

	a := qdf.NewArena()
	dec := qdf.NewStreamDecoder(nil)
	dec.SetArena(a)   // copied string bodies live in the arena, not the heap
	defer dec.Close() //nolint:errcheck // example

	a.Reset()
	dec.Reset(bytes.NewReader(stream))
	var ev Event
	for dec.Decode(&ev) == nil {
		fmt.Println(ev.Service)
	}
}

Output:
billing
auth

func (*StreamDecoder) SetNoCopy 

func (s *StreamDecoder) SetNoCopy(v bool)

SetNoCopy makes Decode return string / []byte values that alias the decoder's internal window buffer instead of copying them — eliminating the per-value copy that dominates decode allocation.

Unlike the one-shot Decoder.SetNoCopy (which aliases the caller's input buffer and is unsafe the moment that buffer is reused — the typical server recycles its read buffer right after the handler returns), the stream owns its window buffer, so aliasing is safe for the lifetime of the stream:

  • A decoded value stays valid until Close. Close returns the window to a pool, after which any retained aliased value is undefined — copy (strings.Clone / append) anything you need past Close.
  • The window grows but is never compacted mid-stream (the Dense intern table already aliases it for cross-message back-references), so a growth does not invalidate earlier values, and noCopy adds no extra memory: the bytes are retained either way.

Because the window is retained for the whole stream regardless, this is the safe home for zero-copy decode — process or copy each value before Close and you pay zero per-value allocation across every message. No-op after Close.

Example

ExampleStreamDecoder_SetNoCopy decodes a multi-message stream without copying the decoded strings out of the input buffer. The decoded values alias the io.Reader's payload; the caller must not retain the values past the next Decode call. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
	}

	// Build a 2-message stream.
	var sink bytes.Buffer
	enc := qdf.NewStreamEncoderWith(&sink, qdf.OptBalanced)
	_ = enc.Encode(Event{Service: "auth"})
	_ = enc.Encode(Event{Service: "auth"})
	_ = enc.Flush()
	_ = enc.Close()

	dec := qdf.NewStreamDecoder(&sink)
	dec.SetNoCopy(true) // aliased reads — zero allocations
	defer dec.Close()   //nolint:errcheck // example
	var ev Event
	for dec.Decode(&ev) == nil {
		fmt.Println(ev.Service)
	}
}

Output:
auth
auth

type StreamEncoder 

type StreamEncoder struct {
	// contains filtered or unexported fields
}

StreamEncoder writes a sequence of values into an io.Writer. The header is emitted once before the first value; in Dense mode the intern table survives across Encode calls so back-references span the whole stream.

Example

ExampleStreamEncoder writes a batch of messages into a single io.Writer where the Dense-mode intern table, shape table, and predictors survive across calls. The first message of every shape pays for the key intern records; every subsequent message of the same shape emits values only.

This is the form to reach for when piping events to a file or a network socket — per-message Marshal would reset the state table each call and lose the cross-message compression. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	var sink bytes.Buffer
	enc := qdf.NewStreamEncoderWith(&sink, qdf.OptBalanced)
	defer enc.Close() //nolint:errcheck // example

	events := []Event{
		{"billing", 200},
		{"billing", 200},
		{"billing", 500},
	}
	for _, e := range events {
		if err := enc.Encode(e); err != nil {
			panic(err)
		}
	}
	if err := enc.Flush(); err != nil {
		panic(err)
	}

	dec := qdf.NewStreamDecoder(&sink)
	for {
		var out Event
		if err := dec.Decode(&out); err != nil {
			break
		}
		fmt.Printf("%s=%d\n", out.Service, out.Status)
	}
}

Output:
billing=200
billing=200
billing=500

func NewStreamEncoder 

func NewStreamEncoder(w io.Writer, mode Mode) *StreamEncoder

NewStreamEncoder returns a stream encoder backed by w. Dense mode activates the full balanced codec set (OptBalanced); Fast mode emits raw tagged bytes (OptSpeed). For finer per-stream tuning, build the StreamEncoder via NewStreamEncoderWith.

func NewStreamEncoderWith 

func NewStreamEncoderWith(w io.Writer, opts Options) *StreamEncoder

NewStreamEncoderWith returns a stream encoder with the given Options bit-mask. The intern table (when OptDense is set) survives across Encode calls so back-references span the whole stream.

func (*StreamEncoder) Close 

func (s *StreamEncoder) Close() error

Close flushes pending data and releases the scratch buffer to the pool. The underlying writer is not closed. Idempotent: subsequent calls are a safe no-op.

func (*StreamEncoder) Encode 

func (s *StreamEncoder) Encode(v any) error

Encode writes v as the next value in the stream. Each message is framed with a uvarint byte-length prefix so the decoder can buffer a whole message — of any size — before decoding it. The 5-byte stream header is written once, before the first frame, and is not itself framed. The encoder flushes internally when its buffer crosses 16 KiB; call Flush to push earlier.

func (*StreamEncoder) Flush 

func (s *StreamEncoder) Flush() error

Flush writes any buffered bytes to the underlying writer.

func (*StreamEncoder) Reset added in v0.0.8 

func (s *StreamEncoder) Reset(w io.Writer)

Reset prepares the encoder to write a NEW, independent stream to w, reusing the encoder, its grown intern / shape state, and the scratch buffer instead of allocating a fresh one. The configured Options (Dense, codecs) are kept; the cross-message state is cleared so the new stream's back-references start from scratch and the next Encode re-emits the one-per-stream header. A no-op after Close.

This is the streaming counterpart of the encoder pool behind Marshal: rather than construct (and re-allocate the intern table for) a StreamEncoder per independent batch, construct one and Reset it between batches — the heavy newEncState() is paid once.

Example

ExampleStreamEncoder_Reset reuses ONE StreamEncoder across two independent streams via Reset, instead of constructing a fresh one per stream. The heavy per-stream construction (the intern table) is paid once; Reset clears the cross-message state so each stream stays independent, while keeping the configured Options and the grown buffers for reuse. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	enc := qdf.NewStreamEncoderWith(nil, qdf.OptBalanced)
	defer enc.Close() //nolint:errcheck // example

	encodeBatch := func(evs []Event) []byte {
		var sink bytes.Buffer
		enc.Reset(&sink) // new independent stream, reuse encoder + intern table
		for _, e := range evs {
			_ = enc.Encode(e)
		}
		_ = enc.Flush()
		return append([]byte(nil), sink.Bytes()...)
	}

	a := encodeBatch([]Event{{"billing", 200}, {"billing", 500}})
	b := encodeBatch([]Event{{"auth", 200}})

	dec := qdf.NewStreamDecoder(nil)
	defer dec.Close() //nolint:errcheck // example
	for _, stream := range [][]byte{a, b} {
		dec.Reset(bytes.NewReader(stream)) // reuse decoder + window across streams
		var ev Event
		for dec.Decode(&ev) == nil {
			fmt.Println(ev.Service, ev.Status)
		}
	}
}

Output:
billing 200
billing 500
auth 200

type Unmarshaler 

type Unmarshaler interface {
	UnmarshalQDF(src []byte) (n int, err error)
}

Unmarshaler is implemented by types that know how to deserialize themselves from a QDF wire-format slice. Implementations should consume exactly one value from src and return the number of bytes consumed.

A custom UnmarshalQDF reads the Fast wire format. Marshal honours this: a type implementing Marshaler always emits its Fast-format body and is framed as Fast regardless of the Options passed, so Marshaler+Unmarshaler types round-trip under any Options. A type that implements Unmarshaler WITHOUT also implementing Marshaler is encoded structurally; under a Dense/QPack tier that produces a Dense body its Fast-only UnmarshalQDF cannot read. Implement both interfaces (or neither) to avoid this — generated code from cmd/qdfgen always implements both.

type UnmarshalerArena added in v0.0.6 

type UnmarshalerArena interface {
	UnmarshalerOpts
	UnmarshalQDFArena(src []byte, noCopy bool, a *Arena) (n int, err error)
}

UnmarshalerArena is an optional extension that also accepts a decode Arena, so the implementation packs copied string/[]byte fields into it (see WithArena) instead of one allocation per field. Generated code from cmd/qdfgen implements this; UnmarshalQDFOpts delegates to it with a nil arena. Decoders honor it only when the caller passed an arena.

type UnmarshalerOpts added in v0.0.2 

type UnmarshalerOpts interface {
	Unmarshaler
	UnmarshalQDFOpts(src []byte, noCopy bool) (n int, err error)
}

UnmarshalerOpts is an optional extension of Unmarshaler that accepts the noCopy flag. When noCopy is true, the implementation should decode string and []byte fields as aliases of src (see WithNoCopy) instead of copying. Generated code from cmd/qdfgen implements this; the plain UnmarshalQDF delegates to it with noCopy=false. Decoders honor it only when the caller opted into noCopy.

type VectorBudget added in v0.1.1 

type VectorBudget struct {
	// contains filtered or unexported fields
}

VectorBudget expresses the fidelity target for the lossy vector codec. Construct it with MaxRelError, MinCosine, or TargetSNR.

func MaxRelError added in v0.1.1 

func MaxRelError(eps float64) VectorBudget

MaxRelError bounds the per-vector relative L2 error (e.g. 1e-3).

Example

ExampleMaxRelError shows the relative-error budget. MaxRelError(eps) bounds the per-vector relative L2 error; a tighter eps spends more bytes for a closer reconstruction. NaN/Inf values are preserved bit-exactly via an exception list, independent of the lossy budget. 

package main

import (
	"fmt"
	"math"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		V []float64
	}

	v := make([]float64, 128)
	for i := range v {
		v[i] = float64(i) * 0.5
	}
	v[3] = math.NaN()
	v[7] = math.Inf(1)
	row := []Row{{V: v}}

	enc := qdf.NewEncoderWith(qdf.OptBalanced | qdf.OptLossyVec)
	enc.SetVectorBudget(qdf.MaxRelError(0.01)) // <= 1% relative L2 error
	if err := enc.EncodeValue(row); err != nil {
		panic(err)
	}

	var out []Row
	if err := qdf.Unmarshal(enc.Bytes(), &out); err != nil {
		panic(err)
	}

	fmt.Println("NaN preserved:", math.IsNaN(out[0].V[3]))
	fmt.Println("+Inf preserved:", math.IsInf(out[0].V[7], 1))

	// finite values within the 1% budget
	var se, ne float64
	for i := range v {
		if math.IsNaN(v[i]) || math.IsInf(v[i], 0) {
			continue
		}
		d := v[i] - out[0].V[i]
		se += d * d
		ne += v[i] * v[i]
	}
	fmt.Println("rel error <= 1%:", math.Sqrt(se/ne) <= 0.01)

}

Output:
NaN preserved: true
+Inf preserved: true
rel error <= 1%: true

func MinCosine added in v0.1.1 

func MinCosine(c float64) VectorBudget

MinCosine bounds the minimum cosine similarity (e.g. 0.999) — for embeddings.

func TargetSNR added in v0.1.1 

func TargetSNR(db float64) VectorBudget

TargetSNR targets a signal-to-noise ratio in dB (e.g. 40).

Source Files

View all Source files

Directories

Path Synopsis
examples
decodealloc command
decodealloc shows the decode-side allocation lever.
 decodealloc shows the decode-side allocation lever.
embeddings command
embeddings demonstrates the opt-in lossy vector codec for AI embeddings.
 embeddings demonstrates the opt-in lossy vector codec for AI embeddings.
query command
query demonstrates predicate pushdown: filtering a columnar batch without fully decoding it.
 query demonstrates predicate pushdown: filtering a columnar batch without fully decoding it.
streaming command
streaming shows the multi-message win: a StreamEncoder keeps ONE intern dictionary across every message in the stream, so a string that repeats from one message to the next (service names, region codes, status labels) is written in full once and referenced by a 1-byte id thereafter.
 streaming shows the multi-message win: a StreamEncoder keeps ONE intern dictionary across every message in the stream, so a string that repeats from one message to the next (service names, region codes, status labels) is written in full once and referenced by a 1-byte id thereafter.
telemetry command
telemetry demonstrates qdf's core win: it compresses *across* records.
 telemetry demonstrates qdf's core win: it compresses *across* records.
internal
bitflag
Package bitflag provides tiny generic helpers for bit-flag enums over any unsigned integer type.
 Package bitflag provides tiny generic helpers for bit-flag enums over any unsigned integer type.
bitpack
Package bitpack implements the pure bit-packing and SIMD kernel layer shared by qdf's integer codecs (FOR, Delta+FOR, dict, PFOR, ALP).
 Package bitpack implements the pure bit-packing and SIMD kernel layer shared by qdf's integer codecs (FOR, Delta+FOR, dict, PFOR, ALP).
bufpool
Package bufpool implements size-classed, sharded byte-slice pools.
 Package bufpool implements size-classed, sharded byte-slice pools.
bumparena
Package bumparena is a monotonic bump allocator for decoded string bodies.
 Package bumparena is a monotonic bump allocator for decoded string bodies.
endian
Package endian exposes a build-time constant for native byte order.
 Package endian exposes a build-time constant for native byte order.
fsst
Package fsst implements FSST (Fast Static Symbol Table) string compression: a learned table of up to 255 substrings (each 1..8 bytes) replaced by 1-byte codes, with code 255 reserved as an escape for a following literal byte.
 Package fsst implements FSST (Fast Static Symbol Table) string compression: a learned table of up to 255 substrings (each 1..8 bytes) replaced by 1-byte codes, with code 255 reserved as an escape for a following literal byte.
hadamard
Package hadamard implements a randomized fast Walsh–Hadamard rotation R = (1/√n)·H·D, where D is a seed-driven random ±1 diagonal and H is the Walsh–Hadamard matrix.
 Package hadamard implements a randomized fast Walsh–Hadamard rotation R = (1/√n)·H·D, where D is a seed-driven random ±1 diagonal and H is the Walsh–Hadamard matrix.
intern
Package intern provides a small fixed-size string cache used by the decoder to deduplicate repeated map keys.
 Package intern provides a small fixed-size string cache used by the decoder to deduplicate repeated map keys.
internarena
Package internarena is a bytes-only bump-pointer allocator for the Dense-mode intern table.
 Package internarena is a bytes-only bump-pointer allocator for the Dense-mode intern table.
lattice
E8 lattice quantizer.
 E8 lattice quantizer.
mapsgen command
internal/mapsgen — code generator for typed map encode/decode fast paths.
 internal/mapsgen — code generator for typed map encode/decode fast paths.
rans
Package rans is a static order-0 byte range Asymmetric Numeral System coder (the canonical rans_byte: 32-bit state, byte renormalization, a 12-bit normalized frequency table).
 Package rans is a static order-0 byte range Asymmetric Numeral System coder (the canonical rans_byte: 32-bit state, byte renormalization, a 12-bit normalized frequency table).
reflectutil
Package reflectutil holds the reflect-based helpers the qdf codec uses to allocate slices and maps without going through reflect.Value.
 Package reflectutil holds the reflect-based helpers the qdf codec uses to allocate slices and maps without going through reflect.Value.
unsafestr
Package unsafestr provides a zero-copy conversion from []byte to string.
 Package unsafestr provides a zero-copy conversion from []byte to string.
vecquant
Package vecquant orchestrates the lossy vector pipeline: it maps a user quality budget to a quantization step, runs Hadamard rotation → scalar quantization → rANS entropy coding of the integer coordinates, and verifies the achieved error on the real data.
 Package vecquant orchestrates the lossy vector pipeline: it maps a user quality budget to a quantization step, runs Hadamard rotation → scalar quantization → rANS entropy coding of the integer coordinates, and verifies the achieved error on the real data.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.