qdf

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 29, 2026 License: MIT Imports: 19 Imported by: 0

README

qdf

ci codeql codecov benchmarks Go Reference Go Report Card Go License Status

Quantum-inspired data serialization format designed for ultra-fast and lightweight data exchange.

qdf gives you encoding/json-style ergonomics (Marshal / Unmarshal / NewEncoder / NewDecoder) and a compact tagged binary wire format with an optional inline string-interning table. The reflect path ships with specialised encoders for the common slice / map shapes. A go:generate tool emits reflection-free MarshalQDF / UnmarshalQDF methods when you need the last 10–15 %.

Where the name comes from

Quantum Density Format is a borrowed metaphor, not physics. A classical CPU cannot store qubits; what qdf borrows from quantum information theory is a way of thinking about repetitive structured data.

The starting observation: in a payload like

{ "users": [
    {"country": "LT"},
    {"country": "LT"},
    {"country": "LT"} ] }

the byte "LT" is written three times even though, from an information-theoretic point of view, it is a single state with high probability. Classical formats (JSON, MessagePack, CBOR, Protobuf) treat data as tree → bytes. qdf treats it as

data = (state set, references, entropy weights)

— a finite set of distinct values plus a stream that "collapses" each position onto one of them. The wire dialect that ships today realises the practical subset of that idea:

Quantum-inspired concept What it maps to in qdf today
State set — the discrete values a position can take. The Dense-mode intern table. The first occurrence of a string or []byte value writes it in full and assigns it a stable ID.
Wavefunction collapse — a single observation picks one state. Every subsequent occurrence emits a state_ref tag plus a varint ID. Decoding "collapses" the reference back to its stored value.
Density — keep only what carries information; minimise entropy H(D | C) against the existing context. Repeated keys / values across a message (and across a Dense stream) cost 1–3 bytes rather than their full length. Numeric and bool slices use QPack codecs (FOR, Delta+FOR, RLE, dictionary, Gorilla XOR, ALP decimal, raw-LE bulk) that auto-select the smallest predicted form per slice; slices of homogeneous structs transpose to per-column codecs. Field-name headers in generated code are pre-encoded once per type and concatenated without further work.
Probabilistic / residual coding — predict, then store only the deviation. QPack's Delta+FOR codec is exactly this for monotonic integer sequences: encode the first value plus the bit-packed residual against a aᵢ = aᵢ₋₁ + minΔ predictor. Gorilla does the same for floats by XOR-ing each sample against the previous one and storing the differing bits only.
Entanglement — correlated values that constrain each other (e.g. city = "Vilnius" ⇒ country = "Lithuania"). Three stacked predictors on the Dense state stream. Markov-0 (tagStateRepeat, 0xE8) collapses an immediate repeat of the previously emitted state-ref to a single byte. MTF rank (tagStateMTF, 0xE9) encodes the LRU rank of the touched ID when that varuint is shorter than the raw id. Markov-1 pair (tagStatePair, 0xEA) keeps the last 4 successors observed after each prev ID and encodes a hit as 0xEA + 1-byte rank — wins when the prev → curr transition is predictable and the raw id needs a multi-byte varuint. The encoder picks the shortest of the four variants per emission, so the wire is never larger than the plain tagStateRef encoding. A full conditional-probability table beyond order-1 stays in the reserved 0xEB / 0xED..0xEF block.
Shape interning — repeated structure means the layout itself is information. Dense mode emits structs (and map[string]any of stable shape via the reflect-struct path) through tagMapShape (0xEC). First occurrence declares the shape inline: 0xEC, 0, varuint(N), N × key. Subsequent occurrences of the same shape emit 0xEC + varuint(shapeID) + N × value — keys are not re-emitted. Per-record saving on an array of identical-shape structs is N × 2 bytes for the elided state-refs plus the map header. The shape table is per-stream, addressed by *typeDesc on the encoder and by sequential ID on the wire. Shapes never collide across types because the encoder keys the binding on the descriptor pointer.
Arithmetic / range coding (rANS) — push the encoded stream to its Shannon limit. Shipped behind OptRANS (in OptCompression): a static order-0 rANS pass over the whole body, applied only when it shrinks (never larger). Squeezes the residual byte-entropy the structural codecs leave — e.g. −37 % on trace batches — at ~4–6× CPU, so it stays in the opt-in compression tier.

The conceptual roadmap (state table → entanglement graph → predictive encoder) is preserved in the codebase: tag space and the encoder interface are designed to let those layers slot in above the current one without a format break. The MVP that ships now is the part of the idea that has a positive CPU / size tradeoff on day-one workloads — logs, traces, columnar rows, embedding vectors.

The realistic conclusion: the real shift is not the "quantum" framing but the move from "data as bytes" to "data as a model of the world that produced it".

Why another format

encoding/json is verbose and slow. vmihailenco/msgpack/v5 is fast but allocates a lot on decode and has no built-in deduplication for repeated keys / values — which is the dominant cost on real telemetry, log, and trace payloads where the same handful of strings (service names, log levels, region codes, span attributes) appears thousands of times per batch.

qdf attacks two costs at once:

  1. Wire size on repetitive data. A Dense-mode encoder maintains an inline interning table: the first time "region":"eu-west-1" appears it is written in full; subsequent occurrences emit a 2-byte back-reference. Streams share that table across messages, so a 1000-event log batch with eight distinct region codes carries each code once. The decoder follows the same protocol — no out-of-band dictionary, no two-pass encode.

  2. CPU and allocations on decode. The reflect path uses a cached per-type descriptor with unsafe.Pointer + field offset access instead of reflect.Value.Field(i), plus specialised type-specific decoders for []string, []int*, []float*, map[string]string, map[string]int, map[string]any, etc. A direct-mapped key interner deduplicates map keys across decode calls without a pool or chained hash table. On a map[string]any decode with repeated keys the allocator runs 3 times per call versus json's 37.

The format is single-pass and streaming-safe in both directions. There is no schema language, no IDL, no compile-time registration — types are picked up via Go struct tags (qdf:"name", falling back to json:"name").

Wire layout
+-----+-----+-----+-----+-----+
| 'Q' | 'D' | 'F' | ver |flags|
+-----+-----+-----+-----+-----+
| body (tagged values)        |
+-----------------------------+

A 5-byte header (3-byte magic + 1-byte version + 1-byte flags) and a tagged body. The flag bit distinguishes Fast from Dense; the decoder auto-detects.

Tag space is msgpack-inspired with a few additions:

Range Purpose
0x00..0x7F positive fixint (0..127)
0x80..0x9F fixstr (len 0..31)
0xA0..0xBF fixarr (len 0..31)
0xC0..0xCF nil, bool, int / uint / float of width 8–64
0xCD..0xCF str8 / str16 / str32
0xD0..0xD2 bin8 / bin16 / bin32
0xD3..0xD4 arr16 / arr32
0xD5..0xD7 map8 / map16 / map32
0xD8..0xDF negfixint (-1..-8)
0xE0..0xE2 intern_str / state_ref / intern_bin (Dense)
0xE3 QPack: bit-packed []bool
0xE4 QPack: raw-LE numeric slice
0xE5 QPack: Frame-of-Reference bit-packed ints
0xE6 QPack: Delta + zigzag + FOR (monotonic ints)
0xE7 QPack: Gorilla XOR-coded float slice
0xE8 Dense: state-ref repeat (Markov-0 predictor)
0xE9 Dense: state-ref MTF rank (Move-To-Front)
0xEA Dense: state-ref pair rank (Markov-1)
0xEB QPack: run-length encoded integer slice
0xEC Dense: struct/map shape declare / reuse
0xED QPack: dictionary-coded integer slice
0xEE QPack: Patched FOR integer slice (outliers)
0xEF QPack: columnar []struct container
0xF0..0xF3 ext / timestamp
0xF4 QPack: ALP decimal-coded []float64 slice
0xF5..0xFF reserved (rANS, n-gram graph, future)

The 5th header byte holds two flag bits: FlagDense (0x01) for the intern dialect, and FlagQPack (0x02) as an early hint that the body may carry codec tags from the QPack codec range (0xE3..0xEF, 0xF4). A reader that does not implement the QPack tags fails with ErrBadTag on first contact; it never decodes a packed payload as scalar by accident.

Alpha note: qdf is pre-1.0. The wire format is still being shaped — tags may shift, version may bump, and "older" decoders here mean "earlier in alpha", not stable releases. No backwards-compat promise yet.

All multi-byte integers and floats are little-endian. amd64 and arm64 are the supported targets.

Quick start

go get github.com/alex60217101990/qdf

Requires Go 1.26.

package main

import (
    "fmt"

    "github.com/alex60217101990/qdf"
)

type Event struct {
    ID      int               `qdf:"id"`
    Source  string            `qdf:"source"`
    Payload []byte            `qdf:"payload"`
    Attrs   map[string]string `qdf:"attrs"`
}

func main() {
    in := Event{
        ID:     42,
        Source: "ingest",
        Attrs:  map[string]string{"region": "eu-west-1", "version": "v3"},
    }

    b, err := qdf.Marshal(in, qdf.OptSpeed)
    if err != nil {
        panic(err)
    }
    fmt.Printf("wire: %d bytes\n", len(b))

    var out Event
    if err := qdf.Unmarshal(b, &out); err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", out)
}
Encode profile is per-call

There is one encode entry point — Marshal(v, opts) — and the opts bit-mask picks which codecs run for that specific call. The convenience bundles cover the common tradeoffs:

Bundle When to use
qdf.OptSpeed Fast path. Tightest CPU cost; size comparable to msgpack. Drop-in for encoding/json behaviour.
qdf.OptBalanced Repetitive payloads — logs, telemetry, columnar rows. Strings intern once; numeric and bool slices use QPack codecs; struct shapes intern; Markov-1 + MTF run on state-refs.
qdf.OptCompression OptBalanced plus the heavier wire-size codecs: Gorilla XOR for smooth float series, ALP for quantized/decimal []float64, and a final order-0 rANS entropy pass over the whole body (never larger). Trades encode CPU for smaller wire — pick it for backup / cold storage.
custom mix Or-combine individual bits (OptDense | OptQPack | OptShapeIntern …) when one of the bundles is one click off the desired tradeoff. 
b, _ := qdf.Marshal(event,    qdf.OptSpeed)        // hot path
b, _ := qdf.Marshal(batch,    qdf.OptBalanced)     // telemetry / logs
b, _ := qdf.Marshal(snapshot, qdf.OptCompression)  // backup / archive
b, _ := qdf.Marshal(payload,                       // tuned mix
    qdf.OptDense|qdf.OptQPack|qdf.OptShapeIntern)

A single qdf.Unmarshal reads everything. The wire header is self- describing; the receiver never has to know which option mix the sender picked.

Picking the right combo for a concrete workload — hot path, telemetry, metric series, embeddings, backup — is covered in docs/CHOOSING.md, with head-to-head numbers vs json and msgpack per scenario.

Dense mode internals — what each bit buys you

OptDense activates the inline intern table. The four codec bits that compose OptBalanced layer on top of it. They share one rule: the encoder picks the shortest variant per emission, so Dense wire is never larger than the same payload at OptSpeed. If the predictors do not pay, the byte cost is identical to a plain state-ref.

Tag Predictor Wins when Doesn't help when
0xE8 Markov-0: same ID as the previous emission Runs of the same value (region, region, region). Distinct values in a row.
0xE9 MTF: encode LRU rank instead of raw id Heavy reuse of a small hot subset of strings that were interned early. Random access pattern with no recency.
0xEA Markov-1: top-1 predictor of the previous emission's successor Correlated pairs — country → city, service → region. Only emits when the raw id needs ≥ 2 varuint bytes, so the wire never grows on small state tables. Unique transitions, low-cardinality state.
0xEC Shape interning: declare struct shape once, reuse by id Arrays / streams of the same struct type. Per-record saving is ≈ N × 2 bytes (elided key state-refs + map header). One-shot encodes of a unique struct type.

The state table is shared per stream in StreamEncoder / StreamDecoder, so a batch of 1 000 log events with eight distinct region codes ships each region once. Marshal* calls do not share — each call starts with an empty table.

Where Dense is worth it: logs, traces, telemetry, columnar rows, event batches, audit streams, snapshot dumps — anything where a small vocabulary of strings and a stable struct shape repeat thousands of times. Expected envelope: 25 – 55 % smaller than JSON at 2–6× the encode speed.

Where Dense is not worth it: single-shot encodes of small unique payloads (Markov / shape predictors never reach their amortisation point, you pay ≤ 2 bytes of prelude for nothing), or strict-throughput paths where decode CPU is the bottleneck — Dense decode is ~15 % slower than Fast because of the state-table lookups. Reach for OptSpeed or OptQPack there.

Where it is wrong: cryptographic / forensic contexts that need byte-stable wire across implementations and versions. Dense embeds predictor state, intern-ID ordering, and shape IDs that depend on emission order. Use OptSpeed if you hash or sign the wire.

Bit reference
Bit What it gates
OptDense Inline intern table; required by everything below.
OptQPack Numeric / bool slice codecs (FOR, Delta+FOR, Gorilla, bit-pack).
OptShapeIntern tagMapShape for struct emissions (declare once, reuse by id).
OptPairPred Markov-1 successor predictor (tagStatePair, 0xEA).
OptMTF Move-to-Front rank coding on state-refs (tagStateMTF, 0xE9).
Bundle Composition
OptSpeed 0 — Fast mode, no codecs.
OptBalanced OptDense | OptQPack | OptShapeIntern | OptPairPred | OptMTF.
OptCompression OptBalanced + Gorilla XOR + ALP decimal floats + an order-0 rANS entropy pass (never larger). Trades CPU for wire size; for backup / cold storage.

Options is a uint32 carried by value, so Marshal and AppendMarshal add zero per-call allocations over the pool / output-clone overhead. Encoders are pooled in a single shared pool (encPool) keyed only by buffer; the configuration is applied on each acquire via applyOpts. Markov-0 (tagStateRepeat, 0xE8) is always on inside OptDense because it costs nothing on the wire when the predictor misses.

A note on tuning knobs that do not live on the bit-mask: the intern threshold (SetIntern) and the cycle-depth ceiling (SetMaxDepth) stay on the *Encoder itself. Use the low-level encoder API for those.

Generic API (Go 1.18+ generics)

Marshal(v any, opts) boxes the argument through interface{} and then makes one reflect copy for value-typed inputs. The generic helpers skip both: T is fixed at the call site, reflect.TypeFor[T]() resolves at compile time, and unsafe.Pointer(&v) points directly at the caller's stack.

Generic Equivalent to
MarshalT[T any] Marshal (typed)
AppendMarshalT[T any] AppendMarshal (typed)
UnmarshalT[T any] Unmarshal (typed)

Wire output is byte-identical. The win is on the encode side: one fewer allocation per call (-80 B) and 25–40 % faster on small/medium payloads.

buf, _ := qdf.MarshalT(event, qdf.OptSpeed)
var out Event
_ = qdf.UnmarshalT(buf, &out)
Direct entry points (skip reflection entirely)

MarshalT still goes through descOf to walk the type. Types that already implement MarshalQDF / UnmarshalQDF (either hand-written or generated by cmd/qdfgen) can bypass the descriptor cache and the runtime type assertion via the Direct generics. The type parameter is constrained to the Marshaler / Unmarshaler interface, so the method call is resolved at compile time and the compiler inlines it.

Direct When to use
MarshalDirect[T Marshaler](v T) ([]byte, error) Hot paths where the type is known to have generated codecs.
AppendMarshalDirect[T Marshaler](dst []byte, v T) Same, when the caller owns the destination buffer.
UnmarshalDirect[T Unmarshaler](data []byte, out T) Inverse of MarshalDirect. Falls back to Unmarshal on FlagDense because generated code does not maintain the intern table.

Wire is identical to Marshal / Unmarshal; the path is Fast-mode only (Dense or QPack stay on the reflect path so the intern table is correctly resolved). On the cmd/qdfgen Sample fixture (11-field struct):

Encode ns/op B/op allocs
json 1800 576 8
qdf_reflect 580 480 3
qdf_codegen 530 504 6
qdf_direct 364 160 1

Decode-side performance is bounded by the receiver's UnmarshalQDF. The reflect path is heavily pooled (Decoder pool + per-decoder key intern cache); generated code from qdfgen uses Decoder.InternKey and matches it. Hand-rolled receivers that allocate a new Decoder per call will not.

buf, _ := qdf.MarshalDirect(&event)
var out Event
_ = qdf.UnmarshalDirect(buf, &out)
QPack: math-driven slice codecs

QPack auto-selects, per slice, the codec with the smallest predicted wire form. Round-trip is lossless for every type (NaN/±Inf survive on floats).

Codec Trigger Math
Bitpack []bool 1 bit per element, LSB-first per byte. 8× smaller for free.
Raw-LE numeric slice, large delta range Unsafe-slice cast → single memmove of LE bytes. 10-50× faster.
FOR numeric slice, clustered values Frame-of-Reference: store min and ceil(log₂(max-min+1))-bit deltas.
Delta+FOR monotonic / near-monotonic integers Δᵢ = aᵢ - aᵢ₋₁, zigzag bias, FOR over the deltas.
Patched FOR integer slice with rare outliers (latency spikes) FOR body at a reduced width b + an exception list for the few values that don't fit. ~50% smaller than FOR on spiky columns.
Gorilla float slices (explicit opt-in via low-level API) XOR with previous, run-length leading/meaningful-bit window. (Facebook VLDB 2015.)

Head-to-head on a mixed 256-bool / 512-monotonic-u64 / 512-i64 / 256-f64 payload (Intel i7-9750H):

Format Bytes Encode (ns/op) Decode (ns/op)
json 10,739 48,000 200,000
msgpack 11,808 64,000 80,000
qdf_fast 6,694 6,500 14,000
qdf_qpack 2,132 2,300 2,600
qdf_dense 2,134 2,500 2,500

For sequences where the deltas alone collapse, the gain is much larger — e.g. 1024 consecutive Unix-second timestamps shrink from 8201 bytes (raw) to 16 bytes (Delta+FOR), a 512× reduction.

Streaming
var w bytes.Buffer
enc := qdf.NewStreamEncoder(&w, qdf.Dense)
for _, ev := range events {
    if err := enc.Encode(ev); err != nil {
        return err
    }
}
enc.Close()

dec := qdf.NewStreamDecoder(&w)
defer dec.Close()
for {
    var ev Event
    if err := dec.Decode(&ev); err == io.EOF {
        break
    } else if err != nil {
        return err
    }
    // ...
}

Dense streams preserve the intern table across messages — the second occurrence of "region":"eu-west-1" in the batch is a 2-byte reference rather than a 13-byte string.

Zero-extra-copy encode

AppendMarshal lets callers own the destination buffer:

out, err := qdf.AppendMarshal(out[:0], v)

Pair with a goroutine-local buffer to drop the per-call allocation entirely.

Code generation

For fixed-schema types where every nanosecond matters, generate reflection-free methods:

go install github.com/alex60217101990/qdf/cmd/qdfgen@latest

//go:generate qdfgen -type Event,User .
type Event struct {
    ID     int    `qdf:"id"`
    Source string `qdf:"source"`
    // ...
}

qdfgen writes <package>_qdf.go with concrete MarshalQDF / UnmarshalQDF methods using only the public qdf API. No reflect.* at runtime. See cmd/qdfgen/README.md for flags and supported types.

On the test fixture (11-field struct with nested struct, slice, map, pointer, fixed array, []byte, time.Time) the generated code is 2.65× faster than encoding/json on encode and 8.49× faster on decode.

Benchmarks

Darwin amd64 / Intel i7-9750H @ 2.6 GHz, Go 1.26.0, -benchtime=2s, -race cross-check. Full numbers, realistic / unique-data scenarios, memory tables and reproduction commands are in docs/BENCH.md.

May 2026 perf series (commits ada9fd7, 2ea3b48, 02d6aac, 7090e25, c0517e8, 95d3c21, 001864b) rebuilt the encode + decode hot paths: MRU ring side-cache for MTF rank, flat open-addressed intern table replacing map[string]uint32, packed lruLink, large-payload buffer probe-and-grow, cached decode interning (decState.stringValues), 4-way mruRank unroll, opt-in Encoder.PreIntern API, and the qdfgen codegen path wired into the bench matrix. qdf is now strictly faster than msgpack on encode AND decode for every workload in the matrix (-5 % HotPath, -40 % TelemetryBatch, -96 % MetricSeries on encode; -55…-98 % on decode). See docs/GUIDE.md for the up-to-date Profile_* matrix.

Encode — ns/op
Payload json msgpack qdf_fast qdf_dense vs json vs msgpack
Tiny ({id,name}) 192 286 227 398 0.85× 1.26×
Flat (20 primitive fields) 1182 1262 480 948 2.46× 2.63×
Nested (4 deep) 446 793 331 786 1.35× 2.40×
Deep linked-list (16) 1157 3060 477 814 2.43× 6.42×
Wide × 1000 991 k 991 k 213 k 289 k 4.66× 4.66×
Log batch × 1000 998 k 624 k 186 k 365 k 5.36× 3.35×
Map-heavy (40 entries) 8446 5282 1391 2019 6.07× 3.80×
[]float32 × 512 30 054 18 270 1821 – 16.5× 10.0×
[]float64 × 512 39 819 23 856 2450 – 16.3× 9.74×
UniqueLog (fresh per iter) 2419 2080 1510 – 1.60× 1.38×
Decode — ns/op
Payload json msgpack qdf_fast vs json vs msgpack
Tiny 729 383 170 4.29× 2.25×
Flat 4411 1862 1122 3.93× 1.66×
Nested 2483 1206 425 5.84× 2.84×
Deep16 7363 4170 1972 3.73× 2.11×
Wide × 1000 4.40 M 1.83 M 990 k 4.44× 1.85×
Log batch × 1000 3.29 M 1.32 M 449 k 7.31× 2.93×
Map-heavy (40 entries) 16 621 8096 3108 5.35× 2.61×
[]float32 × 512 72 928 28 521 3960 18.4× 7.20×
UniqueLog (fresh bytes) 3569 1296 509 7.01× 2.55×
UniqueLog (RunParallel) 667 682 487 1.37× 1.40×
Memory — bytes per decode
Payload json msgpack qdf_fast vs json vs msgpack
Tiny 248 77 29 0.12× 0.38×
Nested 664 160 112 0.17× 0.70×
Log batch × 1000 442 536 407 698 251 838 0.57× 0.62×
Map-heavy 4912 3089 2359 0.48× 0.76×
[]float32 × 512 4384 4282 2113 0.48× 0.49×
MapStringAny (rep. keys) 790 – 345 0.44× –
Allocations per decode
Payload json msgpack qdf_fast
Nested 15 6 5
Map-heavy (40 entries) 124 112 32
MapStringAny (rep. keys) 37 – 3
[]float32 × 512 16 8 3
Encoded size
Payload json msgpack qdf_fast qdf_dense dense vs json
Flat 210 134 132 138 0.66×
Deep16 239 139 166 63 0.26×
Wide × 1000 212 901 135 626 128 632 66 702 0.31×
Log batch × 1000 251 902 185 639 185 649 85 440 0.34×

On a 1000-entry Dense stream the same log batch encodes to 85 bytes per entry, against 251 for json and 186 for msgpack — shared intern table across messages plus per-record struct headers collapsed via shape interning (tagMapShape).

Realistic corpus

Numbers from realistic_corpus_test.go builders that mirror real telemetry workloads. Full breakdown plus encode latency in docs/BENCH.md.

TelemetryBatch (1 000 log events, repeating service / region / level)
Format bytes vs json encode ns/op
json 252 497 1.00× —
qdf_fast 186 674 0.74× 272 k
qdf_qpack 186 674 0.74× 261 k
qdf_dense 50 129 0.20× 1.0 M

Dense pays ~4× CPU for 5.0× smaller wire vs JSON — string-intern

  • Markov-0 + MTF + Markov-1 pair + shape interning crush the repeating service / region / level / host fields AND elide per-record struct headers. QPack does not help much here (per-event numeric fields are scalar, not slice-shaped).
MetricSeries (1 024 numeric timestamps + values)
Format bytes vs json
json 30 043 1.00×
qdf_fast 14 442 0.48×
qdf_qpack 8 307 0.28×
qdf_dense 8 315 0.28×

Here QPack pulls its weight: []int64 timestamp column is monotonic and Delta+FOR compresses it to near-zero bytes per element; the []float64 value column rides raw-LE bulk. Dense and QPack converge because string overhead is tiny.

Large payload (~150 MiB, 200 000 records, every supported type)

bench/largepayload_test.go builds a 200 000-record corpus that exercises every qdf-supported field type (scalars, hot/cold strings, nested map, []int32 / []float64, []byte, UUIDs) and measures size + encode/decode latency + working-set memory delta across json, msgpack, qdf_fast, qdf_qpack, qdf_dense.

Sizes (200 000 records):

Format bytes (MiB) vs json
json 142.10 1.00×
msgpack 92.99 0.65×
qdf_fast 91.99 0.65×
qdf_qpack 89.65 0.63×
qdf_dense 71.19 0.50×

Latency + memory (100 000 records):

Format encode (ms) decode (ms) encode heap delta
json 1 070 1 744 199 MiB
msgpack 296 597 64 MiB
qdf_fast 142 300 94 MiB
qdf_qpack 147 231 93 MiB
qdf_dense 169 216 9.7 MiB

Reproduce:

go test -C bench -run TestSizes_LargePayload -count=1 -timeout=10m
go test -C bench -run TestMem_LargePayload   -count=1 -timeout=10m

Both helpers skip under -short; the size test allocates ~600 MiB during the run, the mem test ~400 MiB.

Reproduce
# Default build
cd bench
go test -bench=. -benchmem -benchtime=2s -timeout=10m

# QPack head-to-head: Marshal(_, OptSpeed) vs Marshal(_, OptQPack)
# vs Marshal(_, OptBalanced) vs json / msgpack on the same payload
go test -bench='BenchmarkQPack_' -benchmem

# QPack micro-benchmarks (codec internals, in the root module)
cd ..
go test -bench='BenchmarkQPack' -benchmem

# AVX2 bit-unpack (asm, requires CPUID AVX2 at run time)
go test -tags qdf_simd -bench='BenchmarkBitUnpackFast' -benchmem

# Realistic unique-data scenarios
cd bench
go test -bench='BenchmarkEncode_UniqueLog|BenchmarkDecode_UniqueLog|BenchmarkEncode_MixedTypes|BenchmarkEncode_RandomSize' -benchmem

# Encoded sizes
go test -run TestSizes -v

# Codegen vs reflect on the Sample fixture
cd ../internal/codegen_test
go test -run TestGenerate .
go test -bench=. -benchmem -benchtime=2s
Profile-guided optimisation (downstream)

Go's PGO applies to the main package being built, so a library like qdf cannot ship its own profile. If you build a service that imports qdf, collect a representative profile of your workload and drop it next to your main:

# Run your service / load test under cpuprofile.
go test -bench=. -cpuprofile=cpu.pprof -benchtime=10s ./...
mv cpu.pprof default.pgo            # same dir as your main package
go build .                          # auto-picks up default.pgo

The Go toolchain will then recompile the qdf functions on the hot path with PGO-driven inlining and devirtualisation. Typical gain is 5–15 % across the encode/decode pipeline on top of the numbers above.

Build tags

Opt-in fast paths. None are required for the headline numbers; the defaults already include the specialised slice / map encoders, the QPack codecs, and a 128-bit sliding-window bit-unpacker for FOR / Delta+FOR.

Tag Effect Build prerequisite
qdf_reflect2 Swap reflect.MakeSlice / MakeMapWithSize for modern-go/reflect2 unsafe equivalents. Smaller decode allocations on map / slice heavy workloads. none — pure Go
qdf_simd SIMD fast paths for the QPack integer/bool codecs. amd64 (AVX2): decode bits ∈ {8,16,32} (VPMOVZX) and every width 1–28 (VPBROADCASTQ+VPSRLVQ), encode {8,16,32} (VPSHUFB) and {10,12,14,20} (VPSLLVQ+lane-OR), []bool pack; runtime CPUID gate, non-AVX2 falls back. arm64 (NEON): decode 1–28 plus 32 (encode/bool scalar for now), baseline. ~3-11× over pure-Go on accelerated widths. Output byte-identical to scalar; other arches compile a stub. See docs/USAGE.md. amd64 (AVX2) / arm64 (NEON) 
go build -tags qdf_reflect2 ./...
go build -tags qdf_simd ./...
go build -tags "qdf_simd qdf_reflect2" ./...   # combined

The qdf_simd path only changes the speed of QPack-encoded numeric and bool slices at the accelerated widths; string / map / struct paths, the float codecs, and other bit widths run the pure-Go path unchanged. It is a pure speed switch — the wire format and output are identical.

Correctness

The test suite runs under -race and covers:

  • Primitive round-trip across every wire-tag boundary.
  • Boundary integers from math.MinInt64 to math.MaxUint64.
  • Strings at the fixstr / str8 / str16 / str32 boundaries (0, 1, 31, 32, 255, 256, 65535, 65536, 1 MiB) plus Unicode and invalid UTF-8.
  • Every specialised fast path against the generic reflect path.
  • Truncated input — every prefix from 0 to N-1 decodes to an error, never a panic.
  • Bad magic / bad version — errors, not panics.
  • Cross-mode interop (Fast bytes decoded via Dense decoder and vice versa).
  • Streaming with mixed message types under Dense intern.
  • Concurrency: 32 × 500 marshal+unmarshal goroutines under -race.
  • QPack codecs: each codec (bitpack-bool, raw-LE, FOR, Delta+FOR, Gorilla) has its own round-trip suite covering edge cases — empty slices, single elements, near-MaxUint64, MinInt64, NaN / ±Inf / ±0 / denormals, monotonic / constant / mixed-direction sequences.
  • TestCompleteness_AllModes runs one big payload (every QPack- eligible type, every edge case, nested structs, maps, string interning) through OptSpeed, OptQPack, and OptBalanced, then asserts bit-for-bit IEEE-754 equality for floats and reflect.DeepEqual elsewhere.
  • TestCompleteness_StreamingDense exercises three messages through NewStreamEncoder in Dense mode (carries QPack + shared intern table); TestCompleteness_FuzzRandomStructsQPack generates 50 random struct shapes and round-trips them through all three encoders.
  • AVX2 bit-unpack (under qdf_simd) has parity tests against scalar zero-extend for n in {0,1,2,3,4,5,7,8,9,15,16,17,1000,4096} at bits ∈ {8, 16, 32}, plus a bitPackU64LE → bitUnpackU64LE round- trip.
  • tagStateRepeat (Markov-0 state-ref predictor): tests that the predictor fires on runs of identical interned values, does not fire on alternation, invalidates correctly across an inline-string emission, stays in sync with Decoder.Skip, and round-trips through the public Marshal(v, OptBalanced) / Unmarshal boundary.
  • Property-based round-trip fuzzers (fuzz_property_test.go) drive a deterministic value generator from the fuzz bytes, encode through every Marshal entry point, and assert Unmarshal(Marshal(v)) == v. Targets cover Int64Slice, Uint64Slice, Float64Slice, BoolSlice, MapStringInt, StructTriad, and an AllModesAgree fuzzer that asserts the three encoder dialects decode to the same Go value. Caught one Markov-0 predictor bug in encodeStruct (commit dfc30ae).
  • Golden-file wire pinning (testdata/golden/*.bin, golden_test.go): every representative payload has its OptSpeed / OptQPack / OptBalanced bytes committed to disk. A later wire-format change either matches the bytes byte-for-byte or fails the test. Map-shaped cases skip the byte-pin half (Go map iteration is randomised) but keep the decode-and-compare half. Regenerate intentionally via go test -run TestGolden -update.
  • Deterministic truncation matrix (truncation_test.go, ~2700 sub-tests): for every representative payload, every prefix payload[:i] is fed through Unmarshal into six destination types, every 4th byte is mutated to one of nine "bad" alternatives (00, FF, 80, 55, AA, and every QPack tag), and malformed 5-byte headers are exhaustively rejected. The decoder must return a typed error in every case; panic budget is zero.
  • Unknown-field skip (unknown_field_test.go): encoder writes a 10-field struct, decoder declares only 3; subsequent declared fields still decode correctly. Each of the 10 fields is also pulled out alone via its own single-field destination type.
  • Pathological-shape stress (stress_nesting_test.go): 256-deep linked list, 512-key map, balanced binary tree (depth 12), 1 MiB string + 1 MiB []byte + 100k-element u64/f64 vectors, and a 2000-deep chain under debug.SetMaxStack(64 MiB).
  • Realistic payload corpus (realistic_corpus_test.go): telemetry batches with repeating service/region/level fields, metric time-series, wide column-store rows with recursive Children — round-tripped through every Marshal entry point and cross-checked for three-way agreement.
  • Allocation-budget tests (alloc_budget_test.go): each hot entry point has an upper-bound allocation budget enforced via testing.AllocsPerRun. A regression in alloc count fails CI rather than silently bloating downstream callers.
  • Tag and named-type matrix (tag_matrix_test.go): every primitive type is exercised through a named alias (type MyInt int64), the tag fallback chain qdf → json → field name is verified, the qdf:"-" skip directive is checked via a map[string]any decode, and embedded-struct (non-)flattening is pinned.
  • Differential testing (bench/diff_test.go): qdf round-trips agree semantically with encoding/json and github.com/vmihailenco/msgpack/v5 on a shared payload, including NaN / ±Inf / ±0 preservation against msgpack.
  • Pointer-cycle detection via depth counter: the encoder increments Encoder.depth on every pointer dereference and returns ErrCycleDetected once the count exceeds DefaultMaxDepth (10000). Cheaper than a per-pointer set (no allocation per call) and catches both genuine *T → *T cycles and pathologically deep payloads. Override the cap via Encoder.SetMaxDepth.
  • OOM protection on every length-prefixed read path: a hostile varuint encoding a value > 2^62 cannot drive the decoder cursor negative because the byte-count check happens in uint64 against (len(d.buf) - d.i) * 8 / bitsPer before any signed cast. Applies to tagPackBool/Raw/For/DeltaFor/Gorilla in both the slice-read and Skip paths.
  • Stream Close idempotency: both StreamEncoder.Close and StreamDecoder.Close are safe to call multiple times.
  • Fuzz: FuzzDecoder_NeverPanics, FuzzRoundTrip_StringSlice, FuzzQPackBool, FuzzQPackRawUint64, FuzzRoundTrip_Int64Slice, FuzzRoundTrip_Uint64Slice, FuzzRoundTrip_Float64Slice, FuzzRoundTrip_BoolSlice, FuzzRoundTrip_MapStringInt, FuzzRoundTrip_StructTriad, FuzzRoundTrip_AllModesAgree — persistent corpus under testdata/fuzz/. 10 M+ executions clean across the suite after the Skip-overflow and encodeStruct-predictor fixes; both repros are saved under testdata/fuzz/.

Length prefixes are validated against the remaining buffer (Decoder.CheckLength) before any make, so a hostile payload claiming a multi-billion-element map cannot OOM the process. The QPack-tag Skip paths perform the same overflow-safe check on uint64 element counts before any signed cast, so a 10-byte varuint encoding a value > 2^62 cannot drive the decoder cursor negative.

go test -race -count=1 ./...

# Property-based round-trip fuzz (each fuzzer asserts
# Unmarshal(Marshal(v)) == v on randomly generated values)
go test -run=^$ -fuzz=FuzzRoundTrip_StructTriad     -fuzztime=60s
go test -run=^$ -fuzz=FuzzRoundTrip_AllModesAgree   -fuzztime=60s
go test -run=^$ -fuzz=FuzzRoundTrip_Int64Slice      -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_Uint64Slice     -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_Float64Slice    -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_BoolSlice       -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_MapStringInt    -fuzztime=30s

# Decoder safety fuzz (asserts never-panic on hostile input)
go test -run=^$ -fuzz=FuzzDecoder_NeverPanics       -fuzztime=30s
go test -run=^$ -fuzz=FuzzRoundTrip_StringSlice     -fuzztime=30s
go test -run=^$ -fuzz=FuzzQPackBool                 -fuzztime=30s
go test -run=^$ -fuzz=FuzzQPackRawUint64            -fuzztime=30s

# Differential vs msgpack and encoding/json
go test -C bench -run=TestDiff -count=1

# Regenerate wire-format golden fixtures (intentional after a wire
# bump only)
go test -run=TestGolden -update

# Build-tag combinations
go test -tags qdf_reflect2          -race ./...
go test -tags qdf_simd              -race ./...
go test -tags "qdf_simd qdf_reflect2" -race ./...

Status

Alpha. The wire format is stable for the 0x01 version byte; future versions will bump it. The public API may change before the first tagged release — pin a commit if you depend on it.

License

MIT. See LICENSE.

Documentation

Overview

Package qdf is a compact, streaming-friendly binary serialization format.

Quick start

One encode entry point: Marshal(v, opts). The Options bit-mask picks which codecs run for that call. Convenience bundles cover the common tradeoffs: 

OptSpeed       — Fast mode, no codecs (matches encoding/json shape)
OptBalanced    — Dense + QPack + shape interning + Markov-1 + MTF
OptCompression — OptBalanced + heavier float codecs: Gorilla XOR for
                 smooth time-series and ALP for quantized/decimal
                 float64 (large wire reduction at higher encode CPU)

A single decoder handles every variant; the wire header self- describes the dialect. 

b, err := qdf.Marshal(v, qdf.OptSpeed)        // hot path
b, err := qdf.Marshal(v, qdf.OptBalanced)     // telemetry / logs
b, err := qdf.Marshal(v, qdf.OptCompression)  // backup / archive

err := qdf.Unmarshal(b, &v)

enc := qdf.NewStreamEncoder(w, qdf.OptBalanced)
dec := qdf.NewStreamDecoder(r)

Marshal returns a freshly-allocated slice owned by the caller. AppendMarshal is the zero-extra-copy variant.

Data shapes and codecs

Under OptBalanced the encoder picks a codec per slice from the data itself: integer slices use Frame-of-Reference, delta, run-length or dictionary coding; bool slices bit-pack; repeated strings are interned and back-referenced. A slice of homogeneous flat structs is transposed and compressed column-by-column automatically — numeric columns get the integer codecs, repeated string columns collapse — with no flag, and a per-array probe falls back to the plain row encoding when columnar would not help. The float codecs that trade encode CPU for size live behind OptCompression: Gorilla XOR on smooth series, ALP on quantized/decimal float64 grids (quantized telemetry, prices, latencies).

Concurrency

Marshal, Unmarshal and AppendMarshal are safe for concurrent use: each call leases its own encoder/decoder from a pool. A single Encoder, Decoder or StreamEncoder value is not safe to share across goroutines — use one per goroutine. A Dense document is a sequential stream, so one document cannot be encoded or decoded in parallel; to parallelize a large dataset, split it into independent shards and Marshal each separately.

See docs/USAGE.md in the repository for a fuller guide.

Public API surface

The package contract is split across four layers; everything else under internal/ is implementation detail and may change between releases without notice.

Top-level entry points (file: qdf.go): 

func Marshal(v any, opts Options) ([]byte, error)
func AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)
func Unmarshal(data []byte, out any) error
type Options uint32   // bit-mask of OptDense, OptQPack, OptMTF, …

Typed convenience wrappers — generic, zero-extra-reflection (file: qdf_generic.go): 

func MarshalT[T any](v T, opts Options) ([]byte, error)
func AppendMarshalT[T any](dst []byte, v T, opts Options) ([]byte, error)
func UnmarshalT[T any](data []byte) (T, error)

Low-level encoder / decoder for callers driving the wire directly or interoperating with the qdfgen code generator (files: encoder.go, decoder.go, stream.go): 

type Encoder struct{ … }
    NewEncoder(mode Mode) *Encoder
    NewEncoderWith(opts Options) *Encoder
    NewEncoderOnBuf(buf []byte, mode Mode) *Encoder
    (e *Encoder) WriteString / WriteBytes / WriteInt / WriteUint /
                WriteFloat32 / WriteFloat64 / WriteBool / WriteNil /
                WriteArrayHeader / WriteMapHeader /
                WriteTimestampNano / WriteStringInline
    (e *Encoder) AppendBytes / EnsureHeader / Bytes / Take /
                Reset / SetBuffer / AdoptBuffer / SetIntern /
                SetMaxDepth / SetQPack / QPack / ApplyOpts /
                EncodeValue / PreIntern
type Decoder struct{ … }
    NewDecoder() *Decoder
    NewDecoderOnBuf(buf []byte) *Decoder
    (d *Decoder) ReadString / ReadStringBytes / ReadBytes /
                ReadBool / ReadInt / ReadUint / ReadFloat32 /
                ReadFloat64 / ReadNil / ReadArrayHeader /
                ReadMapHeader / ReadTimestampNano / Skip /
                PeekTag / IsNil / Pos / Remaining / RemainingBytes /
                Advance / SetInput / SetNoCopy / MarkHeaderRead /
                CheckLength / InternKey
type StreamEncoder, type StreamDecoder         // io.Writer / io.Reader

User-side hook points (file: marshaler.go): 

type Marshaler interface   { MarshalQDF(dst []byte) ([]byte, error) }
type Unmarshaler interface { UnmarshalQDF(src []byte) (int, error) }

The qdfgen code generator (cmd/qdfgen) emits MarshalQDF / UnmarshalQDF methods for user struct types; the codegen path uses the same Encoder / Decoder primitives listed above and requires no reflection at runtime.

Example

Example demonstrates the simplest possible round-trip: Marshal a value into a freshly-allocated byte slice and Unmarshal it back into a typed receiver. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		ID     int    `qdf:"id"`
		Source string `qdf:"source"`
	}

	in := Event{ID: 42, Source: "ingest"}
	buf, err := qdf.Marshal(in, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var out Event
	if err := qdf.Unmarshal(buf, &out); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", out)
}

Output:
{ID:42 Source:ingest}
Example (RoundTripSlice)

Example_roundTripSlice demonstrates encoding a slice of structs. QDF's reflect path resolves the element type once per typeDesc; every element after the first emits its tag stream straight from the cached descriptor. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	rows := []Row{
		{"auth", 200},
		{"billing", 503},
		{"auth", 200},
	}

	buf, _ := qdf.Marshal(rows, qdf.OptBalanced)

	var got []Row
	_ = qdf.Unmarshal(buf, &got)
	for _, r := range got {
		fmt.Printf("%s=%d\n", r.Service, r.Status)
	}
}

Output:
auth=200
billing=503
auth=200

Index

Examples

Constants

View Source 
const (
	Magic0   byte = 'Q'
	Magic1   byte = 'D'
	Magic2   byte = 'F'
	Version1 byte = 0x01
)
View Source 
const (
	FlagDense byte = 1 << 0
	// FlagQPack signals that the encoder may emit QPack codec tags
	// (0xE3..0xEF). A decoder that does not recognize the tags will fail
	// with ErrBadTag on the first packed slice; the flag is an early hint
	// so callers can refuse the buffer up front.
	FlagQPack byte = 1 << 1

	// FlagRANS signals that the body (everything after the 5-byte header) is
	// rANS-compressed: varuint(origLen) + 256-entry frequency table +
	// rANS stream. The decoder reverses it before reading tags. Set only
	// when the rANS form is strictly smaller than the plain body.
	FlagRANS byte = 1 << 2
)

Flag bits in the 5th header byte.

View Source 
const DefaultMaxDepth = 10_000

DefaultMaxDepth caps reflect-path pointer/struct recursion. Set large enough for any legitimate payload (10 000) while still rejecting genuine cycles before the goroutine stack overflows.

View Source 
const TagNil = tagNil

TagNil exposes the nil-value tag for comparison with PeekTag.

Variables

View Source 
var (
	ErrShortBuffer    = errors.New("qdf: short buffer")
	ErrBadMagic       = errors.New("qdf: bad magic / not a qdf stream")
	ErrBadVersion     = errors.New("qdf: unsupported wire version")
	ErrBadTag         = errors.New("qdf: unknown tag")
	ErrTypeMismatch   = errors.New("qdf: type mismatch on decode")
	ErrInvalidLength  = errors.New("qdf: invalid length prefix")
	ErrUnknownStateID = errors.New("qdf: unknown state-table id")
	ErrUnsupported    = errors.New("qdf: unsupported type")
	ErrCycleDetected  = errors.New("qdf: pointer cycle detected (max depth exceeded)")
)

Functions

func AppendMarshal 

func AppendMarshal(dst []byte, v any, opts Options) ([]byte, error)

AppendMarshal encodes v and appends the result to dst. Reuse the returned slice as dst on the next call to avoid per-message allocations.

Example

ExampleAppendMarshal shows the zero-extra-copy form. The caller keeps a reusable byte buffer and asks Marshal to append into it, returning the extended slice. Combined with a sync.Pool of buffers this drives encode allocations to zero on the hot path. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Tick struct {
		Ts    int64   `qdf:"ts"`
		Price float64 `qdf:"price"`
	}

	buf := make([]byte, 0, 64) // caller-owned, reusable
	t := Tick{Ts: 1_700_000_000, Price: 99.5}

	var err error
	buf, err = qdf.AppendMarshal(buf, t, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var out Tick
	if err := qdf.Unmarshal(buf, &out); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", out)
}

Output:
{Ts:1700000000 Price:99.5}

func AppendMarshalDirect 

func AppendMarshalDirect[T Marshaler](dst []byte, v T) ([]byte, error)

AppendMarshalDirect appends the serialisation of v to dst.

func AppendMarshalT 

func AppendMarshalT[T any](dst []byte, v T, opts Options) ([]byte, error)

AppendMarshalT is the generic equivalent of AppendMarshal.

func Marshal 

func Marshal(v any, opts Options) ([]byte, error)

Marshal encodes v with the given option bit-mask. Combine bits with | to opt into individual codecs, or use one of the OptSpeed / OptBalanced / OptCompression bundles. 

b, _ := qdf.Marshal(event,    qdf.OptSpeed)         // hot path
b, _ := qdf.Marshal(batch,    qdf.OptBalanced)      // telemetry
b, _ := qdf.Marshal(snapshot, qdf.OptCompression)   // backup
b, _ := qdf.Marshal(payload,                        // tuned
    qdf.OptDense|qdf.OptQPack|qdf.OptShapeIntern)

func MarshalDirect 

func MarshalDirect[T Marshaler](v T) ([]byte, error)

MarshalDirect serialises a value through its MarshalQDF method. Skips the public Marshal entry point's any-boxing, reflect.New, and descriptor lookup. Roughly 2-4× faster than Marshal for generated types on small payloads, with one fewer allocation per call.

func MarshalT 

func MarshalT[T any](v T, opts Options) ([]byte, error)

MarshalT is the generic equivalent of Marshal. T is fixed at the call site; opts copies by value, so the call adds zero heap allocations over MarshalT itself.

Example

ExampleMarshalT shows the generic typed wrapper. T is fixed at the call site, so the compiler skips the reflect.TypeOf step that Marshal(v any, opts) pays inside; encode/decode of value types stays on the stack. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Point struct {
		X int32 `qdf:"x"`
		Y int32 `qdf:"y"`
	}

	buf, err := qdf.MarshalT(Point{X: 3, Y: 4}, qdf.OptSpeed)
	if err != nil {
		panic(err)
	}

	var p Point
	if err := qdf.UnmarshalT(buf, &p); err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", p)
}

Output:
{X:3 Y:4}

func Unmarshal 

func Unmarshal(data []byte, out any) error

Unmarshal decodes data into out, which must be a non-nil pointer. The wire dialect is detected from the header — the same Unmarshal reads any output produced by Marshal regardless of the encode-side option bits.

func UnmarshalDirect 

func UnmarshalDirect[T Unmarshaler](data []byte, out T) error

UnmarshalDirect dispatches to out.UnmarshalQDF after validating the 5-byte header. It is the inverse of MarshalDirect; the same wire constraints apply (Fast-mode only).

If the input carries the FlagDense bit, UnmarshalDirect falls back to the full Unmarshal path because generated code cannot resolve state-ref tags without the decoder's intern table.

Decode-side performance is bounded by the user's UnmarshalQDF implementation: the reflect path is heavily pooled (Decoder pool + per-decoder key intern cache) and tends to win against naive hand-rolled receivers. Generated code from cmd/qdfgen uses Decoder.InternKey for map / struct keys and matches or beats the reflect path; ad-hoc UnmarshalQDF methods that call NewDecoderOnBuf + ReadString in a tight loop will not.

func UnmarshalT 

func UnmarshalT[T any](data []byte, out *T) error

UnmarshalT is the generic equivalent of Unmarshal. The destination pointer must not be nil.

Types

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads QDF wire data from a single input buffer. Call SetInput to bind a buffer and the typed Read* methods to walk it. Behaviour is undefined if the input is mutated while the decoder holds it.

func NewDecoder 

func NewDecoder() *Decoder

NewDecoder constructs a Decoder. Bind input via SetInput.

func NewDecoderOnBuf 

func NewDecoderOnBuf(buf []byte) *Decoder

NewDecoderOnBuf binds a Decoder to buf. Equivalent to NewDecoder followed by SetInput(buf), without the SetInput branch checks.

func (*Decoder) Advance 

func (d *Decoder) Advance(n int)

Advance moves the read cursor forward by n bytes.

func (*Decoder) CheckLength 

func (d *Decoder) CheckLength(n int, perElem int) error

CheckLength returns ErrShortBuffer when n claimed elements cannot fit in the remaining input. Use before allocating per-element storage.

func (*Decoder) InternKey 

func (d *Decoder) InternKey(b []byte) string

InternKey returns a string equal to b, sharing storage with prior identical keys when the cache has them.

func (*Decoder) IsNil 

func (d *Decoder) IsNil() (bool, error)

IsNil reports whether the next value is the nil tag, consuming it on true. Returns (false, nil) for any other tag.

Example

ExampleDecoder_IsNil consumes a nil tag if present, returning false (without advancing) for any other tag. Useful when a field is optional on the wire and the caller wants to keep the read cursor stable on a non-nil branch. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	buf, _ := qdf.Marshal([]any{nil, "hello"}, qdf.OptSpeed)
	dec := qdf.NewDecoderOnBuf(buf)

	n, _ := dec.ReadArrayHeader()
	for range n {
		isNil, _ := dec.IsNil()
		if isNil {
			fmt.Println("<nil>")
			continue
		}
		s, _ := dec.ReadString()
		fmt.Println(s)
	}
}

Output:
<nil>
hello

func (*Decoder) MarkHeaderRead 

func (d *Decoder) MarkHeaderRead()

MarkHeaderRead tells the decoder the magic+version header is already consumed (e.g. the buffer is a tail slice from a parent decoder). The next read will skip the header check.

func (*Decoder) PeekTag 

func (d *Decoder) PeekTag() (byte, error)

PeekTag returns the next tag byte without advancing the cursor.

Example

ExampleDecoder_PeekTag inspects the next wire tag without consuming it. Handy when implementing a dispatch loop over a schemaless payload, or when an Unmarshaler needs to branch on the upcoming type. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	buf, _ := qdf.Marshal(int64(-42), qdf.OptSpeed)
	dec := qdf.NewDecoderOnBuf(buf)
	tag, _ := dec.PeekTag()
	if tag == qdf.TagNil {
		fmt.Println("nil value")
		return
	}
	v, _ := dec.ReadInt()
	fmt.Printf("int=%d (tag=0x%02x)\n", v, tag)
}

Output:
int=-42 (tag=0xc7)

func (*Decoder) Pos 

func (d *Decoder) Pos() int

Pos returns the current read offset.

func (*Decoder) ReadArrayHeader 

func (d *Decoder) ReadArrayHeader() (int, error)

ReadArrayHeader returns the element count.

func (*Decoder) ReadBool 

func (d *Decoder) ReadBool() (bool, error)

func (*Decoder) ReadBytes 

func (d *Decoder) ReadBytes() ([]byte, error)

ReadBytes returns a []byte value. With noCopy = true the result aliases the input.

func (*Decoder) ReadFloat32 

func (d *Decoder) ReadFloat32() (float32, error)

func (*Decoder) ReadFloat64 

func (d *Decoder) ReadFloat64() (float64, error)

func (*Decoder) ReadInt 

func (d *Decoder) ReadInt() (int64, error)

func (*Decoder) ReadMapHeader 

func (d *Decoder) ReadMapHeader() (int, error)

ReadMapHeader returns the key/value pair count.

func (*Decoder) ReadNil 

func (d *Decoder) ReadNil() error

func (*Decoder) ReadString 

func (d *Decoder) ReadString() (string, error)

ReadString returns the next string. If the decoder is in noCopy mode the returned string aliases the input buffer; otherwise a copy is made.

When the tag at the cursor resolves through the intern table — tagInternStr, tagStateRef, tagStateMTF, tagStatePair, tagStateRepeat — readStringBytes leaves d.state.lastID set to the entry it just touched. We return the pre-materialised d.state.stringValues[id] from that path instead of allocating a fresh `string(b)` copy on every state-ref hit. Inline reads (fixstr, str8/16/32) set lastID = lruInvalidID, fall through, and pay the copy as before.

func (*Decoder) ReadStringBytes 

func (d *Decoder) ReadStringBytes() ([]byte, error)

ReadStringBytes returns the next string or []byte value without copying. The returned slice aliases either the input buffer or the decoder's state-table storage. Callers that retain it beyond the input's lifetime must copy.

func (*Decoder) ReadTimestampNano 

func (d *Decoder) ReadTimestampNano() (int64, error)

func (*Decoder) ReadUint 

func (d *Decoder) ReadUint() (uint64, error)

ReadUint reads a value that was encoded with WriteUint or WriteInt with a non-negative argument. Returns an error if the next value is negative.

func (*Decoder) Remaining 

func (d *Decoder) Remaining() int

Remaining returns the number of unread bytes.

func (*Decoder) RemainingBytes 

func (d *Decoder) RemainingBytes() []byte

RemainingBytes returns the unread portion of the input buffer. The result aliases the buffer and is invalidated by the next SetInput.

func (*Decoder) SetInput 

func (d *Decoder) SetInput(buf []byte)

SetInput rebinds the decoder to buf, dropping any prior state table.

func (*Decoder) SetNoCopy 

func (d *Decoder) SetNoCopy(v bool)

SetNoCopy switches the decoder into aliasing mode: string and []byte reads return slices that share storage with the input buffer. Faster, but the caller must not retain the result past the lifetime of the input.

Example

ExampleDecoder_SetNoCopy returns string / []byte values that alias the input buffer instead of allocating per-value copies. Use this for read-heavy hot paths where the input buffer outlives every value the decoder hands back; do not retain the values past the buffer's lifetime. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	in := []string{"alpha", "beta", "gamma"}
	buf, _ := qdf.Marshal(in, qdf.OptSpeed)

	dec := qdf.NewDecoderOnBuf(buf)
	dec.SetNoCopy(true)

	n, _ := dec.ReadArrayHeader()
	for range n {
		s, _ := dec.ReadString() // alias into buf
		fmt.Println(s)
	}
}

Output:
alpha
beta
gamma

func (*Decoder) Skip 

func (d *Decoder) Skip() error

Skip advances past one value without materializing it.

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes a single QDF value into a growing internal buffer. Reset drops the buffer contents and (in Dense mode) the intern table so the encoder can be reused.

Example

ExampleEncoder drives the encoder by hand instead of via the top-level Marshal pool. Use this when you need to:

  • keep a reusable buffer alive across many encodes;
  • call PreIntern with a known hot string pool;
  • mix WriteX primitives with reflect-driven EncodeValue.
package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	enc := qdf.NewEncoderWith(qdf.OptSpeed)

	// Primitives + struct fields can be mixed by hand.
	enc.WriteMapHeader(2)
	enc.WriteString("name")
	enc.WriteString("alice")
	enc.WriteString("age")
	enc.WriteInt(33)

	dec := qdf.NewDecoderOnBuf(enc.Bytes())
	n, _ := dec.ReadMapHeader()
	for range n {
		k, _ := dec.ReadString()
		switch k {
		case "name":
			v, _ := dec.ReadString()
			fmt.Printf("%s=%s\n", k, v)
		case "age":
			v, _ := dec.ReadInt()
			fmt.Printf("%s=%d\n", k, v)
		}
	}
}

Output:
name=alice
age=33

func NewEncoder 

func NewEncoder(mode Mode) *Encoder

NewEncoder returns an Encoder. The internal buffer is allocated lazily on first write.

func NewEncoderOnBuf 

func NewEncoderOnBuf(buf []byte, mode Mode) *Encoder

NewEncoderOnBuf returns an Encoder that appends to buf at its current length. The buffer is NOT truncated; pass an empty slice for a fresh encoding. Call SetBuffer afterwards to truncate.

func NewEncoderWith 

func NewEncoderWith(opts Options) *Encoder

NewEncoderWith returns an Encoder configured by the option bit-mask directly. Dense state machinery is allocated only when OptDense is set. Defaults for intern threshold / depth match NewEncoder.

func (*Encoder) AdoptBuffer 

func (e *Encoder) AdoptBuffer(b []byte)

AdoptBuffer installs b as the working buffer, preserving its current length. Used to continue writing into a buffer that already carries data — for example, after a nested type returned its extended slice. headerOut is left unchanged.

func (*Encoder) AppendBytes 

func (e *Encoder) AppendBytes(p []byte)

AppendBytes appends raw, already-valid wire bytes. Bypasses tag dispatch; used by generated code to emit pre-encoded field-name prefixes.

func (*Encoder) ApplyOpts 

func (e *Encoder) ApplyOpts(opts Options)

ApplyOpts reconfigures the encoder via the options bit-mask. Equivalent to recreating the encoder with NewEncoderWith but preserves the existing pool buffer and (Dense-mode) intern state. The pool-backed Marshal* entry points call this on every acquire; callers driving an Encoder directly use it to switch between OptSpeed and OptBalanced without re-allocating.

func (*Encoder) Bytes 

func (e *Encoder) Bytes() []byte

Bytes returns the encoded payload. It aliases the encoder's buffer and is only valid until the next write or Reset.

func (*Encoder) EncodeValue 

func (e *Encoder) EncodeValue(v any) error

EncodeValue runs the reflect-driven Marshal pipeline on v against this Encoder. Convenience for callers driving an Encoder directly (e.g. with PreIntern) — equivalent to what the pool-backed Marshal does internally.

Example

ExampleEncoder_EncodeValue drives the reflect-based encode path against any Go value directly through the encoder, bypassing the pool used by qdf.Marshal. Combine with PreIntern, custom SetBuffer, or SetIntern tuning for a long-lived encoder that shapes its state for a specific workload. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Config struct {
		Name  string `qdf:"name"`
		Limit int    `qdf:"limit"`
	}

	enc := qdf.NewEncoderWith(qdf.OptBalanced)
	if err := enc.EncodeValue(Config{Name: "service.cpu", Limit: 1000}); err != nil {
		panic(err)
	}

	var out Config
	_ = qdf.Unmarshal(enc.Bytes(), &out)
	fmt.Printf("%+v\n", out)
}

Output:
{Name:service.cpu Limit:1000}

func (*Encoder) EnsureHeader 

func (e *Encoder) EnsureHeader()

EnsureHeader forces a header write if one has not been emitted yet.

func (*Encoder) MarkHeaderWritten 

func (e *Encoder) MarkHeaderWritten()

MarkHeaderWritten tells the encoder the QDF header is already present in its buffer (e.g. left there by a parent encoder). The next write will skip the header emission.

func (*Encoder) PreIntern 

func (e *Encoder) PreIntern(strs ...string)

PreIntern registers the given strings against the encoder's intern table up front. Subsequent WriteString / WriteBytes calls that pass the SAME backing string header (i.e. the same underlying byte pointer and length) skip the hash + slot probe and emit a state-ref directly against the cached intern id.

Intended for power users who know their hot string pool ahead of time — service names, region codes, enum-like values drawn from a fixed slice. Real telemetry payloads draw 90 %+ of their dense intern hits from such pools. Skipping the hash/probe on those calls is the main per-emit cost left on the encode hot path.

Requires OptDense to be applied first (otherwise the call is a no-op). The registered identities are dropped on Reset so a pooled encoder does not carry caller-supplied pointers across recycles.

Safety: the caller must keep the backing memory of every PreIntern'd string alive for the lifetime of the next encode call. For literals embedded in a slice / global / struct field this is automatic; for short-lived stack strings the caller is responsible.

Example

ExampleEncoder_PreIntern registers a known string pool against the encoder so subsequent WriteString calls with the same backing pointer skip the intern table's hash + slot probe and emit a state-ref directly. Use this when the caller knows the hot vocabulary up front (service names, region codes, enum- like fields).

The caller must keep the registered strings' backing memory alive for the lifetime of the next encode call. String literals embedded in a slice, global, or struct field stay alive automatically; short-lived stack strings do not. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Region  string `qdf:"region"`
		Status  int    `qdf:"status"`
	}

	// Stable hot pool — backing slices live for the program's
	// lifetime, so the registered pointers stay valid across
	// many encode calls on the same encoder.
	services := []string{"billing", "auth", "ingest"}
	regions := []string{"eu-west-1", "us-east-1"}

	enc := qdf.NewEncoderWith(qdf.OptBalanced)
	enc.PreIntern(services...)
	enc.PreIntern(regions...)

	rows := []Row{
		{services[0], regions[0], 200},
		{services[1], regions[1], 500},
		{services[2], regions[0], 200},
	}
	if err := enc.EncodeValue(rows); err != nil {
		panic(err)
	}

	var out []Row
	_ = qdf.Unmarshal(enc.Bytes(), &out)
	for _, r := range out {
		fmt.Printf("%s/%s=%d\n", r.Service, r.Region, r.Status)
	}
}

Output:
billing/eu-west-1=200
auth/us-east-1=500
ingest/eu-west-1=200

func (*Encoder) QPack 

func (e *Encoder) QPack() bool

QPack reports whether QPack codec emission is enabled.

func (*Encoder) Reset 

func (e *Encoder) Reset()

Reset truncates the buffer and resets the intern table. Capacities are preserved. opts / mode / qpack are also reset to OptSpeed so a pooled encoder does not leak its previous configuration into the next caller — apply the desired options explicitly via applyOpts / SetQPack after Reset.

func (*Encoder) SetBuffer 

func (e *Encoder) SetBuffer(b []byte)

SetBuffer installs a caller-owned buffer (truncated to length 0).

func (*Encoder) SetIntern 

func (e *Encoder) SetIntern(min int, cap int)

SetIntern overrides the Dense-mode tuning knobs. Zero values keep the current setting.

func (*Encoder) SetMaxDepth 

func (e *Encoder) SetMaxDepth(d int)

SetMaxDepth caps reflect-path pointer/struct recursion. The default (DefaultMaxDepth = 10000) is sufficient for any normal payload and rejects pointer cycles before they stack-overflow the goroutine. Set to 0 to disable the check entirely — only safe when the caller can prove the input graph is acyclic.

func (*Encoder) SetQPack 

func (e *Encoder) SetQPack(v bool)

SetQPack toggles QPack codec emission. When true, slice fast paths produce packed/encoded forms (bitpacked bools, FOR-packed integers, Gorilla-encoded floats, raw-LE bulk) instead of per-element tag streams. Setting must happen before the first write of the value (the header is emitted lazily and carries the FlagQPack hint when this is on).

func (*Encoder) Take 

func (e *Encoder) Take() []byte

Take returns the encoded payload and detaches it from the encoder. The caller takes ownership.

func (*Encoder) WriteArrayHeader 

func (e *Encoder) WriteArrayHeader(n int)

WriteArrayHeader writes the header for an array of n elements. The caller must follow with exactly n element writes.

func (*Encoder) WriteBool 

func (e *Encoder) WriteBool(v bool)

func (*Encoder) WriteBytes 

func (e *Encoder) WriteBytes(b []byte)

WriteBytes writes a []byte. In Dense mode, eligible payloads are intern-encoded. The intern table is keyed on the byte sequence, so a string and a []byte with identical content share an ID.

func (*Encoder) WriteFloat32 

func (e *Encoder) WriteFloat32(v float32)

func (*Encoder) WriteFloat64 

func (e *Encoder) WriteFloat64(v float64)

func (*Encoder) WriteInt 

func (e *Encoder) WriteInt(v int64)

func (*Encoder) WriteMapHeader 

func (e *Encoder) WriteMapHeader(n int)

WriteMapHeader writes the header for a map of n entries. The caller must follow with exactly n key/value pairs.

func (*Encoder) WriteNil 

func (e *Encoder) WriteNil()

func (*Encoder) WriteString 

func (e *Encoder) WriteString(s string)

WriteString writes s. In Dense mode (OptDense set), eligible strings are intern-encoded.

func (*Encoder) WriteStringInline 

func (e *Encoder) WriteStringInline(s string)

WriteStringInline forces an in-line encoding even when Dense intern would be eligible. Use for fields known to be unique per message.

func (*Encoder) WriteTimestampNano 

func (e *Encoder) WriteTimestampNano(ns int64)

WriteTimestampNano writes a Unix-nanoseconds timestamp.

func (*Encoder) WriteUint 

func (e *Encoder) WriteUint(v uint64)

type Marshaler 

type Marshaler interface {
	MarshalQDF(dst []byte) ([]byte, error)
}

Marshaler is implemented by types that know how to serialize themselves into the QDF wire format. Implementations should append exactly one value to dst and return the extended slice.

Example

ExampleMarshaler shows a custom MarshalQDF / UnmarshalQDF pair. Use the interface when you want a compact wire form for a type — e.g. packing a small struct into a single primitive, versioning a payload, or pre-validating bytes before they hit the wire. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

// Color is a 24-bit RGB value stored as a single uint32 on the
// wire. It implements Marshaler / Unmarshaler so qdf encodes it
// as a primitive integer instead of recursing into reflect.
type Color struct{ R, G, B uint8 }

// MarshalQDF packs the three bytes into a uint32 and writes it
// using the encoder primitives from a temporary qdf.Encoder
// over the caller's buffer. This shape — temporary encoder over
// the caller's dst slice — is what qdfgen produces by default.
func (c Color) MarshalQDF(dst []byte) ([]byte, error) {
	enc := qdf.NewEncoderOnBuf(dst, qdf.Fast)
	enc.MarkHeaderWritten() // dst already contains the parent header
	enc.WriteUint(uint64(c.R)<<16 | uint64(c.G)<<8 | uint64(c.B))
	return enc.Bytes(), nil
}

// UnmarshalQDF reverses the pack and returns the number of bytes
// consumed from src.
func (c *Color) UnmarshalQDF(src []byte) (int, error) {
	dec := qdf.NewDecoderOnBuf(src)
	dec.MarkHeaderRead()
	v, err := dec.ReadUint()
	if err != nil {
		return 0, err
	}
	c.R = uint8(v >> 16)
	c.G = uint8(v >> 8)
	c.B = uint8(v)
	return dec.Pos(), nil
}

// ExampleMarshaler shows a custom MarshalQDF / UnmarshalQDF pair.
// Use the interface when you want a compact wire form for a
// type — e.g. packing a small struct into a single primitive,
// versioning a payload, or pre-validating bytes before they hit
// the wire.
func main() {
	in := Color{R: 0x12, G: 0x34, B: 0x56}
	buf, _ := qdf.MarshalDirect(&in)

	var out Color
	_ = qdf.UnmarshalDirect(buf, &out)
	fmt.Printf("R=%02x G=%02x B=%02x\n", out.R, out.G, out.B)
}

Output:
R=12 G=34 B=56

type Mode 

type Mode uint8

Mode selects the wire dialect. 

const (
	// Fast writes strings and []byte in line. No intern bookkeeping.
	Fast Mode = 0

	// Dense writes repeated strings/bytes once and references them by ID
	// thereafter. Smaller output on repetitive payloads; slightly slower
	// per call.
	Dense Mode = 1
)

type Options 

type Options uint32

Options is a bit-mask of per-call encoder feature toggles. A zero value (OptSpeed) gives the fastest path: Fast mode, no codecs, no predictors. Or-combine bits with | to opt in to individual codecs. Pre-built bundles cover the common "max speed", "balanced", and "max compression" use cases without having to remember the layout.

Options carry by value (uint32) so Marshal / AppendMarshal add zero allocations over the pool / output-clone overhead. The encoder checks each bit with a single AND on the hot path.

Example

ExampleOptions shows how the Options bit-mask picks which codec layers run for a Marshal call. The same input encodes to a progressively smaller wire as more codecs opt in, with a small CPU penalty per added codec. The decoder reads any dialect off the wire header so changing Options is a producer-side decision. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Region  string `qdf:"region"`
		Status  int    `qdf:"status"`
	}
	// Repetitive batch — every row shares the same service and region.
	in := make([]Row, 50)
	for i := range in {
		in[i] = Row{Service: "billing", Region: "eu-west-1", Status: 200}
	}

	for _, p := range []struct {
		name string
		opts qdf.Options
	}{
		{"OptSpeed", qdf.OptSpeed},
		{"OptBalanced", qdf.OptBalanced},
	} {
		buf, _ := qdf.Marshal(in, p.opts)
		fmt.Printf("%-12s wire=%d bytes\n", p.name, len(buf))
	}
}

Output:
OptSpeed     wire=2208 bytes
OptBalanced  wire=161 bytes
Example (StreamingDictShared)

ExampleOptions_streamingDictShared shows the headline Dense-mode win: across many calls of a StreamEncoder the intern table / shape table / predictors all survive, so the second record onwards trades long strings for 1-3 byte state-refs. Per-message Marshal does not share the dictionary across calls; only the stream API does. 

package main

import (
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Row struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	// 100 rows, all the same Service literal. Marshal-per-call
	// reuses the qdf encoder pool but resets state every call.
	one, _ := qdf.Marshal(Row{Service: "billing", Status: 200}, qdf.OptBalanced)
	fmt.Printf("per-message wire = %d bytes\n", len(one))

}

Output:
per-message wire = 36 bytes

const (
	// OptDense activates the inline intern table. First occurrences of
	// strings and []byte payloads are stored once and back-referenced
	// by ID (1-3 bytes per reuse). Required for any of the state-ref
	// codecs (Repeat / MTF / Pair) and for tagMapShape.
	OptDense Options = 1 << iota

	// OptQPack enables the numeric / boolean slice codecs. Bools
	// bit-pack; integer slices try Frame-of-Reference and Delta+FOR;
	// float slices stay on raw-LE. Auto-selected per slice; the
	// encoder falls back to raw-LE when nothing wins. Gorilla XOR
	// for floats lives behind OptGorillaFloat (bit 5) — see
	// OptCompression for the bundle that turns it on.
	OptQPack

	// OptShapeIntern routes struct emissions through tagMapShape: each
	// distinct struct type declares its field ordering on first emit;
	// subsequent emits write only the shape ID + values. Requires
	// OptDense (the shape table lives on the intern table side).
	OptShapeIntern

	// OptPairPred enables the Markov-1 successor predictor
	// (tagStatePair, 0xEA). Per previous state-ref ID the encoder keeps
	// a ring of the last 4 successors; a hit emits the ring rank in a
	// single byte. Requires OptDense.
	OptPairPred

	// OptMTF enables Move-to-Front rank coding (tagStateMTF, 0xE9).
	// When the LRU rank of a state-ref ID needs fewer varuint bytes
	// than the raw id, the rank is emitted instead. Requires OptDense.
	OptMTF

	// OptGorillaFloat opts in to the Gorilla XOR codec for []float64
	// (and []float32) slices. Gorilla collapses smooth time-series
	// data dramatically (~75 % wire reduction on quantised metric
	// streams) but trades that for ~10× more CPU on the encode/decode
	// path because the body is bit-level. Off by default in
	// OptBalanced for that reason; included in OptCompression for
	// archive-style workloads where wire size dominates. Requires
	// OptQPack.
	OptGorillaFloat

	// OptRANS adds a final static order-0 rANS entropy pass over the whole
	// encoded body. Opt-in (bundled into OptCompression) because it trades
	// encode/decode CPU for wire size. The encoder applies it only when the
	// rANS form is strictly smaller than the plain body, so it never makes a
	// buffer larger. Whole-buffer, so the streaming encoder ignores it.
	OptRANS

	// OptSpeed is the zero-bit preset: Fast mode, no codecs, no
	// predictors. Maximum throughput, smallest CPU footprint.
	OptSpeed Options = 0

	// OptBalanced bundles every codec that does not trade CPU for
	// compression beyond its sweet spot. The right default for
	// telemetry, log batches, and any payload with repetitive
	// strings or numeric slices. Notably excludes OptGorillaFloat —
	// reach for OptCompression when the float slices in the payload
	// are smooth time-series and wire size matters more than encode
	// latency.
	OptBalanced Options = OptDense | OptQPack | OptShapeIntern | OptPairPred | OptMTF

	// OptCompression bundles every codec that the encoder will spend
	// CPU on for wire-size gains. On top of OptBalanced it adds
	// OptGorillaFloat (Gorilla XOR for float slices), which trades
	// encode/decode CPU for wire size, so it stays out of the
	// OptBalanced default; future heavy codecs (rANS, dictionary
	// preloading) will land in this bundle without breaking the name.
	OptCompression Options = OptBalanced | OptGorillaFloat | OptRANS
)

func (Options) Has 

func (o Options) Has(bit Options) bool

Has reports whether the named bit is set. Compiles to a single AND + compare. Use on the encoder hot path to gate codec emission.

type StreamDecoder 

type StreamDecoder struct {
	// contains filtered or unexported fields
}

StreamDecoder reads a sequence of values from an io.Reader. The intern table is preserved across Decode calls to match StreamEncoder.

The decoder grows its window buffer as needed. State-table entries alias the buffer, so the window is never compacted during a stream; total memory tracks the stream length. For unbounded streams partition into envelopes and create a fresh StreamDecoder per envelope.

func NewStreamDecoder 

func NewStreamDecoder(r io.Reader) *StreamDecoder

NewStreamDecoder returns a stream decoder reading from r.

func (*StreamDecoder) Close 

func (s *StreamDecoder) Close() error

Close releases the scratch buffer to the pool. The underlying reader is not closed. Idempotent: subsequent calls are a safe no-op.

func (*StreamDecoder) Decode 

func (s *StreamDecoder) Decode(out any) error

Decode reads the next value into out. out must be a pointer.

func (*StreamDecoder) SetNoCopy 

func (s *StreamDecoder) SetNoCopy(v bool)

SetNoCopy mirrors Decoder.SetNoCopy.

Example

ExampleStreamDecoder_SetNoCopy decodes a multi-message stream without copying the decoded strings out of the input buffer. The decoded values alias the io.Reader's payload; the caller must not retain the values past the next Decode call. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
	}

	// Build a 2-message stream.
	var sink bytes.Buffer
	enc := qdf.NewStreamEncoderWith(&sink, qdf.OptBalanced)
	_ = enc.Encode(Event{Service: "auth"})
	_ = enc.Encode(Event{Service: "auth"})
	_ = enc.Flush()
	_ = enc.Close()

	dec := qdf.NewStreamDecoder(&sink)
	dec.SetNoCopy(true) // aliased reads — zero allocations
	defer dec.Close()   //nolint:errcheck // example
	var ev Event
	for dec.Decode(&ev) == nil {
		fmt.Println(ev.Service)
	}
}

Output:
auth
auth

type StreamEncoder 

type StreamEncoder struct {
	// contains filtered or unexported fields
}

StreamEncoder writes a sequence of values into an io.Writer. The header is emitted once before the first value; in Dense mode the intern table survives across Encode calls so back-references span the whole stream.

Example

ExampleStreamEncoder writes a batch of messages into a single io.Writer where the Dense-mode intern table, shape table, and predictors survive across calls. The first message of every shape pays for the key intern records; every subsequent message of the same shape emits values only.

This is the form to reach for when piping events to a file or a network socket — per-message Marshal would reset the state table each call and lose the cross-message compression. 

package main

import (
	"bytes"
	"fmt"

	"github.com/alex60217101990/qdf"
)

func main() {
	type Event struct {
		Service string `qdf:"service"`
		Status  int    `qdf:"status"`
	}

	var sink bytes.Buffer
	enc := qdf.NewStreamEncoderWith(&sink, qdf.OptBalanced)
	defer enc.Close() //nolint:errcheck // example

	events := []Event{
		{"billing", 200},
		{"billing", 200},
		{"billing", 500},
	}
	for _, e := range events {
		if err := enc.Encode(e); err != nil {
			panic(err)
		}
	}
	if err := enc.Flush(); err != nil {
		panic(err)
	}

	dec := qdf.NewStreamDecoder(&sink)
	for {
		var out Event
		if err := dec.Decode(&out); err != nil {
			break
		}
		fmt.Printf("%s=%d\n", out.Service, out.Status)
	}
}

Output:
billing=200
billing=200
billing=500

func NewStreamEncoder 

func NewStreamEncoder(w io.Writer, mode Mode) *StreamEncoder

NewStreamEncoder returns a stream encoder backed by w. Dense mode activates the full balanced codec set (OptBalanced); Fast mode emits raw tagged bytes (OptSpeed). For finer per-stream tuning, build the StreamEncoder via NewStreamEncoderWith.

func NewStreamEncoderWith 

func NewStreamEncoderWith(w io.Writer, opts Options) *StreamEncoder

NewStreamEncoderWith returns a stream encoder with the given Options bit-mask. The intern table (when OptDense is set) survives across Encode calls so back-references span the whole stream.

func (*StreamEncoder) Close 

func (s *StreamEncoder) Close() error

Close flushes pending data and releases the scratch buffer to the pool. The underlying writer is not closed. Idempotent: subsequent calls are a safe no-op.

func (*StreamEncoder) Encode 

func (s *StreamEncoder) Encode(v any) error

Encode writes v as the next value in the stream. The encoder flushes internally when its buffer crosses 16 KiB; call Flush to push earlier.

func (*StreamEncoder) Flush 

func (s *StreamEncoder) Flush() error

Flush writes any buffered bytes to the underlying writer.

type Unmarshaler 

type Unmarshaler interface {
	UnmarshalQDF(src []byte) (n int, err error)
}

Unmarshaler is implemented by types that know how to deserialize themselves from a QDF wire-format slice. Implementations should consume exactly one value from src and return the number of bytes consumed.

Source Files

View all Source files

Directories

Path Synopsis
internal
bufpool
Package bufpool implements size-classed, sharded byte-slice pools.
 Package bufpool implements size-classed, sharded byte-slice pools.
endian
Package endian exposes a build-time constant for native byte order.
 Package endian exposes a build-time constant for native byte order.
intern
Package intern provides a small fixed-size string cache used by the decoder to deduplicate repeated map keys.
 Package intern provides a small fixed-size string cache used by the decoder to deduplicate repeated map keys.
internarena
Package internarena is a bytes-only bump-pointer allocator for the Dense-mode intern table.
 Package internarena is a bytes-only bump-pointer allocator for the Dense-mode intern table.
mapsgen command
internal/mapsgen — code generator for typed map encode/decode fast paths.
 internal/mapsgen — code generator for typed map encode/decode fast paths.
rans
Package rans is a static order-0 byte range Asymmetric Numeral System coder (the canonical rans_byte: 32-bit state, byte renormalization, a 12-bit normalized frequency table).
 Package rans is a static order-0 byte range Asymmetric Numeral System coder (the canonical rans_byte: 32-bit state, byte renormalization, a 12-bit normalized frequency table).
reflectutil
Package reflectutil holds the reflect-based helpers the qdf codec uses to allocate slices and maps without going through reflect.Value.
 Package reflectutil holds the reflect-based helpers the qdf codec uses to allocate slices and maps without going through reflect.Value.
unsafestr
Package unsafestr provides zero-copy conversions between string and []byte.
 Package unsafestr provides zero-copy conversions between string and []byte.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.