utlpfor

package module
v0.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2026 License: Apache-2.0 Imports: 5 Imported by: 0

README

UTL-PFOR

UTL-PFOR is an integer compression library for Go using native SIMD support.

It is the successor to fastpfor-go, which uses PFOR [1] based on the FastPFOR [2] library with SSE2 assembly.

UTL-PFOR replaces the SSE2-only lane layout for bitpacking with the FastLanes [4] Unified Transposed Layout (UTL): 16 lanes x 8 values in 64-byte super-words. This single wire format works across SSE2, AVX2, and AVX-512 without data transposition.

In addition to the fastpfor compression scheme, UTL-PFOR uses real Frame-of-Reference (the for in pfor) encoding, storing a minimum value per block and compressing only the residuals, if beneficial. Outliers, that would harm bitpacking, are stored as exceptions, that are patched on decompression (the p in pfor). Exceptions are encoded using StreamVByte [3], a variable-byte encoding scheme optimized for SIMD (specifically SSE2). Delta-Encoding allows to only store the difference between values. Zigzag-Encoding is used to encode negative values after Delta-Encoding.

Status

Work in progress. The library is functional but not yet final.

API

func PackUint32(flag Flag, values []uint32, dst []byte, scratch []uint32) ([]byte, error)
func UnpackUint32(src []byte, values []uint32, scratch []uint32) ([]uint32, int, error)
func GetUint32(pos int, src []byte, scratch []uint32) (uint32, error)
func BlockLength(src []byte) (int, error)
func MaxBlockLength32(flag Flag) int
func Header(src []byte) (count, bitWidth, excCount int, hasDelta, hasFOR, hasZigZag, hasSpecial bool, err error)

And for []uint64 handling:

func PackUint64(flag Flag, values []uint64, dst []byte, scratch []uint32) ([]byte, error)
func UnpackUint64(src []byte, values []uint64, scratch []uint32) ([]uint64, int, error)
func GetUint64(pos int, src []byte, scratch []uint32) (uint64, error)
func MaxBlockLength64(flag Flag) int

The uint64 functions use the same block format and reuse the entire uint32 pipeline internally. 64-bit values are either range-reduced to 32 bits (via 64-bit frame of reference) or split into lower/upper 32-bit halves encoded as two consecutive uint32 sub-blocks.

Compressor

The compressor objects own a reusable scratch buffer and provide a simplified API for compression, decompression, and random access. Callers no longer need to allocate or pass scratch buffers.

// uint32
c := utlpfor.NewUint32()
packed, err := c.Compress(utlpfor.Delta, dst, values)
unpacked, consumed, err := c.Decompress(dst, packed)
val, err := c.Get(pos, packed)

// uint64
c64 := utlpfor.NewUint64()
packed, err := c64.Compress(utlpfor.Delta, dst, values)
unpacked, consumed, err := c64.Decompress(dst, packed)
val, err := c64.Get(pos, packed)

A single compressor handles both compression and decompression. It is not safe for concurrent use; each goroutine should create its own.

Flag Constants
Flag Description
Delta Delta-encode values before packing
NoFOR Skip Frame-of-Reference analysis
NoPatch Skip exception analysis (no patching)
Special Set the SPECIAL header bit
Append Append packed block after existing dst content (implies NoInPlace)
NoInPlace Guarantee the input values slice is not modified

Flags can be combined with bitwise OR, e.g. Delta | NoFOR.

The Append and NoInPlace flags are pack-time control flags only and are never stored in the on-disk block header.

When Append is set, the packed block is written after the existing content instead of overwriting from index 0.

By default, PackUint32 may modify the input values slice in-place during FOR subtraction and delta encoding. When NoInPlace is set (or implied by Append), the library uses a scratch work buffer instead, leaving the original values untouched.

For zero-allocation operation with NoInPlace, provide a scratch buffer with capacity >= ScratchLenNoInPlace (256 elements). If scratch is smaller, the library allocates internally.

Requirements

This library uses Go's native SIMD support (simd/archsimd) introduced in Go 1.26 (via GOEXPERIMENT=simd).

Runtime dispatch selects the best path at startup: AVX-512 > AVX2 > SSE2 > scalar fallback.

  • Go 1.26+ with GOEXPERIMENT=simd for SIMD acceleration
  • Scalar fallback works without the experiment flag on any architecture
  • Recommended: Use gotip (Go 1.27-devel) for optimal AVX2/AVX-512 performance until Go 1.27 is released. Go 1.26.x compilers have a known (and fixed) issue generating suboptimal code, resulting in performance degradation on AVX2 and AVX-512 bitpacking kernels.

Quick Start

# Scalar path (any Go 1.26+)
gotip test ./...

# With SIMD acceleration
GOEXPERIMENT=simd gotip test ./...

# Benchmarks
GOEXPERIMENT=simd gotip test -bench=. -benchmem -count=5 ./...

Data Format

A Kaitai Struct definition file is part of this repository.

Disclaimer

This library was developed with AI assistance (Claude Opus 4.6 and Codex 5.3).

Literature

  • [1] Zukowski, M., Heman, S., Nes, N., & Boncz, P. (2006). Super-Scalar RAM-CPU Cache Compression. 22nd International Conference on Data Engineering (ICDE'06), 59-59. https://doi.org/10.1109/ICDE.2006.150
  • [2] Lemire, D., & Boytsov, L. (2015). Decoding billions of integers per second through vectorization. Software: Practice and Experience, 45(1), 1-29. https://doi.org/10.1002/spe.2203
  • [3] Lemire, D., Kurz, N. & Rupp, C. (2018). Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters 130.
  • [4] Afroozeh, A. & Muehlbauer, P. (2023). FastLanes: An SIMD-Friendly Layout for Analytic Databases. Proceedings of the VLDB Endowment, 16(9), 2132-2144. https://www.vldb.org/pvldb/vol16/p2132-afroozeh.pdf

Documentation

Overview

Package utlpfor implements a UTL-PFOR integer compression codec optimized for Go 1.26+ native SIMD.

The codec operates on fixed blocks of up to 128 unsigned integers using the Unified Transposed Layout (UTL, aka FastLanes) for SIMD-width independence.

The library automatically selects the best SIMD path at startup: AVX-512 -> AVX2 -> SSE2 -> scalar fallback. No build-time configuration is required.

Index

Examples

Constants

View Source 
const (

	// IntTypeUint8 is the header integer type for uint8 (reserved, not yet supported).
	IntTypeUint8 = 0
	// IntTypeUint16 is the header integer type for uint16 blocks.
	IntTypeUint16 = 1
	// IntTypeUint32 is the header integer type for uint32 blocks.
	IntTypeUint32 = 2
	// IntTypeUint64 is the header integer type for uint64 blocks.
	IntTypeUint64 = 3
)
View Source 
const ScratchLen = blockSize

ScratchLen is the minimum scratch buffer capacity (in uint32 elements) for zero-allocation Pack and Unpack operations.

View Source 
const ScratchLen64 = 3 * blockSize

ScratchLen64 is the minimum scratch buffer capacity (in uint32 elements) for zero-allocation uint64 Pack and Unpack operations.

Three regions of blockSize (128) uint32 elements each: 

scratch[0*blockSize : 1*blockSize]  - Block 1 values (lower halves / FOR64 work)
scratch[1*blockSize : 2*blockSize]  - exception workspace for current block
scratch[2*blockSize : 3*blockSize]  - Block 2 values (upper halves)

Keeping lower and upper halves in separate, non-overlapping regions lets the two-block unpack path combine them in a single fused loop (dst[i] = upper[i]<<32 | lower[i]) instead of two passes.

View Source 
const ScratchLenNoInPlace = 2 * blockSize

ScratchLenNoInPlace is the minimum scratch buffer capacity (in uint32 elements) for zero-allocation Pack operations when NoInPlace or Append is set. The first blockSize elements hold the values work buffer and the second blockSize elements hold exception highBits.

Variables

View Source 
var ErrInvalidBlockLength = errors.New("UTLpfor: invalid block length")

ErrInvalidBlockLength is returned when the block length in the header is invalid.

View Source 
var ErrInvalidBuffer = errors.New("UTLpfor: invalid buffer")

ErrInvalidBuffer is returned when the input buffer is nil, empty, or truncated.

View Source 
var ErrInvalidFlags = errors.New("UTLpfor: invalid header flags")

ErrInvalidFlags is returned when the header flags contain an invalid combination.

View Source 
var ErrNotImplemented = errors.New("UTLpfor: not implemented")

ErrNotImplemented is returned by stub functions not yet implemented.

View Source 
var ErrPositionOutOfRange = errors.New("UTLpfor: position out of range")

ErrPositionOutOfRange is returned when the requested position exceeds the block count.

View Source 
var ErrUnsupportedType = errors.New("UTLpfor: unsupported integer type")

ErrUnsupportedType is returned when the header integer type is not supported.

Functions

func BlockLength 

func BlockLength(src []byte) (int, error)

BlockLength returns the total byte length of the encoded block starting at the beginning of buf. Only the first few bytes of the buffer are read, enabling efficient block skipping in MMAP-backed files. For uint64 blocks with combine-with-next set, the returned length includes both Block 1 and Block 2.

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := []uint32{10, 20, 30, 40, 50} // 5 x 4 = 20 bytes unpacked
	packed, _ := utlpfor.PackUint32(0, values, nil, nil)

	blockLen, err := utlpfor.BlockLength(packed)
	if err != nil {
		panic(err)
	}
	fmt.Printf("block length: %d bytes\n", blockLen)
}

Output:
block length: 18 bytes
Example (SkipBlocks) 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	block1 := []uint32{1, 2, 3}
	block2 := []uint32{100, 200, 300}

	var buf []byte
	buf, _ = utlpfor.PackUint32(utlpfor.Append, block1, buf, nil)
	buf, _ = utlpfor.PackUint32(utlpfor.Append, block2, buf, nil)

	bl, _ := utlpfor.BlockLength(buf)
	fmt.Printf("first block: %d bytes\n", bl)

	unpacked, _, _ := utlpfor.UnpackUint32(buf[bl:], nil, nil)
	fmt.Printf("second block values: %v\n", unpacked)
}

Output:
first block: 13 bytes
second block values: [100 200 300]

func GetUint32 

func GetUint32(pos int, src []byte, scratch []uint32) (uint32, error)

GetUint32 extracts a single value at the given position from the packed block. scratch with capacity >= ScratchLen enables zero-allocation operation for delta-encoded blocks; pass nil if zero-alloc is not required. Single code path; SIMD acceleration is applied internally via dispatched UnpackUint32 (full-unpack fallback for delta blocks with high posInLane).

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := []uint32{100, 200, 300, 400, 500}
	packed, _ := utlpfor.PackUint32(0, values, nil, nil)

	val, err := utlpfor.GetUint32(2, packed, nil)
	if err != nil {
		panic(err)
	}
	fmt.Printf("value at position 2: %d\n", val)
}

Output:
value at position 2: 300

func GetUint64 

func GetUint64(pos int, src []byte, scratch []uint32) (uint64, error)

GetUint64 extracts a single uint64 value at the given position from the packed block. scratch with capacity >= ScratchLen64 enables zero-allocation operation; pass nil if zero-alloc is not required. SIMD acceleration is applied internally when the delta full-unpack fallback is triggered (via getFullUnpackViaBlock dispatch).

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := []uint64{1_000_000_000_000, 1_000_000_000_001, 1_000_000_000_002} // 3 x 8 = 24 bytes unpacked
	packed, _ := utlpfor.PackUint64(0, values, nil, nil)

	val, err := utlpfor.GetUint64(1, packed, nil)
	if err != nil {
		panic(err)
	}
	fmt.Printf("value at position 1: %d\n", val)
}

Output:
value at position 1: 1000000000001

func Header(src []byte) (count, bitWidth, excCount int,
	hasDelta, hasFOR, hasZigZag, hasSpecial bool, err error)

Header reads the 4-byte block header from src and returns the decoded fields.

EXPERIMENTAL: This function's return values may change in future versions.

Example 
package main

import (
	"fmt"
	"slices"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := make([]uint32, 128)
	for i := range values {
		values[i] = uint32(i * 10)
	}
	packed, _ := utlpfor.PackUint32(utlpfor.Delta|utlpfor.NoPatch, slices.Clone(values), nil, nil)

	count, bitWidth, excCount, hasDelta, hasFOR, _, _, err := utlpfor.Header(packed)
	if err != nil {
		panic(err)
	}
	fmt.Printf("count=%d, bitWidth=%d, excCount=%d, delta=%v, FOR=%v\n",
		count, bitWidth, excCount, hasDelta, hasFOR)
}

Output:
count=128, bitWidth=8, excCount=0, delta=true, FOR=false

func MaxBlockLength32 

func MaxBlockLength32(flag Flag) int

MaxBlockLength32 returns the maximum byte length of a single packed uint32 block for the given encoding flags. This is useful for pre-allocating destination buffers to avoid allocations during PackUint32.

The returned value is a conservative upper bound. Actual block lengths are typically much smaller.

Flags that affect the worst-case length:

  • NoPatch: disables exceptions, reducing the maximum length.
  • NoFOR: disables frame-of-reference, removing the FOR base bytes.

Flags that do NOT affect the worst-case length (ignored):

  • Delta, Special, Append.

Examples: 

MaxBlockLength32(0)              // 1018 (general case, exceptions possible)
MaxBlockLength32(NoPatch)        // 520  (no exceptions)
MaxBlockLength32(NoPatch|NoFOR)  // 516  (no exceptions, no FOR base)
MaxBlockLength32(NoFOR)          // 1014 (exceptions possible, no FOR base)

Usage for buffer pre-allocation: 

dst := make([]byte, 0, utlpfor.MaxBlockLength32(0))
dst, _ = utlpfor.PackUint32(utlpfor.Append, block, dst, scratch)
Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	plain := utlpfor.MaxBlockLength32(0)
	noPatch := utlpfor.MaxBlockLength32(utlpfor.NoPatch)

	fmt.Printf("max block (default): %d bytes\n", plain)
	fmt.Printf("max block (NoPatch): %d bytes\n", noPatch)
}

Output:
max block (default): 1018 bytes
max block (NoPatch): 520 bytes

func MaxBlockLength64 

func MaxBlockLength64(flag Flag) int

MaxBlockLength64 returns the maximum byte length of a single packed uint64 block for the given encoding flags. This is useful for pre-allocating destination buffers to avoid allocations during PackUint64.

The returned value is a conservative upper bound that accounts for the worst-case two-block encoding (Block 1 + Block 2).

Flags that affect the worst-case length:

  • NoPatch: disables exceptions, reducing the maximum length.
  • NoFOR: disables frame-of-reference, removing FOR base bytes.

Flags that do NOT affect the worst-case length (ignored):

  • Delta, Special, Append.
Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	plain := utlpfor.MaxBlockLength64(0)
	noPatch := utlpfor.MaxBlockLength64(utlpfor.NoPatch)

	fmt.Printf("max block64 (default): %d bytes\n", plain)
	fmt.Printf("max block64 (NoPatch): %d bytes\n", noPatch)
}

Output:
max block64 (default): 2038 bytes
max block64 (NoPatch): 1042 bytes

func NewUint32 added in v0.3.0 

func NewUint32() *compressor

NewUint32 creates a [compressor] with a pre-allocated scratch buffer for uint32 block operations. The scratch buffer is sized to support NoInPlace and Append without internal allocation.

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	c := utlpfor.NewUint32()

	values := []uint32{10, 20, 30, 40, 50}

	packed, _ := c.Compress(0, nil, values)

	unpacked, _, _ := c.Decompress(nil, packed)

	fmt.Printf("round-trip: %v\n", unpacked)
}

Output:
round-trip: [10 20 30 40 50]
Example (Delta) 
package main

import (
	"fmt"
	"slices"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	c := utlpfor.NewUint32()

	values := make([]uint32, 128)
	for i := range values {
		values[i] = uint32(1000 + i*4)
	}

	packed, _ := c.Compress(utlpfor.Delta, nil, slices.Clone(values))

	fmt.Printf("delta-compressed 128 values into %d bytes\n", len(packed))

	unpacked, _, _ := c.Decompress(nil, packed)

	fmt.Printf("first 5 values: %v\n", unpacked[:5])
}

Output:
delta-compressed 128 values into 170 bytes
first 5 values: [1000 1004 1008 1012 1016]
Example (Get) 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	c := utlpfor.NewUint32()

	values := []uint32{100, 200, 300, 400, 500}
	packed, _ := c.Compress(0, nil, values)

	val, _ := c.Get(2, packed)

	fmt.Printf("value at position 2: %d\n", val)
}

Output:
value at position 2: 300

func NewUint64 added in v0.3.0 

func NewUint64() *compressor64

NewUint64 creates a [compressor64] with a pre-allocated scratch buffer for uint64 block operations.

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	c := utlpfor.NewUint64()

	values := []uint64{1_000_000_000_000, 1_000_000_000_001, 1_000_000_000_002}

	packed, _ := c.Compress(0, nil, values)

	unpacked, _, _ := c.Decompress(nil, packed)

	fmt.Printf("round-trip: %v\n", unpacked)
}

Output:
round-trip: [1000000000000 1000000000001 1000000000002]
Example (Get) 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	c := utlpfor.NewUint64()

	values := []uint64{1_000_000_000_000, 2_000_000_000_000, 3_000_000_000_000}
	packed, _ := c.Compress(0, nil, values)

	val, _ := c.Get(1, packed)

	fmt.Printf("value at position 1: %d\n", val)
}

Output:
value at position 1: 2000000000000

func PackUint32 

func PackUint32(flag Flag, values []uint32, dst []byte, scratch []uint32) ([]byte, error)

PackUint32 encodes uint32 values into a packed block. If dst has sufficient capacity, it is reused; otherwise a new slice is allocated. If scratch has capacity >= ScratchLen (128), no heap allocations occur (assuming dst also has sufficient capacity). Pass nil for scratch to use internal allocations. The flag parameter controls encoding options (e.g. Delta for delta encoding). When Append is set, the packed block is written after the existing content of dst (starting at len(dst)) instead of overwriting from index 0. The returned slice includes the preserved prefix followed by the new block.

When NoInPlace or Append is set, the values slice is guaranteed unmodified after the call. The library uses scratch as a work buffer for FOR subtraction and delta encoding. For zero-allocation operation with NoInPlace/Append, provide scratch with capacity >= ScratchLenNoInPlace (256).

Without NoInPlace or Append, the values slice may be modified in-place (FOR subtraction, delta encoding). Callers that need the original values must either set NoInPlace or copy them before calling PackUint32.

To pre-allocate dst for zero-allocation packing, use MaxBlockLength32: 

dst := make([]byte, 0, MaxBlockLength32(flag))
dst, err = PackUint32(flag|Append, values, dst, scratch)
Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := []uint32{10, 20, 30, 40, 50} // 5 x 4 = 20 bytes unpacked

	packed, err := utlpfor.PackUint32(0, values, nil, nil)
	if err != nil {
		panic(err)
	}
	fmt.Printf("packed %d values into %d bytes\n", len(values), len(packed))
}

Output:
packed 5 values into 18 bytes
Example (Append) 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	block1 := []uint32{1, 2, 3, 4}         // 4 x 4 = 16 bytes unpacked
	block2 := []uint32{100, 200, 300, 400} // 4 x 4 = 16 bytes unpacked

	var dst []byte
	var err error
	dst, err = utlpfor.PackUint32(utlpfor.Append, block1, dst, nil)
	if err != nil {
		panic(err)
	}
	firstBlockLen := len(dst)
	dst, err = utlpfor.PackUint32(utlpfor.Append, block2, dst, nil)
	if err != nil {
		panic(err)
	}
	fmt.Printf("two blocks concatenated: %d + %d = %d bytes\n",
		firstBlockLen, len(dst)-firstBlockLen, len(dst))
}

Output:
two blocks concatenated: 15 + 17 = 32 bytes
Example (Delta) 
package main

import (
	"fmt"
	"slices"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := make([]uint32, 128) // 128 x 4 = 512 bytes unpacked
	for i := range values {
		values[i] = uint32(100000 + i*4)
	}

	plain, _ := utlpfor.PackUint32(0, slices.Clone(values), nil, nil)
	delta, _ := utlpfor.PackUint32(utlpfor.Delta, slices.Clone(values), nil, nil)

	fmt.Printf("plain: %d bytes, delta: %d bytes\n", len(plain), len(delta))
}

Output:
plain: 200 bytes, delta: 136 bytes
Example (Flags) 
package main

import (
	"fmt"
	"slices"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := make([]uint32, 128) // 128 x 4 = 512 bytes unpacked
	for i := range values {
		values[i] = uint32(4_000_000 + i%16)
	}
	values[10] = 4_500_000
	values[90] = 4_600_000

	plain, _ := utlpfor.PackUint32(0, slices.Clone(values), nil, nil)
	noFOR, _ := utlpfor.PackUint32(utlpfor.NoFOR, slices.Clone(values), nil, nil)
	noPatch, _ := utlpfor.PackUint32(utlpfor.NoPatch, slices.Clone(values), nil, nil)
	both, _ := utlpfor.PackUint32(utlpfor.NoFOR|utlpfor.NoPatch, slices.Clone(values), nil, nil)

	fmt.Printf("default: %d bytes\n", len(plain))
	fmt.Printf("NoFOR: %d bytes\n", len(noFOR))
	fmt.Printf("NoPatch: %d bytes\n", len(noPatch))
	fmt.Printf("NoFOR|NoPatch: %d bytes\n", len(both))
}

Output:
default: 81 bytes
NoFOR: 388 bytes
NoPatch: 328 bytes
NoFOR|NoPatch: 388 bytes
Example (NoInPlace) 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	// A shared fixed-size buffer packed multiple times without cloning.
	var buf [128]uint32
	for i := range buf {
		buf[i] = uint32(500 + i*3)
	}

	scratch := make([]uint32, utlpfor.ScratchLenNoInPlace)
	var dst []byte
	var err error

	// Append implies NoInPlace: the buf array is never modified.
	for range 3 {
		dst, err = utlpfor.PackUint32(utlpfor.Append, buf[:], dst, scratch)
		if err != nil {
			panic(err)
		}
	}
	fmt.Printf("packed 3 blocks (%d bytes), buf[0]=%d (unchanged)\n",
		len(dst), buf[0])
}

Output:
packed 3 blocks (588 bytes), buf[0]=500 (unchanged)
Example (ZeroAlloc) 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := make([]uint32, 128) // 128 x 4 = 512 bytes unpacked
	for i := range values {
		values[i] = uint32(i)
	}

	// Pre-allocate dst with MaxBlockLength32 to avoid allocation during packing.
	dst := make([]byte, 0, utlpfor.MaxBlockLength32(0))

	// ScratchLenNoInPlace (256) is the minimum scratch buffer capacity
	// (in uint32 elements) for zero-allocation packing with Append or
	// NoInPlace. Use ScratchLen (128) when neither flag is needed.
	// For uint64 operations, use ScratchLen64 (384) instead.
	scratch := make([]uint32, utlpfor.ScratchLenNoInPlace)

	packed, err := utlpfor.PackUint32(utlpfor.Append, values, dst, scratch)
	if err != nil {
		panic(err)
	}
	fmt.Printf("packed %d values with zero allocations\n", len(values))
	_ = packed
}

Output:
packed 128 values with zero allocations

func PackUint64 

func PackUint64(flag Flag, values []uint64, dst []byte, scratch []uint32) ([]byte, error)

PackUint64 encodes uint64 values into a packed block using the double-block strategy: values are split into lower and upper 32-bit halves, each encoded as a standard uint32 UTL block. If all values fit in 32 bits, a single block is produced. If dst has sufficient capacity, it is reused; otherwise a new slice is allocated. scratch with capacity >= ScratchLen64 (384) enables zero-allocation operation. Pass nil for scratch to use internal allocations. The values slice is never modified (the uint64 pack path reads values into internal scratch buffers). NoInPlace and Append are accepted for API consistency but have no additional effect on value preservation.

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	values := []uint64{1_000_000_000_000, 1_000_000_000_001, 1_000_000_000_002} // 3 x 8 = 24 bytes unpacked

	packed, err := utlpfor.PackUint64(0, values, nil, nil)
	if err != nil {
		panic(err)
	}
	fmt.Printf("packed %d uint64 values into %d bytes\n", len(values), len(packed))
}

Output:
packed 3 uint64 values into 19 bytes

func UnpackUint32 

func UnpackUint32(src []byte, values []uint32, scratch []uint32) ([]uint32, int, error)

UnpackUint32 decodes a packed block into values, using scratch as workspace. Returns the populated values slice, the number of bytes consumed, and any error.

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	original := []uint32{10, 20, 30, 40, 50} // 5 x 4 = 20 bytes unpacked
	packed, _ := utlpfor.PackUint32(0, original, nil, nil)

	unpacked, consumed, err := utlpfor.UnpackUint32(packed, nil, nil)
	if err != nil {
		panic(err)
	}
	fmt.Printf("unpacked %d values, consumed %d bytes\n", len(unpacked), consumed)
	fmt.Printf("values: %v\n", unpacked)
}

Output:
unpacked 5 values, consumed 18 bytes
values: [10 20 30 40 50]

func UnpackUint64 

func UnpackUint64(src []byte, values []uint64, scratch []uint32) ([]uint64, int, error)

UnpackUint64 decodes a packed uint64 block into values, using scratch as workspace. Returns the populated values slice, the number of bytes consumed, and any error.

Example 
package main

import (
	"fmt"

	utlpfor "github.com/Akron/utlpfor"
)

func main() {
	original := []uint64{1_000_000_000_000, 1_000_000_000_001, 1_000_000_000_002} // 3 x 8 = 24 bytes unpacked
	packed, _ := utlpfor.PackUint64(0, original, nil, nil)

	unpacked, consumed, err := utlpfor.UnpackUint64(packed, nil, nil)
	if err != nil {
		panic(err)
	}
	fmt.Printf("unpacked %d values, consumed %d bytes\n", len(unpacked), consumed)
	fmt.Printf("values: %v\n", unpacked)
}

Output:
unpacked 3 values, consumed 19 bytes
values: [1000000000000 1000000000001 1000000000002]

Types

type ErrOverflow 

type ErrOverflow struct {
	Position int
}

ErrOverflow is returned when delta decoding overflows uint32.

func (*ErrOverflow) Error 

func (e *ErrOverflow) Error() string

Error returns the overflow error message.

type Flag 

type Flag byte

Flag controls encoding options passed to PackUint32. 

const Append Flag = 1 << 4

Append makes PackUint32 append the packed block after the existing content of dst (starting at len(dst)) instead of overwriting from index 0. The returned slice includes the preserved prefix. This is a pack-time control flag only, never stored on disk. Append automatically implies NoInPlace: the input values slice is guaranteed unmodified after the call. 

dst = dst[:0]
dst, _ = PackUint32(Delta|Append, block1, dst, scratch)
dst, _ = PackUint32(Delta|Append, block2, dst, scratch)
// dst now contains both blocks concatenated

const Delta Flag = 1 << 0

Delta indicates that the values should be delta-encoded before packing. 

const NoFOR Flag = 1 << 1

NoFOR disables Frame-of-Reference analysis during packing. When set, the encoder skips the min/max scan and FOR cost evaluation, going directly to bitpacking + patching. This improves packing speed when it is known in advance that FOR will not be beneficial. 

const NoInPlace Flag = 1 << 5

NoInPlace guarantees that PackUint32 (and PackUint64) will not modify the input values slice. The library internally uses a work buffer from scratch for FOR subtraction and delta encoding. This is a pack-time control flag only, never stored on disk.

For zero-allocation operation, provide scratch with capacity >= ScratchLenNoInPlace (256). If scratch is smaller, a temporary buffer is allocated internally.

Append automatically implies NoInPlace. 

const NoPatch Flag = 1 << 2

NoPatch disables exception analysis and patching during packing. The encoder uses the minimum step bitwidth that fits all values, producing zero exceptions. This yields faster packing at the cost of potentially larger output when a few outlier values inflate the bitwidth. Ideal for dictionary-compressed data where all values share a known maximum range. NoPatch also benefits random access by guaranteeing no exception data needs to be decoded. 

const Special Flag = 1 << 3

Special bit written in header during packing. The current decoder does not interpret this bit yet.

Source Files

View all Source files

Directories

Path Synopsis
internal
benchfmt command
Command benchfmt reads raw Go benchmark output files (one per SIMD level) and prints a Markdown comparison table with median ns/op values.
 Command benchfmt reads raw Go benchmark output files (one per SIMD level) and prints a Markdown comparison table with median ns/op values.
gen command
Generator for per-bitwidth specialized SIMD pack/unpack functions.
 Generator for per-bitwidth specialized SIMD pack/unpack functions.
threshold command
Command threshold reads BenchmarkGetUint32_Approaches output files (one per SIMD level) and suggests deltaFullUnpackThreshold values.
 Command threshold reads BenchmarkGetUint32_Approaches output files (one per SIMD level) and suggests deltaFullUnpackThreshold values.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.