svb

README

Stream VByte in Go with SIMD

This is another pure Go implementation of Stream VByte: Faster Byte-Oriented Integer Compression.

It uses avo by Michael McLoughlin to generate Go assembler code. This Go implementation has referenced https://github.com/lemire/streamvbyte

Random Access

Codec.Get random access implements idea based on Partitioned Elias-Fano Indexes

Speed Test

Uint32Decoder performance benchmark measures the latency to decode 1 million uint32. Codec.Get measures the latency of random access 1000 members among 1 million uint32.

svb/perf ❯❯❯ go test -bench .

Function	Cascade Lake	Skylake
Uint32Decode32	13562586ns	17394533ns
Uint32Decode128	331368ns	411102ns
Uint32Decode256	327100ns	406230ns
Uint32Decode512	470571ns	569497ns
Codec.Get	ns	144818ns

Other

Forked from https://github.com/rleiwang/svb for better testing and local management. License intact.

Documentation

Index

• Variables
• func HasXmm() bool
• func HasYmm() bool
• func HasZmm() bool
• func Uint32Decode(masks, data []byte, out []uint32)
• func Uint32Decode128(masks []byte, data []byte, out []uint32)
• func Uint32Decode256(masks []byte, data []byte, out []uint32)
• func Uint32Decode32(masks, data []byte, out []uint32)
• func Uint32Decode512(masks []byte, data []byte, out []uint32)
• func Uint32Encode(value []uint32) (mask, data []byte)
• type Codec
  • func NewFromBytes(data []byte) *Codec
  • func NewFromUint32(data []uint32) *Codec
  • func (c *Codec) Bytes() []byte
  • func (c *Codec) Get(i int) uint32
  • func (c *Codec) Len() int

Variables

var ShuffleTable = [256][16]byte{}/* 256 elements not displayed */

https://github.com/lemire/streamvbyte/blob/master/src/streamvbyte_shuffle_tables_decode.h#L25

Functions

func HasXmm

func HasXmm() bool

func HasYmm

func HasYmm() bool

func HasZmm

func HasZmm() bool

func Uint32Decode

func Uint32Decode(masks, data []byte, out []uint32)

func Uint32Decode128

func Uint32Decode128(masks []byte, data []byte, out []uint32)

Uint32Decode128 32 bits integer using XMM register, AVX

func Uint32Decode256

func Uint32Decode256(masks []byte, data []byte, out []uint32)

Uint32Decode256 32 bits integer using YMM register, AVX2

func Uint32Decode32

func Uint32Decode32(masks, data []byte, out []uint32)

func Uint32Decode512

func Uint32Decode512(masks []byte, data []byte, out []uint32)

Uint32Decode512 32 bits integer using ZMM register, AVX512

func Uint32Encode

func Uint32Encode(value []uint32) (mask, data []byte)

Uint32Encode encode []uint32 with stream vbytes codec as LittleEndian

Types

type Codec

type Codec struct {
	// contains filtered or unexported fields
}

func NewFromBytes

func NewFromBytes(data []byte) *Codec

func NewFromUint32

func NewFromUint32(data []uint32) *Codec

func (*Codec) Bytes

func (c *Codec) Bytes() []byte

func (*Codec) Get

func (c *Codec) Get(i int) uint32

Get 0-based ith uint32

func (*Codec) Len

func (c *Codec) Len() int