serialize

package module
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2026 License: BSD-3-Clause Imports: 5 Imported by: 0

README

Introduction

CI Go Reference

serialize.go is a simple bitpacking serializer for Go.

It is a pure Go port of the C++ serialize library, with no native code. The two libraries produce bit-for-bit identical output, so streams written by one language can be read by the other. This is pinned down by a golden wire format test whose bytes are copied verbatim from the C++ test suite.

It has the following features:

  • Serialize a bool with only one bit
  • Serialize any integer value from [1,64] bits writing only that number of bits to the buffer
  • Serialize signed integer values with [min,max] writing only the required bits to the buffer
  • Serialize floats, doubles, compressed floats, strings, byte arrays, and integers relative to another integer
  • Alignment support so you can align your bitstream to a byte boundary whenever you want
  • Unified serialization through the Stream interface, so you can write one function that handles read, write and measure
  • Zero allocations on every serialization path
  • Every read is bounds checked and range validated, so maliciously crafted packets fail with errors instead of panicking

Usage

go get github.com/mas-bandwidth/serialize.go

The package name is serialize, so no import alias is needed:

import "github.com/mas-bandwidth/serialize.go"

You can use the bitpacker directly:

buffer := make([]byte, 256)

writer := serialize.NewBitWriter(buffer)

writer.WriteBits(0, 1)
writer.WriteBits(1, 1)
writer.WriteBits(10, 8)
writer.WriteBits(255, 8)
writer.WriteBits(1000, 10)
writer.WriteBits(50000, 16)
writer.WriteBits(9999999, 32)
writer.FlushBits()

reader := serialize.NewBitReader(writer.Data())

a := reader.ReadBits(1)
b := reader.ReadBits(1)
c := reader.ReadBits(8)
d := reader.ReadBits(8)
e := reader.ReadBits(10)
f := reader.ReadBits(16)
g := reader.ReadBits(32)

Or you can write serialize methods for your types:

type Vector struct {
    X, Y, Z float32
}

func (v *Vector) Serialize(stream serialize.Stream) error {
    stream.SerializeFloat32(&v.X)
    stream.SerializeFloat32(&v.Y)
    stream.SerializeFloat32(&v.Z)
    return stream.Err()
}

type RigidBody struct {
    Position        Vector
    Orientation     Quaternion
    LinearVelocity  Vector
    AngularVelocity Vector
    AtRest          bool
}

func (b *RigidBody) Serialize(stream serialize.Stream) error {
    stream.SerializeObject(&b.Position)
    stream.SerializeObject(&b.Orientation)
    stream.SerializeBool(&b.AtRest)
    if !b.AtRest {
        stream.SerializeObject(&b.LinearVelocity)
        stream.SerializeObject(&b.AngularVelocity)
    } else if stream.IsReading() {
        b.LinearVelocity = Vector{}
        b.AngularVelocity = Vector{}
    }
    return stream.Err()
}

One serialize function handles write, read and measure:

// write
writeStream := serialize.NewWriteStream(buffer)
if err := body.Serialize(writeStream); err != nil {
    // handle error
}
writeStream.Flush()
packet := writeStream.Data()

// read
readStream := serialize.NewReadStream(packet)
if err := body.Serialize(readStream); err != nil {
    // packet is truncated, corrupt or malicious
}

// measure
measureStream := serialize.NewMeasureStream()
body.Serialize(measureStream)
bytesRequired := measureStream.BytesProcessed()

Errors are sticky: the first failure latches on the stream and every later serialize call returns it without touching the stream. That is why serialize functions can simply call one serialize method per field and return stream.Err() at the end — or check every call, if you prefer early exits.

If you want separate read and write functions instead of unified ones, use the concrete *ReadStream and *WriteStream types directly: they have the same methods as the Stream interface, with no dynamic dispatch.

Reading untrusted data

Packets come from the network and can be truncated or maliciously crafted. Every read is bounds checked and range validated, and the first failure latches an error on the stream: from then on every serialize call is a no-op that returns the same error and leaves values unmodified.

That last property implies one rule: a value that controls how much more work your serialize function does — a loop count or a continuation bit — must have its error checked before you use it. A loop that waits for a serialized value to change will wait forever on a truncated packet, because failed reads never update values. That is a denial of service vector.

For sentinel-driven loops, use serialize.Continue (a true bit before each element) or serialize.Until (a true bit terminating the sequence), which fold the stream error state into the loop condition in the style of bufio.Scanner:

hasNext := len(items) > 0 // when writing: true if there is a first element
i := 0
for serialize.Continue(stream, &hasNext) {
    // ... serialize element i ...
    i++
    if stream.IsWriting() {
        hasNext = i < len(items)
    }
}
if err := stream.Err(); err != nil {
    return err
}

For count-driven loops, check the error on the count before looping:

if err := stream.SerializeInt(&numItems, 0, MaxItems); err != nil {
    return err
}

The failure modes, why both sentinel polarities exist, and why loops that follow these rules are always bounded by the packet size are covered in docs/reading_untrusted_data.md.

Performance

All serialization paths are zero allocation. On an Apple M3 Ultra the bitpacker writes and reads mixed width values at around 2.1ns per value, and a representative 133 byte game network packet serializes at around 10 million packets per second per core. Interface dispatch through Stream costs only 6-8% over the concrete stream types.

The C++ serialize library is 2-6× faster on the same benchmarks, mostly because its release builds compile away the safety checks that serialize.go deliberately keeps on in every build. Full benchmark numbers and the cross language comparison are in docs/performance.md.

Limitations

  • Write buffer sizes must be a multiple of 8 bytes, because the bit writer flushes qwords to memory. Bytes past the end of the written data are only ever written as zeros. Buffers do not need any particular alignment.
  • Read buffers may be any number of bytes. For the fastest reads, keep at least 7 bytes of slack in the backing array beyond the packet data — for example, read packets into a large buffer and slice the packet out of it. The reader detects the slack via cap() and uses fully branchless window loads; without slack, reads near the end of the buffer assemble the window from the remaining bytes instead. Slack bytes are loaded but never interpreted.
  • Buffer sizes are effectively unlimited, because bit counts are stored in 64 bit signed integers.
  • SerializeWideString stores 32 bits per code point and is wire compatible with serialize_wstring in the C++ library. Code points that are not valid (surrogates or values above 0x10FFFF) fail on read.

Author

The author of this library is Glenn Fiedler.

Open source libraries by the same author include: serialize, netcode, reliable and yojimbo

If you find this software useful, please consider sponsoring it. Thanks!

License

BSD 3-Clause license.

Documentation

Overview

Package serialize implements fast bitpacked binary serialization for Go.

It is a pure Go port of the C++ serialize library (github.com/mas-bandwidth/serialize) and produces bit-for-bit identical output, so streams written by one language can be read by the other.

It has the following features:

  • Serialize a bool with only one bit
  • Serialize any integer value from [1,64] bits writing only that number of bits to the buffer
  • Serialize signed integer values in [min,max] writing only the required bits to the buffer
  • Serialize floats, doubles, compressed floats, strings, byte arrays, and integers relative to another integer
  • Alignment support so you can align your bitstream to a byte boundary whenever you want
  • Unified serialization through the Stream interface, so you can write one function that handles read, write and measure

The bit stream is written to memory in little endian order, which is considered network byte order for this library.

Values read from a stream are untrusted network data: every read is bounds checked and range validated, and the first failure latches an error on the stream. Serialize methods return that error and become no-ops once it is set, so you can either check each call or serialize an entire object and check Stream.Err once at the end — with one rule: a value that controls a loop must have its error checked before the loop uses it, because after an error values are never updated again and a loop waiting for one spins forever on a truncated or malicious packet. Use Continue for sentinel-driven loops, and check the error on serialized loop counts.

Example (UnifiedSerialization)

Write one serialize function per type and use it for write, read and measure. 

package main

import (
	"fmt"

	"github.com/mas-bandwidth/serialize.go"
)

type Vector struct {
	X, Y, Z float32
}

func (v *Vector) Serialize(stream serialize.Stream) error {
	stream.SerializeFloat32(&v.X)
	stream.SerializeFloat32(&v.Y)
	stream.SerializeFloat32(&v.Z)
	return stream.Err()
}

type RigidBody struct {
	Position       Vector
	LinearVelocity Vector
	AtRest         bool
}

// Serialize is a unified serialize function: the same code writes, reads and measures.
func (b *RigidBody) Serialize(stream serialize.Stream) error {
	stream.SerializeObject(&b.Position)
	stream.SerializeBool(&b.AtRest)
	if !b.AtRest {
		stream.SerializeObject(&b.LinearVelocity)
	} else if stream.IsReading() {
		b.LinearVelocity = Vector{}
	}
	return stream.Err()
}

func main() {
	buffer := make([]byte, 256)

	body := RigidBody{
		Position:       Vector{X: 10, Y: 20, Z: 30},
		LinearVelocity: Vector{X: 1, Y: 2, Z: 3},
	}

	writeStream := serialize.NewWriteStream(buffer)
	if err := body.Serialize(writeStream); err != nil {
		panic(err)
	}
	writeStream.Flush()

	packet := writeStream.Data()
	fmt.Printf("wrote %d bytes\n", len(packet))

	var received RigidBody
	readStream := serialize.NewReadStream(packet)
	if err := received.Serialize(readStream); err != nil {
		panic(err)
	}

	fmt.Println(received.Position, received.AtRest)

}

Output:
wrote 25 bytes
{10 20 30} false
Example (VariantSerialization)

Serialize a discriminated union: a type field followed by fields that depend on it. 

package main

import (
	"fmt"

	"github.com/mas-bandwidth/serialize.go"
)

// An Address is a discriminated union: the type field decides which fields follow on
// the wire. Ported from example.cpp in the C++ serialize library.
const (
	addressNone = iota
	addressIPv4
	addressIPv6
	numAddressTypes
)

type Address struct {
	Type int32
	IPv4 [4]uint8
	IPv6 [8]uint16
	Port uint16
}

func (a *Address) Serialize(stream serialize.Stream) error {

	if err := stream.SerializeInt(&a.Type, addressNone, numAddressTypes-1); err != nil {
		return err
	}
	switch a.Type {
	case addressIPv4:
		for i := range a.IPv4 {
			stream.SerializeUint8(&a.IPv4[i])
		}
	case addressIPv6:
		for i := range a.IPv6 {
			stream.SerializeUint16(&a.IPv6[i])
		}
	}
	stream.SerializeUint16(&a.Port)
	return stream.Err()
}

func main() {
	buffer := make([]byte, 64)

	address := Address{Type: addressIPv4, IPv4: [4]uint8{127, 0, 0, 1}, Port: 40000}

	writeStream := serialize.NewWriteStream(buffer)
	if err := address.Serialize(writeStream); err != nil {
		panic(err)
	}
	writeStream.Flush()

	var received Address
	readStream := serialize.NewReadStream(writeStream.Data())
	if err := received.Serialize(readStream); err != nil {
		panic(err)
	}

	fmt.Printf("%d.%d.%d.%d:%d\n", received.IPv4[0], received.IPv4[1], received.IPv4[2], received.IPv4[3], received.Port)

}

Output:
127.0.0.1:40000

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrOverflow is returned when a read would go past the end of the buffer, or a write
	// would go past the end of the buffer.
	ErrOverflow = errors.New("serialize: stream overflow")

	// ErrValueOutOfRange is returned when a value is outside the range it is serialized with.
	// On read this typically means the packet is corrupt or maliciously crafted.
	ErrValueOutOfRange = errors.New("serialize: value out of range")

	// ErrAlign is returned when the zero pad bits read by an align are not zero.
	// This typically means the read and write serialize functions don't match.
	ErrAlign = errors.New("serialize: nonzero padding bits in align")
)

Functions

func BitsRequired 

func BitsRequired(min, max uint32) int

BitsRequired returns the number of bits required to serialize an integer in range [min,max].

func BitsRequired64 

func BitsRequired64(min, max uint64) int

BitsRequired64 returns the number of bits required to serialize a 64 bit integer in range [min,max]. The result is in [0,64].

func Continue 

func Continue(stream Stream, more *bool) bool

Continue serializes *more as a single continuation bit and reports whether a sentinel-driven loop should proceed, folding the stream error state into the loop condition in the style of bufio.Scanner: it returns false as soon as the stream has an error, so loops of this form always terminate on truncated or malicious data, bounded by the size of the packet. 

hasNext := ... // when writing: true if there is a first element
for serialize.Continue(stream, &hasNext) {
    // serialize one element
    // when writing: set hasNext = true if there is another element
}
if err := stream.Err(); err != nil {
    return err
}

Never write the loop as `for hasNext { stream.SerializeBool(&hasNext); ... }`: once the stream has an error the failed read leaves hasNext unmodified and the loop never exits. For wire formats with the opposite bit polarity — a termination bit rather than a continuation bit — use Until. See the README section on reading untrusted data.

Example

Serialize a variable length sequence with a continuation bit per element. Continue folds the stream error state into the loop condition, so the loop always terminates on truncated or malicious data. 

package main

import (
	"fmt"

	"github.com/mas-bandwidth/serialize.go"
)

func main() {
	buffer := make([]byte, 64)
	items := []uint32{10, 20, 30}

	writeStream := serialize.NewWriteStream(buffer)
	i := 0
	hasNext := len(items) > 0
	for serialize.Continue(writeStream, &hasNext) {
		writeStream.SerializeBits(&items[i], 8)
		i++
		hasNext = i < len(items)
	}
	writeStream.Flush()

	readStream := serialize.NewReadStream(writeStream.Data())
	var received []uint32
	hasNext = true
	for serialize.Continue(readStream, &hasNext) {
		var item uint32
		readStream.SerializeBits(&item, 8)
		received = append(received, item)
	}
	if err := readStream.Err(); err != nil {
		panic(err)
	}

	fmt.Println(received)

}

Output:
[10 20 30]

func SignedToUnsigned 

func SignedToUnsigned(n int32) uint32

SignedToUnsigned converts a signed integer to an unsigned integer with zig-zag encoding. 0,-1,+1,-2,+2... becomes 0,1,2,3,4...

func UnsignedToSigned 

func UnsignedToSigned(n uint32) int32

UnsignedToSigned converts an unsigned integer to a signed integer with zig-zag encoding. 0,1,2,3,4... becomes 0,-1,+1,-2,+2...

func Until 

func Until(stream Stream, done *bool) bool

Until serializes *done as a single termination bit and reports whether a sentinel-driven loop should proceed. It is the inverse of Continue, for wire formats that mark the end of a sequence with a true bit instead of marking each element with a continuation bit — the polarity cannot be flipped without changing the wire format, so both helpers exist. Like Continue, it returns false as soon as the stream has an error, so loops of this form always terminate on truncated or malicious data, bounded by the size of the packet. 

done := ... // when writing: true if the sequence is empty
for serialize.Until(stream, &done) {
    // serialize one element
    // when writing: set done = true after the last element
}
if err := stream.Err(); err != nil {
    return err
}

Never write the loop as `for !done { stream.SerializeBool(&done); ... }`: once the stream has an error the failed read leaves done unmodified and the loop never exits.

Types

type BitReader 

type BitReader struct {
	// contains filtered or unexported fields
}

BitReader reads bitpacked integer values from a buffer.

The reader relies on the user reconstructing the exact same set of bit reads as bit writes when the buffer was written. This is an unattributed bitpacked binary stream!

Reads are effectively branchless: each read loads a 64 bit little endian window at the current byte position and shifts by the bit remainder. There is no scratch state and no refill loop, so reads carry no dependency between calls other than advancing the bit index.

Any buffer size is supported. For the fastest reads, keep at least 7 bytes of slack in the backing array beyond the data — for example, read packets into a large buffer and slice the packet out of it. The reader detects the slack via cap() and uses the fully branchless window load everywhere; without slack, reads near the end of the buffer assemble the window from the remaining bytes instead. Slack bytes are loaded but never interpreted: bits past the end of the data cannot reach the output of a read.

The zero value is an exhausted reader: create one with NewBitReader, or Reset one onto a buffer before use.

func NewBitReader 

func NewBitReader(data []byte) *BitReader

NewBitReader creates a bit reader that reads the bitpacked data in the given slice. The slice does not need any particular alignment. Buffer sizes are effectively unlimited, because bit counts are stored in 64 bit signed integers.

func (*BitReader) AlignBits 

func (r *BitReader) AlignBits() int

AlignBits returns the number of align bits that would be read, if an align was read right now. The result is in [0,7], where 0 means the stream is already byte aligned.

func (*BitReader) BitsRead 

func (r *BitReader) BitsRead() int64

BitsRead returns the number of bits read from the buffer so far.

func (*BitReader) BitsRemaining 

func (r *BitReader) BitsRemaining() int64

BitsRemaining returns the number of bits still available to read. For example, if the buffer size is 8 bytes and 10 bits have been read, 54 bits remain.

func (*BitReader) ReadAlign 

func (r *BitReader) ReadAlign() bool

ReadAlign reads an align, corresponding to a WriteAlign call when the buffer was written, and skips ahead to the next byte boundary. As a safety check, it verifies that the padding bits are zero and returns false if they are not; this typically aborts the packet read.

func (*BitReader) ReadBits 

func (r *BitReader) ReadBits(bits int) uint32

ReadBits reads bits from the buffer and returns the integer value read, in range [0,(1<<bits)-1]. bits must be in [1,32]. Panics if the read would go past the end of the buffer: check WouldReadPastEnd first when reading untrusted data, or use ReadStream, which performs all checks and returns errors instead.

func (*BitReader) ReadBytes 

func (r *BitReader) ReadBytes(data []byte)

ReadBytes reads len(data) bytes from the bit stream into data, corresponding to a WriteBytes call when the buffer was written. The reader must be aligned to a byte boundary. Panics if the read would go past the end of the buffer: bounds check with BitsRemaining first when reading untrusted data, or use ReadStream.

func (*BitReader) Reset 

func (r *BitReader) Reset(data []byte)

Reset points the bit reader at a data slice and clears all read state, allowing a single reader to be reused without allocation.

func (*BitReader) WouldReadPastEnd 

func (r *BitReader) WouldReadPastEnd(bits int) bool

WouldReadPastEnd returns true if reading the given number of bits would read past the end of the buffer.

type BitWriter 

type BitWriter struct {
	// contains filtered or unexported fields
}

BitWriter bitpacks unsigned integer values to a buffer.

Integer bit values are written to a 64 bit scratch value from right to left. Once the scratch fills to 64 bits it is stored to memory as a qword and the handful of bits that spilled past 64 carry over into the next scratch. Flushing half as often as a dword design makes writes significantly faster.

The bit stream is written to memory in little endian order, which is considered network byte order for this library. The output is bit-for-bit identical to the C++ serialize library.

IMPORTANT: The buffer size must be a multiple of 8 bytes, because words are stored to memory 8 bytes at a time. Bytes past the end of the written data are only ever written as zeros. The buffer does not need any particular alignment.

The zero value is not usable: create a BitWriter with NewBitWriter, or Reset one onto a buffer before use.

Example

Use the bitpacker directly when you want full control over each bit. 

package main

import (
	"fmt"

	"github.com/mas-bandwidth/serialize.go"
)

func main() {
	buffer := make([]byte, 256)

	writer := serialize.NewBitWriter(buffer)
	writer.WriteBits(0, 1)
	writer.WriteBits(1, 1)
	writer.WriteBits(10, 8)
	writer.WriteBits(255, 8)
	writer.WriteBits(1000, 10)
	writer.WriteBits(50000, 16)
	writer.WriteBits(9999999, 32)
	writer.FlushBits()

	fmt.Printf("wrote %d bytes\n", writer.BytesWritten())

	reader := serialize.NewBitReader(writer.Data())
	fmt.Println(reader.ReadBits(1))
	fmt.Println(reader.ReadBits(1))
	fmt.Println(reader.ReadBits(8))
	fmt.Println(reader.ReadBits(8))
	fmt.Println(reader.ReadBits(10))
	fmt.Println(reader.ReadBits(16))
	fmt.Println(reader.ReadBits(32))

}

Output:
wrote 10 bytes
0
1
10
255
1000
50000
9999999

func NewBitWriter 

func NewBitWriter(buffer []byte) *BitWriter

NewBitWriter creates a bit writer that fills the given buffer with bitpacked data. The buffer size must be a multiple of 8 bytes. Buffer sizes are effectively unlimited, because bit counts are stored in 64 bit signed integers.

func (*BitWriter) AlignBits 

func (w *BitWriter) AlignBits() int

AlignBits returns the number of align bits that would be written, if an align was written right now. The result is in [0,7], where 0 means the stream is already byte aligned.

func (*BitWriter) BitsAvailable 

func (w *BitWriter) BitsAvailable() int64

BitsAvailable returns the number of bits still available to write. For example, if the buffer size is 8 bytes and 10 bits have been written, 54 bits are available.

func (*BitWriter) BitsWritten 

func (w *BitWriter) BitsWritten() int64

BitsWritten returns the number of bits written so far.

func (*BitWriter) BytesWritten 

func (w *BitWriter) BytesWritten() int64

BytesWritten returns the number of bytes flushed to memory, which is the number of bits written rounded up to the next byte. This is effectively the size of the packet you should send after you have finished bitpacking values with this writer.

IMPORTANT: Call FlushBits first, otherwise you risk missing the last word of data.

func (*BitWriter) Data 

func (w *BitWriter) Data() []byte

Data returns the written portion of the buffer: the first BytesWritten bytes of the buffer passed to the writer.

IMPORTANT: Call FlushBits first, otherwise you risk missing the last word of data.

func (*BitWriter) FlushBits 

func (w *BitWriter) FlushBits()

FlushBits flushes any remaining bits in the scratch to memory. Call this once after you have finished writing bits. The flush stores a full qword: the buffer size is a multiple of 8 so this stays in bounds, and bytes past the written data are only ever written as zeros.

FlushBits ends the write: writing more bits after a mid-stream flush corrupts the stream, because the flushed partial word cannot be resumed.

func (*BitWriter) Reset 

func (w *BitWriter) Reset(buffer []byte)

Reset points the bit writer at a buffer and clears all write state, allowing a single writer to be reused without allocation. The buffer size must be a multiple of 8 bytes.

func (*BitWriter) WriteAlign 

func (w *BitWriter) WriteAlign()

WriteAlign pads the bit stream with zeros so the bit index becomes a multiple of 8. This is useful if you want to write data that should be byte aligned, for example an array of bytes or a string. If the current bit index is already a multiple of 8, nothing is written.

func (*BitWriter) WriteBits 

func (w *BitWriter) WriteBits(value uint32, bits int)

WriteBits writes the low order bits of value to the buffer, without padding to the nearest byte. A boolean value writes just 1 bit, a value in [0,31] can be written with just 5 bits and so on. bits must be in [1,32]; bits of value above that count are ignored. Panics if the write would go past the end of the buffer.

IMPORTANT: When you have finished writing, call FlushBits, otherwise the last word of data will not get flushed to memory!

func (*BitWriter) WriteBytes 

func (w *BitWriter) WriteBytes(data []byte)

WriteBytes writes an array of bytes to the bit stream. Use this when you have to copy a large block of data into your bitstream. Faster than writing each byte via WriteBits(value, 8), because the aligned middle of the data is block copied into the buffer without bitpacking. The writer must be aligned to a byte boundary: call WriteAlign first.

type MeasureStream 

type MeasureStream struct {
	// contains filtered or unexported fields
}

MeasureStream counts how many bits it would take to serialize something, without writing any data. It acts like a write stream (IsWriting is true), so a unified serialize function measures the exact same fields it would write.

When the serialization includes alignment to byte boundaries, the measurement is an estimate rather than exact, because the true pad depends on where the object lands in the final bit stream. The estimate is guaranteed to be conservative: every align is counted as the worst case 7 bits.

Example

Measure how many bytes an object needs before allocating a buffer and writing it. 

package main

import (
	"fmt"

	"github.com/mas-bandwidth/serialize.go"
)

type Vector struct {
	X, Y, Z float32
}

func (v *Vector) Serialize(stream serialize.Stream) error {
	stream.SerializeFloat32(&v.X)
	stream.SerializeFloat32(&v.Y)
	stream.SerializeFloat32(&v.Z)
	return stream.Err()
}

type RigidBody struct {
	Position       Vector
	LinearVelocity Vector
	AtRest         bool
}

// Serialize is a unified serialize function: the same code writes, reads and measures.
func (b *RigidBody) Serialize(stream serialize.Stream) error {
	stream.SerializeObject(&b.Position)
	stream.SerializeBool(&b.AtRest)
	if !b.AtRest {
		stream.SerializeObject(&b.LinearVelocity)
	} else if stream.IsReading() {
		b.LinearVelocity = Vector{}
	}
	return stream.Err()
}

func main() {
	body := RigidBody{
		Position:       Vector{X: 10, Y: 20, Z: 30},
		LinearVelocity: Vector{X: 1, Y: 2, Z: 3},
	}

	measureStream := serialize.NewMeasureStream()
	if err := body.Serialize(measureStream); err != nil {
		panic(err)
	}

	// round up to a multiple of 8 bytes, as the write stream requires
	bufferSize := int(measureStream.BytesProcessed()+7) / 8 * 8

	writeStream := serialize.NewWriteStream(make([]byte, bufferSize))
	if err := body.Serialize(writeStream); err != nil {
		panic(err)
	}
	writeStream.Flush()

	fmt.Printf("measured %d bytes, wrote %d bytes\n", measureStream.BytesProcessed(), writeStream.BytesProcessed())

}

Output:
measured 25 bytes, wrote 25 bytes

func NewMeasureStream 

func NewMeasureStream() *MeasureStream

NewMeasureStream creates a measure stream. The zero value is also ready to use.

func (*MeasureStream) AlignBits 

func (s *MeasureStream) AlignBits() int

AlignBits returns the worst case align of 7 bits. The number of bits required for alignment depends on where an object lands in the final bit stream, so the measurement is conservative.

func (*MeasureStream) BitsProcessed 

func (s *MeasureStream) BitsProcessed() int64

BitsProcessed returns the number of bits measured so far.

func (*MeasureStream) BytesProcessed 

func (s *MeasureStream) BytesProcessed() int64

BytesProcessed returns the number of bits measured so far, rounded up to the next byte.

func (*MeasureStream) Context 

func (s *MeasureStream) Context() any

Context returns the context value set on the stream. It may be nil.

func (*MeasureStream) Err 

func (s *MeasureStream) Err() error

Err returns the first error latched on the stream, or nil.

func (*MeasureStream) IsReading 

func (s *MeasureStream) IsReading() bool

IsReading returns false.

func (*MeasureStream) IsWriting 

func (s *MeasureStream) IsWriting() bool

IsWriting returns true: a measure stream behaves like a write stream so that unified serialize functions measure exactly what they would write.

func (*MeasureStream) Reset 

func (s *MeasureStream) Reset()

Reset clears the measured bit count and any latched error. The context is kept.

func (*MeasureStream) SerializeAlign 

func (s *MeasureStream) SerializeAlign() error

SerializeAlign measures the conservative worst case align of 7 bits.

func (*MeasureStream) SerializeBits 

func (s *MeasureStream) SerializeBits(value *uint32, bits int) error

SerializeBits measures bits, which must be in [1,32].

func (*MeasureStream) SerializeBits64 

func (s *MeasureStream) SerializeBits64(value *uint64, bits int) error

SerializeBits64 measures bits, which must be in [1,64].

func (*MeasureStream) SerializeBool 

func (s *MeasureStream) SerializeBool(value *bool) error

SerializeBool measures 1 bit.

func (*MeasureStream) SerializeBytes 

func (s *MeasureStream) SerializeBytes(data []byte) error

SerializeBytes measures a worst case align plus the data bytes.

func (*MeasureStream) SerializeCompressedFloat32 

func (s *MeasureStream) SerializeCompressedFloat32(value *float32, min, max, resolution float32) error

SerializeCompressedFloat32 measures the bits required for the quantized range.

func (*MeasureStream) SerializeFloat32 

func (s *MeasureStream) SerializeFloat32(value *float32) error

SerializeFloat32 measures 32 bits.

func (*MeasureStream) SerializeFloat64 

func (s *MeasureStream) SerializeFloat64(value *float64) error

SerializeFloat64 measures 64 bits.

func (*MeasureStream) SerializeInt 

func (s *MeasureStream) SerializeInt(value *int32, min, max int32) error

SerializeInt measures the bits required for the range [min,max]. Like a write, the value must be in range or ErrValueOutOfRange is returned.

func (*MeasureStream) SerializeInt64 

func (s *MeasureStream) SerializeInt64(value *int64, min, max int64) error

SerializeInt64 measures the bits required for the range [min,max]. Like a write, the value must be in range or ErrValueOutOfRange is returned.

func (*MeasureStream) SerializeIntRelative 

func (s *MeasureStream) SerializeIntRelative(previous int32, current *int32) error

SerializeIntRelative measures the encoding of *current relative to previous, exactly as SerializeIntRelative on a write stream would encode it. previous must be less than *current or ErrValueOutOfRange is returned.

func (*MeasureStream) SerializeObject 

func (s *MeasureStream) SerializeObject(object Serializer) error

SerializeObject measures an object that implements Serializer.

func (*MeasureStream) SerializeString 

func (s *MeasureStream) SerializeString(value *string, bufferSize int) error

SerializeString measures the length prefix, a worst case align, and the string bytes. Like a write, the string must fit in bufferSize-1 bytes or ErrValueOutOfRange is returned.

func (*MeasureStream) SerializeUint8 

func (s *MeasureStream) SerializeUint8(value *uint8) error

SerializeUint8 measures 8 bits.

func (*MeasureStream) SerializeUint16 

func (s *MeasureStream) SerializeUint16(value *uint16) error

SerializeUint16 measures 16 bits.

func (*MeasureStream) SerializeUint32 

func (s *MeasureStream) SerializeUint32(value *uint32) error

SerializeUint32 measures 32 bits.

func (*MeasureStream) SerializeUint64 

func (s *MeasureStream) SerializeUint64(value *uint64) error

SerializeUint64 measures 64 bits.

func (*MeasureStream) SerializeWideString 

func (s *MeasureStream) SerializeWideString(value *string, bufferSize int) error

SerializeWideString measures the length prefix plus 32 bits per code point. Like a write, the string must fit in bufferSize-1 code points or ErrValueOutOfRange is returned.

func (*MeasureStream) SetContext 

func (s *MeasureStream) SetContext(context any)

SetContext sets a context value that serialize functions can retrieve with Context.

type ReadStream 

type ReadStream struct {
	// contains filtered or unexported fields
}

ReadStream reads bitpacked data from a buffer. It wraps BitReader with bounds and range checking on every read, so maliciously crafted packets fail with errors instead of panicking or smuggling out of range values, and implements the Stream interface so unified serialize functions can read with it.

The zero value is an exhausted stream: create a ReadStream with NewReadStream, or Reset one onto a buffer before use.

func NewReadStream 

func NewReadStream(data []byte) *ReadStream

NewReadStream creates a read stream that reads the bitpacked data in the given slice. Any slice length is supported. For the fastest reads, keep at least 7 bytes of slack in the backing array beyond the data: see BitReader for details.

func (*ReadStream) AlignBits 

func (s *ReadStream) AlignBits() int

AlignBits returns the number of bits required to align the stream to the next byte boundary, in [0,7].

func (*ReadStream) BitsProcessed 

func (s *ReadStream) BitsProcessed() int64

BitsProcessed returns the number of bits read so far.

func (*ReadStream) BytesProcessed 

func (s *ReadStream) BytesProcessed() int64

BytesProcessed returns the number of bits read so far, rounded up to the next byte.

func (*ReadStream) Context 

func (s *ReadStream) Context() any

Context returns the context value set on the stream. It may be nil.

func (*ReadStream) Err 

func (s *ReadStream) Err() error

Err returns the first error latched on the stream, or nil.

func (*ReadStream) IsReading 

func (s *ReadStream) IsReading() bool

IsReading returns true.

func (*ReadStream) IsWriting 

func (s *ReadStream) IsWriting() bool

IsWriting returns false.

func (*ReadStream) Reset 

func (s *ReadStream) Reset(data []byte)

Reset points the stream at a data slice and clears all read state including any latched error, allowing a single stream to be reused without allocation. The context is kept.

func (*ReadStream) SerializeAlign 

func (s *ReadStream) SerializeAlign() error

SerializeAlign skips ahead to the next byte boundary, verifying that the padding bits are zero. Nonzero padding fails with ErrAlign, which typically means the read and write serialize functions don't match.

func (*ReadStream) SerializeBits 

func (s *ReadStream) SerializeBits(value *uint32, bits int) error

SerializeBits reads bits into *value. bits must be in [1,32]. On success *value is in [0,(1<<bits)-1]; on failure it is left unmodified.

func (*ReadStream) SerializeBits64 

func (s *ReadStream) SerializeBits64(value *uint64, bits int) error

SerializeBits64 reads bits into *value. bits must be in [1,64]. Values wider than 32 bits are read as the low dword first, then the high remainder.

func (*ReadStream) SerializeBool 

func (s *ReadStream) SerializeBool(value *bool) error

SerializeBool reads a boolean value from one bit.

func (*ReadStream) SerializeBytes 

func (s *ReadStream) SerializeBytes(data []byte) error

SerializeBytes aligns the stream to a byte boundary and block copies len(data) bytes into data.

func (*ReadStream) SerializeCompressedFloat32 

func (s *ReadStream) SerializeCompressedFloat32(value *float32, min, max, resolution float32) error

SerializeCompressedFloat32 reads a quantized floating point value. On success *value is guaranteed to be in [min,max]; quantized values smuggled into the bit headroom fail with ErrValueOutOfRange.

func (*ReadStream) SerializeFloat32 

func (s *ReadStream) SerializeFloat32(value *float32) error

SerializeFloat32 reads an uncompressed 32 bit floating point value.

func (*ReadStream) SerializeFloat64 

func (s *ReadStream) SerializeFloat64(value *float64) error

SerializeFloat64 reads an uncompressed 64 bit floating point value.

func (*ReadStream) SerializeInt 

func (s *ReadStream) SerializeInt(value *int32, min, max int32) error

SerializeInt reads a signed integer into *value. On success *value is guaranteed to be in [min,max]; values smuggled into the bit headroom of the range fail with ErrValueOutOfRange.

func (*ReadStream) SerializeInt64 

func (s *ReadStream) SerializeInt64(value *int64, min, max int64) error

SerializeInt64 reads a signed 64 bit integer into *value. On success *value is guaranteed to be in [min,max]; values smuggled into the bit headroom of the range fail with ErrValueOutOfRange.

func (*ReadStream) SerializeIntRelative 

func (s *ReadStream) SerializeIntRelative(previous int32, current *int32) error

SerializeIntRelative reads *current relative to previous. The value is reconstructed in the unsigned domain, so it wraps rather than overflowing when previous is near the top of the int32 range. The absolute fallback encoding validates that the decoded value is greater than previous.

func (*ReadStream) SerializeObject 

func (s *ReadStream) SerializeObject(object Serializer) error

SerializeObject reads an object that implements Serializer.

func (*ReadStream) SerializeString 

func (s *ReadStream) SerializeString(value *string, bufferSize int) error

SerializeString reads a string of fewer than bufferSize bytes into *value with a single allocation. On failure *value is left unmodified.

func (*ReadStream) SerializeUint8 

func (s *ReadStream) SerializeUint8(value *uint8) error

SerializeUint8 reads an unsigned 8 bit integer.

func (*ReadStream) SerializeUint16 

func (s *ReadStream) SerializeUint16(value *uint16) error

SerializeUint16 reads an unsigned 16 bit integer.

func (*ReadStream) SerializeUint32 

func (s *ReadStream) SerializeUint32(value *uint32) error

SerializeUint32 reads an unsigned 32 bit integer.

func (*ReadStream) SerializeUint64 

func (s *ReadStream) SerializeUint64(value *uint64) error

SerializeUint64 reads an unsigned 64 bit integer as the low dword then the high dword.

func (*ReadStream) SerializeWideString 

func (s *ReadStream) SerializeWideString(value *string, bufferSize int) error

SerializeWideString reads a string stored as 32 bits per code point into *value. Code points that are not valid (surrogates or values above 0x10FFFF) fail with ErrValueOutOfRange. On failure *value is left unmodified.

func (*ReadStream) SetContext 

func (s *ReadStream) SetContext(context any)

SetContext sets a context value that serialize functions can retrieve with Context.

type Serializer 

type Serializer interface {
	Serialize(stream Stream) error
}

Serializer is the interface implemented by objects that serialize themselves to a stream. Write one Serialize method per type and it works for write, read and measure.

Return an error to abort serialization: the standard pattern is to call serialize methods for each field and return stream.Err() at the end, adding your own validation errors where needed.

IMPORTANT: A value that controls how much more work your serialize function does — a loop count or a continuation bit — must have its error checked before you use it. Once an error latches, serialize calls are no-ops that leave values unmodified, so a loop waiting for a serialized value to change spins forever on a truncated or malicious packet. Use Continue or Until for sentinel-driven loops.

type Stream 

type Stream interface {
	// IsWriting returns true if the stream writes or measures values (WriteStream and
	// MeasureStream), and false if it reads them.
	IsWriting() bool

	// IsReading returns true if the stream reads values (ReadStream).
	IsReading() bool

	// SerializeBits serializes the low order bits of an unsigned integer. bits must be
	// in [1,32]. A value in [0,31] can be serialized with just 5 bits and so on.
	SerializeBits(value *uint32, bits int) error

	// SerializeBits64 serializes the low order bits of a 64 bit unsigned integer.
	// bits must be in [1,64].
	SerializeBits64(value *uint64, bits int) error

	// SerializeInt serializes a signed integer in [min,max], writing only the number of
	// bits required to represent the range. On read the value is guaranteed to be in
	// [min,max] if the call succeeds.
	SerializeInt(value *int32, min, max int32) error

	// SerializeInt64 serializes a signed 64 bit integer in [min,max], writing only the
	// number of bits required to represent the range. The full 64 bit range is supported.
	SerializeInt64(value *int64, min, max int64) error

	// SerializeUint8 serializes an unsigned 8 bit integer.
	SerializeUint8(value *uint8) error

	// SerializeUint16 serializes an unsigned 16 bit integer.
	SerializeUint16(value *uint16) error

	// SerializeUint32 serializes an unsigned 32 bit integer.
	SerializeUint32(value *uint32) error

	// SerializeUint64 serializes an unsigned 64 bit integer.
	SerializeUint64(value *uint64) error

	// SerializeBool serializes a boolean value with one bit.
	SerializeBool(value *bool) error

	// SerializeFloat32 serializes an uncompressed 32 bit floating point value.
	SerializeFloat32(value *float32) error

	// SerializeFloat64 serializes an uncompressed 64 bit floating point value.
	SerializeFloat64(value *float64) error

	// SerializeCompressedFloat32 serializes a floating point value in [min,max] with the
	// given resolution, writing only the bits required for the quantized range. On write
	// the value is clamped into [min,max]; on read it is guaranteed to be in [min,max]
	// if the call succeeds.
	SerializeCompressedFloat32(value *float32, min, max, resolution float32) error

	// SerializeBytes serializes an array of bytes. The stream aligns to a byte boundary
	// first, then block copies the data. Both sides must know the length: it is not sent.
	SerializeBytes(data []byte) error

	// SerializeString serializes a string of fewer than bufferSize bytes: the length is
	// serialized in [0,bufferSize-1], the stream aligns to a byte boundary, then the
	// string bytes are block copied. bufferSize mirrors the C++ API, where a string with
	// its terminating null character must fit into the buffer, keeping streams
	// compatible between the two languages.
	SerializeString(value *string, bufferSize int) error

	// SerializeWideString serializes a string as 32 bits per code point, wire compatible
	// with serialize_wstring in the C++ library. The length is serialized in
	// [0,bufferSize-1] code points. On read, code points that are not valid (surrogates
	// or values above 0x10FFFF) fail with ErrValueOutOfRange.
	SerializeWideString(value *string, bufferSize int) error

	// SerializeAlign pads the stream with zero bits to the next byte boundary. On read
	// the padding is verified to be zero.
	SerializeAlign() error

	// SerializeObject serializes an object that implements Serializer.
	SerializeObject(object Serializer) error

	// SerializeIntRelative serializes an integer relative to a previous integer, using
	// fewer bits the closer the two values are. previous must be less than current.
	SerializeIntRelative(previous int32, current *int32) error

	// AlignBits returns the number of bits required to align the stream to the next byte
	// boundary, in [0,7]. MeasureStream always returns the conservative worst case 7.
	AlignBits() int

	// BitsProcessed returns the number of bits written to, read from or measured on
	// the stream.
	BitsProcessed() int64

	// BytesProcessed returns the number of bits processed rounded up to the next byte.
	// After writing, this is effectively the packet size.
	BytesProcessed() int64

	// Err returns the first error latched on the stream, or nil.
	Err() error

	// SetContext sets a context value on the stream. The context lets you pass data
	// through to your serialize functions, for example lookup tables or min/max ranges
	// needed to read and write values. It mirrors the context pointer in the C++
	// library and is unrelated to context.Context.
	SetContext(context any)

	// Context returns the context value set on the stream. It may be nil.
	Context() any
}

Stream is the unified serialization interface implemented by WriteStream, ReadStream and MeasureStream. It is the Go equivalent of the templated stream parameter in the C++ serialize library: write one serialize function against Stream and it handles write, read and measure.

All serialize methods take pointers so the same call reads or writes the value depending on the stream. Use IsWriting and IsReading inside serialize functions for logic that differs between the two, such as initializing fields that are not sent.

Errors are sticky: the first failure latches on the stream and every later serialize call returns it without touching the stream, so you can either check every call or serialize a whole object and check Err once at the end. The one rule: a value that controls a loop must have its error checked before the loop runs, because after an error the value is never updated again. See Continue and the Serializer documentation.

type WriteStream 

type WriteStream struct {
	// contains filtered or unexported fields
}

WriteStream writes bitpacked data to a buffer. It wraps BitWriter with overflow and range checking that returns errors instead of panicking, and implements the Stream interface so unified serialize functions can write with it.

The zero value is not usable: create a WriteStream with NewWriteStream, or Reset one onto a buffer before use.

func NewWriteStream 

func NewWriteStream(buffer []byte) *WriteStream

NewWriteStream creates a write stream that writes to the given buffer. The buffer size must be a multiple of 8 bytes, because the bit writer stores qwords to memory.

func (*WriteStream) AlignBits 

func (s *WriteStream) AlignBits() int

AlignBits returns the number of bits required to align the stream to the next byte boundary, in [0,7].

func (*WriteStream) BitsProcessed 

func (s *WriteStream) BitsProcessed() int64

BitsProcessed returns the number of bits written so far.

func (*WriteStream) BytesProcessed 

func (s *WriteStream) BytesProcessed() int64

BytesProcessed returns the number of bits written so far, rounded up to the next byte. This is effectively the packet size.

func (*WriteStream) Context 

func (s *WriteStream) Context() any

Context returns the context value set on the stream. It may be nil.

func (*WriteStream) Data 

func (s *WriteStream) Data() []byte

Data returns the written portion of the buffer: the packet you should send.

IMPORTANT: Call Flush first.

func (*WriteStream) Err 

func (s *WriteStream) Err() error

Err returns the first error latched on the stream, or nil.

func (*WriteStream) Flush 

func (s *WriteStream) Flush()

Flush flushes the last word of bits to memory. Always call this after you finish writing and before you call Data, or you risk truncating the last word of data. The flush ends the write: do not serialize more values after it.

func (*WriteStream) IsReading 

func (s *WriteStream) IsReading() bool

IsReading returns false.

func (*WriteStream) IsWriting 

func (s *WriteStream) IsWriting() bool

IsWriting returns true.

func (*WriteStream) Reset 

func (s *WriteStream) Reset(buffer []byte)

Reset points the stream at a buffer and clears all write state including any latched error, allowing a single stream to be reused without allocation. The context is kept.

func (*WriteStream) SerializeAlign 

func (s *WriteStream) SerializeAlign() error

SerializeAlign pads the stream with zero bits to the next byte boundary.

func (*WriteStream) SerializeBits 

func (s *WriteStream) SerializeBits(value *uint32, bits int) error

SerializeBits writes the low order bits of *value. bits must be in [1,32].

func (*WriteStream) SerializeBits64 

func (s *WriteStream) SerializeBits64(value *uint64, bits int) error

SerializeBits64 writes the low order bits of *value. bits must be in [1,64]. Values wider than 32 bits are written as the low dword first, then the high remainder.

func (*WriteStream) SerializeBool 

func (s *WriteStream) SerializeBool(value *bool) error

SerializeBool writes a boolean value with one bit.

func (*WriteStream) SerializeBytes 

func (s *WriteStream) SerializeBytes(data []byte) error

SerializeBytes aligns the stream to a byte boundary and block copies data into it.

func (*WriteStream) SerializeCompressedFloat32 

func (s *WriteStream) SerializeCompressedFloat32(value *float32, min, max, resolution float32) error

SerializeCompressedFloat32 quantizes *value into [min,max] at the given resolution and writes only the bits required for the quantized range. The value is clamped into [min,max] before quantization; the !>= / !<= clamp form forces NaN into range too.

func (*WriteStream) SerializeFloat32 

func (s *WriteStream) SerializeFloat32(value *float32) error

SerializeFloat32 writes an uncompressed 32 bit floating point value.

func (*WriteStream) SerializeFloat64 

func (s *WriteStream) SerializeFloat64(value *float64) error

SerializeFloat64 writes an uncompressed 64 bit floating point value.

func (*WriteStream) SerializeInt 

func (s *WriteStream) SerializeInt(value *int32, min, max int32) error

SerializeInt writes *value, which must be in [min,max], using only the bits required to represent the range. Returns ErrValueOutOfRange if it is not.

func (*WriteStream) SerializeInt64 

func (s *WriteStream) SerializeInt64(value *int64, min, max int64) error

SerializeInt64 writes *value, which must be in [min,max], using only the bits required to represent the range. Returns ErrValueOutOfRange if it is not.

func (*WriteStream) SerializeIntRelative 

func (s *WriteStream) SerializeIntRelative(previous int32, current *int32) error

SerializeIntRelative writes *current relative to previous, using fewer bits the closer the two values are. previous must be less than *current or ErrValueOutOfRange is returned. The difference is computed in the unsigned domain, so gaps wider than 2^31 wrap and fall through to the absolute 32 bit encoding.

func (*WriteStream) SerializeObject 

func (s *WriteStream) SerializeObject(object Serializer) error

SerializeObject writes an object that implements Serializer.

func (*WriteStream) SerializeString 

func (s *WriteStream) SerializeString(value *string, bufferSize int) error

SerializeString writes the length of *value in [0,bufferSize-1], aligns the stream to a byte boundary, then block copies the string bytes. Returns ErrValueOutOfRange if the string does not fit in bufferSize-1 bytes.

func (*WriteStream) SerializeUint8 

func (s *WriteStream) SerializeUint8(value *uint8) error

SerializeUint8 writes an unsigned 8 bit integer.

func (*WriteStream) SerializeUint16 

func (s *WriteStream) SerializeUint16(value *uint16) error

SerializeUint16 writes an unsigned 16 bit integer.

func (*WriteStream) SerializeUint32 

func (s *WriteStream) SerializeUint32(value *uint32) error

SerializeUint32 writes an unsigned 32 bit integer.

func (*WriteStream) SerializeUint64 

func (s *WriteStream) SerializeUint64(value *uint64) error

SerializeUint64 writes an unsigned 64 bit integer as the low dword then the high dword.

func (*WriteStream) SerializeWideString 

func (s *WriteStream) SerializeWideString(value *string, bufferSize int) error

SerializeWideString writes the length of *value in code points in [0,bufferSize-1], then each code point as 32 bits. Wire compatible with serialize_wstring in the C++ library. Returns ErrValueOutOfRange if the string does not fit in bufferSize-1 code points.

func (*WriteStream) SetContext 

func (s *WriteStream) SetContext(context any)

SetContext sets a context value that serialize functions can retrieve with Context.

Source Files

View all Source files

Directories

Path Synopsis
Command compat is the Go half of the cross language wire compatibility harness.
 Command compat is the Go half of the cross language wire compatibility harness.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.