msgpck

package module
v0.3.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 30, 2026 License: MIT Imports: 8 Imported by: 0

README

msgpck

CI Coverage Status Go Report Card Go Reference Quality Gate Status Vulnerabilities

A high-performance msgpack library for Go, optimized for database and serialization-heavy workloads. Zero external dependencies.

Why Another Msgpack Library?

Gives how standards proliferate vibes. s/standards/msgpack libraries/g

I built msgpck for tinykvs, a key-value store where msgpack encoding/decoding is on the hot path. I found issues with fuzz testing where vmihailenco would allocate excessively on big maps/arrays, and decided to check out the alternatives: shamaton/msgpack, hashicorp/go-msgpack, and tinylib/msgp. I didn't want code generation so tinylib/msgp was out. hashicorp/go-msgpack was slow. shamaton/msgpack was better but didn't compete with vmihailenco/msgpack on map[string]any performance. So I built msgpck focused on the common case: decoding known struct types and map[string]any with minimal allocations.

Performance

Benchmarks vs vmihailenco/msgpack (Apple M3 Max):

Struct Operations (using cached GetStructEncoder/GetStructDecoder)
Operation msgpck allocs
SmallStruct Encode 61 ns 1
MediumStruct Encode 226 ns 1
SmallStruct Decode 65 ns 2
SmallStruct Decode (zeroCopy: true) 50 ns 1
MediumStruct Decode 446 ns 13
MediumStruct Decode (zeroCopy: true) 338 ns 4
Large Struct Operations
Operation msgpck allocs
LargeStruct Encode (cached) 768 ns 9
LargeStruct Decode (cached) 1169 ns 30
LargeStruct Decode (zeroCopy: true) 839 ns 6
Generic Slice (1M items - simulates database column storage)
Operation msgpck allocs
1M int64 Encode 8.7 ms 1
1M int64 Decode 7.2 ms 3
1M uint16 Encode 8.6 ms 1
1M uint16 Decode 6.2 ms 2
Map Operations
Operation msgpck allocs
SmallMap Encode (Marshal) 82 ns 1
MediumMap Encode (Marshal) 251 ns 1
SmallMap Decode (Unmarshal) 336 ns 10
MediumMap Decode (Unmarshal) 1235 ns 40
StringMap Decode (Unmarshal) 532 ns 19

Run benchmarks yourself:

go test -bench=. -benchmem

Quick Start

import "github.com/freeeve/msgpck"

// Encode any value
data, _ := msgpck.Marshal(map[string]any{"name": "Alice", "age": 30})

// Decode to map
var m map[string]any
msgpck.Unmarshal(data, &m)

// Decode to struct
var user User
msgpck.Unmarshal(data, &user)

Key Features

Cached Struct Codecs

For hot paths, use cached codecs that avoid reflection on every call:

// Get cached encoder/decoder (created on first use, reused forever)
enc := msgpck.GetStructEncoder[User]()
dec := msgpck.GetStructDecoder[User](false)

// Encode - 0 allocations with pooled buffer
data, _ := enc.Encode(&user)

// Decode - minimal allocations
var user User
dec.Decode(data, &user)
Zero-Copy Mode

When your input buffer outlives the decoded result (common in databases), skip string allocations entirely:

// Get cached zero-copy decoder
dec := msgpck.GetStructDecoder[User](true)

// Strings point directly into 'data' - no copies
dec.Decode(data, &user)

Warning: Zero-copy strings are only valid while the input buffer exists. Copy strings if you need them to outlive the buffer.

Generic Type Support

The struct encoder/decoder fully supports Go generics:

type SortColumn[T cmp.Ordered] struct {
    Values   []T    `msgpack:"values"`
    MaxDocID uint32 `msgpack:"max_doc_id"`
}

// Works with any concrete type
enc := msgpck.GetStructEncoder[SortColumn[int64]]()
dec := msgpck.GetStructDecoder[SortColumn[int64]](false)
Supported Field Types

The cached struct codecs support:

  • Primitives: all int/uint sizes (8/16/32/64), float32/64, bool, string
  • Slices: []T for all primitive types, []string, []byte
  • Maps: map[string]string, map[string]any, nested maps
  • Nested structs: automatically handled
  • Pointers: pointer fields are supported

API Reference

Encoding
// Encode any Go value to msgpack (safe to retain, concurrent-safe)
msgpck.Marshal(v any) ([]byte, error)

// For hot paths: cached struct encoder
enc := msgpck.GetStructEncoder[MyStruct]()
enc.Encode(&src)         // safe to retain (1 alloc)
enc.EncodeWith(e, &src)  // zero-alloc with your own Encoder
Decoding

The API matches encoding/json:

// Decode to any type - structs, maps, slices, primitives
var user User
msgpck.Unmarshal(data, &user)

var m map[string]any
msgpck.Unmarshal(data, &m)

var s map[string]string
msgpck.Unmarshal(data, &s)

// For hot paths: cached struct decoder
dec := msgpck.GetStructDecoder[MyStruct](false)
dec.Decode(data, &dst)

// Zero-copy cached decoder (strings point into input buffer)
dec := msgpck.GetStructDecoder[MyStruct](true)
dec.Decode(data, &dst)
Timestamps

msgpck supports the msgpack timestamp extension type (-1). Times are encoded using the most compact format and decoded to UTC:

// Encode a time.Time
data := msgpck.MarshalTimestamp(time.Now())

// Decode back to time.Time (UTC)
t, _ := msgpck.UnmarshalTimestamp(data)

// Streaming API
enc := msgpck.NewEncoder(nil)
enc.EncodeTimestamp(time.Now())

dec := msgpck.NewDecoder(data)
t, _ := dec.DecodeTimestamp()

// Convert extension values
ext, _ := dec.DecodeExt()
if msgpck.IsTimestamp(ext) {
    t, _ := msgpck.ExtToTimestamp(ext)
}

Concurrency

All public APIs are concurrent-safe:

  • Marshal and Unmarshal functions use internal pools
  • GetStructEncoder[T](), GetStructDecoder[T](zeroCopy) return cached, thread-safe codecs
  • StructEncoder and StructDecoder instances are safe to use from multiple goroutines

When to Use msgpck vs vmihailenco/msgpack

Use msgpck when:

  • Encoding/decoding is on your hot path
  • You decode the same struct types repeatedly
  • You can benefit from zero-copy (database, network buffers)
  • You need minimal allocations

Use vmihailenco/msgpack when:

  • You need custom encoders/decoders for complex types
  • You're decoding unknown/dynamic schemas
  • Convenience matters more than raw speed

Compatibility

msgpck produces standard msgpack bytes. Data encoded with vmihailenco/msgpack decodes correctly with msgpck and vice versa.

License

MIT

Documentation

Index

Constants

View Source 
const (
	DefaultMaxStringLen = math.MaxInt32 // ~2GB, spec allows 2^32-1
	DefaultMaxBinaryLen = math.MaxInt32 // ~2GB, spec allows 2^32-1
	DefaultMaxArrayLen  = math.MaxInt32 // ~2B elements, spec allows 2^32-1
	DefaultMaxMapLen    = math.MaxInt32 // ~2B pairs, spec allows 2^32-1
	DefaultMaxExtLen    = math.MaxInt32 // ~2GB, spec allows 2^32-1
	DefaultMaxDepth     = 10000         // practical limit for nested structures
)

Default limits - aligned with msgpack spec (max 2^32-1) We use MaxInt32 for cross-platform safety (32-bit systems)

View Source 
const TimestampExtType int8 = -1

TimestampExtType is the extension type for msgpack timestamps.

Variables

View Source 
var (
	// ErrUnexpectedEOF is returned when the input ends before a complete value
	ErrUnexpectedEOF = errors.New("msgpck: unexpected end of input")

	// ErrInvalidFormat is returned when an invalid format byte is encountered
	ErrInvalidFormat = errors.New("msgpck: invalid format byte")

	// ErrStringTooLong is returned when a string exceeds MaxStringLen
	ErrStringTooLong = errors.New("msgpck: string exceeds maximum length")

	// ErrBinaryTooLong is returned when binary data exceeds MaxBinaryLen
	ErrBinaryTooLong = errors.New("msgpck: binary data exceeds maximum length")

	// ErrArrayTooLong is returned when an array exceeds MaxArrayLen
	ErrArrayTooLong = errors.New("msgpck: array exceeds maximum length")

	// ErrMapTooLong is returned when a map exceeds MaxMapLen
	ErrMapTooLong = errors.New("msgpck: map exceeds maximum length")

	// ErrExtTooLong is returned when ext data exceeds MaxExtLen
	ErrExtTooLong = errors.New("msgpck: ext data exceeds maximum length")

	// ErrMaxDepthExceeded is returned when nesting exceeds MaxDepth
	ErrMaxDepthExceeded = errors.New("msgpck: maximum nesting depth exceeded")

	// ErrTypeMismatch is returned when decoding into an incompatible type
	ErrTypeMismatch = errors.New("msgpck: type mismatch")

	// ErrNotPointer is returned when DecodeStruct is called with a non-pointer
	ErrNotPointer = errors.New("msgpck: decode target must be a pointer")

	// ErrNotStruct is returned when DecodeStruct is called with a non-struct pointer
	ErrNotStruct = errors.New("msgpck: decode target must be a pointer to struct")

	// ErrUnsupportedType is returned when encoding an unsupported Go type
	ErrUnsupportedType = errors.New("msgpck: unsupported type")
)

Functions

func ExtToTimestamp 

func ExtToTimestamp(ext Ext) (time.Time, error)

ExtToTimestamp converts an Ext value to time.Time. Returns ErrTypeMismatch if the extension type is not a timestamp. Returns the time in UTC.

func IsTimestamp 

func IsTimestamp(ext Ext) bool

IsTimestamp checks if an Ext value is a timestamp.

func Marshal 

func Marshal(v any) ([]byte, error)

Marshal encodes a Go value to msgpack bytes. The returned bytes are a copy and safe to retain.

func MarshalTimestamp 

func MarshalTimestamp(t time.Time) []byte

MarshalTimestamp encodes a time.Time to msgpack timestamp bytes. Returns a copy of the encoded bytes (safe to retain). The time is converted to UTC before encoding.

func Unmarshal 

func Unmarshal(data []byte, v any) error

Unmarshal decodes msgpack data into v. v must be a pointer to the target value.

Supported target types:

  • *struct: decoded from msgpack map
  • *map[K]V: decoded from msgpack map
  • *[]T: decoded from msgpack array
  • *string, *int64, *float64, *bool, *[]byte: decoded from msgpack primitives
  • *any: decoded to map[string]any, []any, or primitive types
  • *time.Time: decoded from msgpack timestamp extension

func UnmarshalTimestamp 

func UnmarshalTimestamp(data []byte) (time.Time, error)

UnmarshalTimestamp decodes msgpack timestamp bytes to time.Time. Returns the time in UTC. If the input has no timezone info, it is treated as UTC.

func UnmarshalWithConfig 

func UnmarshalWithConfig(data []byte, v any, cfg Config) error

UnmarshalWithConfig decodes msgpack data into v with custom config.

Types

type Config 

type Config struct {
	// MaxStringLen is the maximum allowed string length in bytes
	MaxStringLen int

	// MaxBinaryLen is the maximum allowed binary data length in bytes
	MaxBinaryLen int

	// MaxArrayLen is the maximum allowed array length (number of elements)
	MaxArrayLen int

	// MaxMapLen is the maximum allowed map length (number of key-value pairs)
	MaxMapLen int

	// MaxExtLen is the maximum allowed ext data length in bytes
	MaxExtLen int

	// MaxDepth is the maximum allowed nesting depth for arrays and maps
	MaxDepth int
}

Config controls decoder/encoder behavior and security limits

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults

func (Config) WithMaxArrayLen 

func (c Config) WithMaxArrayLen(n int) Config

WithMaxArrayLen returns a new Config with the specified MaxArrayLen

func (Config) WithMaxBinaryLen 

func (c Config) WithMaxBinaryLen(n int) Config

WithMaxBinaryLen returns a new Config with the specified MaxBinaryLen

func (Config) WithMaxDepth 

func (c Config) WithMaxDepth(n int) Config

WithMaxDepth returns a new Config with the specified MaxDepth

func (Config) WithMaxExtLen 

func (c Config) WithMaxExtLen(n int) Config

WithMaxExtLen returns a new Config with the specified MaxExtLen

func (Config) WithMaxMapLen 

func (c Config) WithMaxMapLen(n int) Config

WithMaxMapLen returns a new Config with the specified MaxMapLen

func (Config) WithMaxStringLen 

func (c Config) WithMaxStringLen(n int) Config

WithMaxStringLen returns a new Config with the specified MaxStringLen

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads MessagePack data from a byte slice. It provides zero-copy decoding where possible.

func NewDecoder 

func NewDecoder(data []byte) *Decoder

NewDecoder creates a new Decoder for the given data with default config

func NewDecoderWithConfig 

func NewDecoderWithConfig(data []byte, cfg Config) *Decoder

NewDecoderWithConfig creates a new Decoder with custom config

func (*Decoder) Decode 

func (d *Decoder) Decode() (Value, error)

Decode decodes a single MessagePack value. Returns a Value with zero-copy references to strings/binary data in the source.

func (*Decoder) DecodeAny 

func (d *Decoder) DecodeAny() (any, error)

DecodeAny decodes a MessagePack value directly into a Go native type. Returns: nil, bool, int64, uint64, float64, string, []byte, []any, map[string]any This is optimized to avoid intermediate Value structs.

func (*Decoder) DecodeFloat64Array added in v0.3.0 

func (d *Decoder) DecodeFloat64Array() ([]float64, error)

DecodeFloat64Array decodes an array of float64 values. More efficient than decoding elements individually.

func (*Decoder) DecodeInt64Array added in v0.3.0 

func (d *Decoder) DecodeInt64Array() ([]int64, error)

DecodeInt64Array decodes an array of int64 values. More efficient than decoding elements individually.

func (*Decoder) DecodeStringArray added in v0.3.0 

func (d *Decoder) DecodeStringArray() ([]string, error)

DecodeStringArray decodes an array of strings (with copy). More efficient than decoding elements individually.

func (*Decoder) DecodeStruct 

func (d *Decoder) DecodeStruct(v any) error

DecodeStruct decodes a msgpack map into a struct. v must be a pointer to a struct.

func (*Decoder) DecodeTimestamp 

func (d *Decoder) DecodeTimestamp() (time.Time, error)

DecodeTimestamp decodes a msgpack timestamp extension to time.Time. Returns the time in UTC. Returns ErrTypeMismatch if the value is not a timestamp extension.

func (*Decoder) DecodeUint64Array added in v0.3.0 

func (d *Decoder) DecodeUint64Array() ([]uint64, error)

DecodeUint64Array decodes an array of uint64 values. More efficient than decoding elements individually.

func (*Decoder) Position 

func (d *Decoder) Position() int

Position returns the current position in the data

func (*Decoder) Remaining 

func (d *Decoder) Remaining() int

Remaining returns the number of unread bytes

func (*Decoder) Reset 

func (d *Decoder) Reset(data []byte)

Reset resets the decoder to decode new data

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes MessagePack data to a byte buffer.

func NewEncoder 

func NewEncoder(capacity int) *Encoder

NewEncoder creates a new Encoder with the given initial capacity.

func NewEncoderBuffer 

func NewEncoderBuffer(buf []byte) *Encoder

NewEncoderBuffer creates an Encoder that writes to an existing buffer. The buffer will be grown as needed.

func (*Encoder) Bytes 

func (e *Encoder) Bytes() []byte

Bytes returns the encoded bytes.

func (*Encoder) Encode 

func (e *Encoder) Encode(v any) error

Encode encodes any Go value to msgpack.

func (*Encoder) EncodeArrayHeader 

func (e *Encoder) EncodeArrayHeader(length int)

EncodeArrayHeader writes the header for an array of the given length. Call this, then encode each element.

func (*Encoder) EncodeBinary 

func (e *Encoder) EncodeBinary(v []byte)

EncodeBinary writes binary data

func (*Encoder) EncodeBool 

func (e *Encoder) EncodeBool(v bool)

EncodeBool writes a boolean value

func (*Encoder) EncodeExt 

func (e *Encoder) EncodeExt(extType int8, data []byte)

EncodeExt writes extension data

func (*Encoder) EncodeFloat32 

func (e *Encoder) EncodeFloat32(v float32)

EncodeFloat32 writes a float32 value

func (*Encoder) EncodeFloat64 

func (e *Encoder) EncodeFloat64(v float64)

EncodeFloat64 writes a float64 value

func (*Encoder) EncodeFloat64Array added in v0.3.0 

func (e *Encoder) EncodeFloat64Array(arr []float64)

EncodeFloat64Array encodes a slice of float64 values. More efficient than encoding elements individually.

func (*Encoder) EncodeInt 

func (e *Encoder) EncodeInt(v int64)

EncodeInt writes an int64 using the most compact format

func (*Encoder) EncodeInt64Array added in v0.3.0 

func (e *Encoder) EncodeInt64Array(arr []int64)

EncodeInt64Array encodes a slice of int64 values. More efficient than encoding elements individually.

func (*Encoder) EncodeMapHeader 

func (e *Encoder) EncodeMapHeader(length int)

EncodeMapHeader writes the header for a map of the given length. Call this, then encode each key-value pair.

func (*Encoder) EncodeNil 

func (e *Encoder) EncodeNil()

EncodeNil writes a nil value

func (*Encoder) EncodeString 

func (e *Encoder) EncodeString(v string)

EncodeString writes a string value (zero-copy using unsafe)

func (*Encoder) EncodeStringArray added in v0.3.0 

func (e *Encoder) EncodeStringArray(arr []string)

EncodeStringArray encodes a slice of strings. More efficient than encoding elements individually.

func (*Encoder) EncodeStringBytes 

func (e *Encoder) EncodeStringBytes(v []byte)

EncodeStringBytes writes a string from bytes

func (*Encoder) EncodeTimestamp 

func (e *Encoder) EncodeTimestamp(t time.Time)

EncodeTimestamp encodes a time.Time as a msgpack timestamp extension. Uses the most compact format that can represent the timestamp:

  • Timestamp 32: seconds only (no nanoseconds), fits in uint32
  • Timestamp 64: nanoseconds + seconds fits in 34-bit unsigned
  • Timestamp 96: full range with 64-bit signed seconds

The time is converted to UTC before encoding.

func (*Encoder) EncodeUint 

func (e *Encoder) EncodeUint(v uint64)

EncodeUint writes a uint64 using the most compact format

func (*Encoder) EncodeUint64Array added in v0.3.0 

func (e *Encoder) EncodeUint64Array(arr []uint64)

EncodeUint64Array encodes a slice of uint64 values. More efficient than encoding elements individually.

func (*Encoder) EncodeValue 

func (e *Encoder) EncodeValue(v *Value)

EncodeValue writes a Value

func (*Encoder) Len 

func (e *Encoder) Len() int

Len returns the length of encoded data.

func (*Encoder) Reset 

func (e *Encoder) Reset()

Reset resets the encoder for reuse.

type Ext 

type Ext struct {
	Type int8
	Data []byte // points into source
}

Ext represents MessagePack extension data

type KV 

type KV struct {
	Key   []byte // points into source
	Value Value
}

KV represents a key-value pair in a map. Key is a []byte pointing into the source buffer for zero-copy.

type StructDecoder 

type StructDecoder[T any] struct {
	// contains filtered or unexported fields
}

StructDecoder is a pre-registered decoder for a specific struct type. Create once, reuse many times for best performance.

func GetStructDecoder 

func GetStructDecoder[T any](zeroCopy bool) *StructDecoder[T]

GetStructDecoder returns a cached decoder for type T. Creates one if it doesn't exist. If zeroCopy is true, strings point directly into the input buffer and are only valid while the buffer exists.

func (*StructDecoder[T]) Decode 

func (sd *StructDecoder[T]) Decode(data []byte, dst *T) error

Decode decodes msgpack data into the struct. This is the fast path - single pass, no intermediate allocations.

func (*StructDecoder[T]) DecodeWith added in v0.3.0 

func (sd *StructDecoder[T]) DecodeWith(d *Decoder, data []byte, dst *T) error

DecodeWith decodes msgpack data using a provided decoder. Use this to avoid pool overhead when you manage your own decoder. The decoder will be reset with the provided data before decoding.

func (*StructDecoder[T]) ZeroCopy 

func (sd *StructDecoder[T]) ZeroCopy() *StructDecoder[T]

ZeroCopy returns a zero-copy version of this decoder. Strings will point directly into the input buffer - no allocations! WARNING: Decoded strings are only valid while the input buffer exists. Use this when the input buffer outlives the decoded struct (e.g., from database).

type StructEncoder 

type StructEncoder[T any] struct {
	// contains filtered or unexported fields
}

StructEncoder is a pre-registered encoder for a specific struct type. Create once, reuse many times for best performance.

func GetStructEncoder 

func GetStructEncoder[T any]() *StructEncoder[T]

GetStructEncoder returns a cached encoder for type T.

func (*StructEncoder[T]) Encode 

func (se *StructEncoder[T]) Encode(src *T) ([]byte, error)

Encode encodes the struct to msgpack bytes. The returned bytes are safe to retain. This method is safe for concurrent use. For zero-allocation encoding, use EncodeWith with your own Encoder.

func (*StructEncoder[T]) EncodeWith added in v0.2.0 

func (se *StructEncoder[T]) EncodeWith(e *Encoder, src *T) error

EncodeWith encodes using a provided encoder. Use this to avoid pool overhead when you manage your own encoder. The encoder is NOT reset - call e.Reset() before if needed.

type Type 

type Type uint8

Type represents the type of a decoded MessagePack value 

const (
	TypeNil Type = iota
	TypeBool
	TypeInt
	TypeUint
	TypeFloat32
	TypeFloat64
	TypeString
	TypeBinary
	TypeArray
	TypeMap
	TypeExt
)

func (Type) String 

func (t Type) String() string

String returns the string representation of the type

type Value 

type Value struct {
	Type    Type
	Bool    bool
	Int     int64
	Uint    uint64
	Float32 float32
	Float64 float64
	Bytes   []byte  // string/binary - points into source
	Array   []Value // array elements
	Map     []KV    // map key-value pairs
	Ext     Ext     // extension data
}

Value represents a decoded MessagePack value. For strings and binary data, Bytes points directly into the source buffer (zero-copy). The caller must not modify the source buffer while Value is in use.

func (*Value) AsBool 

func (v *Value) AsBool() bool

AsBool returns the value as bool. Panics if Type != TypeBool.

func (*Value) AsBytes 

func (v *Value) AsBytes() []byte

AsBytes returns Bytes directly (zero-copy reference to source).

func (*Value) AsFloat64 

func (v *Value) AsFloat64() float64

AsFloat64 returns the value as float64. Works for float32/64 and int/uint.

func (*Value) AsInt 

func (v *Value) AsInt() int64

AsInt returns the value as int64. Works for TypeInt and TypeUint.

func (*Value) AsString 

func (v *Value) AsString() string

AsString returns Bytes as a string. This allocates.

func (*Value) AsUint 

func (v *Value) AsUint() uint64

AsUint returns the value as uint64. Works for TypeInt and TypeUint.

func (*Value) Get 

func (v *Value) Get(key []byte) *Value

Get returns the value for a key in a map (linear search). Returns nil if not found or not a map.

func (*Value) GetString 

func (v *Value) GetString(key string) *Value

GetString is a convenience for Get with a string key.

func (*Value) Index 

func (v *Value) Index(i int) *Value

Index returns the i-th element of an array. Panics if out of bounds or not an array.

func (*Value) IsNil 

func (v *Value) IsNil() bool

IsNil returns true if the value is nil

func (*Value) Len 

func (v *Value) Len() int

Len returns the length of array/map/string/binary.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.