fastmsgpack

README

fastmsgpack

Fastmsgpack is a Golang msgpack decoder. It is fast, but lacks features you might need. It can be used in combination with other msgpack libraries for e.g. encoding.

Pros:

• It is very fast to query a list of fields from a nested msgpack structure.
• It is zero copy for strings and []byte.
• No reflection usage when decoding.

Cons:

• It can only encode Go builtin types.
• The return value might contain pointers to the original data, so you can't modify the input data until you're done with the return value.
• It uses unsafe (to cast []byte to string without copying).
• It can't deserialize into your structs.
• It only supports strings as map keys.
• It decodes all ints as a Go int, including 64 bit ones, so it doesn't work on 32-bit platforms.

Supported extensions:

• It supports extension -1 and decodes such values into a time.Time.
• It supports extension -128 (interned strings). I didn't find any documentation for it, but I've based it on https://github.com/vmihailenco/msgpack. You can pass a dict to the decoder and it will replace indexes into that dict with the strings from the dict.
• It introduces extension 17 which wraps map and arrays. Because extension headers contain the byte-length, partial decoding can efficiently skip over those values.
• It introduces extension 18 which is like a Switch statement inside the data. (Whether that's a good idea is up for debate.) We use this to pack data for multiple locales in one value.
• It introduces extension 19 which encodes void. When decoding a void as for example a map value the key and value are treated as non-existent.
• It introduces extension 20 which injects a piece of msgpack in that place. It's a placeholder for data to be specified when decoding.

Returned types

Decode returns an any, which is either int, float32, float64, string, []byte, []any, map[string]any or time.Time.

(*Resolver).Resolve returns a list of such anys, one for each field requested.

Example

r := fastmsgpack.NewResolver([]string{"person.properties.firstName", "person.properties.age"}, nil)
found, err := r.Resolve(data)
firstName, ok := found[0].(string)
age, ok := found[1].(int)

Documentation

Overview

Package fastmsgpack is a msgpack decoder. See the README.

Index

• Variables
• func Canonical(dst, data []byte, eo EncodeOptions, opts ...DecodeOption) ([]byte, error)
• func Decode(data []byte, opts ...DecodeOption) (any, error)
• func Encode(dst []byte, v any) ([]byte, error)
• func LengthEncode(dst, data []byte) ([]byte, error)
• func Size(data []byte) (int, error)
• func SplitArray(data []byte) ([][]byte, error)
• func SplitMap(data []byte, dict *Dict) ([]string, [][]byte, error)
• type DecodeOption
  • func WithDict(dict *Dict) DecodeOption
  • func WithFlavorSelector(field, value uint) DecodeOption
  • func WithInjection(field uint, msgpack []byte) DecodeOption
• type Decoder
  • func NewDecoder(data []byte, opts ...DecodeOption) *Decoder
  • func (d *Decoder) Break() error
  • func (d *Decoder) DecodeArrayLen() (int, error)
  • func (d *Decoder) DecodeBool() (bool, error)
  • func (d *Decoder) DecodeFloat32() (float32, error)
  • func (d *Decoder) DecodeFloat64() (float64, error)
  • func (d *Decoder) DecodeInt() (int, error)
  • func (d *Decoder) DecodeMapLen() (int, error)
  • func (d *Decoder) DecodeRaw() ([]byte, error)
  • func (d *Decoder) DecodeString() (string, error)
  • func (d *Decoder) DecodeTime() (time.Time, error)
  • func (d *Decoder) DecodeValue() (any, error)
  • func (d *Decoder) PeekType() ValueType
  • func (d *Decoder) Skip() error
• type Dict
  • func MakeDict(dict []string) *Dict
• type EncodeOptions
  • func (o EncodeOptions) Encode(dst []byte, v any) ([]byte, error)
• type Extension
  • func (e Extension) AppendMsgpack(dst []byte) ([]byte, error)
  • func (e Extension) MarshalMsgpack() ([]byte, error)
• type FlavorBuilder
  • func NewFlavorBuilder(field uint) FlavorBuilder
  • func (f *FlavorBuilder) AddCase(match uint, b []byte)
  • func (f FlavorBuilder) MarshalMsgpack() ([]byte, error)
  • func (f *FlavorBuilder) SetElse(b []byte)
• type Resolver
  • func NewResolver(fields []string, opts ...DecodeOption) (*Resolver, error)
  • func (r *Resolver) AddArrayResolver(field string, sub *Resolver) (int, error)
  • func (r *Resolver) Describe() ([]string, map[string]SubresolverDescription)
  • func (r *Resolver) Resolve(data []byte, opts ...DecodeOption) (foundFields []any, retErr error)
  • func (r *Resolver) Select(dst, data []byte) (_ []byte, retErr error)
• type SubresolverDescription
• type ValueType
  • func DecodeType(data []byte) ValueType
  • func (t ValueType) String() string

Variables

var ErrVoid = internal.ErrVoid

Functions

func Canonical

func Canonical(dst, data []byte, eo EncodeOptions, opts ...DecodeOption) ([]byte, error)

func Decode

func Decode(data []byte, opts ...DecodeOption) (any, error)

Decode the given data (with the optional given dictionary). Any []byte and string in the return value might point into memory from the given data. Don't modify the input data until you're done with the return value. The dictionary is optional and can be nil.

func Encode

func Encode(dst []byte, v any) ([]byte, error)

Encode calls EncodeOptions.Decode with the default options.

func LengthEncode

func LengthEncode(dst, data []byte) ([]byte, error)

LengthEncode injects a length-encoding extension before every map and array to make skipping over it faster. The result is appended to dst and returned. dst can be nil.

func Size

func Size(data []byte) (int, error)

Size returns the number of bytes the first entry in the given msgpack data is.

func SplitArray

func SplitArray(data []byte) ([][]byte, error)

SplitArray splits a msgpack array into the msgpack chunks of its components. The returned slices point into the given data.

func SplitMap

func SplitMap(data []byte, dict *Dict) ([]string, [][]byte, error)

SplitMap splits a msgpack map into string-keys and the msgpack-values. It does not decode the values. The returned slices point into the given data.

Types

type DecodeOption

type DecodeOption func(*internal.DecodeOptions)

func WithDict

func WithDict(dict *Dict) DecodeOption

func WithFlavorSelector

func WithFlavorSelector(field, value uint) DecodeOption

func WithInjection

func WithInjection(field uint, msgpack []byte) DecodeOption

WithInjection replaces any encountered extension 20 encoding number $field with the given msgpack data.

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder gives a low-level api for stepping through msgpack data. Any []byte and string in return values might point into memory from the given data. Don't modify the input data until you're done with the return value.

func NewDecoder

func NewDecoder(data []byte, opts ...DecodeOption) *Decoder

NewDecoder initializes a new Decoder.

func (*Decoder) Break

func (d *Decoder) Break() error

Break out of the map or array we're currently in. This can only be called before the last element of the array/map is read, because otherwise you'd break out one level higher.

func (*Decoder) DecodeArrayLen

func (d *Decoder) DecodeArrayLen() (int, error)

func (*Decoder) DecodeBool

func (d *Decoder) DecodeBool() (bool, error)

func (*Decoder) DecodeFloat32

func (d *Decoder) DecodeFloat32() (float32, error)

func (*Decoder) DecodeFloat64

func (d *Decoder) DecodeFloat64() (float64, error)

func (*Decoder) DecodeInt

func (d *Decoder) DecodeInt() (int, error)

func (*Decoder) DecodeMapLen

func (d *Decoder) DecodeMapLen() (int, error)

func (*Decoder) DecodeRaw

func (d *Decoder) DecodeRaw() ([]byte, error)

DecodeRaw decodes the next value enough to know its length and returns the msgpack data for it while skipping over it.

func (*Decoder) DecodeString

func (d *Decoder) DecodeString() (string, error)

func (*Decoder) DecodeTime

func (d *Decoder) DecodeTime() (time.Time, error)

func (*Decoder) DecodeValue

func (d *Decoder) DecodeValue() (any, error)

DecodeValue decodes the next value in the msgpack data. Return types are: nil, bool, int, float32, float64, string, []byte, time.Time, []any, map[string]any or Extension.

func (*Decoder) PeekType

func (d *Decoder) PeekType() ValueType

PeekType returns the type of next entry without changing the state of the Decoder. PeekType returning another value than TypeInvalid does not guarantee decoding it will succeed.

func (*Decoder) Skip

func (d *Decoder) Skip() error

type Dict

type Dict struct {
	Strings []string
	// contains filtered or unexported fields
}

Dict is a dictionary for smaller msgpack. Instead of putting the string into the binary data, we use the number of the entry in the dictionary. Dictionaries should be the same between encoders and decoders. Adding new entries at the end is safe, as long as all decoders have the new dict before trying to decode newly encoded msgpack.

func MakeDict

func MakeDict(dict []string) *Dict

MakeDict prepares a dictionary.

type EncodeOptions

type EncodeOptions struct {
	CompactInts bool
	Dict        map[string]int
}

func (EncodeOptions) Encode

func (o EncodeOptions) Encode(dst []byte, v any) ([]byte, error)

Encode appends the msgpack representation to dst and returns the result.

type Extension

type Extension struct {
	Data []byte
	Type int8
}

func (Extension) AppendMsgpack

func (e Extension) AppendMsgpack(dst []byte) ([]byte, error)

func (Extension) MarshalMsgpack

func (e Extension) MarshalMsgpack() ([]byte, error)

type FlavorBuilder

type FlavorBuilder struct {
	// contains filtered or unexported fields
}

FlavorBuilder helps create an encoded extension 18. The flavor extension is like a switch statement inside your data.

When decoding with WithFlavorSelector(1, 2) we will return x if it decodes the result of NewFlavorBuilder(1).AddCase(2, x).

You are expected to cover all possible cases when building. Using WithFlavorSelector(1, 5) without having called AddCase(5, x) or SetElse() is undefined behavior.

func NewFlavorBuilder

func NewFlavorBuilder(field uint) FlavorBuilder

func (*FlavorBuilder) AddCase

func (f *FlavorBuilder) AddCase(match uint, b []byte)

func (FlavorBuilder) MarshalMsgpack

func (f FlavorBuilder) MarshalMsgpack() ([]byte, error)

func (*FlavorBuilder) SetElse

func (f *FlavorBuilder) SetElse(b []byte)

type Resolver

type Resolver struct {
	// contains filtered or unexported fields
}

func NewResolver

func NewResolver(fields []string, opts ...DecodeOption) (*Resolver, error)

NewResolver prepares a new resolver. It can be reused for multiple Resolve calls. You can't query the same field twice. You can't even query a child of something else you request (e.g. both "person.properties" and "person.properties.age"). This is the only reason NewResolver might return an error. The dictionary is optional and can be nil.

func (*Resolver) AddArrayResolver

func (r *Resolver) AddArrayResolver(field string, sub *Resolver) (int, error)

AddArrayResolver allows resolving inside array fields. For example like this pseudocode: `r.AddArrayResolve("person.addresses", NewResolver(["street"]))`. It returns the offset in the return value from Resolve(), which will be of type [][]any. AddArrayResolver can not be called concurrently with itself or Resolve. The dict that was given to the subresolver is not used.

r, err := NewResolver([]string{"person.properties.age"}, nil)
sub, err := NewResolver([]string{"street", "number"}, nil)
addrOffset, err := r.AddArrayResolver("person.addresses", sub)
found, err := r.Resolve(msgpackData)
age := found[0] // e.g. 5
addresses := found[addrOffset] // e.g. [][]any{[]any{"Main Street", 1}, []any{"Second Street", 2}}

func (*Resolver) Describe

func (r *Resolver) Describe() ([]string, map[string]SubresolverDescription)

Describe returns which fields and subresolvers were registered to this Resolver. The returned values should not be modified.

func (*Resolver) Resolve

func (r *Resolver) Resolve(data []byte, opts ...DecodeOption) (foundFields []any, retErr error)

Resolve scans through the given data and returns an array with the fields you've requested from this Resolver. Any []byte and string in the return value might point into memory from the given data. Don't modify the input data until you're done with the return value.

func (*Resolver) Select

func (r *Resolver) Select(dst, data []byte) (_ []byte, retErr error)

Select returns a new msgpack containing only the requested fields. The result is appended to dst and returned. dst can be nil.

type SubresolverDescription

type SubresolverDescription struct {
	Fields       []string
	Subresolvers map[string]SubresolverDescription
	Index        int
}

type ValueType

type ValueType uint8

const (
	TypeInvalid ValueType = iota
	TypeNil
	TypeBool
	TypeInt
	TypeFloat32
	TypeFloat64
	TypeString
	TypeBinary
	TypeArray
	TypeMap
	TypeTimestamp
	TypeFlavorSelector
	TypeVoid
	TypeInjection
	TypeUnknownExtension
)

func DecodeType

func DecodeType(data []byte) ValueType

func (ValueType) String

func (t ValueType) String() string