Help us shape the future of Go! Take the 2026 Go Developer Survey

msgpraw

package module
v0.0.0-...-039af76 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2026 License: MIT Imports: 4 Imported by: 0

README

msgpraw

CI Go Reference Go Report Card

A low-level MessagePack reader and writer for Go, designed for zero-allocation decoding. msgpraw does not unmarshal values into Go types — instead it returns the raw payload bytes alongside the format tag, leaving big-endian decoding to the caller. That keeps the reader allocation-free and lets callers decode lazily, only paying for what they actually use.

Features

  • Zero allocations on the reader's success path (verified by TestReader_NoAllocs and benchmarks).
  • Minimal allocations on the writer (zero with a pre-sized buffer).
  • Full MessagePack format coverage: nil, bool, all int/uint widths, float32/float64, fixstr/str8/str16/str32, bin8/bin16/bin32, fixarray/array16/array32, fixmap/map16/map32, fixext1..16, ext8/ext16/ext32, posfixint, negfixint.
  • Both auto-sized writers (smallest format that fits) and explicit fixed-format writers.
  • Pre-allocated sentinel errors — error paths don't allocate either.

Install

go get github.com/marino39/msgpraw

Requires Go 1.20 or later.

Reader

import "github.com/marino39/msgpraw"

r := &msgpraw.MsgpReader{Buff: payload}
for {
    tag, count, data, err := r.Read()
    if err == msgpraw.EOF {
        break
    }
    if err != nil {
        // ErrTruncated or ErrUnknownType
        return err
    }
    switch tag {
    case msgpraw.Int64:
        // data is the 8-byte big-endian payload
        v := int64(binary.BigEndian.Uint64(data))
        _ = v
    case msgpraw.Array16, msgpraw.Array32:
        // count is the number of elements; call Read again `count` times
        _ = count
    case msgpraw.Map16, msgpraw.Map32:
        // count is the pair count; call Read 2*count times
        _ = count
    }
}
Read return values

Read() returns (Type, int, []byte, error):

Tag family int []byte
Nil, True, False, PosFixInt, NegFixInt 0 nil (value carried by tag byte)
Scalar (Int*, Uint*, Float*) 0 raw big-endian payload bytes
String/binary (FixStr, Str*, Bin*) 0 raw bytes
Ext (FixExt*, Ext*) 0 [extType, dataBytes...] — caller does data[0], data[1:]
Collections (FixArray/Array*, FixMap/Map*) element/pair count remainder of buffer; caller calls Read() repeatedly

The returned []byte is a sub-slice of the input buffer — not a copy. Don't write to it; copy if the caller needs ownership.

Skip
err := r.Skip()

Skip() advances past the next value without inspecting it (still O(1) per scalar; for collections you must call Skip once per element to fully consume them).

Errors
Error When
EOF (io.EOF) The buffer is exhausted between values.
ErrTruncated The buffer ends mid-value (truncated length prefix or data).
ErrUnknownType The leading byte is not a defined MessagePack format.

All errors are pre-allocated package-level sentinels. Compare with errors.Is.

Writer

w := &msgpraw.MsgpWriter{Buff: make([]byte, 0, 1024)}

// Auto-sized: pick the smallest spec format that fits.
_ = w.WriteString("hello")        // FixStr
_ = w.WriteArray(2)               // FixArray
_ = w.WriteInt(42)                // Int64
_ = w.WriteFloat64(3.14159)
_ = w.WriteExt(0x05, []byte{1,2}) // picks FixExt2

// Explicit: pin the wire format.
_ = w.WriteStr16("longer payload")
_ = w.WriteArray32(1_000_000)
_ = w.WriteFixExt4(0x07, []byte{1, 2, 3, 4})

send(w.Buff)
Auto-sized vs. explicit
Auto-sized Explicit variants
WriteString(s) WriteFixStr / WriteStr8 / WriteStr16 / WriteStr32
WriteBytes(b) WriteBin8 / WriteBin16 / WriteBin32
WriteArray(n) WriteFixArray / WriteArray16 / WriteArray32
WriteMap(n) WriteFixMap / WriteMap16 / WriteMap32
WriteExt(t, d) WriteFixExt1/2/4/8/16 / WriteExt8 / WriteExt16 / WriteExt32

Auto-sized methods choose the smallest format that fits the input and return a sentinel error if the input exceeds the largest variant. Explicit methods return a range error when the input doesn't fit the named format.

WriteInt / WriteUint always emit Int64 / Uint64 — use WritePosFixInt, WriteNegFixInt, or WriteInt8..64 for compact forms. WriteNegFixInt accepts only -32..-1 per spec.

Zero-alloc writes

For zero-allocation writes, pre-size Buff so it doesn't have to grow:

buf := make([]byte, 0, 4096)
w := msgpraw.MsgpWriter{Buff: buf}
// ... writes ...
buf = w.Buff // grown if you wrote more than 4096 bytes total

The numeric writers use binary.BigEndian.AppendUint* (Go 1.19+) directly on Buff — no temporary slices.

Benchmarks

On an Apple M4 Max (go test -bench . -benchmem -run=^$):

BenchmarkReader_AllTags-16    7,376,100    139.6 ns/op    0 B/op    0 allocs/op
BenchmarkWriter_AllTags-16   65,632,200     18.4 ns/op    0 B/op    0 allocs/op

BenchmarkReader_AllTags exercises every supported tag in a single pass. BenchmarkWriter_AllTags writes a representative mix of tags to a pre-sized buffer.

Testing

go test ./...                                # unit tests
go test -race -cover ./...                   # race detector + coverage
go test -run TestReader_NoAllocs ./...       # zero-alloc assertion
go test -bench . -benchmem -run=^$ ./...     # benchmarks
go test -run=^$ -fuzz=FuzzReader -fuzztime=30s .     # fuzz the reader (malformed input)
go test -run=^$ -fuzz=FuzzRoundTrip -fuzztime=30s .  # fuzz writer -> reader roundtrip

The fuzz targets assert the reader never panics, returns only io.EOF / ErrTruncated / ErrUnknownType, and never hands back a payload outside its input buffer; crashers land in testdata/fuzz/.

Non-goals

  • Marshalling/unmarshalling Go types. This is a raw codec — bring your own binary.BigEndian.Uint* calls, or layer a typed codec on top.
  • Streaming io.Reader/io.Writer. The buffer-based API is what makes zero-allocation reads possible. Streaming would require an internal buffer.
  • Predefined extension types (Timestamp, etc.). Easy to layer on top of WriteExt / the ext payload format.

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	EOF            = io.EOF
	ErrTruncated   = errors.New("msgpraw: truncated input")
	ErrUnknownType = errors.New("msgpraw: unknown msgp type")
)
View Source 
var (
	ErrPosFixIntRange = errors.New("msgpraw: value out of PosFixInt range (0..127)")
	ErrNegFixIntRange = errors.New("msgpraw: value out of NegFixInt range (-32..-1)")
	ErrFixStrRange    = errors.New("msgpraw: string length out of FixStr range (0..31)")
	ErrFixArrayRange  = errors.New("msgpraw: array length out of FixArray range (0..15)")
	ErrFixMapRange    = errors.New("msgpraw: map length out of FixMap range (0..15)")
	ErrFixExtSize     = errors.New("msgpraw: data length does not match FixExt format size")
	ErrStr8Range      = errors.New("msgpraw: string length out of Str8 range (0..255)")
	ErrStr16Range     = errors.New("msgpraw: string length out of Str16 range (0..65535)")
	ErrStr32Range     = errors.New("msgpraw: string length out of Str32 range (0..2^32-1)")
	ErrBin8Range      = errors.New("msgpraw: data length out of Bin8 range (0..255)")
	ErrBin16Range     = errors.New("msgpraw: data length out of Bin16 range (0..65535)")
	ErrBin32Range     = errors.New("msgpraw: data length out of Bin32 range (0..2^32-1)")
	ErrArray16Range   = errors.New("msgpraw: array length out of Array16 range (0..65535)")
	ErrArray32Range   = errors.New("msgpraw: array length out of Array32 range (0..2^32-1)")
	ErrMap16Range     = errors.New("msgpraw: map length out of Map16 range (0..65535)")
	ErrMap32Range     = errors.New("msgpraw: map length out of Map32 range (0..2^32-1)")
	ErrExt8Range      = errors.New("msgpraw: data length out of Ext8 range (0..255)")
	ErrExt16Range     = errors.New("msgpraw: data length out of Ext16 range (0..65535)")
	ErrExt32Range     = errors.New("msgpraw: data length out of Ext32 range (0..2^32-1)")
	ErrStringTooLong  = errors.New("msgpraw: string too long for any Str format")
	ErrBytesTooLong   = errors.New("msgpraw: byte slice too long for any Bin format")
	ErrArrayTooLong   = errors.New("msgpraw: array too long for any Array format")
	ErrMapTooLong     = errors.New("msgpraw: map too long for any Map format")
	ErrExtTooLong     = errors.New("msgpraw: ext data too long for any Ext format")
)

Functions

This section is empty.

Types

type IMsgpReader 

type IMsgpReader interface {
	// Read reads the next msgp value.
	//
	// Return values:
	//   Type   - the msgp tag byte
	//   int    - element count for FixArray/Array16/Array32/FixMap/Map16/Map32; otherwise 0
	//   []byte - raw payload (sub-slice of Buff, no copy):
	//              scalars         : the big-endian-encoded value bytes
	//              FixExt*/Ext*    : [extType, dataBytes...] - caller does data[0], data[1:]
	//              FixArray/Array* : remainder of buffer; caller calls Read repeatedly
	//              FixMap/Map*     : remainder of buffer; caller calls Read 2*count times
	//              Nil/True/False/PosFixInt/NegFixInt: nil
	//   error  - io.EOF when buffer exhausted, ErrTruncated when input ends mid-value,
	//            ErrUnknownType when the tag is not a valid msgp format
	Read() (Type, int, []byte, error)
	Skip() error
}

type IMsgpWriter 

type IMsgpWriter interface {
	// Scalars
	WritePosFixInt(uint8) error
	WriteNegFixInt(int8) error
	WriteInt(int) error
	WriteInt8(int8) error
	WriteInt16(int16) error
	WriteInt32(int32) error
	WriteInt64(int64) error
	WriteUint(uint) error
	WriteUint8(uint8) error
	WriteUint16(uint16) error
	WriteUint32(uint32) error
	WriteUint64(uint64) error
	WriteFloat32(float32) error
	WriteFloat64(float64) error
	WriteNil() error
	WriteBool(bool) error

	// Auto-sized variable-length writers (pick the smallest format that fits).
	WriteString(string) error
	WriteBytes([]byte) error
	WriteArray(int) error
	WriteMap(int) error
	WriteExt(extType int8, data []byte) error

	// Explicit fixed-format variants. Each errors if the input does not fit
	// the named format.
	WriteFixStr(string) error
	WriteStr8(string) error
	WriteStr16(string) error
	WriteStr32(string) error

	WriteBin8([]byte) error
	WriteBin16([]byte) error
	WriteBin32([]byte) error

	WriteFixArray(int) error
	WriteArray16(int) error
	WriteArray32(int) error

	WriteFixMap(int) error
	WriteMap16(int) error
	WriteMap32(int) error

	WriteFixExt1(extType int8, data []byte) error
	WriteFixExt2(extType int8, data []byte) error
	WriteFixExt4(extType int8, data []byte) error
	WriteFixExt8(extType int8, data []byte) error
	WriteFixExt16(extType int8, data []byte) error
	WriteExt8(extType int8, data []byte) error
	WriteExt16(extType int8, data []byte) error
	WriteExt32(extType int8, data []byte) error
}

type MsgpReader 

type MsgpReader struct {
	Buff []byte
	Idx  int
}

func (*MsgpReader) Read 

func (r *MsgpReader) Read() (Type, int, []byte, error)

Read reads the next msgp value. See IMsgpReader for return value semantics.

func (*MsgpReader) Skip 

func (r *MsgpReader) Skip() error

Skip skips the next msgp value.

type MsgpWriter 

type MsgpWriter struct {
	Buff []byte
}

MsgpWriter appends msgpack-encoded values to Buff. Buff is exposed so callers can pre-size it: w := &MsgpWriter{Buff: make([]byte, 0, n)}.

func (*MsgpWriter) WriteArray 

func (w *MsgpWriter) WriteArray(n int) error

func (*MsgpWriter) WriteArray16 

func (w *MsgpWriter) WriteArray16(n int) error

func (*MsgpWriter) WriteArray32 

func (w *MsgpWriter) WriteArray32(n int) error

func (*MsgpWriter) WriteBin8 

func (w *MsgpWriter) WriteBin8(b []byte) error

func (*MsgpWriter) WriteBin16 

func (w *MsgpWriter) WriteBin16(b []byte) error

func (*MsgpWriter) WriteBin32 

func (w *MsgpWriter) WriteBin32(b []byte) error

func (*MsgpWriter) WriteBool 

func (w *MsgpWriter) WriteBool(b bool) error

func (*MsgpWriter) WriteBytes 

func (w *MsgpWriter) WriteBytes(b []byte) error

func (*MsgpWriter) WriteExt 

func (w *MsgpWriter) WriteExt(extType int8, data []byte) error

func (*MsgpWriter) WriteExt8 

func (w *MsgpWriter) WriteExt8(extType int8, data []byte) error

func (*MsgpWriter) WriteExt16 

func (w *MsgpWriter) WriteExt16(extType int8, data []byte) error

func (*MsgpWriter) WriteExt32 

func (w *MsgpWriter) WriteExt32(extType int8, data []byte) error

func (*MsgpWriter) WriteFixArray 

func (w *MsgpWriter) WriteFixArray(n int) error

func (*MsgpWriter) WriteFixExt1 

func (w *MsgpWriter) WriteFixExt1(extType int8, data []byte) error

func (*MsgpWriter) WriteFixExt2 

func (w *MsgpWriter) WriteFixExt2(extType int8, data []byte) error

func (*MsgpWriter) WriteFixExt4 

func (w *MsgpWriter) WriteFixExt4(extType int8, data []byte) error

func (*MsgpWriter) WriteFixExt8 

func (w *MsgpWriter) WriteFixExt8(extType int8, data []byte) error

func (*MsgpWriter) WriteFixExt16 

func (w *MsgpWriter) WriteFixExt16(extType int8, data []byte) error

func (*MsgpWriter) WriteFixMap 

func (w *MsgpWriter) WriteFixMap(n int) error

func (*MsgpWriter) WriteFixStr 

func (w *MsgpWriter) WriteFixStr(s string) error

func (*MsgpWriter) WriteFloat32 

func (w *MsgpWriter) WriteFloat32(f float32) error

func (*MsgpWriter) WriteFloat64 

func (w *MsgpWriter) WriteFloat64(f float64) error

func (*MsgpWriter) WriteInt 

func (w *MsgpWriter) WriteInt(i int) error

func (*MsgpWriter) WriteInt8 

func (w *MsgpWriter) WriteInt8(i int8) error

func (*MsgpWriter) WriteInt16 

func (w *MsgpWriter) WriteInt16(i int16) error

func (*MsgpWriter) WriteInt32 

func (w *MsgpWriter) WriteInt32(i int32) error

func (*MsgpWriter) WriteInt64 

func (w *MsgpWriter) WriteInt64(i int64) error

func (*MsgpWriter) WriteMap 

func (w *MsgpWriter) WriteMap(n int) error

func (*MsgpWriter) WriteMap16 

func (w *MsgpWriter) WriteMap16(n int) error

func (*MsgpWriter) WriteMap32 

func (w *MsgpWriter) WriteMap32(n int) error

func (*MsgpWriter) WriteNegFixInt 

func (w *MsgpWriter) WriteNegFixInt(i int8) error

func (*MsgpWriter) WriteNil 

func (w *MsgpWriter) WriteNil() error

func (*MsgpWriter) WritePosFixInt 

func (w *MsgpWriter) WritePosFixInt(i uint8) error

func (*MsgpWriter) WriteStr8 

func (w *MsgpWriter) WriteStr8(s string) error

func (*MsgpWriter) WriteStr16 

func (w *MsgpWriter) WriteStr16(s string) error

func (*MsgpWriter) WriteStr32 

func (w *MsgpWriter) WriteStr32(s string) error

func (*MsgpWriter) WriteString 

func (w *MsgpWriter) WriteString(s string) error

func (*MsgpWriter) WriteUint 

func (w *MsgpWriter) WriteUint(u uint) error

func (*MsgpWriter) WriteUint8 

func (w *MsgpWriter) WriteUint8(u uint8) error

func (*MsgpWriter) WriteUint16 

func (w *MsgpWriter) WriteUint16(u uint16) error

func (*MsgpWriter) WriteUint32 

func (w *MsgpWriter) WriteUint32(u uint32) error

func (*MsgpWriter) WriteUint64 

func (w *MsgpWriter) WriteUint64(u uint64) error

type Type 

type Type byte
const (
	PosFixInt    Type = 0x00
	PosFixIntMax Type = 0x7f
	FixMap       Type = 0x80
	FixMapMax    Type = 0x8f
	FixArray     Type = 0x90
	FixArrayMax  Type = 0x9f
	FixStr       Type = 0xa0
	FixStrMax    Type = 0xbf
	Nil          Type = 0xc0
	False        Type = 0xc2
	True         Type = 0xc3
	Bin8         Type = 0xc4
	Bin16        Type = 0xc5
	Bin32        Type = 0xc6
	Ext8         Type = 0xc7
	Ext16        Type = 0xc8
	Ext32        Type = 0xc9
	Float32      Type = 0xca
	Float64      Type = 0xcb
	Uint8        Type = 0xcc
	Uint16       Type = 0xcd
	Uint32       Type = 0xce
	Uint64       Type = 0xcf
	Int8         Type = 0xd0
	Int16        Type = 0xd1
	Int32        Type = 0xd2
	Int64        Type = 0xd3
	FixExt1      Type = 0xd4
	FixExt2      Type = 0xd5
	FixExt4      Type = 0xd6
	FixExt8      Type = 0xd7
	FixExt16     Type = 0xd8
	Str8         Type = 0xd9
	Str16        Type = 0xda
	Str32        Type = 0xdb
	Array16      Type = 0xdc
	Array32      Type = 0xdd
	Map16        Type = 0xde
	Map32        Type = 0xdf
	NegFixInt    Type = 0xe0
	NegFixIntMax Type = 0xff
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.