README
¶
Marshal and Unmarshal Binary Formats in Go
The Go standard library features packages for converting between structs
and various message and file formats.
Packages encoding/json and encoding/xml are well known
for providing convenient functions Marshal() and Unmarshal()
that leverage the declarative nature of struct tags.
These functions are however missing from
encoding/binary,
leaving developers without an accessible and intuitive way
to work with binary formats.
This module supplies the ubiquitous Marshal() and Unmarshal()
function signatures
so that developers can define custom binary formats using only struct tags and
avoid custom bit manipulation.
User Story
As a Go developer implementing a binary message or file format,
I want a pair of functions "Marshal/Unmarshal" like those in "encoding/json"
that convert a struct into a series of bits in a byte slice and vice versa,
so that I can avoid the complexities of custom bit manipulation.
See the rest of the behaviour-driven specifications of this module below.
Binary Formats
Message and file formats specify how bits are arranged to encode information. Control over individual bits or groups smaller than a byte is often required to put together and take apart these binary structures.
Message Format Examples
Describing the anatomy of TCP/IP headers at the beginning of every internet datagram ("packet") are
- Section 3.1 of RFC 791 Internet Protocol, and
- Section 3.1 of RFC 793 Transmission Control Protocol.
File Format Examples
Binary file formats are not significantly different from message formats from an application developer's perspective. RFC 1952 describes the GZIP File Format Specification.
Working with Binary Formats in Go
The smallest data structures Go provides are
the basic type byte (alias of uint8, an unsigned 8-bit integer), and bool,
both eight bits long.
To manipulate data at a scale smaller than eight bits
would require the use of bitwise logical and shift operators such as
AND (&),
OR (|),
left shift (<<),
and right shift (>>).
Relevant Questions Posted on StackOverflow
Suggestions on StackOverflow are limited to the use of bitwise operators.
- Golang: Parse bit values from a byte
- Creating 8 bit binary data from 4,3, and 1 bit data in Golang
- How to pack the C bit field struct via encoding package in GO?
Behaviour-Driven Specifications
Feature: Marshal and Unmarshal
As a Go developer implementing a binary message or file format,
I want a pair of functions "Marshal/Unmarshal" like those in "encoding/json"
that convert a struct into a series of bits in a byte slice and vice versa,
so that I can avoid the complexities of custom bit manipulation.
Background:
# Ubiquitous language
Given a message or file "format"
"""
A format specifies how bits are arranged to encode information.
"""
And the format is a series of "bit fields"
"""
A bit field is one or more adjacent bits representing a value,
and should not be confused with struct fields.
"""
And adjacent bit fields are grouped into "words"
"""
A word is a series of bits that can be simultaneously processed
by a given computer architecture and programming language.
"""
# Define format-structs
And a format is represented by a type definition of a "format-struct"
And the format-struct nests one or more exported "word-structs"
And the words are tagged to indicate their lengths in number of bits
type RFC791InternetHeaderFormatWithoutOptions struct {
RFC791InternetHeaderFormatWord0 `word:"32"`
RFC791InternetHeaderFormatWord1 `word:"32"`
RFC791InternetHeaderFormatWord2 `word:"32"`
RFC791InternetHeaderFormatWord3 `word:"32"`
RFC791InternetHeaderFormatWord4 `word:"32"`
}
And the length of each word is a multiple of eight in the range [8, 64]
# Define word-structs
And each word-struct has exported field(s) corresponding to bit field(s)
And the fields are of unsigned integer or boolean types
And the fields are tagged to indicate the lengths of those bit fields
type RFC791InternetHeaderFormatWord0 struct {
Version uint8 `bitfield:"4"`
IHL uint8 `bitfield:"4"`
Precedence uint8 `bitfield:"3"`
Delay bool `bitfield:"1"`
Throughput bool `bitfield:"1"`
Reliability bool `bitfield:"1"`
Reserved uint8 `bitfield:"2"`
TotalLength uint16 `bitfield:"16"`
}
And the length of each bit field does not overflow the type of the field
"""
A bit field overflows a type
when it is long enough to represent values
outside the set of values of the type.
"""
And the sum of lengths of all fields is equal to the length of that word
Marshal
Scenario: Marshal a struct into a byte slice
Given a format-struct variable representing a binary message or file
internetHeader = RFC791InternetHeaderFormatWithoutOptions{
RFC791InternetHeaderFormatWord0{
Version: 4,
IHL: 5,
// ...
},
// ...
}
And the struct field values do not overflow corresponding bit fields
"""
A struct field value overflows its corresponding bit field
when it falls outside the range of values
that can be represented by that bit field given its length.
"""
When I pass to function Marshal() a pointer to that struct variable
var (
bytes []byte
e error
)
bytes, e = binary.Marshal(&internetHeader)
Then Marshal() should return a slice of bytes and a nil error
And I should see struct field values reflected as bits in those bytes
log.Printf("%08b", bytes)
// [01000101 ...]
log.Println(e == nil)
// true
And I should see that the lengths of the slice and the format are equal
"""
The length of a format is the sum of lengths of the words in it.
The length of a word is the sum of lengths of the bit fields in it.
"""
Unmarshal
Scenario: Unmarshal a byte slice into a struct
Given a format-struct type representing a binary message or file format
var internetHeader RFC791InternetHeaderFormatWithoutOptions
And a slice of bytes containing a binary message or file
var bytes []byte
// ...
log.Printf("%08b", bytes)
// [01000101 ...]
And the lengths of the slice and the format (measured in bits) are equal
When I pass to function Unmarshal() the slice of bytes as an argument
And I pass to the function a pointer to the struct as a second argument
e = binary.Unmarshal(bytes, &internetHeader)
Then Unmarshal() should return a nil error
And I should see struct field values matching the bits in those bytes
log.Println(e == nil)
// true
log.Println(internetHeader.RFC791InternetHeaderFormatWord0.Version)
// 4
log.Println(internetHeader.RFC791InternetHeaderFormatWord0.IHL)
// 5
Performance and Optimisation
This module is optimised for performance.
$ go test -cpuprofile cpu.prof -memprofile mem.prof -bench . -benchmem
goos: linux
goarch: arm64
pkg: github.com/encodingx/binary
BenchmarkMarshal-2 3435181 349.9 ns/op 64 B/op 6 allocs/op
BenchmarkUnmarshal-2 3004425 396.4 ns/op 64 B/op 8 allocs/op
PASS
ok github.com/encodingx/binary 3.288s
Documentation
¶
Index ¶
- Constants
- Variables
- func Marshal(iface interface{}) (bytes []byte, e error)
- func PutUvarint(buf []byte, x uint64) int
- func PutVarint(buf []byte, x int64) int
- func Read(r io.Reader, order ByteOrder, data interface{}) error
- func ReadUvarint(r io.ByteReader) (uint64, error)
- func ReadVarint(r io.ByteReader) (int64, error)
- func Size(v interface{}) int
- func Unmarshal(bytes []byte, iface interface{}) (e error)
- func Uvarint(buf []byte) (uint64, int)
- func Varint(buf []byte) (int64, int)
- func Write(w io.Writer, order ByteOrder, data interface{}) error
- type ByteOrder
Constants ¶
const ( MaxVarintLen16 = stdlib.MaxVarintLen16 MaxVarintLen32 = stdlib.MaxVarintLen32 MaxVarintLen64 = stdlib.MaxVarintLen64 )
Variables ¶
var ( BigEndian = stdlib.BigEndian LittleEndian = stdlib.LittleEndian )
Functions ¶
func PutUvarint ¶
func ReadUvarint ¶
func ReadUvarint(r io.ByteReader) (uint64, error)
func ReadVarint ¶
func ReadVarint(r io.ByteReader) (int64, error)
Source Files
Directories