voxa

README

Voxa

Voxa is a binary-compact message format suitable for delivery Go types over the wire with minimal memory usage. It removes all meta-data and encodes into a binary format where a struct fields are simply marked by a id value, this makes it highly unsuitable for map types.

Voxa uses id tags as the means of identifying fields to be encoded and fields which would receive said encoding, where associated types must match.

Supported Types

Voxa has working support for most Go types as listed below:

• bool
• string
• int/uint
• int8/uint8/byte
• int16/uint16
• int32/uint32
• int64/uint64
• float32/float64
• Struct
• []byte
• []{string, uint8/16/32/64, int8/16/32/64, float32/64, Struct}

In voxa, Maps are special in time, they do not contain any meta-data like structs about the fields, hence when voxa decodes an encoded map, it uses the id values has keys. Hence its requires more work to get such information properly, which makes the use of struct's more suitable.

Install

go get -u github.com/wirekit/voxa

Example

record := struct {
    Age        int      `id:"1"`
    Name       string   `id:"2"`
    Address    string   `id:"3"`
    OtherNames []string `id:"4"`
}{
    Age:        20,
    Name:       "bob",
    Address:    "20. Classy Street",
    OtherNames: []string{"Rick Woss", "Ross Rics", "Frilino Felioi"},
}

var codec codecs.RecordCodec
encoded, err := codec.NativeToBinary(record, []byte{})
if err != nil {
    log.Fatal(err)
}

res := &(struct {
    Age        int      `id:"1"`
    Name       string   `id:"2"`
    Address    string   `id:"3"`
    OtherNames []string `id:"4"`
}{})

err = codec.BinaryToNative(encoded, reflect.ValueOf(res))
if err != nil {
    log.Fatal(err)
}

if !reflect.DeepEqual(*res, record) {
    log.Fatal("not matching")
}

Documentation

Index

• Constants
• Variables
• type Atom
  • func (a Atom) String() string
• type Codec
• type CodecTextual
• type FieldID
• type HeaderCodec

Constants

const (
	// IDTagName specifies the giving name for tagging struct fields
	// with, to mark a field as matching a giving id from a struct to
	// be converted or one to be deserialized with binary stream.
	IDTagName = "id"
)

Variables

var (
	// MaxBlockCount is the maximum number of data items allowed in a single
	// block that will be decoded from a binary stream, whether when reading
	// blocks to decode an array or a map, or when reading blocks from an OCF
	// stream. This check is to ensure decoding binary data will not cause the
	// library to over allocate RAM, potentially creating a denial of service on
	// the system.
	//
	// If a particular application needs to decode binary Voxa data that
	// potentially has more data items in a single block, then this variable may
	// be modified at your discretion.
	MaxBlockCount = int64(math.MaxInt32)

	// MaxBlockSize is the maximum number of bytes that will be allocated for a
	// single block of data items when decoding from a binary stream. This check
	// is to ensure decoding binary data will not cause the library to over
	// allocate RAM, potentially creating a denial of service on the system.
	//
	// If a particular application needs to decode binary Voxa data that
	// potentially has more bytes in a single block, then this variable may be
	// modified at your discretion.
	MaxBlockSize = int64(math.MaxInt32)

	// MaxIBU8Count is the maximum number of items a []uint8/[]int8/[]byte/[]bool can contain
	// per block, with respect to the maximum allowed block size in MaxBlockSize.
	MaxIBU8Count = MaxBlockSize / 1

	// MaxIBU16Count is the maximum number of items a []uint16/[]int16 can contain
	// per block, with respect to the maximum allowed block size in MaxBlockSize.
	MaxIBU16Count = MaxBlockSize / 2

	// MaxIBU32Count is the maximum number of items a []uint32/[]int32/[]float32 can contain
	// per block, with respect to the maximum allowed block size in MaxBlockSize.
	MaxIBU32Count = MaxBlockSize / 4

	// MaxIBU64 is the maximum number of items a []uint64/[]int64/[]float64 can contain
	// per block, with respect to the maximum allowed block size in MaxBlockSize.
	MaxIBU64Count = MaxBlockSize / 8

	// MaxICU128 is the maximum number of items a []complex128 can contain
	// per block, with respect to the maximum allowed block size in MaxBlockSize.
	MaxICU128Count = MaxBlockSize / 16
)

Types

type Atom

type Atom uint8

Atom is a int8 type declaration to represent different acceptable and convertible data types.

const (
	Invalid Atom = iota
	Text
	Bit
	Int
	Int8
	Int16
	Int32
	Int64
	UInt
	UInt8
	UInt16
	UInt32
	UInt64
	Boolean
	Float32
	Float64
	Bytes
	List
	Record
	Time
)

constants of all Type types.

func (Atom) String

func (a Atom) String() string

type Codec

type Codec interface {
	// BinaryToNative takes giving byte slice and attempts to
	// convert binary into native.
	BinaryToNative([]byte) (interface{}, FieldID, error)

	// NativeToBinary will receive a native value, which it encodes
	// into the provided byte slice, returning provided byte slice
	// with new length.
	NativeToBinary(interface{}, FieldID, []byte) ([]byte, error)
}

Codec defines a interface type which exposes conversion methods for types.

type CodecTextual

type CodecTextual interface {
	Codec

	// TextualToNative takes giving byte slice containing text version
	// and attempts to convert into native.
	TextualToNative([]byte) (interface{}, error)

	// MarshalTextualToNative takes giving byte slice and attempts to
	// unmarshal/deserialize it's content into provided type. This exists
	// to allow underline implementation specify the native type to be
	// converted to from a textual representation.
	MarshalTextualToNative([]byte, interface{}) error

	// NativeToTextual takes giving native value and converts into bytes,
	// writing into provided byte slice and returns provided byte slice
	// and returning provided byte slice with new length.
	NativeToTextual(interface{}, []byte) ([]byte, error)
}

CodecTextual defines a type which include textual to native and vice-versal conversion methods.

type FieldID

type FieldID int8

FieldID sets a int8 type which is used to represent the ID attached to a field name.

type HeaderCodec

type HeaderCodec interface {
	// FieldToBinary transforms giving field name string and it's unique id
	// into desired representation containing a type bit stored in
	// the provided slice which is returned.
	FieldToBinary(string, FieldID, Atom, []byte) ([]byte, error)

	// BinaryToField attempts to transforms provided byte slice
	// into a field name and Atom flag to both represent the
	// type desired and its type.
	BinaryToField([]byte) (string, FieldID, Atom, error)
}

HeaderCodec defines a interface which exposes two methods to derive the representation of giving field name in a readable format. It is meant to have types define how they wish the meta to represent their associated field with type details.