dynobuffers

README

Dyno Buffers

Codegen-less wrapper for FlatBuffers with get\set by name feature

Features

• Uses FlatBuffer to read\write values from\to byte array converting to the required type described in the Schema.
• No codegen, no compilers, no (de)serialization. Just fields description and get\set by name.
• In contrast to FlatBuffers tracks if the field was unset or initially not set
• Supported types
  • int32, int64, float32, float64, bool, string, byte
• Schema versioning
  • Any data written with Schema of any version will be correctly read using Schema of any other version
    • Written in old Schema, read in New Schema -> nil result on new field read, field considered as unset
    • Written in new Schema, read in old Schema -> no errors

Limitations

• Only 2 cases of schema modification are allowed: field rename and append fields to the end. This is necessary to have ability to read byte buffers in Schema of any version
• Written in New -> read in Old -> write in Old -> New fields are lost (todo)
• No arrays (todo)
• No nested objects (todo)
• No blobs (use strings?)?

Installation

go get github.com/untillpro/dynobuffers

Usage

• Describe Schema
  • By yaml. Field types:
    • int -> int32
    • long -> int64
    • float -> float32
    • double -> float64
    • bool -> bool
    • string -> string
    • byte -> byte

    var schemaStr = `
    name: string
    price: float
    quantity: int
    `
    schema, err := dynobuffers.YamlToSchema(schemaStr)
    if err != nil {
    	panic(err)
    }
    

  • Manually

    schema := dynobuffers.NewSchema()
    schema.AddField("name", dynobuffers.FieldTypeString)
    schema.AddField("price", dynobuffers.FieldTypeFloat)
    schema.AddField("quantity", dynobuffers.FieldTypeInt)
    

• Create Dyno Buffer using Schema

  b := dynobuffers.NewBuffer(schema)
  

• Set\modify fields according to the Schema

  b.Set("price", float32(0.123))
  b.Set("name", nil) // unset field
  

• To bytes array

  bytes := b.ToBytes()
  

• To JSON key-value go jsonStr := b.ToJSON()
• Read Buffer from bytes using Schema

  b = dynobuffers.ReadBuffer(bytes, schema)
  

• Work with Buffer

  value, ok := b.GetFloat32("price") // read typed. !ok -> field is unset or no such field in the schema. Works faster and takes less memory allocations than Get()
  b.Get("price") // read untyped. nil -> field is unset or no such field in the schema
  b.Set("price", nil) // set to nil means unset
  bytes = b.ToBytes()
  

• See dynobuffers_test.go for usage examples

Benchmarks

Description

• benchmarks\benchRead_test.go read benchmarks comparing to Avro, FlatBuffers, JSON
• benchmarks\benchReadWrite_test.go read and write benchmarks comparing to Avro, FlatBuffers, JSON
• Benchmarks naming
  • BenchmarkWrite... - benchmark read, change 1 field and write
  • BenchmarkRead... - benchmark read only
  • ...Simple... - schema with 3 fields, read and multiply 2 fields
  • ...Article... - schema with 123 fields
  • ...ReadFewFields... - read and multiply 2 fields
  • ...ReadAllFields... - read all fields

Results

• Schema of 123 fields, read and multiply 2 fields, change 1 field and write

DynoBuffers    	  	    200000	      8763 ns/op	    5240 B/op	      41 allocs/op
FlatBuffers         	    300000	      5702 ns/op	    3672 B/op	      19 allocs/op
Json                	     10000	    201575 ns/op	   29096 B/op	     857 allocs/op
LinkedInAvro                100000	     25173 ns/op	   11929 B/op	     176 allocs/op

• Schema of 3 fields, read and multiply 2 fields

DynoBuffers               20000000	      65.0 ns/op	       0 B/op	       0 allocs/op
FlatBuffers               50000000	      33.2 ns/op	       0 B/op	       0 allocs/op
Json                        500000	      2660 ns/op	     760 B/op	      21 allocs/op
LinkedInAvro               2000000	       859 ns/op	     744 B/op	      10 allocs/op

• Schema of 123 fields, read and multiple 2 fields

DynoBuffers     	  20000000	      83.2 ns/op	       0 B/op	       0 allocs/op
FlatBuffers          	 100000000	      26.5 ns/op	       0 B/op	       0 allocs/op
Json                 	     10000	    108663 ns/op	   13730 B/op	     604 allocs/op
LinkedInAvro         	    100000	     14936 ns/op	   11257 B/op	     149 allocs/op

• Schema of 123 fields, simply read all fields

DynoBuffers      	    300000	      4935 ns/op	      35 B/op	      11 allocs/op
FlatBuffers          	   1000000	      1421 ns/op	       0 B/op	       0 allocs/op
Json                 	     10000	    116814 ns/op	   14130 B/op	     603 allocs/op
LinkedInAvro         	    100000	     19187 ns/op	   11257 B/op	     149 allocs/op

To do

• Lists?
• Written in New -> read in Old -> write in Old -> New fields are kept.

Documentation

Index

• type Buffer
  • func NewBuffer(schema *Schema) *Buffer
  • func ReadBuffer(bytes []byte, schema *Schema) *Buffer
  • func (b *Buffer) Get(name string) interface{}
  • func (b *Buffer) GetBool(name string) (bool, bool)
  • func (b *Buffer) GetByte(name string) (byte, bool)
  • func (b *Buffer) GetDouble(name string) (float64, bool)
  • func (b *Buffer) GetFloat(name string) (float32, bool)
  • func (b *Buffer) GetInt(name string) (int32, bool)
  • func (b *Buffer) GetLong(name string) (int64, bool)
  • func (b *Buffer) GetString(name string) (string, bool)
  • func (b *Buffer) Set(name string, value interface{})
  • func (b *Buffer) ToBytes() []byte
  • func (b *Buffer) ToJSON() string
• type FieldType
• type Schema
  • func NewSchema() *Schema
  • func YamlToSchema(yamlStr string) (*Schema, error)
  • func (s *Schema) AddField(name string, ft FieldType)
  • func (s *Schema) HasField(name string) bool

Types

type Buffer

type Buffer struct {
	// contains filtered or unexported fields
}

Buffer is wrapper for FlatBuffers

func NewBuffer

func NewBuffer(schema *Schema) *Buffer

NewBuffer creates new empty Buffer

func ReadBuffer

func ReadBuffer(bytes []byte, schema *Schema) *Buffer

ReadBuffer creates Buffer from bytes using provided Schema

func (*Buffer) Get

func (b *Buffer) Get(name string) interface{}

Get returns stored field value by name. nil -> field is unset or no such field in the schema

func (*Buffer) GetBool

func (b *Buffer) GetBool(name string) (bool, bool)

GetBool returns bool value by name and if the schema contains the field and if the value was set to non-nil

func (*Buffer) GetByte

func (b *Buffer) GetByte(name string) (byte, bool)

GetByte returns byte value by name and if the schema contains the field and if the value was set to non-nil

func (*Buffer) GetDouble

func (b *Buffer) GetDouble(name string) (float64, bool)

GetDouble returns float64 value by name and if the schema contains the field and if the value was set to non-nil

func (*Buffer) GetFloat

func (b *Buffer) GetFloat(name string) (float32, bool)

GetFloat returns float32 value by name and if the schema contains the field and if the value was set to non-nil

func (*Buffer) GetInt

func (b *Buffer) GetInt(name string) (int32, bool)

GetInt returns int32 value by name and if the schema contains the field and the value was set to non-nil

func (*Buffer) GetLong

func (b *Buffer) GetLong(name string) (int64, bool)

GetLong returns int64 value by name and if the schema contains the field and if the value was set to non-nil

func (*Buffer) GetString

func (b *Buffer) GetString(name string) (string, bool)

GetString returns string value by name and if the schema contains the field and if the value was set to non-nil

func (*Buffer) Set

func (b *Buffer) Set(name string, value interface{})

Set sets field value by name. Byte array is not modified. Call ToBytes() to get modified byte array

func (*Buffer) ToBytes

func (b *Buffer) ToBytes() []byte

ToBytes returns new FlatBuffer byte array with fields modified by Set() and fields which initially had values Note: initial byte array and current modifications are kept

func (*Buffer) ToJSON

func (b *Buffer) ToJSON() string

ToJSON returns JSON flat key->value string

type FieldType

type FieldType int

FieldType s.e.

const (
	// FieldTypeUnspecified - wrong type
	FieldTypeUnspecified FieldType = iota
	// FieldTypeInt int32
	FieldTypeInt
	// FieldTypeLong int64
	FieldTypeLong
	// FieldTypeFloat float32
	FieldTypeFloat
	// FieldTypeDouble float64
	FieldTypeDouble
	// FieldTypeString variable length
	FieldTypeString
	// FieldTypeBool s.e.
	FieldTypeBool
	// FieldTypeByte byte
	FieldTypeByte
)

type Schema

type Schema struct {
	// contains filtered or unexported fields
}

Schema s.e.

func NewSchema

func NewSchema() *Schema

NewSchema creates new empty schema

func YamlToSchema

func YamlToSchema(yamlStr string) (*Schema, error)

YamlToSchema creates Schema by provided yaml `fieldName: yamlFieldType`

Field types:

• `int` -> `int32`
• `long` -> `int64`
• `float` -> `float32`
• `double` -> `float64`
• `bool` -> `bool`
• `string` -> `string`
• `byte` -> `byte`

See [dynobuffers_test.go](dynobuffers_test.go) for examples

func (*Schema) AddField

func (s *Schema) AddField(name string, ft FieldType)

AddField appends schema with new field

func (*Schema) HasField

func (s *Schema) HasField(name string) bool

HasField returns if the Schema contains the specified field