byframe

README

byframe

It's a low overhead length header format with dynamic header length. So you don't waste resource on the header itself when framing data. The algorithm is based on LEB128.

This lib also contains functions to encode and decode the header, so you have the full flexibility to decide how to use it, such as streaming TCP frames, indexing database, etc.

This lib is not suit for high level usage, for extensible or debugging friendly it's better to use lib like FlatBuffers, Protobuf or Msagepack.

Format

Each frame has two parts: the header and body.

|     frame     |
| header | body |

Header

Each byte (8 bits) in the header has two parts, "continue" and "fraction":

bit index |    0     | 1 2 3 4 5 6 7 |
sections  | continue |   fraction    |

If the "continue" is 0, the header ends. If the "continue" is 1, then the followed byte should also be part of the header.

Sum all the fractions together, we will get the size of the message.

For example:

|                         frame                              |
|                      header                         | body |
| continue |   fraction    | continue |   fraction    |      |
|    0     | 1 0 0 0 0 0 0 |    1     | 1 1 0 1 0 0 0 | ...  |

So the size of the body is 0b1101000,1000000 bytes.

Documentation

Index

• Variables
• func Decode(data []byte) ([]byte, int, error)
• func DecodeHeader(raw []byte) (int, int, bool)
• func DecodeTuple(data []byte, items ...*[]byte) error
• func Encode(data []byte) []byte
• func EncodeHeader(l int) (header []byte)
• func EncodeTuple(items ...*[]byte) []byte
• type Scanner
  • func NewScanner(r io.Reader) *Scanner
  • func (s *Scanner) Err() error
  • func (s *Scanner) Frame() []byte
  • func (s *Scanner) Limit(size int) *Scanner
  • func (s *Scanner) Scan() bool

Examples

• Encode
• EncodeHeader
• EncodeTuple
• Scanner

Variables

var ErrHeaderInsufficient = errors.New("data is not sufficient to construct the header")

ErrHeaderInsufficient ...

var ErrInsufficient = errors.New("data is not sufficient to construct the body")

ErrInsufficient ...

Functions

func Decode

func Decode(data []byte) ([]byte, int, error)

Decode decode frame into data, decoded bytes and error

func DecodeHeader

func DecodeHeader(raw []byte) (int, int, bool)

DecodeHeader decode bytes into data length, header length and whether it's sufficient to parse the header from raw.

func DecodeTuple

func DecodeTuple(data []byte, items ...*[]byte) error

DecodeTuple zero copy decode

func Encode

func Encode(data []byte) []byte

Encode encode data into frame format

Example

package main

import (
	"fmt"

	"github.com/ysmood/byframe"
)

func main() {
	frame := byframe.Encode([]byte("test"))

	data, _, _ := byframe.Decode(frame)

	fmt.Println(string(data))
}

Output:

test

func EncodeHeader

func EncodeHeader(l int) (header []byte)

EncodeHeader encode data length into header

Example

package main

import (
	"fmt"

	"github.com/ysmood/byframe"
)

func main() {
	header := byframe.EncodeHeader(1000)

	dataLen, headerLen, sufficient := byframe.DecodeHeader(header)

	fmt.Println(headerLen, dataLen, sufficient)
}

Output:

2 1000 true

func EncodeTuple

func EncodeTuple(items ...*[]byte) []byte

EncodeTuple encode an array of []byte into an []byte. The format:

| itemA length | itemA bytes | ... | itemN length | itemN bytes | itemLast bytes |

The last item doesn't need length header. The protection for data curruption can be done on other levels of data handling, this algorithm is designed for small size.

Example

package main

import (
	"fmt"

	"github.com/ysmood/byframe"
)

func main() {
	type Name struct {
		First []byte
		Last  []byte
	}

	name := Name{[]byte("Jack"), []byte("Black")}

	data := byframe.EncodeTuple(&name.First, &name.Last)

	var newName Name
	_ = byframe.DecodeTuple(data, &newName.First, &newName.Last)

	fmt.Println(string(newName.First), string(newName.Last))
}

Output:

Jack Black

Types

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner scan frames based on the length header

Example

package main

import (
	"bytes"
	"fmt"

	"github.com/ysmood/byframe"
)

func main() {
	frame := byframe.Encode([]byte("test"))
	s := byframe.NewScanner(bytes.NewReader(frame))

	for s.Scan() {
		fmt.Println(string(s.Frame()))
	}
}

Output:

test

func NewScanner

func NewScanner(r io.Reader) *Scanner

NewScanner just like line scanner

func (*Scanner) Err

func (s *Scanner) Err() error

Err the error

func (*Scanner) Frame

func (s *Scanner) Frame() []byte

Frame current frame

func (*Scanner) Limit

func (s *Scanner) Limit(size int) *Scanner

Limit of frame buffer, panic if exceeds. Zero means no limit.

func (*Scanner) Scan

func (s *Scanner) Scan() bool

Scan scan next frame