jsonpack

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 16, 2020 License: MIT Imports: 9 Imported by: 0

README

JSONPack

Fast and space efficiency JSON serialization golang library. It is a schema oriented design which leverages schema definition to encode JSON document into compact binary encoded format, and decodes back into JSON document.

Introduction

When we want to exchange data between services or over network, the JSON is most popular format to do it. In golang world, the most convenient way is using official encoding/json package to marshal and unmarshal JSON document from/to struct or map. For usual RESTful web service scenario, JSON format is quite convenience and representative, but for real-time message exchanging or other scenarios that has small footprint data and low-letency requirement, JSON is a bit too heavyweight, not only data footprint is not space saving, but also has heavy loading in encode/decode procedure.

So if we want to a compact, small footprint data for exchanging over network, and also leverages the convenience of JSON, we need an encoding format that removes "property name" and other notations likes ':', '[', '{'...etc from original JSON document, and leaves "value" only.

To achieve this goal, we need a schematic to define our JSON document and provide enough information for serialization engine to know sequence and data type of every properties in document. it's the reason why JSONPack is a schema oriented design library.

Key Features

  • Similar Marshal / Unmarshal API to standard encoding/json package.
  • Space saving encoded format, the size of encoded data is similar to Protocol Buffers, can be 30-80% of original JSON document, depends on data.
  • Blazing fast, provides about 3.x decoding speed compared to protobuf and many times than other JSON packages.
  • Memory saving design, avoids any un-neccessary memory allocations, suitable for embedded environment.
  • Has production ready javascript implementation Buffer Plus, can be used in node.js and Web browser environment.
  • No need to write schema definition by hand, jsonpack will generate schema definition from golang struct automatically.

How to Get

go get https://github.com/arloliu/jsonpack

Usage

Example of add schema definition

import "github.com/arloliu/jsonpack"

type Info struct {
	Name string `json:"name"`
	Area uint32 `json:"area"`
	// omit this field
	ExcludeField string `-`
}

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("Info", Info{}, jsonpack.LittleEndian)

Example of encoding data with Info struct

infoStruct := map[string]interface{} {
	"name": "example name",
	"area": uint32(888),
}
// encodedResult1 contains encoded data,
encodedResult1, err := jsonPack.Marshal("Info", infoStruct)

Example of encoding data with golang map

infoMap := map[string]interface{} {
	"name": "example name",
	"area": uint32(888),
}

encodedResult2, err := jsonPack.Marshal("Info", infoMap)

Example of decoding data

decodeInfoStruct = Info{}
err := jsonPack.Decode("Info", encodedResult1, &decodeInfoStruct)

decodeInfoMap = make(map[string]interface{})
err := jsonPack.Decode("Info", encodedResult2, &decodeInfoMap)

Benchmark

The benchmark result is a important reference but not always suitable for every scenarios.

Test environment: Intel i7-9700K CPU@3.60GHz.

Sorts from fastest to slowest in the following.

Encode from golang map

ns/op allocation bytes allocation times
jsonpack 1933 ns/op 752 B/op 2 allocs/op
jsoniter 10134 ns/op 3320 B/op 46 allocs/op
std. json 23560 ns/op 8610 B/op 171 allocs/op
goccy 75298 ns/op 82639 B/op 651 allocs/op

Decode into golang map

ns/op allocation bytes allocation times
jsonpack 6461 ns/op 6512 B/op 96 allocs/op
jsoniter 17436 ns/op 9666 B/op 290 allocs/op
std. json 18949 ns/op 8864 B/op 228 allocs/op
goccy 19985 ns/op 15900 B/op 316 allocs/op

Encode from golang struct

ns/op allocation bytes allocation times
jsonpack 1834 ns/op 800 B/op 3 allocs/op
protobuf 1972 ns/op 896 B/op 1 allocs/op
goccy 2166 ns/op 1280 B/op 1 allocs/op
jsoniter 3372 ns/op 1296 B/op 3 allocs/op
std. json 3578 ns/op 1280 B/op 1 allocs/op

Decode into golang struct

ns/op allocation bytes allocation times
jsonpack 1475 ns/op 96 B/op 2 allocs/op
goccy 3284 ns/op 2215 B/op 5 allocs/op
jsoniter 4680 ns/op 1072 B/op 79 allocs/op
protobuf 5075 ns/op 3152 B/op 84 allocs/op
std. json 18378 ns/op 1232 B/op 69 allocs/op

The benchmark result indicates jsonpack keeps constant performance on both encoding and encoding side, and keeps very low memory allocation size and times.

The benchmark result also delivers an important message.

The performance of operating with golang map sucks

So it's better to use struct if possible. :)

Documentation

Overview

Fast and space efficiency JSON serialization golang library. It is a schema oriented design which leverages schema definition to encode JSON document into compact binary encoded format, and decodes back into JSON document.

Introduction

When we want to exchange data between services or over network, the JSON is most popular format to do it. In golang world, the most convenient way is using official `encoding/json` package to marshal and unmarshal JSON document from/to struct or map. For usual RESTful web service scenario, JSON format is quite convenience and representative, but for real-time message exchanging or other scenarios that has small footprint data and low-letency requirement, JSON is a bit too heavyweight, not only data footprint is not space saving, but also has heavy loading in encode/decode procedure.

So if we want to a compact, small footprint data for exchanging over network, and also leverages the convenience of JSON, we need an encoding format that removes "property name" and other notations likes ':', '[', '{'...etc from original JSON document, and leaves "value" only.

To achieve this goal, we need a schematic to define our JSON document and provide enough information for serialization engine to know sequence and data type of every properties in document. it's the reason why jsonpack is a schema oriented design library.

Key Features

* Similar Marshal / Unmarshal API to standard `encoding/json` package.

* Space saving encoded format, the size of encoded data is similar to Protocol Buffers, can be 30-80% of original JSON document, depends on data.

* Blazing fast, provides about 3.x decoding speed compared to `protobuf` and many times than other JSON packages.

* Memory saving design, avoids any un-neccessary memory allocations, suitable for embedded environment.

* Has production ready javascript implementation https://github.com/arloliu/buffer-plus, can be used in node.js and Web browser environment.

* No need to write schema definition by hand, `jsonpack` will generate schema definition from golang struct automatically.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ByteOrder 

type ByteOrder int
const (
	LittleEndian ByteOrder = 0
	BigEndian    ByteOrder = 1
)

type CompileError 

type CompileError struct {
	Err error
	// contains filtered or unexported fields
}

CompileError indicates an error that occurred while attempting to compile schema definition

func (*CompileError) Error 

func (e *CompileError) Error() string

func (*CompileError) Unwrap 

func (e *CompileError) Unwrap() error

type DecodeError 

type DecodeError struct {
	Err error
	// contains filtered or unexported fields
}

DecodeError indicates an error that occurred while attempting to decode packed binary data with pre-compiled schema definition

func (*DecodeError) Error 

func (e *DecodeError) Error() string

func (*DecodeError) Unwrap 

func (e *DecodeError) Unwrap() error

type EncodeError 

type EncodeError struct {
	Err error
	// contains filtered or unexported fields
}

EncodeError indicates an error that occurred while attempting to encode data with pre-compiled schema definition

func (*EncodeError) Error 

func (e *EncodeError) Error() string

func (*EncodeError) Unwrap 

func (e *EncodeError) Unwrap() error

type InvalidPropValueError 

type InvalidPropValueError struct {
	// contains filtered or unexported fields
}

InvalidPropValueError indicates an error the that occurred when reading or writing property got invalid value

func (*InvalidPropValueError) Error 

func (e *InvalidPropValueError) Error() string

type JSONPack 

type JSONPack struct {
	// contains filtered or unexported fields
}

JSONPack JSON packer structure

func NewJSONPack 

func NewJSONPack() *JSONPack

NewJSONPack Create instance of JSONPacker

func (*JSONPack) AddSchema 

func (p *JSONPack) AddSchema(schemaName string, v ...interface{}) (*Schema, error)

AddSchema will compile schema definition and stores compiled result in internal schema manager.

It's a variadic function which accept two types of input parameters in the following.

AddSchema(schemaName string, v interface{})

The v is schema definition which want to compile. The value of v can be a JSON format of text data with []byte/string type, a map represents JSON format of schema definition, or a SchemaDef struct represents schema definition.

Example of add new schema from JSON text string: 

schDef := `
{
	"type": "object",
	"properties": {
		"name": {"type": "string"},
		"area": {"type": "uint32le"}
	},
	"order": ["name", "area"]
}
`

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("info", schDef)

Example of adding new schema from map of schema definition: 

schDef := map[string]interface{}{
	"type": "object",
	"properties": map[string]interface{}{
		"name": map[string]interface{}{"type": "string"},
		"area": map[string]interface{}{"type": "uint32le"},
	},
	"order": []string{"name", "area"},
}

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("info", schDef)

Example of adding new schema from SchemaDef struct: 

schDef := jsonpack.SchemaDef{
	Type: "object",
	Properties: map[string]*jsonpack.SchemaDef{
		"name": {Type: "string"},
		"area": {Type: "uint32le"},
	},
	Order: []string{"name", "area"},
}
jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("info", schDef)

AddSchema(schemaName string, v interface{}, byteOrder jsonpack.ByteOrder)

For fast prototyping, AddSchema method supports generate schema definition from existing struct without writing schema definition by hand.

In this scenario, the value of v is the source struct which to generated, and byteOrder parameter indicates the byte order, can be either jsonpack.LittleEndian or jsonpack.BigEndian, it defaults to little-endian if not specified.

This method supports struct tag, use the same format as standard encoding/json excepts "omitempty" option, the "omitempty" option will be ignored.

Example of adding new schema and build schema definition from struct: 

type Info struct {
	Name string `json:"name"`
	// "omitempty" option will be ignore, so this field will be not be omitted
	Area uint32 `json:"area,omitempty"`
	ExcludeField string `-` // this field is ignored
}

jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("Info", Info{}, jsonpack.BigEndian)

func (*JSONPack) Decode 

func (p *JSONPack) Decode(schemaName string, data []byte, v interface{}) error

Decode is a wrapper of Schema.Decode, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) Encode 

func (p *JSONPack) Encode(schemaName string, v interface{}) ([]byte, error)

Encode is a wrapper of Schema.Encode, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) EncodeTo 

func (p *JSONPack) EncodeTo(schemaName string, v interface{}, dataPtr *[]byte) error

EncodeTo is a wrapper of Schema.EncodeTo, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) GetAllSchemaDefTexts 

func (p *JSONPack) GetAllSchemaDefTexts() map[string][]byte

GetAllSchemaDefTexts returns a map which contains all existed schema text definitions, key of map it schema name, and value of map is text format of schema defintion which presented as []byte.

func (*JSONPack) GetAllSchemaDefs 

func (p *JSONPack) GetAllSchemaDefs() map[string]*SchemaDef

GetAllSchemaDefs returns a map which contains all existed schema definitions, key of map it schema name, and value of map is *SchemaDef.

func (*JSONPack) GetAllSchemas 

func (p *JSONPack) GetAllSchemas() map[string]*Schema

GetAllSchemas returns a map which contains all existed schema instances, key of map it schema name, and value of map is *Schema.

func (*JSONPack) GetSchema 

func (p *JSONPack) GetSchema(schemaName string) *Schema

GetSchema returns schema instance by schemaName, returns nil if schema not found.

func (*JSONPack) GetSchemaDef 

func (p *JSONPack) GetSchemaDef(schemaName string) (*SchemaDef, error)

GetSchemaDef is a wrapper of Schema.GetSchemaDef, it gets a Schema instance by schemaName, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) GetSchemaDefText 

func (p *JSONPack) GetSchemaDefText(schemaName string) ([]byte, error)

GetSchemaDefText is a wrapper of Schema.GetSchemaDefText, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) Marshal 

func (p *JSONPack) Marshal(schemaName string, v interface{}) ([]byte, error)

Marshal is an alias to Encode function, provides familiar interface of json package

func (*JSONPack) RemoveSchema 

func (p *JSONPack) RemoveSchema(schemaName string) error

RemoveSchema removes schema by schemaName, it returns *SchemaNonExistError error if schema not found.

func (*JSONPack) Reset 

func (p *JSONPack) Reset()

Reset removes all schema instances

func (*JSONPack) Unmarshal 

func (p *JSONPack) Unmarshal(schemaName string, data []byte, v interface{}) error

Unmarshal is an alias to Decode function, provides familiar interface of json package

type NotImplementedError 

type NotImplementedError struct {
	// contains filtered or unexported fields
}

NotImplemented means this method/operation is not implemented

func (*NotImplementedError) Error 

func (e *NotImplementedError) Error() string

type Schema 

type Schema struct {
	// schema name
	Name string
	// contains filtered or unexported fields
}

Schema of jsonpack, each Schema instance represents a schema added by AddSchema function

func (*Schema) Decode 

func (s *Schema) Decode(data []byte, v interface{}) (err error)

Decode reads encoded data with compiled schema definition and stores the result in the value pointed to v.

If type of v is not a pointer type that pointed to a map or struct, Decode function will return DecodeError.

The valid type of v is either a *map[string]interface{} or a pointer to the struct which added by AddSchema function.

func (*Schema) Encode 

func (s *Schema) Encode(d interface{}) ([]byte, error)

Encode returns packed binary data of v with compiled schema definition. The compiled schema definition is specified by schemaName and return data will be nil if error occurs.

The type of v can be a map[string]interface{} which represents valid JSON data, or a struct instance which added by AddSchema function.

The return data contains encoded binary data that can then be decoded by Decode function.

Example of encoding a map 

data := map[string]interface{} {
	"name": "example name",
	"area": uint32(888),
}

jsonPack := jsonpack.NewJSONPack()
// call jsonPack.AddSchema to register schema
sch := jsonPack.GetSchema("info")
result, err := sch.Encode(data)

Example of encoding struct 

type Info struct

func (*Schema) EncodeTo 

func (s *Schema) EncodeTo(d interface{}, dataPtr *[]byte) (err error)

EncodeTo is similar to Encode function, but passing a pointer to []byte to store encoded data instead of returning new allocated []byte encoded data.

func (*Schema) GetSchemaDef 

func (s *Schema) GetSchemaDef() (*SchemaDef, error)

GetSchemaDef returns schema definition instance, returns nil and error if error occurs.

func (*Schema) GetSchemaDefText 

func (s *Schema) GetSchemaDefText() []byte

GetSchemaDefText returns JSON format document of schema definition, which presetned as []byte.

func (*Schema) Marshal 

func (s *Schema) Marshal(d interface{}) ([]byte, error)

Marshal is an alias to Encode function, provides familiar interface of json package

func (*Schema) SetDecodeBufSize 

func (s *Schema) SetDecodeBufSize(size int64)

SetDecodeBufSize set default decode buffer allocation bytes

func (*Schema) SetEncodeBufSize 

func (s *Schema) SetEncodeBufSize(size int64)

SetEncodeBufSize set default encode buffer allocation bytes

func (*Schema) Unmarshal 

func (s *Schema) Unmarshal(data []byte, v interface{}) (err error)

Unmarshal is an alias to Decode function, provides familiar interface of json package

type SchemaDef 

type SchemaDef struct {
	Type       string                `json:"type"`
	Properties map[string]*SchemaDef `json:"properties,omitempty"`
	Items      *SchemaDef            `json:"items,omitempty"`
	Order      []string              `json:"order,omitempty"`
}

The schema defintion represents the definition that define the structure of JSON data.

type SchemaNonExistError 

type SchemaNonExistError struct {
	// contains filtered or unexported fields
}

SchemaNonExistError indicates an error that occurred while pre-compiled schema defintion does not exist

func (*SchemaNonExistError) Error 

func (e *SchemaNonExistError) Error() string

type StructFieldNonExistError 

type StructFieldNonExistError struct {
	// contains filtered or unexported fields
}

StructFieldNonExistError indicates an error the that occurred when structure doesn't contain required field

func (*StructFieldNonExistError) Error 

func (e *StructFieldNonExistError) Error() string

type TypeAssertionError 

type TypeAssertionError struct {
	// contains filtered or unexported fields
}

TypeAssertionError indicates an error the that occurred when doing data data type asserrion

func (*TypeAssertionError) Error 

func (e *TypeAssertionError) Error() string

type UnknownTypeError 

type UnknownTypeError struct {
	// contains filtered or unexported fields
}

UnknownTypeError means there has a un-supported data type when compiling schema definition

func (*UnknownTypeError) Error 

func (e *UnknownTypeError) Error() string

type WrongTypeError 

type WrongTypeError struct {
	// contains filtered or unexported fields
}

WrongTypeError means there has a wrong data type when encoding or decoding data with pre-compiled schema definition

func (*WrongTypeError) Error 

func (e *WrongTypeError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
jsonpack-parser
internal
logger

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.