data

package
Version: v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2019 License: AGPL-3.0 Imports: 9 Imported by: 0

README

data

import "github.com/wzbox/bytom/pkg/go-wire/data"

Overview

Data is designed to provide a standard interface and helper functions to easily allow serialization and deserialization of your data structures in both binary and json representations.

This is commonly needed for interpreting transactions or stored data in the abci app, as well as accepting json input in the light-client proxy. If we can standardize how we pass data around the app, we can also allow more extensions, like data storage that can interpret the meaning of the []byte passed in, and use that to index multiple fields for example.

Serialization of data is pretty automatic using standard json and go-wire encoders. The main issue is deserialization, especially when using interfaces where there are many possible concrete types.

go-wire handles this by registering the types and providing a custom deserializer:

var _ = wire.RegisterInterface(
  struct{ PubKey }{},
  wire.ConcreteType{PubKeyEd25519{}, PubKeyTypeEd25519},
  wire.ConcreteType{PubKeySecp256k1{}, PubKeyTypeSecp256k1},
)

func PubKeyFromBytes(pubKeyBytes []byte) (pubKey PubKey, err error) {
  err = wire.ReadBinaryBytes(pubKeyBytes, &pubKey)
  return
}

func (pubKey PubKeyEd25519) Bytes() []byte {
  return wire.BinaryBytes(struct{ PubKey }{pubKey})
}

This prepends a type-byte to the binary representation upon serialization and using that byte to switch between various representations on deserialization. go-wire also supports something similar in json, but it leads to kind of ugly mixed-types arrays, and requires using the go-wire json parser, which is limited relative to the standard library encoding/json library.

In json, the typical idiom is to use a type string and message data:

{
  "type": "this part tells you how to interpret the message",
  "data": ...the actual message is here, in some kind of json...
}

I took inspiration from two blog posts, that demonstrate how to use this to build (de)serialization in a go-wire like way.

This package unifies these two in a single Mapper.

You app needs to do three things to take full advantage of this:

  1. For every interface you wish to serialize, define a holder struct with some helper methods, like FooerS wraps Fooer in common_test.go
  2. In all structs that include this interface, include the wrapping struct instead. Functionally, this also fulfills the interface, so except for setting it or casting it to a sub-type it works the same.
  3. Register the interface implementations as in the last init of common_test.go. If you are currently using go-wire, you should be doing this already

The benefits here is you can now run any of the following methods, both for efficient storage in our go app, and a common format for rpc / humans.

orig := FooerS{foo}

// read/write binary a la tendermint/go-wire
bparsed := FooerS{}
err := wire.ReadBinaryBytes(
  wire.BinaryBytes(orig), &bparsed)

// read/write json a la encoding/json
jparsed := FooerS{}
j, err := json.MarshalIndent(orig, "", "\t")
err = json.Unmarshal(j, &jparsed)

See https://github.com/wzbox/bytom/pkg/go-wire/data/blob/master/common_test.go to see how to set up your code to use this.

Index

Package files

binary.go bytes.go docs.go json.go wrapper.go

Variables

var (
    Encoder       ByteEncoder = hexEncoder{}
    HexEncoder                = hexEncoder{}
    B64Encoder                = base64Encoder{base64.URLEncoding}
    RawB64Encoder             = base64Encoder{base64.RawURLEncoding}
)

Encoder is a global setting for all byte encoding This is the default. Please override in the main()/init() of your program to change how byte slices are presented

type ByteEncoder

type ByteEncoder interface {
    Marshal(bytes []byte) ([]byte, error)
    Unmarshal(dst *[]byte, src []byte) error
}

ByteEncoder handles both the marshalling and unmarshalling of an arbitrary byte slice.

All Bytes use the global Encoder set in this package. If you want to use this encoding for byte arrays, you can just implement a simple custom marshaller for your byte array

type Dings [64]byte

func (d Dings) MarshalJSON() ([]byte, error) {
  return data.Encoder.Marshal(d[:])
}

func (d *Dings) UnmarshalJSON(data []byte) error {
  ref := (*d)[:]
  return data.Encoder.Unmarshal(&ref, data)
}

type Bytes

type Bytes []byte

Bytes is a special byte slice that allows us to control the serialization format per app.

Thus, basecoin could use hex, another app base64, and a third app base58...

func (Bytes) MarshalJSON
func (b Bytes) MarshalJSON() ([]byte, error)
func (*Bytes) UnmarshalJSON
func (b *Bytes) UnmarshalJSON(data []byte) error

type JSONMapper

type JSONMapper struct {
    // contains filtered or unexported fields
}
func (*JSONMapper) FromJSON
func (m *JSONMapper) FromJSON(data []byte) (interface{}, error)

FromJSON will deserialize the output of ToJSON for every registered implementation of the interface

func (*JSONMapper) ToJSON
func (m *JSONMapper) ToJSON(data interface{}) ([]byte, error)

ToJson will serialize a registered implementation into a format like:

{
  "type": "foo",
  "data": {
    "name": "dings"
  }
}

this allows us to properly deserialize with FromJSON

type Mapper

type Mapper struct {
    *JSONMapper
    // contains filtered or unexported fields
}

Mapper is the main entry point in the package.

On init, you should call NewMapper() for each interface type you want to support flexible de-serialization, and then RegisterInterface() in the init() function for each implementation of these interfaces.

Note that unlike go-wire, you can call RegisterInterface separately from different locations with each implementation, not all in one place. Just be careful not to use the same key or byte, of init will panic

func NewMapper
func NewMapper(base interface{}) Mapper

NewMapper creates a Mapper.

If you have:

type Foo interface {....}
type FooS struct { Foo }

then you should pass in FooS{} in NewMapper, and implementations of Foo in RegisterInterface

func (Mapper) RegisterInterface
func (m Mapper) RegisterInterface(kind string, b byte, data interface{}) Mapper

RegisterInterface should be called once for each implementation of the interface that we wish to support.

kind is the type string used in the json representation, while b is the type byte used in the go-wire representation. data is one instance of this concrete type, like Bar{}


Generated by godoc2md

Documentation

Overview

Data is designed to provide a standard interface and helper functions to easily allow serialization and deserialization of your data structures in both binary and json representations.

This is commonly needed for interpreting transactions or stored data in the abci app, as well as accepting json input in the light-client proxy. If we can standardize how we pass data around the app, we can also allow more extensions, like data storage that can interpret the meaning of the []byte passed in, and use that to index multiple fields for example.

Serialization of data is pretty automatic using standard json and go-wire encoders. The main issue is deserialization, especially when using interfaces where there are many possible concrete types.

go-wire handles this by registering the types and providing a custom deserializer:

var _ = wire.RegisterInterface(
  struct{ PubKey }{},
  wire.ConcreteType{PubKeyEd25519{}, PubKeyTypeEd25519},
  wire.ConcreteType{PubKeySecp256k1{}, PubKeyTypeSecp256k1},
)

func PubKeyFromBytes(pubKeyBytes []byte) (pubKey PubKey, err error) {
  err = wire.ReadBinaryBytes(pubKeyBytes, &pubKey)
  return
}

func (pubKey PubKeyEd25519) Bytes() []byte {
  return wire.BinaryBytes(struct{ PubKey }{pubKey})
}

This prepends a type-byte to the binary representation upon serialization and using that byte to switch between various representations on deserialization. go-wire also supports something similar in json, but it leads to kind of ugly mixed-types arrays, and requires using the go-wire json parser, which is limited relative to the standard library encoding/json library.

In json, the typical idiom is to use a type string and message data:

{
  "type": "this part tells you how to interpret the message",
  "data": ...the actual message is here, in some kind of json...
}

I took inspiration from two blog posts, that demonstrate how to use this to build (de)serialization in a go-wire like way.

* http://eagain.net/articles/go-dynamic-json/ * http://eagain.net/articles/go-json-kind/

This package unifies these two in a single Mapper.

You app needs to do three things to take full advantage of this:

1. For every interface you wish to serialize, define a holder struct with some helper methods, like FooerS wraps Fooer in common_test.go 2. In all structs that include this interface, include the wrapping struct instead. Functionally, this also fulfills the interface, so except for setting it or casting it to a sub-type it works the same. 3. Register the interface implementations as in the last init of common_test.go. If you are currently using go-wire, you should be doing this already

The benefits here is you can now run any of the following methods, both for efficient storage in our go app, and a common format for rpc / humans.

orig := FooerS{foo}

// read/write binary a la tendermint/go-wire
bparsed := FooerS{}
err := wire.ReadBinaryBytes(
  wire.BinaryBytes(orig), &bparsed)

// read/write json a la encoding/json
jparsed := FooerS{}
j, err := json.MarshalIndent(orig, "", "\t")
err = json.Unmarshal(j, &jparsed)

See https://github.com/wzbox/bytom/pkg/go-wire/data/blob/master/common_test.go to see how to set up your code to use this.

Index

Constants

This section is empty.

Variables

View Source
var (
	Encoder       ByteEncoder = hexEncoder{}
	HexEncoder                = hexEncoder{}
	B64Encoder                = base64Encoder{base64.URLEncoding}
	RawB64Encoder             = base64Encoder{base64.RawURLEncoding}
)

Encoder is a global setting for all byte encoding This is the default. Please override in the main()/init() of your program to change how byte slices are presented

In addition to these implementation, you can also find BTCEncoder and FlickrEncoder that use base58 variants in github.com/wzbox/bytom/pkg/go-wire/data/base58

Functions

func FromJSON

func FromJSON(d []byte, o interface{}) error

FromJSON is a convenience method to deserialize with encoding/json

func FromWire

func FromWire(d []byte, o interface{}) error

FromWire is a convenience method to deserialize with go-wire

func ToJSON

func ToJSON(o interface{}) ([]byte, error)

ToJSON is a convenience method to serialize with encoding/json

func ToText

func ToText(o interface{}) (string, error)

ToText is a rather special-case serialization for cli, especially for []byte interfaces

It tries to serialize as json, and the result looks like:

{ "type": "string", "data": "string" }

Then it will return "<type>:<data>"

Main usecase is serializing eg. crypto.PubKeyS as "ed25119:a1b2c3d4..." for displaying in cli tools.

It also supports encoding data.Bytes to a string using the proper codec (or anything else that has a marshals to a string)

func ToWire

func ToWire(o interface{}) ([]byte, error)

ToWire is a convenience method to serialize with go-wire error is there to keep the same interface as json, but always nil

Types

type ByteEncoder

type ByteEncoder interface {
	Marshal(bytes []byte) ([]byte, error)
	Unmarshal(dst *[]byte, src []byte) error
}

ByteEncoder handles both the marshalling and unmarshalling of an arbitrary byte slice.

All Bytes use the global Encoder set in this package. If you want to use this encoding for byte arrays, you can just implement a simple custom marshaller for your byte array

type Dings [64]byte

func (d Dings) MarshalJSON() ([]byte, error) {
  return data.Encoder.Marshal(d[:])
}

func (d *Dings) UnmarshalJSON(enc []byte) error {
  var ref []byte
  err := data.Encoder.Unmarshal(&ref, enc)
  copy(d[:], ref)
  return err
}

type Bytes

type Bytes []byte

Bytes is a special byte slice that allows us to control the serialization format per app.

Thus, basecoin could use hex, another app base64, and a third app base58...

func (Bytes) Bytes

func (b Bytes) Bytes() []byte

Allow it to fulfill various interfaces in light-client, etc...

func (Bytes) MarshalJSON

func (b Bytes) MarshalJSON() ([]byte, error)

func (Bytes) String

func (b Bytes) String() string

String gets a simple string for printing (the json output minus quotes)

func (*Bytes) UnmarshalJSON

func (b *Bytes) UnmarshalJSON(data []byte) error

type Mapper

type Mapper struct {
	// contains filtered or unexported fields
}

Mapper is the main entry point in the package.

On init, you should call NewMapper() for each interface type you want to support flexible de-serialization, and then RegisterImplementation() in the init() function for each implementation of these interfaces.

Note that unlike go-wire, you can call RegisterImplementation separately from different locations with each implementation, not all in one place. Just be careful not to use the same key or byte, of init will *panic*

func NewMapper

func NewMapper(base interface{}) Mapper

NewMapper creates a Mapper.

If you have:

type Foo interface {....}
type FooS struct { Foo }

then you should pass in FooS{} in NewMapper, and implementations of Foo in RegisterImplementation

func (Mapper) FromJSON

func (m Mapper) FromJSON(data []byte) (interface{}, error)

FromJSON will deserialize the output of ToJSON for every registered implementation of the interface

func (Mapper) RegisterImplementation

func (m Mapper) RegisterImplementation(data interface{}, kind string, b byte) Mapper

RegisterImplementation should be called once for each implementation of the interface that we wish to support.

kind is the type string used in the json representation, while b is the type byte used in the go-wire representation. data is one instance of this concrete type, like Bar{}

func (Mapper) ToJSON

func (m Mapper) ToJSON(data interface{}) ([]byte, error)

ToJson will serialize a registered implementation into a format like:

{
  "type": "foo",
  "data": {
    "name": "dings"
  }
}

this allows us to properly deserialize with FromJSON

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL