decoder

README

Decoder

Dynamic decoders based on inspector framework and vector packages, like jsonvector.

Basics

One of the major the problems we ran into was a necessity to convert tons of different response formats from external services into our internal response format. Also, there was a requirement to connect new services at runtime and provide a possibility to decode their responses without rebooting the application.

This package provides a possibility to describe decoding rules in Go-like meta-language and add new decoders or edit existing at runtime.

Decoders is a context-based and all variables should be preliminarily registered in special context before decoding.

Syntax

Recommend reading inspector first.

Each decoder consists of name and body. Name uses only as index and may contain any symbols. There is only one requirement - name should be unique.

Decoder's body is a multi-line string (that calls ruleset) and each line (rule) has the following format

<destination path> = [<source path>|<callback>]

Both <source path> and <destination path> describes a source/destination fields accessed using a dot, example:

dstObj.Name = user.FullName

As mentioned above, both dstObj and user variables should be preliminarily registered in the context like this

ctx := decoder.AcquireCtx()
ctx.Set("dstObj", dst, &DstInspector{})
ctx.Set("user", user, &UserInspector{})

In this example user is a Go struct, but you may use as source raw response, example in JSON

jsonResponse = []byte(`{"a":"foo"}`)
ctx := decoder.AcquireCtx()
ctx.SetJson("response", jsonResponse)
// in decoder body:
// dstObj.Name = response.a

In this way, decoder provides a possibility to describe where source data should be taken and where it should be came.

Enough easy, isn't it?

Modifiers

Sometimes, isn't enough just specify source data address, but need to modify it before assigning to destination.

Especially for that cases was added support of source modifiers. The syntax:

<destination path> = <source path>|<modifier name>(<arg0>, <arg1>, ...)

Example

dstObj.Balance = user.Finance.Rest|default(0)

Default is a built-in modifier, but you may register your own modifiers using modifiers registry:

func modMyCustomMod(ctx *Ctx, buf *interface{}, val interface{}, args []interface{}) error {
    // ...
}

decoder.RegisterModFn("myCustomMod", "customMod", modMyCustomMod)

Modifier arguments:

• ctx is a storage of variables/buffers you may use.
• buf is a type-free buffer that receives result of modifier's work. Please note the type *interface{} is an alloc-free trick.
• val if a value of variable from left side of modifier separator (|).
• args array of modifier arguments, specified in rule.

See mod.go for details and mod_builtin.go for more example of modifiers.

Documentation

Index

• Variables
• func AssignVectorNode(dst, src interface{}, _ inspector.AccumulativeBuffer) (ok bool)
• func Decode(id string, ctx *Ctx) error
• func DecodeRuleset(ruleset Ruleset, ctx *Ctx) (err error)
• func RegisterCallbackFn(name, alias string, cb CallbackFn)
• func RegisterDecoder(id string, rules Ruleset)
• func RegisterGetterFn(name, alias string, cb GetterFn)
• func RegisterModFn(name, alias string, mod ModFn)
• func ReleaseCtx(ctx *Ctx)
• type CallbackFn
  • func GetCallbackFn(name string) *CallbackFn
• type Ctx
  • func AcquireCtx() *Ctx
  • func NewCtx() *Ctx
  • func (ctx *Ctx) AcquireBytes() []byte
  • func (ctx *Ctx) Get(path string) interface{}
  • func (ctx *Ctx) ReleaseBytes(p []byte)
  • func (ctx *Ctx) Reset()
  • func (ctx *Ctx) Set(key string, val interface{}, ins inspector.Inspector)
  • func (ctx *Ctx) SetStatic(key string, val interface{})
  • func (ctx *Ctx) SetVector(key string, data []byte, typ VectorType) (vec vector.Interface, err error)
  • func (ctx *Ctx) SetVectorNode(key string, node *vector.Node) error
• type CtxPool
  • func (p *CtxPool) Get() *Ctx
  • func (p *CtxPool) Put(ctx *Ctx)
• type Decoder
• type GetterFn
  • func GetGetterFn(name string) *GetterFn
• type ModFn
  • func GetModFn(name string) *ModFn
• type Ruleset
  • func Parse(src []byte) (ruleset Ruleset, err error)
  • func ParseFile(fileName string) (rules Ruleset, err error)
  • func (r *Ruleset) HumanReadable() []byte
• type VectorType

Variables

var (
	ErrDecoderNotFound = errors.New("decoder not found")
	ErrEmptyNode       = errors.New("provided node is empty")
	ErrModNoArgs       = errors.New("empty arguments list")
	ErrModPoorArgs     = errors.New("arguments list in modifier is too small")
	ErrCbPoorArgs      = errors.New("arguments list in callback is too small")
	ErrGetterPoorArgs  = errors.New("arguments list in getter callback is too small")
)

Functions

func AssignVectorNode

func AssignVectorNode(dst, src interface{}, _ inspector.AccumulativeBuffer) (ok bool)

AssignVectorNode implements assign callback to convert vector.Node to destination with arbitrary type.

func Decode

func Decode(id string, ctx *Ctx) error

Decode applies decoder rules using given id.

ctx should contain all variables mentioned in the decoder's body.

func DecodeRuleset

func DecodeRuleset(ruleset Ruleset, ctx *Ctx) (err error)

DecodeRuleset applies decoder ruleset without using id.

func RegisterCallbackFn

func RegisterCallbackFn(name, alias string, cb CallbackFn)

RegisterCallbackFn registers new callback to the registry.

func RegisterDecoder

func RegisterDecoder(id string, rules Ruleset)

RegisterDecoder registers decoder ruleset in the registry.

func RegisterGetterFn

func RegisterGetterFn(name, alias string, cb GetterFn)

RegisterGetterFn registers new getter callback to the registry.

func RegisterModFn

func RegisterModFn(name, alias string, mod ModFn)

RegisterModFn registers new modifier function.

func ReleaseCtx

func ReleaseCtx(ctx *Ctx)

ReleaseCtx puts object back to default pool.

Types

type CallbackFn

type CallbackFn func(ctx *Ctx, args []interface{}) error

CallbackFn represents the signature of callback function.

args contains list of all arguments you passed in decoder rule.

func GetCallbackFn

func GetCallbackFn(name string) *CallbackFn

GetCallbackFn returns callback function from the registry.

type Ctx

type Ctx struct {

	// External buffers to use in modifier and condition helpers.
	BufAcc bytebuf.AccumulativeBuf
	// todo remove as unused later
	Buf, Buf1, Buf2 bytebuf.ChainBuf

	Err error
	// contains filtered or unexported fields
}

Ctx represents decoder context object.

Contains list of variables that can be used as source or destination.

func AcquireCtx

func AcquireCtx() *Ctx

AcquireCtx returns object from the default context pool.

func NewCtx

func NewCtx() *Ctx

NewCtx makes new context object.

func (*Ctx) AcquireBytes

func (ctx *Ctx) AcquireBytes() []byte

AcquireBytes returns accumulative buffer.

func (*Ctx) Get

func (ctx *Ctx) Get(path string) interface{}

Get arbitrary value from the context by path.

See Ctx.get(). Path syntax: <ctxVrName>[.<Field>[.<NestedField0>[....<NestedFieldN>]]] Examples: * user.Bio.Birthday * staticVar

func (*Ctx) ReleaseBytes

func (ctx *Ctx) ReleaseBytes(p []byte)

ReleaseBytes updates accumulative buffer with p.

func (*Ctx) Reset

func (ctx *Ctx) Reset()

Reset the context.

Made to use together with pools.

func (*Ctx) Set

func (ctx *Ctx) Set(key string, val interface{}, ins inspector.Inspector)

Set the variable to context. Inspector ins should be corresponded to variable val.

func (*Ctx) SetStatic

func (ctx *Ctx) SetStatic(key string, val interface{})

SetStatic registers static variable in context.

func (*Ctx) SetVector

func (ctx *Ctx) SetVector(key string, data []byte, typ VectorType) (vec vector.Interface, err error)

SetVector parses source data and register it in context under given key.

func (*Ctx) SetVectorNode

func (ctx *Ctx) SetVectorNode(key string, node *vector.Node) error

SetVectorNode directly registers node in context under given key.

type CtxPool

type CtxPool struct {
	// contains filtered or unexported fields
}

CtxPool represents context pool.

var (
	// CP is a default instance of context pool.
	// You may use it directly as decoder.CP.Get()/Put() or using functions AcquireCtx()/ReleaseCtx().
	CP CtxPool
)

func (*CtxPool) Get

func (p *CtxPool) Get() *Ctx

Get context object from the pool or make new object if pool is empty.

func (*CtxPool) Put

func (p *CtxPool) Put(ctx *Ctx)

Put the object to the pool.

type Decoder

type Decoder struct {
	Id string
	// contains filtered or unexported fields
}

Decoder represents main decoder object. Decoder contains only parsed ruleset. All temporary and intermediate data should be store in context logic to make using of decoders thread-safe.

type GetterFn

type GetterFn func(ctx *Ctx, buf *interface{}, args []interface{}) error

GetterFn represents signature of getter callback function.

args contains list of all arguments you passed in decoder rule.

func GetGetterFn

func GetGetterFn(name string) *GetterFn

GetGetterFn returns getter callback function from the registry.

type ModFn

type ModFn func(ctx *Ctx, buf *interface{}, val interface{}, args []interface{}) error

ModFn represents signature of the modifier functions.

Arguments description: * ctx provides access to additional variables and various buffers to reduce allocations. * buf is a storage for final result after finishing modifier work. * val is a left side variable that preceded to call of modifier func, example: {%= val|mod(...) %} * args is a list of all arguments listed on modifier call.

func GetModFn

func GetModFn(name string) *ModFn

GetModFn returns modifier from the registry.

type Ruleset

type Ruleset []rule

Ruleset represents list of rules.

func Parse

func Parse(src []byte) (ruleset Ruleset, err error)

Parse parses the decoder rules.

func ParseFile

func ParseFile(fileName string) (rules Ruleset, err error)

ParseFile parses the file.

func (*Ruleset) HumanReadable

func (r *Ruleset) HumanReadable() []byte

HumanReadable builds human-readable view of the rules list.

type VectorType

type VectorType int

const (
	VectorJSON VectorType = iota
	VectorURL
	VectorXML
	VectorYAML

	VectorsSupported = 4
)