package module
Version: v0.0.0-...-9b5a675 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Apr 2, 2019 License: Apache-2.0 Imports: 14 Imported by: 6



decode/encode thrift message without IDL


  • because IDL generated model is ugly and inflexible, it is seldom used in application directly. instead we define another model, which leads to bad performance.
    • bytes need to be copied twice
    • more objects to gc
  • thrift proxy can not know all possible IDL in advance, in scenarios like api gateway, we need to decode/encode in a generic way to modify embedded header.
  • official thrift library for go is slow, verified in several benchmarks. It is even slower than json-iterator

works like encoding/json

encoding/json has a super simple api to encode/decode json. thrifter mimic the same api.

import ""
// marshal to thrift
thriftEncodedBytes, err := thrifter.Marshal([]int{1, 2, 3})
// unmarshal back
var val []int
err = thrifter.Unmarshal(thriftEncodedBytes, &val)

even struct data binding is supported

import ""

type NewOrderRequest struct {
    Lines []NewOrderLine `thrift:",1"`

type NewOrderLine struct {
    ProductId string `thrift:",1"`
    Quantity int `thrift:",2"`

// marshal to thrift
thriftEncodedBytes, err := thrifter.Marshal(NewOrderRequest{
	Lines: []NewOrderLine{
		{"apple", 1},
		{"orange", 2},
// unmarshal back
var val NewOrderRequest
err = thrifter.Unmarshal(thriftEncodedBytes, &val)

without IDL

you do not need to define IDL. you do not need to use static code generation. you do not event need to define struct.

import ""
import ""

var msg general.Message
err := thrifter.Unmarshal(thriftEncodedBytes, &msg)
// the RPC call method name, type is string
// the RPC call arguments, type is general.Struct

what is general.Struct, it is defined as a map

type FieldId int16
type Struct map[FieldId]interface{}

we can extract out specific argument from deeply nested arguments using one line

productId := msg.MessageArgs.Get(
	protocol.FieldId(1), // lines of request
	0, // the first line
	protocol.FieldId(1), // product id

You can unmarshal any thrift bytes into general objects. And you can marshal them back.

Partial decoding

fully decoding into a go struct consumes substantial resources. thrifter provide option to do partial decoding. You can modify part of the message, with untouched parts in []byte form.

import ""
import ""
import ""

// partial decoding
decoder := thrifter.NewDecoder(reader)
var msgHeader protocol.MessageHeader
var msgArgs raw.Struct

// modify...

// encode back
encoder := thrifter.NewEncoder(writer)

the definition of raw.Struct is

type StructField struct {
	Buffer []byte
	Type protocol.TType

type Struct map[protocol.FieldId]StructField


thrifter does not compromise performance.


5000000	       366 ns/op	     144 B/op	      12 allocs/op


1000000	      1549 ns/op	     528 B/op	       9 allocs/op

thrifter by static codegen

5000000	       389 ns/op	     192 B/op	       6 allocs/op

thrifter by reflection

2000000	       585 ns/op	     192 B/op	       6 allocs/op

You can see the reflection implementation is not bad, much faster than the static code generated by thrift original implementation.

To have best performance, you can choose to use static code generation. The api is unchanged, just need to add extra static codegen in your build steps, and include the generated code in your package. The runtime will automatically use the generated encoder/decoder instead of reflection.

For example of static codegen, checkout

Sync IDL and Go Struct

Keep IDL and your object model is challenging. We do not always like the code generated from thrift IDL. But manually keeping the IDL and model in sync is tedious and error prone.

A separate toolchain to manipulate thrift IDL file, and keeping them bidirectionally in sync will be provided in another project.




This section is empty.


View Source
var DefaultConfig = Config{Protocol: ProtocolBinary, StaticCodegen: false}.Froze()


func Marshal

func Marshal(obj interface{}) ([]byte, error)

func MarshalMessage

func MarshalMessage(msg general.Message) ([]byte, error)

MarshalMessage is just a shortcut to demonstrate message decoded by UnmarshalMessage can be encoded back

func NewIterator

func NewIterator(reader io.Reader, buf []byte) spi.Iterator

func NewStream

func NewStream(writer io.Writer, buf []byte) spi.Stream

func ToJSON

func ToJSON(buf []byte) (string, error)

ToJSON convert the thrift message to JSON string

func Unmarshal

func Unmarshal(buf []byte, obj interface{}) error

func UnmarshalMessage

func UnmarshalMessage(buf []byte) (general.Message, error)

UnmarshalMessage demonstrate how to decode thrift binary without IDL into a general message struct


type API

type API interface {
	// NewStream is low level streaming api
	NewStream(writer io.Writer, buf []byte) spi.Stream
	// NewIterator is low level streaming api
	NewIterator(reader io.Reader, buf []byte) spi.Iterator
	// Unmarshal from []byte
	Unmarshal(buf []byte, obj interface{}) error
	// UnmarshalMessage from []byte
	UnmarshalMessage(buf []byte) (general.Message, error)
	// Marshal to []byte
	Marshal(obj interface{}) ([]byte, error)
	// ToJSON convert thrift message to JSON string
	ToJSON(buf []byte) (string, error)
	// MarshalMessage to []byte
	MarshalMessage(msg general.Message) ([]byte, error)
	// NewDecoder to unmarshal from []byte or io.Reader
	NewDecoder(reader io.Reader, buf []byte) *Decoder
	// NewEncoder to marshal to io.Writer
	NewEncoder(writer io.Writer) *Encoder
	// WillDecodeFromBuffer should only be used in generic.Declare
	WillDecodeFromBuffer(sample ...interface{})
	// WillDecodeFromReader should only be used in generic.Declare
	WillDecodeFromReader(sample ...interface{})
	// WillEncode should only be used in generic.Declare
	WillEncode(sample ...interface{})

type Config

type Config struct {
	Protocol      Protocol
	StaticCodegen bool
	Extensions    spi.Extensions

func (Config) AddExtension

func (cfg Config) AddExtension(extension spi.Extension) Config

func (Config) Froze

func (cfg Config) Froze() API

type Decoder

type Decoder struct {
	// contains filtered or unexported fields

func NewDecoder

func NewDecoder(reader io.Reader, buf []byte) *Decoder

func (*Decoder) Decode

func (decoder *Decoder) Decode(val interface{}) error

func (*Decoder) DecodeMessage

func (decoder *Decoder) DecodeMessage() (general.Message, error)

func (*Decoder) DecodeMessageArguments

func (decoder *Decoder) DecodeMessageArguments() (general.Struct, error)

func (*Decoder) DecodeMessageHeader

func (decoder *Decoder) DecodeMessageHeader() (protocol.MessageHeader, error)

func (*Decoder) Reset

func (decoder *Decoder) Reset(reader io.Reader, buf []byte)

type Encoder

type Encoder struct {
	// contains filtered or unexported fields

func NewEncoder

func NewEncoder(writer io.Writer) *Encoder

func (*Encoder) Buffer

func (encoder *Encoder) Buffer() []byte

func (*Encoder) Encode

func (encoder *Encoder) Encode(val interface{}) error

func (*Encoder) EncodeMessage

func (encoder *Encoder) EncodeMessage(msg general.Message) error

func (*Encoder) EncodeMessageArguments

func (encoder *Encoder) EncodeMessageArguments(msgArgs general.Struct) error

func (*Encoder) EncodeMessageHeader

func (encoder *Encoder) EncodeMessageHeader(msgHeader protocol.MessageHeader) error

func (*Encoder) Reset

func (encoder *Encoder) Reset(writer io.Writer)

type Protocol

type Protocol int
var ProtocolBinary Protocol = 1
var ProtocolCompact Protocol = 2

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL