refmt

README

refmt

refmt is a serialization and object-mapping library.

Why?

Mostly because I have some types which I need to encode in two different ways, and that needs to not suck, and that totally sucks with most serialization libraries I've used. Also, I need to serialize things in different formats, e.g. sometimes JSON and other times CBOR, and that needs to work without me wrestling two different object-serial libraries and configs.

More broadly, I want a single library that can handle my serialization -- with the possibility of different setups on the same types -- and if it can do general object traversals, e.g. a deepcopy, that also seems like... just something that should be natural.

So it seems like there should be some way to define token streams... and a way to define converting objects to and from token streams... and a way to covert token streams to and from serialized bytes... and all of these should be pretty separate!

Thusly was this library thrust into the world: refmt/tok to define the token stream, and refmt/obj to define how to map objects to tokens and back, and refmt/json and refmt/cbor as ways to exchange tokens with serialized formats.

All of these formats can mix-n-match freely, because they communicate values as the standard token stream. Voilà:

• pair obj.Marshaller with json.Encoder to get a json serializer.
• pair cbor.Decoder with obj.Unmarshaller to get a cbor deserializer.
• pair cbor.Decoder with json.Encoder to get a cbor->json streaming transcoder!
• pair obj.Marshaller with obj.Unmarshaller to get a deep-copy system! (Try it with two different types: marshalling a struct and unmarshalling into a freeform map!)

Along the way, we've added a powerful system for defining how exactly the refmt/obj tools should treat your structures: the Atlas system (defined in the refmt/obj/atlas package). Atlases can be used to customize how struct fields are handled, how map keys are sorted, and even define conversions between completely different kinds of objects: serialize arrays as strings, or turn stringy enums into bitfields, no problem. By default, refmt will generate atlases automatically for your structs and types, just like the standard library json marshallers do; if you want more control, atlases give you the power.

An Atlas can be given to each obj.Marshaller and obj.Unmarshaller when it is constructed. This allows great variation in how you wish to handle types -- more than one mapping can be defined for the same concrete type! (This is a killer feature if you need to support multiple versions of an API, for example: you can define 'v1' and 'v2' types, each with their own structs to unmarshal user requests into; then in the backend implement another Marshal/Unmarshal with different atlases which translates the 'v1' requests to 'v2' types, and you only have to implement business logic against the latest types!)

Atlases are significantly more convenient to use than defining custom JSONMarshal() methods. Atlases attach to the type they concern. This means you can use atlases to define custom serialization even for types in packages you can't modify! Atlases also behave better in complex situations: for example, if you have a TypeFoo struct and you wish to serialize it as a string, you don't have to write a custom marshaller for every type that contains a TypeFoo field. Leaking details of custom serialization into the types that contain the interesting objects is a common pitfall when getting into advanced usage of other marshalling libraries; refmt has no such issue.

tl;dr:

• you can swap out atlases for custom serialization on any type;
• and that works for serialization modes even deep in other structures;
• and even on types you don't own!!
• at the same time, you can swap out a cbor encoder for a json encoder, or anything else you want;
• and the mapper part doesn't care -- no complex interactions between the layers.

Come to refmt. It's nicer here.

Where do I start?

If you're already using json.Marshal: switch to json.Marshal (yes, I'm not kidding; just switch your imports!).

If you're already using json.NewEncoder().Encode(): switch to json.NewMarshaller().Marshal().

If you want to get more serial-flexible: try using refmt.Marshal(json.EncodeOptions{}, obj)... then switch to cbor.EncodeOptions{} and see what happens!

If you want to use Atlas to get fancy: go take a peek at the example*.go files in this repo! :)

Happy hacking!

Documentation

Overview

Refmt is a serialization and object-mapping library.

Look to the README on github for more high-level information.

This top-level package exposes simple helper methods for most common operations. You can also compose custom marshallers/unmarshallers and serializer/deserializers by constructing a `refmt.TokenPump` with components from the packages beneath this one. For example, the `refmt.JsonEncode` helper method can be replicated by combining an `obj.Marshaller` with a `json.Encoder`.

Index

• func Clone(src, dst interface{}) error
• func CloneAtlased(src, dst interface{}, atl atlas.Atlas) error
• func Marshal(opts EncodeOptions, v interface{}) ([]byte, error)
• func MarshalAtlased(opts EncodeOptions, v interface{}, atl atlas.Atlas) ([]byte, error)
• func MustClone(src, dst interface{})
• func MustCloneAtlased(src, dst interface{}, atl atlas.Atlas)
• func Unmarshal(opts DecodeOptions, data []byte, v interface{}) error
• func UnmarshalAtlased(opts DecodeOptions, data []byte, v interface{}, atl atlas.Atlas) error
• type Cloner
  • func NewCloner(atl atlas.Atlas) Cloner
• type DecodeOptions
• type EncodeOptions
• type Marshaller
  • func NewMarshaller(opts EncodeOptions, wr io.Writer) Marshaller
  • func NewMarshallerAtlased(opts EncodeOptions, wr io.Writer, atl atlas.Atlas) Marshaller
• type Unmarshaller
  • func NewUnmarshaller(opts DecodeOptions, r io.Reader) Unmarshaller
  • func NewUnmarshallerAtlased(opts DecodeOptions, r io.Reader, atl atlas.Atlas) Unmarshaller

Examples

• NewMarshaller (Json)
• NewMarshaller (Json_prettyprint)
• NewMarshallerAtlased (AutogeneratedAtlas)
• NewMarshallerAtlased (CustomStructMapping)
• NewMarshallerAtlased (TransformFuncs)

Functions

func Clone

func Clone(src, dst interface{}) error

func CloneAtlased

func CloneAtlased(src, dst interface{}, atl atlas.Atlas) error

func Marshal

func Marshal(opts EncodeOptions, v interface{}) ([]byte, error)

func MarshalAtlased

func MarshalAtlased(opts EncodeOptions, v interface{}, atl atlas.Atlas) ([]byte, error)

func MustClone

func MustClone(src, dst interface{})

func MustCloneAtlased

func MustCloneAtlased(src, dst interface{}, atl atlas.Atlas)

func Unmarshal

func Unmarshal(opts DecodeOptions, data []byte, v interface{}) error

func UnmarshalAtlased

func UnmarshalAtlased(opts DecodeOptions, data []byte, v interface{}, atl atlas.Atlas) error

Types

type Cloner

type Cloner interface {
	Clone(src, dst interface{}) error
}

func NewCloner

func NewCloner(atl atlas.Atlas) Cloner

type DecodeOptions

type DecodeOptions interface {
	IsDecodeOptions() // marker method.
}

type EncodeOptions

type EncodeOptions interface {
	IsEncodeOptions() // marker method.
}

type Marshaller

type Marshaller interface {
	Marshal(v interface{}) error
}

func NewMarshaller

func NewMarshaller(opts EncodeOptions, wr io.Writer) Marshaller

Example (Json)

package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
)

func main() {
	var buf bytes.Buffer
	encoder := refmt.NewMarshaller(json.EncodeOptions{}, &buf)
	err := encoder.Marshal(map[string]interface{}{
		"x": "a",
		"y": 1,
	})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{"x":"a","y":1}
<nil>

Example (Json_prettyprint)

package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
)

func main() {
	var buf bytes.Buffer
	encoder := refmt.NewMarshaller(json.EncodeOptions{
		Line:   []byte{'\n'},
		Indent: []byte{'\t'},
	}, &buf)
	err := encoder.Marshal(map[string]interface{}{
		"x": "a",
		"y": 1,
	})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{
	"x": "a",
	"y": 1
}

<nil>

func NewMarshallerAtlased

func NewMarshallerAtlased(opts EncodeOptions, wr io.Writer, atl atlas.Atlas) Marshaller

Example (AutogeneratedAtlas)

package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
	"github.com/polydawn/refmt/obj/atlas"
)

func main() {
	type MyType struct {
		X string
		Y int
	}

	MyType_AtlasEntry := atlas.BuildEntry(MyType{}).
		StructMap().Autogenerate().
		Complete()

	atl := atlas.MustBuild(
		MyType_AtlasEntry,
		// this is a vararg... stack more entries here!
	)

	var buf bytes.Buffer
	encoder := refmt.NewMarshallerAtlased(json.EncodeOptions{}, &buf, atl)
	err := encoder.Marshal(MyType{"a", 1})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{"x":"a","y":1}
<nil>

Example (CustomStructMapping)

package main

import (
	"bytes"
	"fmt"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
	"github.com/polydawn/refmt/obj/atlas"
)

func main() {
	type MyType struct {
		X string
		Y int
	}

	MyType_AtlasEntry := atlas.BuildEntry(MyType{}).
		StructMap().
		AddField("X", atlas.StructMapEntry{SerialName: "overrideName"}).
		// and no "Y" mapping at all!
		Complete()

	atl := atlas.MustBuild(
		MyType_AtlasEntry,
		// this is a vararg... stack more entries here!
	)

	var buf bytes.Buffer
	encoder := refmt.NewMarshallerAtlased(json.EncodeOptions{}, &buf, atl)
	err := encoder.Marshal(MyType{"a", 1})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

{"overrideName":"a"}
<nil>

Example (TransformFuncs)

package main

import (
	"bytes"
	"fmt"
	"strings"

	"github.com/polydawn/refmt"
	"github.com/polydawn/refmt/json"
	"github.com/polydawn/refmt/obj/atlas"
)

func main() {
	type MyType struct {
		X string
		Y string
		Z string
	}

	MyType_AtlasEntry := atlas.BuildEntry(MyType{}).
		Transform().
		TransformMarshal(atlas.MakeMarshalTransformFunc(
			func(x MyType) (string, error) {
				return string(x.X) + ":" + string(x.Y) + ":" + string(x.Z), nil
			})).
		TransformUnmarshal(atlas.MakeUnmarshalTransformFunc(
			func(x string) (MyType, error) {
				ss := strings.Split(x, ":")
				if len(ss) != 3 {
					return MyType{}, fmt.Errorf("parsing MyType: string must have 3 parts, separated by colon")
				}
				return MyType{ss[0], ss[1], ss[2]}, nil
			})).
		Complete()

	atl := atlas.MustBuild(
		MyType_AtlasEntry,
		// this is a vararg... stack more entries here!
	)

	var buf bytes.Buffer
	encoder := refmt.NewMarshallerAtlased(json.EncodeOptions{}, &buf, atl)

	err := encoder.Marshal(MyType{"serializes", "as", "string!"})
	fmt.Println(buf.String())
	fmt.Printf("%v\n", err)

}

Output:

"serializes:as:string!"
<nil>

type Unmarshaller

type Unmarshaller interface {
	Unmarshal(v interface{}) error
}

func NewUnmarshaller

func NewUnmarshaller(opts DecodeOptions, r io.Reader) Unmarshaller

func NewUnmarshallerAtlased

func NewUnmarshallerAtlased(opts DecodeOptions, r io.Reader, atl atlas.Atlas) Unmarshaller