msgp

README

MessagePack Code Generator

This is a code generation tool and serialization library for MessagePack. You can read more about MessagePack in the wiki, or at msgpack.org.

Why?

• Use Go as your schema language
• Performance
• JSON interop
• User-defined extensions
• Type safety
• Encoding flexibility

Quickstart

First install the msgp generator command. Using Go this is done with go install github.com/tinylib/msgp@latest

In a source file, include the following directive:

//go:generate msgp

The msgp command will generate serialization methods for all exported type declarations in the file.

You can read more about the code generation options here.

Use

Field names can be set in much the same way as the encoding/json package. For example:

type Person struct {
	Name       string `msg:"name"`
	Address    string `msg:"address"`
	Age        int    `msg:"age"`
	Hidden     string `msg:"-"` // this field is ignored
	unexported bool             // this field is also ignored
}

By default, the code generator will satisfy msgp.Sizer, msgp.Encodable, msgp.Decodable, msgp.Marshaler, and msgp.Unmarshaler. Carefully-designed applications can use these methods to do marshalling/unmarshalling with zero heap allocations.

While msgp.Marshaler and msgp.Unmarshaler are quite similar to the standard library's json.Marshaler and json.Unmarshaler, msgp.Encodable and msgp.Decodable are useful for stream serialization. (*msgp.Writer and *msgp.Reader are essentially protocol-aware versions of *bufio.Writer and *bufio.Reader, respectively.)

An important thing to note is that msgp operates on individual files. This means if your structs include types defined in other files, these must be processed as well.

Features

• Extremely fast generated code
• Test and benchmark generation
• JSON interoperability (see msgp.CopyToJSON() and msgp.UnmarshalAsJSON())
• Support for complex type declarations
• Native support for Go's time.Time, complex64, and complex128 types
• Generation of both []byte-oriented and io.Reader/io.Writer-oriented methods
• Support for arbitrary type system extensions
• Preprocessor directives
• File-based dependency model means fast codegen regardless of source tree size.

Consider the following:

const Eight = 8
type MyInt int
type Data []byte

type Struct struct {
	Which  map[string]*MyInt `msg:"which"`
	Other  Data              `msg:"other"`
	Nums   [Eight]float64    `msg:"nums"`
}

As long as the declarations of MyInt and Data are in the same file as Struct, the parser will determine that the type information for MyInt and Data can be passed into the definition of Struct before its methods are generated.

Extensions

MessagePack supports defining your own types through "extensions," which are just a tuple of the data "type" (int8) and the raw binary. You can see a worked example in the wiki.

Status

Mostly stable, in that no breaking changes have been made to the /msgp library in more than a year. Newer versions of the code may generate different code than older versions for performance reasons. I (@philhofer) am aware of a number of stability-critical commercial applications that use this code with good results. But, caveat emptor.

You can read more about how msgp maps MessagePack types onto Go types in the wiki.

Here some of the known limitations/restrictions:

• Identifiers from outside the processed source file are assumed (optimistically) to satisfy the generator's interfaces. If this isn't the case, your code will fail to compile.
• Like most serializers, chan and func fields are ignored, as well as non-exported fields.
• Encoding of interface{} is limited to built-ins or types that have explicit encoding methods.
• Maps must have string keys. This is intentional (as it preserves JSON interop.) Although non-string map keys are not forbidden by the MessagePack standard, many serializers impose this restriction. (It also means any well-formed struct can be de-serialized into a map[string]interface{}.) The only exception to this rule is that the deserializers will allow you to read map keys encoded as bin types, due to the fact that some legacy encodings permitted this. (However, those values will still be cast to Go strings, and they will be converted to str types when re-encoded. It is the responsibility of the user to ensure that map keys are UTF-8 safe in this case.) The same rules hold true for JSON translation.

If the output compiles, then there's a pretty good chance things are fine. (Plus, we generate tests for you.) Please, please, please file an issue if you think the generator is writing broken code.

Performance

If you like benchmarks, see here and here.

As one might expect, the generated methods that deal with []byte are faster for small objects, but the io.Reader/Writer methods are generally more memory-efficient (and, at some point, faster) for large (> 2KB) objects.

Documentation

Overview

msgp is a code generation tool for creating methods to serialize and de-serialize Go data structures to and from MessagePack.

This package is targeted at the `go generate` tool. To use it, include the following directive in a go source file with types requiring source generation:

//go:generate msgp

The go generate tool should set the proper environment variables for the generator to execute without any command-line flags. However, the following options are supported, if you need them:

-o = output file name (default is {input}_gen.go)
-file = input file name (or directory; default is $GOFILE, which is set by the `go generate` command)
-io = satisfy the `msgp.Decodable` and `msgp.Encodable` interfaces (default is true)
-marshal = satisfy the `msgp.Marshaler` and `msgp.Unmarshaler` interfaces (default is true)
-tests = generate tests and benchmarks (default is true)

For more information, please read README.md, and the wiki at github.com/tinylib/msgp