README

GoDoc C.I

Molecule

Molecule is a Go library for parsing protobufs in an efficient and zero-allocation manner. The API is loosely based on this excellent Go JSON parsing library.

This library is in alpha and the API could change. The current APIs are fairly low level, but additional helpers may be added in the future to make certain operations more ergonomic.

Rationale

The standard Unmarshal protobuf interface in Go makes it difficult to manually control allocations when parsing protobufs. In addition, its common to only require access to a subset of an individual protobuf's fields. These issues make it hard to use protobuf in performance critical paths.

This library attempts to solve those problems by introducing a streaming, zero-allocation interface that allows users to have complete control over which fields are parsed, and how/when objects are allocated.

The downside, of course, is that molecule is more difficult to use (and easier to misuse) than the standard protobuf libraries so its recommended that it only be used in situations where performance is important. It is not a general purpose replacement for proto.Unmarshal(). It is recommended that users familiarize themselves with the proto3 encoding before attempting to use this library.

Features

  1. Unmarshal all protobuf primitive types with a streaming, zero-allocation API.
  2. Support for iterating through protobuf messages in a streaming fashion.
  3. Support for iterating through packed protobuf repeated fields (arrays) in a streaming fashion.

Not Supported

  1. Proto2 syntax (some things will probably work, but nothing is tested).
  2. Repeated fields encoded not using the "packed" encoding (although in theory they can be parsed using this library, there just aren't any special helpers).
  3. Map fields. It should be possible to parse maps using this library's API, but it would be a bid tedious. I plan on adding better support for this once I settle on a reasonable API.
  4. Probably lots of other things.

Examples

The godocs have numerous runnable examples.

Attributions

This library is mostly a thin wrapper around other people's work:

  1. The interface was inspired by this jsonparser library.
  2. The codec for interacting with protobuf streams was lifted from this protobuf reflection library. The code was manually vendored instead of imported to reduce dependencies.

Dependencies

The core molecule library has zero external dependencies. The go.sum file does contain some dependencies introduced from the tests package, however, those should not be included transitively when using this library.

Documentation

Overview

    Package molecule is a Go library for parsing protobufs in an efficient and zero-allocation manner.

    Example

      Example demonstrates how the molecule library can be used to parse a protobuf message.

      Output:
      
      StringField: hello world!
      Int64Field: 10
      
      Example (Nested)

        Example_nested demonstrates how to use the MessageEach function to decode a nested message.

        Output:
        
        NestedMessage.StringField: Hello world!
        
        Example (Repeated)

          Example_repeated demonstrates how to use the PackedRepeatedEach function to decode a repeated field encoded in the packed (proto 3) format.

          Output:
          
          Int64s: [1 2 3 4 5 6 7]
          

          Index

          Examples

          Constants

          This section is empty.

          Variables

          This section is empty.

          Functions

          func MessageEach

          func MessageEach(buffer *codec.Buffer, fn MessageEachFn) error

            MessageEach iterates over each top-level field in the message stored in buffer and calls fn on each one.

            Example (SelectAField)

              ExampleMessageEach_SelectAField desmonates how the MessageEach function can be used to select an individual field.

              Output:
              
              StringField: hello world!
              

              func PackedRepeatedEach

              func PackedRepeatedEach(buffer *codec.Buffer, fieldType codec.FieldType, fn PackedRepeatedEachFn) error

                PackedRepeatedEach iterates over each value in the packed repeated field stored in buffer and calls fn on each one.

                The fieldType argument should match the type of the value stored in the repeated field.

                PackedRepeatedEach only supports repeated fields encoded using packed encoding.

                Example

                  ExamplePackedRepeatedEach demonstrates how to use the PackedRepeatedEach function to decode a repeated field encoded in the packed (proto 3) format.

                  Output:
                  
                  Int64s: [1 2 3 4 5 6 7]
                  

                  Types

                  type MessageEachFn

                  type MessageEachFn func(fieldNum int32, value Value) (bool, error)

                    MessageEachFn is a function that will be called for each top-level field in a message passed to MessageEach.

                    type PackedRepeatedEachFn

                    type PackedRepeatedEachFn func(value Value) (bool, error)

                      PackedRepeatedEachFn is a function that is called for each value in a repeated field.

                      type Value

                      type Value struct {
                      	// WireType is the protobuf wire type that was used to encode the field.
                      	WireType codec.WireType
                      	// Number will contain the value for any fields encoded with the
                      	// following wire types:
                      	//
                      	// 1. varint
                      	// 2. Fixed32
                      	// 3. Fixed64
                      	Number uint64
                      	// Bytes will contain the value for any fields encoded with the
                      	// following wire types:
                      	//
                      	// 1. bytes
                      	//
                      	// Bytes is an unsafe view over the bytes in the buffer. To obtain a "safe" copy
                      	// call value.AsSafeBytes() or copy Bytes directly.
                      	Bytes []byte
                      }

                        Value represents a protobuf value. It contains the original wiretype that the value was encoded with as well as a variety of helper methods for interpreting the raw value based on the field's actual type.

                        func (*Value) AsBool

                        func (v *Value) AsBool() (bool, error)

                          AsBool interprets the value as a bool.

                          func (*Value) AsBytesSafe

                          func (v *Value) AsBytesSafe() ([]byte, error)

                            AsBytesSafe interprets the value as a byte slice by allocating a safe copy of the underlying data.

                            func (*Value) AsBytesUnsafe

                            func (v *Value) AsBytesUnsafe() ([]byte, error)

                              AsBytesUnsafe interprets the value as a byte slice. The returned []byte is an unsafe view over the underlying bytes. Use AsBytesSafe() to obtain a "safe" [] that is a copy of the underlying data.

                              func (*Value) AsDouble

                              func (v *Value) AsDouble() (float64, error)

                                AsDouble interprets the value as a double.

                                func (*Value) AsFixed32

                                func (v *Value) AsFixed32() (uint32, error)

                                  AsFixed32 interprets the value as a fixed32.

                                  func (*Value) AsFixed64

                                  func (v *Value) AsFixed64() (uint64, error)

                                    AsFixed64 interprets the value as a fixed64.

                                    func (*Value) AsFloat

                                    func (v *Value) AsFloat() (float32, error)

                                      AsFloat interprets the value as a float.

                                      func (*Value) AsInt32

                                      func (v *Value) AsInt32() (int32, error)

                                        AsInt32 interprets the value as an int32.

                                        func (*Value) AsInt64

                                        func (v *Value) AsInt64() (int64, error)

                                          AsInt64 interprets the value as an int64.

                                          func (*Value) AsSFixed32

                                          func (v *Value) AsSFixed32() (int32, error)

                                            AsSFixed32 interprets the value as a SFixed32.

                                            func (*Value) AsSFixed64

                                            func (v *Value) AsSFixed64() (int64, error)

                                              AsSFixed64 interprets the value as a SFixed64.

                                              func (*Value) AsSint32

                                              func (v *Value) AsSint32() (int32, error)

                                                AsSint32 interprets the value as a sint32.

                                                func (*Value) AsSint64

                                                func (v *Value) AsSint64() (int64, error)

                                                  AsSint64 interprets the value as a sint64.

                                                  func (*Value) AsStringSafe

                                                  func (v *Value) AsStringSafe() (string, error)

                                                    AsStringSafe interprets the value as a string by allocating a safe copy of the underlying data.

                                                    func (*Value) AsStringUnsafe

                                                    func (v *Value) AsStringUnsafe() (string, error)

                                                      AsStringUnsafe interprets the value as a string. The returned string is an unsafe view over the underlying bytes. Use AsStringSafe() to obtain a "safe" string that is a copy of the underlying data.

                                                      func (*Value) AsUint32

                                                      func (v *Value) AsUint32() (uint32, error)

                                                        AsUint32 interprets the value as a uint32.

                                                        func (*Value) AsUint64

                                                        func (v *Value) AsUint64() (uint64, error)

                                                          AsUint64 interprets the value as a uint64.

                                                          Directories

                                                          Path Synopsis
                                                          src
                                                          codec
                                                          Package codec contains all the logic required for interacting with the protobuf raw encoding.
                                                          Package codec contains all the logic required for interacting with the protobuf raw encoding.
                                                          proto
                                                          Package simple is a generated protocol buffer package.
                                                          Package simple is a generated protocol buffer package.