csproto

package module
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 23, 2022 License: MIT Imports: 17 Imported by: 1

README

csproto - CrowdStrike's Protocol Buffers library

GoDoc

csproto is a Go module that provides a library for working with Protocol Buffers messages along with a protoc plug-in for generating optimized marshaling and unmarshaling code for those messages.

Like many other companies, CrowdStrike extensively uses Protocol Buffers as an efficient wire format for communicating between disparate processes and services. Protocol Buffers' compatibility guarantees and smaller, more efficient binary encoding made it a natural fit for the problems we needed to solve.

As our data volume continued to grow, CrowdStrike started to run into performance limitations in Google's Protobuf library and transitioned to using Gogo Protobuf in an attempt to overcome the issues. This adjustment proved successful and "Use Gogo Protobuf" became the de facto guidance within our development teams.

Fast forward to 2020 when the maintainers of that library announced that they are looking for people to take over. Unfortunately, this also coincided with Google releasing V2 of their Protobuf API. As new and/or improved functionality was introduced in Google's library, the lack of active maintenance on Gogo inevitably led to incompatibilities.

This created a problem for CrowdStrike. We needed to update our system to no longer depend on Gogo Protobuf but we had a lot of direct dependencies on that code spread throughout our codebase. The solution we arrived at is this library. It provides the "core" pieces of the Protocol Buffers API as used by consumers without having those consumers directly depend on any particular runtime implementation.

If you want to dive right in, there's a handy migration guide. Otherwise, keep reading for all of the technical details.

Disclaimer: csproto is an open source project, not a CrowdStrike product. As such, it carries no formal support, expressed or implied. The project is licensed under the MIT open source license.

Supporting Types Across Runtime Implementations

As part of their V2 API, Google also introduced significant changes to the protoc code generation plug-in, protoc-gen-go. One effect of this change was that code generated using the new plug-in uses the new API internally. An unfortunate side effect is that those types are no longer compatible with Gogo's API.

One technical limitation of Protocol Buffers is that deserializing a message requires knowledge of the actual message type because the encoded field values only contain the integer field tag. Due to this limitation, both Google and Gogo use reflection to read the struct tags on the generated Go types and to dynamically assign field values when unmarshaling Protobuf messages.

This dependence on reflection has created a scenario where passing a type generated by the new plug-in to Gogo's implementation of csproto.Unmarshal() results in failures. Specifically, there are several new fields in the generated code and the reflection-based logic in Gogo's library doesn't know how to treat them. Additionally, several fields that are used by the V1 API, and consequently Gogo's library, are no longer generated.

A Minimal Protocol Buffers API

After a bit of digging, we came up with what we consider the smallest API necessary to support reading and writing Protocol Buffers messages that does not expose any dependence on the runtime implementations.

  • Size(msg interface{}) int
    • Calculate the size, in bytes, required to hold the binary representation of msg
  • Marshal(msg interface{}) ([]byte, error)
    • Convert the contents of msg to the binary representation and return it
  • Unmarshal(p []byte, msg interface{}) error
    • Populate msg with the contents of the binary message in p
  • HasExtension(msg interface{}, ext interface{}) bool
    • Determine if msg contains a proto2 extension field
  • ClearExtension(msg interface{}, ext interface{})
    • Clears a proto2 extension field from msg
  • GetExtension(msg interface{}, ext interface{}) (interface{}, error)
    • Return the value of a proto2 extension field from msg
  • SetExtension(msg interface{}, ext interface{}, val interface{}) error
    • Assign the value of a proto2 extension field on msg

There isn't any common interface shared between Google's two runtimes and Gogo's runtime so our library had to use the empty interface for all message and extension definition parameters.

With this minimal API, services and libraries are able to create and consume Protobuf-encoded messages without being tightly coupled to any specific runtime. Being able to do this was essential for CrowdStrike because it is simply impossible to update everything at once to change which runtime library is in use. Instead, we gradually updated all of our libraries and services to use this new runtime-independent API so that each of our development teams is able to change out their runtime and code generation dependencies independently.

Don't Recreate Everything

Our intent is not to fully recreate the Protocol Buffers runtime. Instead, csproto is built to determine which existing runtime is the "correct" one for a given message and to delegate to that implementation.

We take advantage of the fact that the Go types can't change at runtime to minimize the impact of this indirection. The underlying type of the msg parameter is inspected to determine which of the 3 supported runtimes (Google V1, Gogo, and Google V2) is correct and we store that value in a lookup dictionary so that any given type only has to be inspected once.

Even with this optimization, calling reflect.TypeOf() on the message and performing the lookup has a cost, over 8% in some scenarios! At CrowdStrike's volume even that difference can add up to a non-trivial impact across the system so we needed to find a way to at least break even but, ideally, to better the performance.

The benchmarks below for proto2 marshaling were generated on a 2019 MacBook Pro:

$ go test -run='^$' -bench=. -benchmem
goos: darwin
goarch: amd64
pkg: github.com/CrowdStrike/csproto/example/proto2
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkEncodeGogo-12                   1741834               667.4 ns/op           216 B/op          4 allocs/op
BenchmarkCustomEncodeGogo-12             1785268               669.2 ns/op           216 B/op          4 allocs/op
BenchmarkEncodeGoogleV1-12               1326734               921.7 ns/op           176 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV1-12         1315390               933.5 ns/op           176 B/op          1 allocs/op
BenchmarkEncodeGoogleV2-12               1329092               906.9 ns/op           176 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV2-12         1306638               923.3 ns/op           176 B/op          1 allocs/op

And for proto3:

$ go test -run='^$' -bench=. -benchmem
goos: darwin
goarch: amd64
pkg: github.com/CrowdStrike/csproto/example/proto3
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkEncodeGogo-12                   3008721               394.1 ns/op            88 B/op          2 allocs/op
BenchmarkCustomEncodeGogo-12             2900726               400.1 ns/op            88 B/op          2 allocs/op
BenchmarkEncodeGoogleV1-12               3109386               388.2 ns/op            80 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV1-12         2990907               392.8 ns/op            80 B/op          1 allocs/op
BenchmarkEncodeGoogleV2-12               3290887               367.7 ns/op            80 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV2-12         3003828               398.3 ns/op            80 B/op          1 allocs/op

The table below shows the approximate cost of the indirection across the various combinations of Protobuf runtimes:

Cost proto2 proto3
Gogo +0.27% +1.52%
Google V1 +1.28% +1.18%
Google V2 +1.81% +8.32%

Optimized Protobuf Marshaling and Unmarshaling

The marshaling and unmarshaling implemented by both Google and Gogo necessarily relies on runtime reflection. Both implementations dynamically query the set of fields on the message type and read the associated Protobuf struct tags. This information is then used to match up the field tag and wire type in the encoded data to the corresponding field on the message and to assign the field values. This solution is generic and can be applied to any/all messages without any changes to the implementation but it is necessary slower because it has to inspect each message much more deeply.

Another common source of performance bottlenecks is repeated small allocations. It is, in most cases, far more efficient to allocate one buffer large enough to hold all of the data you need than to incrementally allocate many smaller buffers.

Before moving on, credit must be given to the Vitess team for their vtprotobuf project which they covered in this blog from June of 2021. That project already implements these strategies and more, only with some constraints that didn't work for us. Specifically, vtprotobuf is only compatible with things that are already using Google's V2 API. Given that the inception of this project for CrowdStrike was due to our dependency on Gogo Protobuf we weren't able to make use of their work. We also make significant usage of proto2 extensions, which may or may not be supported by the Vitess tooling.

Protocol Buffers Binary Codec

The first step to improving Protobuf serialization is to implement a binary encoder and decoder that avoids the issues noted in the last section. Additionally, the Protocol Buffer encoding spec has a much smaller surface area than the set of all valid Protobuf messages.

Encoder

The Encoder type wraps a pre-allocated byte slice and sequentially writes encoded field values to it. It is up to the caller to ensure that the provided buffer is large enough to hold the full encoded value. As each encoded field is prefixed by the integer field tag and Protobuf wire type, Encoder's API is provided as a set of EncodeXxx(tag int, val T) methods, one for each supported type of value.

This snippet encodes a boolean true value with a field tag of 1:

// Protobuf binary encoding will require 2 bytes, 1 for the tag/wire type and 1 for the value
buf := make([]byte, 2)
enc := csproto.NewEncoder(buf)
enc.EncodeBool(1, true)
// buf now contains {0x8, 0x1}

Encoding a full message is similar, but using csproto.Size() to calculate the required buffer size.

msg := SomeMessage{
    Name: csproto.String("example"),
    Value: csproto.Int32(42),
    // assign additional fields
}
siz := csproto.Size(msg)
buf := make([]byte, siz)
enc := csproto.NewEncoder(buf)
// encode each field sequentially
enc.EncodeString(1, msg.Name)
enc.EncodeInt32(2, msg.Value)
// ...
Decoder

Like Encoder, the Decoder type wraps a byte slice and sequentially reads field values from it. The Protobuf encoding does not require fields to be in tag order, or present at all for that matter, so decoding a message requires a for loop combined with a switch statement.

func decodeExample(p []byte) (SomeMessage, error) {
    var (
        msg SomeMessage
        s string
        i32 int32
    )
    dec := csproto.NewDecoder(p)
    for dec.More() {
        tag, wireType, err := dec.DecodeTag()
        if err != nil {
            return SomeMessage{}, err
        }
        switch tag {
        case 1: // Name
            if wireType != csproto.WireTypeLengthDelimited {
                return SomeMessage{}, fmt.Errorf("invalid wire type %s, expected %s", wireType, csproto.WireTypeLengthDelimited)
            }
            s, err = dec.DecodeString()
            if err != nil {
                return SomeMessage{}, fmt.Errorf("unable to decode string: %w", err)
            }
            msg.Name = csproto.String(s)
        case 2: // Value
            if wireType != csproto.WireTypeVarint {
                return SomeMessage{}, fmt.Errorf("invalid wire type %s, expected %s", wireType, csproto.WireTypeVarint)
            }
            i32, err = dec.DecodeInt32()
            if err != nil {
                return SomeMessage{}, fmt.Errorf("unable to decode int32: %w", err)
            }
            msg.Value = csproto.Int32(i32)
        default: // unknown/unrecognized field, skip it
            _, _ = dec.Skip(tag, wireType)
        }
    }
}

Notes:

  • The cases in the switch statement use csproto.String() and csproto.Int32() to grab pointers to copies of the decoded values.
  • The example above simply throws away unknown fields which you shouldn't do in practice.
Safe vs Fast

By default, Decoder.DecodeString() will make a full copy of the decoded string. This is the safest, most stable practice but it does come with a small cost in both time and allocations. For scenarios where maximum performance is more desirable, Decoder supports a "fast" mode that uses unsafe to return the bytes of wrapped buffer directly, saving the type conversion and allocation to create a new string value.

...
dec := csproto.NewDecoder(p)
dec.SetMode(proto.DecoderModeFast)
...
s, err := dec.DecodeString()
...

Representative benchmarks from a 2019 MacBook Pro

...> go test -run='^$' -bench=DecodeString -benchmem ./proto
goos: darwin
goarch: amd64
pkg: github.com/CrowdStrike/csproto
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkSafeDecodeString-12            37183212                27.33 ns/op           16 B/op          1 allocs/op
BenchmarkFastDecodeString-12            127437440                9.211 ns/op           0 B/op          0 allocs/op

The trade off for the increased performance is that the behavior is undefined if the wrapped buffer is modified after decoding the field values from it.

Opting In

Now that we have a custom, optimized codec available, we need a way to seamlessly integrate it into the developer workflow. We do that by defining several new interfaces which our API functions will look for when marshaling or unmarshaling messages.

We define 4 single-method interfaces as integration points:

  • csproto.Sizer
    • Size() int: calculates the size, in bytes, needed to hold the encoded contents of the message
      • csproto.Size() will call this method if the message satisfies the interface
  • csproto.Marshaler
    • Marshal() ([]byte, error): returns the binary encoding of the message
      • csproto.Marshal() will call this method if the message satisfies the interface
  • csproto.MarshalerTo
    • MarshalTo([]byte) error: encodes the message into the provided buffer
      • csproto.Marshal() will call this method, after allocating a sufficiently sized buffer, if the messaage satisfies the interface
  • csproto.Unmarshaler
    • Unmarshal([]byte) error: decodes the provided data into the message
      • csproto.Unmarshal() will call this method if the message satisfies the interface

With this in place developers have all of the parts needed to create a fully optimized implementation of Protocol Buffer marshaling and unmarshaling. We can make things even better, though, by capitalizing on the fact that the Protobuf IDL that developers have already written has all of the information we need to generate those optimized implementations.

The protoc plug-in

The final piece of the puzzle is protoc-gen-fastmarshal, a protoc compiler plug-in that reads the Protobuf file descriptor and emits implementations of the Size, Marshal, MarshalTo, and Unmarshal methods for each message defined in the .proto file.

Given this example message

message Example {
  string name   = 1;
  int32  result = 2;
}

the generated code would be roughly as follows

// Size returns the size, in bytes, required to store the contents of m in Protocol Buffers
// binary format.
func (m *Example) Size() int {
  if m == nil {
    return 0
  }
  var (
    sz, l int
  )
  // Name
  if m.Name != nil {
    // key + len + bytes
    l = len(*m.Name)
    sz += csproto.SizeOfVarint(uint64(1)) + csproto.SizeOfVarint(uint64(l)) + l
  }
  // Result
  if m.Result != nil {
    // key + varint
    sz += csproto.SizeOfVarint(uint64(2)) + csproto.SizeOfVarint(uint64(*m.Result))
  }
  // unknown/unrecognized fields
  sz += len(m.unknownFields)
  return sz
}
// Marshal allocates a buffer, writes the contents of m to it using Protocol Buffers binary
// format, then returns the the buffer.
func (m *Example) Marshal() ([]byte, error) {
  sz := m.Size()
  buf := make([]byte, sz)
  err := m.MarshalTo(buf)
  return buf, err
}
// MarshalTo writes the contents of m into dest using Protocol Buffers binary format.
func (m *Example) MarshalTo(dest []byte) error {
  var (
    buf []byte
    err error
  )
  enc := csproto.NewEncoder(dest)
  if m.Name != nil {
    enc.EncodeString(1, *m.Name)
  }
  if m.Result != nil {
    enc.EncodeInt32(2, *m.Result)
  }
  if len(m.unknownFields) > 0 {
    enc.EncodeRaw(m.unknownFields)
  }
  return nil
}
// Unmarshal decodes the Protocol Buffers binary format message in p and populates m with the
// result.
func (m *Example) Unmarshal(p []byte) error {
  if len(p) == 0 {
    return fmt.Errorf("cannot unmarshal from empty buffer")
  }
  var (
    tag int
    wt  csproto.WireType
    err error
  )
  dec := pbtools.NewDecoder(p)
  for dec.More() {
    tag, wt, err = dec.DecodeTag()
    if err != nil {
      return err
    }
    switch tag {
    case 1: // Name
      if wt != csproto.WireTypeLengthDelimited {
        return fmt.Errorf("invalid message data, expected wire type 2 for tag 1, got %v", wt)
      }
      if v, err := dec.DecodeString();  err != nil {
        return fmt.Errorf("unable to decode string value for tag 1: %w", err)
      } else {
        m.Name = csproto.String(v)
      }
    case 2: // Result
      if wt != csproto.WireTypeVarint {
        return fmt.Errorf("invalid message data, expected wire type 0 for tag 2, got %v", wt)
      }
      if v, err := dec.DecodeInt32(); err != nil {
        return fmt.Errorf("unable to decode int32 value for tag 2: %w", err)
      } else {
        m.Result = csproto.Int32(v)
      }
    default: // unrecognized/unknown field
      if skipped, err := dec.Skip(tag, wt); err != nil {
        return fmt.Errorf("invalid operation skipping tag %v: %w", tag, err)
      } else {
        m.unknownFields = append(m.unknownFields, skipped)
      }
    }
  }
  return nil
}
Final Benchmarks

After invoking protoc-gen-fastmarshal, the final benchmarks for our examples are:

$ go test -run='^$' -bench=. -benchmem ./proto2 ./proto3
goos: darwin
goarch: amd64
pkg: github.com/CrowdStrike/csproto/example/proto2
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkEncodeGogo-12                   1357027               858.9 ns/op           352 B/op          2 allocs/op
BenchmarkCustomEncodeGogo-12             2116568               570.4 ns/op           176 B/op          1 allocs/op
BenchmarkEncodeGoogleV1-12               1267740               934.6 ns/op           176 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV1-12         1576399               762.9 ns/op           176 B/op          1 allocs/op
BenchmarkEncodeGoogleV2-12               1308109               913.9 ns/op           176 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV2-12         1641061               738.2 ns/op           176 B/op          1 allocs/op
PASS
ok      github.com/CrowdStrike/csproto/example/proto2   12.247s

goos: darwin
goarch: amd64
pkg: github.com/CrowdStrike/csproto/example/proto3
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkEncodeGogo-12                   3349590               352.8 ns/op           160 B/op          2 allocs/op
BenchmarkCustomEncodeGogo-12             5543668               221.5 ns/op            80 B/op          1 allocs/op
BenchmarkEncodeGoogleV1-12               3090246               389.5 ns/op            80 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV1-12         5504506               215.4 ns/op            80 B/op          1 allocs/op
BenchmarkEncodeGoogleV2-12               3222398               367.3 ns/op            80 B/op          1 allocs/op
BenchmarkCustomEncodeGoogleV2-12         5384648               218.6 ns/op            80 B/op          1 allocs/op
PASS
ok      github.com/CrowdStrike/csproto/example/proto3   9.186s

As you can see in the table below, the optimized code is faster across the board.

Cost proto2 proto3
Gogo -33.6% -37.2%
Google V1 -18.4% -45.0%
Google V2 -19.2% -40.5%

gRPC

To use with gRPC you will need to register csproto as the encoder. NOTE: If messages do not implement Marshaler or Unmarshaler then an error will be returned. An example is below.

For more information, see the gRPC documentation.

import (
  "github.com/CrowdStrike/csproto"
  "google.golang.org/grpc/encoding"
  _ "google.golang.org/grpc/encoding/proto"
)

func init() {
  encoding.RegisterCodec(csproto.GrpcCodec{})
}

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrInvalidFieldTag is returned by the decoder when it fails to read a varint-encoded field tag/wire type value.
	ErrInvalidFieldTag = errors.New("unable to read protobuf field tag")
	// ErrInvalidVarintData is returned by the decoder when it fails to read a varint-encoded value.
	ErrInvalidVarintData = errors.New("unable to read protobuf varint value")
	// ErrValueOverflow is returned by DecodeUInt32() or DecodeInt32() when the decoded value is too large for a 32-bit value.
	ErrValueOverflow = errors.New("value overflow trying to read protobuf varint value")
	// ErrInvalidZigZagData is returned by the decoder when it fails to read a zigzag-encoded value.
	ErrInvalidZigZagData = errors.New("unable to read protobuf zigzag value")
	// ErrInvalidFixed32Data is returned by the decoder when it fails to read a fixed-size 32-bit value.
	ErrInvalidFixed32Data = errors.New("unable to read protobuf fixed 32-bit value")
	// ErrInvalidFixed64Data is returned by the decoder when it fails to read a fixed-size 64-bit value.
	ErrInvalidFixed64Data = errors.New("unable to read protobuf fixed 64-bit value")
	// ErrInvalidPackedData is returned by the decoder when it fails to read a packed repeated value.
	ErrInvalidPackedData = errors.New("unable to read protobuf packed value")
)
View Source 
var (
	// ErrMarshaler is returned the Marshal() function when a message is passed in that does not
	// match any of the supported behaviors
	ErrMarshaler = errors.New("message does not implement csproto.Marshaler")
	// ErrUnmarshaler is returned the Unmarshal() function when a message is passed in that does not
	// match any of the supported behaviors
	ErrUnmarshaler = errors.New("message does not implement csproto.Unmarshaler")
)

Functions

func Bool 

func Bool(v bool) *bool

Bool returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func ClearExtension added in v0.3.0 

func ClearExtension(msg interface{}, ext interface{})

ClearExtension removes a proto2 extension field from msg, if it exists, delegating to the appropriate underlying Protobuf API based on the concrete type of msg.

This function panics if the provded parameters are invalid, rather than returning an error, to be consistent with the signature of the ClearExtension() functions in the underlying Protobuf runtimes.

func EncodeTag 

func EncodeTag(dest []byte, tag int, wireType WireType) int

EncodeTag combines tag and wireType then encodes the result into dest using the Protobuf varint format and returns the number of bytes written.

func EncodeVarint 

func EncodeVarint(dest []byte, v uint64) int

EncodeVarint encodes v into dest using the Protobuf base-128 varint format and returns the number of bytes written

func EncodeZigZag32 

func EncodeZigZag32(dest []byte, v int32) int

EncodeZigZag32 encodes v into dest using the Protobuf zig/zag encoding for more efficient encoding of negative numbers, and returns the number of bytes written.

func EncodeZigZag64 

func EncodeZigZag64(dest []byte, v int64) int

EncodeZigZag64 encodes v into dest using the Protobuf zig/zag encoding for more efficient encoding of negative numbers, and returns the number of bytes written.

func Float32 

func Float32(v float32) *float32

Float32 returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func Float64 

func Float64(v float64) *float64

Float64 returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func GetExtension 

func GetExtension(msg interface{}, ext interface{}) (interface{}, error)

GetExtension returns a proto2 extension field from msg, delegating to the appropriate underlying Protobuf API based on the concrete type of msg.

func HasExtension 

func HasExtension(msg interface{}, ext interface{}) bool

HasExtension returns true if msg contains the specified proto2 extension field, delegating to the appropriate underlying Protobuf API based on the concrete type of msg.

func Int 

func Int(v int) *int32

Int returns a pointer to v as an int32 value (for use when assigning pointer fields on Protobuf message types)

func Int32 

func Int32(v int32) *int32

Int32 returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func Int64 

func Int64(v int64) *int64

Int64 returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func JSONMarshaler added in v0.2.0 

func JSONMarshaler(msg interface{}, opts ...JSONOption) json.Marshaler

JSONMarshaler returns an implementation of the json.Marshaler interface that formats msg to JSON using the specified options.

func Marshal 

func Marshal(msg interface{}) ([]byte, error)

Marshal marshals msg to binary Protobuf format, delegating to the appropriate underlying Protobuf API based on the concrete type of msg.

func PointerTo 

func PointerTo[T NativeTypes](v T) *T

PointerTo makes a copy of v and returns a pointer to that copy

func SetExtension 

func SetExtension(msg interface{}, ext interface{}, val interface{}) error

SetExtension sets a proto2 extension field in msg to the provided value, delegating to the appropriate underlying Protobuf API based on the concrete type of msg.

func Size 

func Size(msg interface{}) int

Size returns the encoded size of msg.

func SizeOfTagKey 

func SizeOfTagKey(k int) int

SizeOfTagKey returns the number of bytes required to hold the Protobuf varint encoding of k.

func SizeOfVarint 

func SizeOfVarint(v uint64) int

SizeOfVarint returns the number of bytes required to hold the Protobuf varint encoding of v.

func SizeOfZigZag 

func SizeOfZigZag(v uint64) int

SizeOfZigZag returns the number of bytes required to hold the zig zag encoding of v.

func String 

func String(v string) *string

String returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func Uint32 

func Uint32(v uint32) *uint32

Uint32 returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func Uint64 

func Uint64(v uint64) *uint64

Uint64 returns a pointer to v (for use when assigning pointer fields on Protobuf message types)

func Unmarshal 

func Unmarshal(data []byte, msg interface{}) error

Unmarshal decodes the specified Protobuf data into msg, delegating to the appropriate underlying Protobuf API based on the concrete type of msg.

Types

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder implements a binary Protobuf Decoder by sequentially reading from a provided []byte.

func NewDecoder 

func NewDecoder(p []byte) *Decoder

NewDecoder initializes a new Protobuf decoder to read the provided buffer.

func (*Decoder) DecodeBool 

func (d *Decoder) DecodeBool() (b bool, err error)

DecodeBool decodes a boolean value from the stream and returns the value.

func (*Decoder) DecodeBytes 

func (d *Decoder) DecodeBytes() ([]byte, error)

DecodeBytes decodes a length-delimited slice of bytes from the stream and returns the value.

func (*Decoder) DecodeFixed32 

func (d *Decoder) DecodeFixed32() (uint32, error)

DecodeFixed32 decodes a 4-byte integer from the stream and returns the value.

func (*Decoder) DecodeFixed64 

func (d *Decoder) DecodeFixed64() (uint64, error)

DecodeFixed64 decodes an 8-byte integer from the stream and returns the value.

func (*Decoder) DecodeFloat32 

func (d *Decoder) DecodeFloat32() (float32, error)

DecodeFloat32 decodes a 4-byte IEEE 754 floating point value from the stream and returns the value.

func (*Decoder) DecodeFloat64 

func (d *Decoder) DecodeFloat64() (float64, error)

DecodeFloat64 decodes an 8-byte IEEE 754 floating point value from the stream and returns the value.

func (*Decoder) DecodeInt32 

func (d *Decoder) DecodeInt32() (int32, error)

DecodeInt32 decodes a varint-encoded 32-bit integer from the stream and returns the value.

func (*Decoder) DecodeInt64 

func (d *Decoder) DecodeInt64() (int64, error)

DecodeInt64 decodes a varint-encoded 64-bit integer from the stream and returns the value.

func (*Decoder) DecodeNested 

func (d *Decoder) DecodeNested(m interface{}) error

DecodeNested decodes a nested Protobuf message from the stream into m. If m satisfies our csproto.Unmarshaler interface its Unmarshal() method will be called. Otherwise, this method delegates to Marshal().

func (*Decoder) DecodePackedBool 

func (d *Decoder) DecodePackedBool() ([]bool, error)

DecodePackedBool decodes a packed encoded list of boolean values from the stream and returns the value.

func (*Decoder) DecodePackedFixed32 

func (d *Decoder) DecodePackedFixed32() ([]uint32, error)

DecodePackedFixed32 decodes a packed encoded list of 32-bit fixed-width integers from the stream and returns the value.

func (*Decoder) DecodePackedFixed64 

func (d *Decoder) DecodePackedFixed64() ([]uint64, error)

DecodePackedFixed64 decodes a packed encoded list of 64-bit fixed-width integers from the stream and returns the value.

func (*Decoder) DecodePackedFloat32 

func (d *Decoder) DecodePackedFloat32() ([]float32, error)

DecodePackedFloat32 decodes a packed encoded list of 32-bit floating point numbers from the stream and returns the value.

func (*Decoder) DecodePackedFloat64 

func (d *Decoder) DecodePackedFloat64() ([]float64, error)

DecodePackedFloat64 decodes a packed encoded list of 64-bit floating point numbers from the stream and returns the value.

func (*Decoder) DecodePackedInt32 

func (d *Decoder) DecodePackedInt32() ([]int32, error)

DecodePackedInt32 decodes a packed encoded list of 32-bit integers from the stream and returns the value.

func (*Decoder) DecodePackedInt64 

func (d *Decoder) DecodePackedInt64() ([]int64, error)

DecodePackedInt64 decodes a packed encoded list of 64-bit integers from the stream and returns the value.

func (*Decoder) DecodePackedSint32 

func (d *Decoder) DecodePackedSint32() ([]int32, error)

DecodePackedSint32 decodes a packed encoded list of 32-bit signed integers from the stream and returns the value.

func (*Decoder) DecodePackedSint64 

func (d *Decoder) DecodePackedSint64() ([]int64, error)

DecodePackedSint64 decodes a packed encoded list of 64-bit signed integers from the stream and returns the value.

func (*Decoder) DecodePackedUint32 

func (d *Decoder) DecodePackedUint32() ([]uint32, error)

DecodePackedUint32 decodes a packed encoded list of unsigned 32-bit integers from the stream and returns the value.

func (*Decoder) DecodePackedUint64 

func (d *Decoder) DecodePackedUint64() ([]uint64, error)

DecodePackedUint64 decodes a packed encoded list of unsigned 64-bit integers from the stream and returns the value.

func (*Decoder) DecodeSInt32 

func (d *Decoder) DecodeSInt32() (int32, error)

DecodeSInt32 decodes a zigzag-encoded 32-bit integer from the stream and returns the value.

func (*Decoder) DecodeSInt64 

func (d *Decoder) DecodeSInt64() (int64, error)

DecodeSInt64 decodes a zigzag-encoded 32-bit integer from the stream and returns the value.

func (*Decoder) DecodeString 

func (d *Decoder) DecodeString() (string, error)

DecodeString decodes a length-delimited string from the stream and returns the value.

func (*Decoder) DecodeTag 

func (d *Decoder) DecodeTag() (tag int, wireType WireType, err error)

DecodeTag decodes a field tag and Protobuf wire type from the stream and returns the values.

func (*Decoder) DecodeUInt32 

func (d *Decoder) DecodeUInt32() (uint32, error)

DecodeUInt32 decodes a varint-encoded 32-bit unsigned integer from the stream and returns the value.

func (*Decoder) DecodeUInt64 

func (d *Decoder) DecodeUInt64() (uint64, error)

DecodeUInt64 decodes a varint-encoded 64-bit unsigned integer from the stream and returns the value.

func (*Decoder) More 

func (d *Decoder) More() bool

More indicates if there is more data to be read in the buffer.

func (*Decoder) Reset 

func (d *Decoder) Reset()

Reset moves the read offset back to the beginning of the encoded data

func (*Decoder) SetMode 

func (d *Decoder) SetMode(m DecoderMode)

SetMode configures the decoding behavior, safe vs fastest.

func (*Decoder) Skip 

func (d *Decoder) Skip(tag int, wt WireType) ([]byte, error)

Skip skips over the encoded field value at the current offset, returning the raw bytes so that the caller can decide what to do with the data.

The tag and wire type are validated against the provided values and a DecoderSkipError error is returned if they do not match. This check is skipped when using "fast" mode.

type DecoderMode 

type DecoderMode int

DecoderMode defines the behavior of the decoder (safe vs fastest). 

const (
	// DecoderModeSafe instructs the decoder to only use safe operations when decoding values.
	DecoderModeSafe DecoderMode = iota
	// DecoderModeFast instructs the decoder to use unsafe operations to avoid allocations and copying data
	// for the fastest throughput.
	//
	// When using DecoderModeFast, the byte slice passed to the decoder must not be modified after
	// using the decoder to extract values.  The behavior is undefined if the slice is modified.
	DecoderModeFast
)

func (DecoderMode) String 

func (m DecoderMode) String() string

String returns a string representation of m, "safe" or "fast".

type DecoderSkipError 

type DecoderSkipError struct {
	ExpectedTag      int
	ExpectedWireType WireType
	ActualTag        int
	ActualWireType   WireType
}

DecoderSkipError defines an error returned by the decoder's Skip() method when the specified tag and wire type do not match the data in the stream at the current decoder offset.

func (*DecoderSkipError) Error 

func (e *DecoderSkipError) Error() string

Error satisfies the error interface

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder implements a binary Protobuf Encoder by sequentially writing to a wrapped []byte.

func NewEncoder 

func NewEncoder(p []byte) *Encoder

NewEncoder initializes a new Protobuf encoder to write to the specified buffer, which must be pre-allocated by the caller with sufficient space to hold the message(s) being written.

func (*Encoder) EncodeBool 

func (e *Encoder) EncodeBool(tag int, v bool)

EncodeBool writes a varint-encoded boolean value to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeBytes 

func (e *Encoder) EncodeBytes(tag int, v []byte)

EncodeBytes writes a length-delimited byte slice to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeFixed32 

func (e *Encoder) EncodeFixed32(tag int, v uint32)

EncodeFixed32 writes a 32-bit unsigned integer value to the buffer using 4 bytes in little endian format, preceded by the varint-encoded tag key.

func (*Encoder) EncodeFixed64 

func (e *Encoder) EncodeFixed64(tag int, v uint64)

EncodeFixed64 writes a 64-bit unsigned integer value to the buffer using 8 bytes in little endian format, preceded by the varint-encoded tag key.

func (*Encoder) EncodeFloat32 

func (e *Encoder) EncodeFloat32(tag int, v float32)

EncodeFloat32 writes a 32-bit IEEE 754 floating point value to the buffer using 4 bytes in little endian format, preceded by the varint-encoded tag key.

func (*Encoder) EncodeFloat64 

func (e *Encoder) EncodeFloat64(tag int, v float64)

EncodeFloat64 writes a 64-bit IEEE 754 floating point value to the buffer using 8 bytes in little endian format, preceded by the varint-encoded tag key.

func (*Encoder) EncodeInt32 

func (e *Encoder) EncodeInt32(tag int, v int32)

EncodeInt32 writes a varint-encoded 32-bit signed integer value to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeInt64 

func (e *Encoder) EncodeInt64(tag int, v int64)

EncodeInt64 writes a varint-encoded 64-bit signed integer value to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeMapEntryHeader 

func (e *Encoder) EncodeMapEntryHeader(tag int, size int)

EncodeMapEntryHeader writes a map entry header into the buffer, which consists of the specified tag with a wire type of WireTypeLengthDelimited followed by the varint encoded entry size.

func (*Encoder) EncodeNested 

func (e *Encoder) EncodeNested(tag int, m interface{}) error

EncodeNested writes a nested message to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedBool 

func (e *Encoder) EncodePackedBool(tag int, vs []bool)

EncodePackedBool writes a list of booleans to the buffer using packed encoding, preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedFixed32 

func (e *Encoder) EncodePackedFixed32(tag int, vs []uint32)

EncodePackedFixed32 writes a list of 32-bit fixed-width unsigned integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedFixed64 

func (e *Encoder) EncodePackedFixed64(tag int, vs []uint64)

EncodePackedFixed64 writes a list of 64-bit fixed-width unsigned integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedFloat32 

func (e *Encoder) EncodePackedFloat32(tag int, vs []float32)

EncodePackedFloat32 writes a list of 32-bit floating point numbers to the buffer using packed encoding, preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedFloat64 

func (e *Encoder) EncodePackedFloat64(tag int, vs []float64)

EncodePackedFloat64 writes a list of 64-bit floating point numbers to the buffer using packed encoding, preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedInt32 

func (e *Encoder) EncodePackedInt32(tag int, vs []int32)

EncodePackedInt32 writes a list of 32-bit integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

This operation is O(n^2) because we have to traverse the list of values to calculate the total encoded size and write that size *before* the actual encoded values.

func (*Encoder) EncodePackedInt64 

func (e *Encoder) EncodePackedInt64(tag int, vs []int64)

EncodePackedInt64 writes a list of 64-bit integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

This operation is O(n^2) because we have to traverse the list of values to calculate the total encoded size and write that size *before* the actual encoded values.

func (*Encoder) EncodePackedSFixed32 

func (e *Encoder) EncodePackedSFixed32(tag int, vs []int32)

EncodePackedSFixed32 writes a list of 32-bit fixed-width signed integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedSFixed64 

func (e *Encoder) EncodePackedSFixed64(tag int, vs []int64)

EncodePackedSFixed64 writes a list of 64-bit fixed-width signed integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

func (*Encoder) EncodePackedSInt32 

func (e *Encoder) EncodePackedSInt32(tag int, vs []int32)

EncodePackedSInt32 writes a list of 32-bit signed integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

This operation is O(n^2) because we have to traverse the list of values to calculate the total encoded size and write that size *before* the actual encoded values.

func (*Encoder) EncodePackedSInt64 

func (e *Encoder) EncodePackedSInt64(tag int, vs []int64)

EncodePackedSInt64 writes a list of 64-bit signed integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

This operation is O(n^2) because we have to traverse the list of values to calculate the total encoded size and write that size *before* the actual encoded values.

func (*Encoder) EncodePackedUInt32 

func (e *Encoder) EncodePackedUInt32(tag int, vs []uint32)

EncodePackedUInt32 writes a list of 32-bit unsigned integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

This operation is O(n^2) because we have to traverse the list of values to calculate the total encoded size and write that size *before* the actual encoded values.

func (*Encoder) EncodePackedUInt64 

func (e *Encoder) EncodePackedUInt64(tag int, vs []uint64)

EncodePackedUInt64 writes a list of 64-bit unsigned integers to the buffer using packed encoding, preceded by the varint-encoded tag key.

This operation is O(n^2) because we have to traverse the list of values to calculate the total encoded size and write that size *before* the actual encoded values.

func (*Encoder) EncodeRaw 

func (e *Encoder) EncodeRaw(d []byte)

EncodeRaw writes the raw bytes of d into the buffer at the current offset

func (*Encoder) EncodeSInt32 

func (e *Encoder) EncodeSInt32(tag int, v int32)

EncodeSInt32 writes a zigzag-encoded 32-bit signed integer value to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeSInt64 

func (e *Encoder) EncodeSInt64(tag int, v int64)

EncodeSInt64 writes a zigzag-encoded 64-bit signed integer value to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeString 

func (e *Encoder) EncodeString(tag int, s string)

EncodeString writes a length-delimited string value to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeUInt32 

func (e *Encoder) EncodeUInt32(tag int, v uint32)

EncodeUInt32 writes a varint-encoded 32-bit unsigned integer value to the buffer preceded by the varint-encoded tag key.

func (*Encoder) EncodeUInt64 

func (e *Encoder) EncodeUInt64(tag int, v uint64)

EncodeUInt64 writes a varint-encoded 64-bit unsigned integer value to the buffer preceded by the varint-encoded tag key.

type GrpcCodec 

type GrpcCodec struct{}

GrpcCodec implements gRPC encoding.Codec. See: https://pkg.go.dev/google.golang.org/grpc/encoding#Codec

func (GrpcCodec) Marshal 

func (GrpcCodec) Marshal(v interface{}) ([]byte, error)

Marshal returns the wire format of v.

func (GrpcCodec) Name 

func (GrpcCodec) Name() string

Name returns the name of the Codec implementation.

func (GrpcCodec) Unmarshal 

func (GrpcCodec) Unmarshal(data []byte, v interface{}) error

Unmarshal parses the wire format into v.

type JSONOption added in v0.2.0 

type JSONOption func(*jsonOptions)

JSONOption defines a function that sets a specific JSON formatting option

func JSONIncludeZeroValues added in v0.2.0 

func JSONIncludeZeroValues(emitZeroValues bool) JSONOption

JSONIncludeZeroValues returns a JSON option that enables or disables including zero-valued fields in the JSON output.

func JSONIndent added in v0.2.0 

func JSONIndent(indent string) JSONOption

JSONIndent returns a JSONOption that configures the JSON indentation.

Passing an empty string disables indentation. If not empty, indent must consist of only spaces or tab characters.

func JSONUseEnumNumbers added in v0.2.0 

func JSONUseEnumNumbers(useNumbers bool) JSONOption

JSONUseEnumNumbers returns a JSON option that enables or disables outputting integer values rather than the enum names for enum fields.

type Marshaler 

type Marshaler interface {
	// Marshal allocates a buffer large enough to hold this message, writes the contents of the
	// message into it, then returns the buffer.
	Marshal() ([]byte, error)
}

Marshaler defines a message that can marshal itself into binary Protobuf format.

type MarshalerTo 

type MarshalerTo interface {
	// MarshalTo writes the contents of this message into dest using the binary Protobuf encoding.
	//
	// It is up to the caller to ensure that the buffer is large enough to hold the serialized message
	// data.
	MarshalTo(dest []byte) error
}

MarshalerTo defines a message that can marshal itself into a provided buffer in binary Protobuf format.

type MessageType 

type MessageType int

MessageType defines the types of Protobuf message implementations this API supports. 

const (
	// MessageTypeUnknown indicates that we don't know which type of Protobuf message a type is.
	MessageTypeUnknown MessageType = iota
	// MessageTypeGogo indicates that a type was generated using Gogo Protobuf.
	MessageTypeGogo
	// MessageTypeGoogleV1 indicates that a type was generated using v1 of Google's Protobuf tools.
	MessageTypeGoogleV1
	// MessageTypeGoogle indicates that a type was generated using v2 of Google's Protobuf tools.
	MessageTypeGoogle
)

func MsgType 

func MsgType(msg interface{}) MessageType

MsgType accepts a protobuf message and returns the corresponding MessageType value.

type NativeTypes 

type NativeTypes interface {
	bool | int32 | int64 | uint32 | uint64 | float32 | float64 | string
}

NativeTypes defines a generic constraint for the native Go types that need to be converted to pointers when storing values in generated Protobuf message structs.

Unsized integers (type int) also need to be converted to a pointer, but Protobuf doesn't support unsized integers. Use csproto.Int(v int) *int32 instead.

type ProtoV1Marshaler 

type ProtoV1Marshaler interface {
	ProtoV1Sizer
	XXX_Marshal(b []byte, deterministic bool) ([]byte, error)
}

ProtoV1Marshaler defines the interface for a type that provides custom Protobuf V1 marshaling logic.

type ProtoV1Sizer 

type ProtoV1Sizer interface {
	XXX_Size() int
}

ProtoV1Sizer defines the interface for a type that provides custom Protobuf V1 sizing logic.

type ProtoV1Unmarshaler 

type ProtoV1Unmarshaler interface {
	XXX_Unmarshal([]byte) error
}

ProtoV1Unmarshaler defines the interface a type that provides custom Protobuf V1 unmarshaling.

type Sizer 

type Sizer interface {
	// Size returns the size, in bytes, required to serialize a message into binary Protobuf format.
	Size() int
}

Sizer defines a message that can pre-calculate the required size for storing the binary Protobuf encoding of its contents.

type Unmarshaler 

type Unmarshaler interface {
	// Unmarshal decodes the binary Protobuf data into this message.
	Unmarshal([]byte) error
}

Unmarshaler defines a message that can unmarshal binary Protobuf data into itself.

type WireType 

type WireType int

WireType defines the supported values for Protobuf wire encodings. 

const (
	// WireTypeVarint denotes a value that is encoded using base128 varint encoding.
	WireTypeVarint WireType = 0
	// WireTypeFixed64 denotes a value that is encoded using 8 bytes.
	WireTypeFixed64 WireType = 1
	// WireTypeLengthDelimited denotes a value that is encoded as a sequence of bytes preceded by a
	// varint-encoded length.
	WireTypeLengthDelimited WireType = 2
	// WireTypeFixed32 denotes a value that is encoded using 4 bytes.
	WireTypeFixed32 WireType = 5
)

func (WireType) String 

func (wt WireType) String() string

String returns a string representation of wt.

Source Files

View all Source files

Directories

Path Synopsis
cmd
protoc-gen-fastmarshal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.