README
¶
Go support for Protocol Buffers
This project hosts the Go implementation for protocol buffers, which is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. The protocol buffer language is a language for specifying the schema for structured data. This schema is compiled into language specific bindings. This project provides both a tool to generate Go code for the protocol buffer language, and also the runtime implementation to handle serialization of messages in Go. See the protocol buffer developer guide for more information about protocol buffers themselves.
This project is comprised of two components:
-
Code generator: The
protoc-gen-go
tool is a compiler plugin toprotoc
, the protocol buffer compiler. It augments theprotoc
compiler so that it knows how to generate Go specific code for a given.proto
file. -
Runtime library: The
protobuf
module contains a set of Go packages that form the runtime implementation of protobufs in Go. This provides the set of interfaces that define what a message is and functionality to serialize message in various formats (e.g., wire, JSON, and text).
See the developer guide for protocol buffers in Go for a general guide for how to get started using protobufs in Go.
This project is the second major revision of the Go protocol buffer API
implemented by the
google.golang.org/protobuf
module. The first major version is implemented by the
github.com/golang/protobuf
module.
Package index
Summary of the packages provided by this module:
proto
: Packageproto
provides functions operating on protobuf messages such as cloning, merging, and checking equality, as well as binary serialization.encoding/protojson
: Packageprotojson
serializes protobuf messages as JSON.encoding/prototext
: Packageprototext
serializes protobuf messages as the text format.encoding/protowire
: Packageprotowire
parses and formats the low-level raw wire encoding. Most users should use packageproto
to serialize messages in the wire format.reflect/protoreflect
: Packageprotoreflect
provides interfaces to dynamically manipulate protobuf messages.reflect/protoregistry
: Packageprotoregistry
provides data structures to register and lookup protobuf descriptor types.reflect/protodesc
: Packageprotodesc
provides functionality for convertingdescriptorpb.FileDescriptorProto
messages to/from the reflectiveprotoreflect.FileDescriptor
.testing/protocmp
: Packageprotocmp
provides protobuf specific options for thecmp
package.testing/protopack
: Packageprotopack
aids manual encoding and decoding of the wire format.testing/prototest
: Packageprototest
exercises the protobuf reflection implementation for concrete message types.types/dynamicpb
: Packagedynamicpb
creates protobuf messages at runtime from protobuf descriptors.types/known/anypb
: Packageanypb
is the generated package forgoogle/protobuf/any.proto
.types/known/timestamppb
: Packagetimestamppb
is the generated package forgoogle/protobuf/timestamp.proto
.types/known/durationpb
: Packagedurationpb
is the generated package forgoogle/protobuf/duration.proto
.types/known/wrapperspb
: Packagewrapperspb
is the generated package forgoogle/protobuf/wrappers.proto
.types/known/structpb
: Packagestructpb
is the generated package forgoogle/protobuf/struct.proto
.types/known/fieldmaskpb
: Packagefieldmaskpb
is the generated package forgoogle/protobuf/field_mask.proto
.types/known/apipb
: Packageapipb
is the generated package forgoogle/protobuf/api.proto
.types/known/typepb
: Packagetypepb
is the generated package forgoogle/protobuf/type.proto
.types/known/sourcecontextpb
: Packagesourcecontextpb
is the generated package forgoogle/protobuf/source_context.proto
.types/known/emptypb
: Packageemptypb
is the generated package forgoogle/protobuf/empty.proto
.types/descriptorpb
: Packagedescriptorpb
is the generated package forgoogle/protobuf/descriptor.proto
.types/pluginpb
: Packagepluginpb
is the generated package forgoogle/protobuf/compiler/plugin.proto
.compiler/protogen
: Packageprotogen
provides support for writing protoc plugins.cmd/protoc-gen-go
: Theprotoc-gen-go
binary is a protoc plugin to generate a Go protocol buffer package.
Reporting issues
The issue tracker for this project is currently located at golang/protobuf.
Please report any issues there with a sufficient description of the bug or feature request. Bug reports should ideally be accompanied by a minimal reproduction of the issue. Irreproducible bugs are difficult to diagnose and fix (and likely to be closed after some period of time). Bug reports must specify the version of the Go protocol buffer module and also the version of the protocol buffer toolchain being used.
Contributing
This project is open-source and accepts contributions. See the contribution guide for more information.
Compatibility
This module and the generated code are expected to be stable over time. However, we reserve the right to make breaking changes without notice for the following reasons:
- Security: A security issue in the specification or implementation may come to light whose resolution requires breaking compatibility. We reserve the right to address such issues.
- Unspecified behavior: There are some aspects of the protocol buffer specification that are undefined. Programs that depend on unspecified behavior may break in future releases.
- Specification changes: It may become necessary to address an inconsistency, incompleteness, or change in the protocol buffer specification, which may affect the behavior of existing programs. We reserve the right to address such changes.
- Bugs: If a package has a bug that violates correctness, a program depending on the buggy behavior may break if the bug is fixed. We reserve the right to fix such bugs.
- Generated additions: We reserve the right to add new declarations to
generated Go packages of
.proto
files. This includes declared constants, variables, functions, types, fields in structs, and methods on types. This may break attempts at injecting additional code on top of what is generated byprotoc-gen-go
. Such practice is not supported by this project. - Internal changes: We reserve the right to add, modify, and remove
internal code, which includes all unexported declarations, the
protoc-gen-go/internal_gengo
package, theruntime/protoimpl
package, and all packages underinternal
.
Any breaking changes outside of these will be announced 6 months in advance to protobuf@googlegroups.com.
Users should use generated code produced by a version of
protoc-gen-go
that is identical to the runtime version provided by the
protobuf module. This
project promises that the runtime remains compatible with code produced by a
version of the generator that is no older than 1 year from the version of the
runtime used, according to the release dates of the minor version. Generated
code is expected to use a runtime version that is at least as new as the
generator used to produce it. Generated code contains references to
protoimpl.EnforceVersion
to statically ensure that the generated code and runtime do not drift
sufficiently far apart.
Historical legacy
This project is the second major revision
(released in 2020)
of the Go protocol buffer API implemented by the
google.golang.org/protobuf
module. The first major version
(released publicly in 2010)
is implemented by the
github.com/golang/protobuf
module.
The first version predates the release of Go 1 by several years. It has a long history as one of the first core pieces of infrastructure software ever written in Go. As such, the Go protobuf project was one of many pioneers for determining what the Go language should even look like and what would eventually be considered good design patterns and “idiomatic” Go (by simultaneously being both positive and negative examples of it).
Consider the changing signature of the proto.Unmarshal
function as an example
of Go language and library evolution throughout the life of this project:
// 2007/09/25 - Conception of Go
// 2008/11/12
export func UnMarshal(r io.Read, pb_e reflect.Empty) *os.Error
// 2008/11/13
export func UnMarshal(buf *[]byte, pb_e reflect.Empty) *os.Error
// 2008/11/24
export func UnMarshal(buf *[]byte, pb_e interface{}) *os.Error
// 2008/12/18
export func UnMarshal(buf []byte, pb_e interface{}) *os.Error
// 2009/01/20
func UnMarshal(buf []byte, pb_e interface{}) *os.Error
// 2009/04/17
func UnMarshal(buf []byte, pb_e interface{}) os.Error
// 2009/05/22
func Unmarshal(buf []byte, pb_e interface{}) os.Error
// 2011/11/03
func Unmarshal(buf []byte, pb_e interface{}) error
// 2012/03/28 - Release of Go 1
// 2012/06/12
func Unmarshal(buf []byte, pb Message) error
These changes demonstrate the difficulty of determining what the right API is for any new technology. It takes time multiplied by many users to determine what is best; even then, “best” is often still somewhere over the horizon.
The change on June 6th, 2012 added a degree of type-safety to Go protobufs by declaring a new interface that all protobuf messages were required to implement:
type Message interface {
Reset()
String() string
ProtoMessage()
}
This interface reduced the set of types that can be passed to proto.Unmarshal
from the universal set of all possible Go types to those with a special
ProtoMessage
marker method. The intention of this change is to limit the
protobuf API to only operate on protobuf data types (i.e., protobuf messages).
For example, there is no sensible operation if a Go channel were passed to the
protobuf API as a channel cannot be serialized. The restricted interface would
prevent that.
This interface does not behaviorally describe what a protobuf message is, but
acts as a marker with an undocumented expectation that protobuf messages must be
a Go struct with a specific layout of fields with formatted tags. This
expectation is not statically enforced by the Go language, for it is an
implementation detail checked dynamically at runtime using Go reflection. Back
in 2012, the only types with this marker were those generated by
protoc-gen-go
. Since protoc-gen-go
would always generate messages with the
proper layout of fields, this was deemed an acceptable and dramatic improvement
over interface{}
.
Over the next 10 years,
use of Go would skyrocket and use of
protobufs in Go would skyrocket as well. With increased popularity also came
more diverse usages and requirements for Go protobufs and an increased number of
custom proto.Message
implementations that were not generated by
protoc-gen-go
.
The increasingly diverse ecosystem of Go types implementing the proto.Message
interface led to incompatibilities, which often occurred when:
-
Passing custom
proto.Message
types to the protobuf APIs: A concrete message implementation might work with some top-level functions (e.g.,proto.Marshal
), but cause others (e.g.,proto.Equal
) to choke and panic. This occurs because the type only had partial support for being an actual message by only implementing theproto.Marshaler
interface or having malformed struct field tags that happened to work with one function, but not another. -
Using Go reflection on any
proto.Message
types: A common desire is to write general-purpose code that operates on any protobuf message. For example, a microservice might want to populate atrace_id
field if it is present in a message. To accomplish this, one would use Go reflection to introspect the message type, and assume it were a pointer to a Go struct with a field namedTraceId
(as would be commonly produced byprotoc-gen-go
). If the concrete message type did not match this expectation, it either failed to work or even resulted in a panic. Such was the case for concrete message types that might be backed by a Go map instead of a Go struct.
Both of these issues are solved by following the idiom that interfaces should describe behavior, not data. This means that the interface itself should provide sufficient functionality through its methods that users can introspect and interact with all aspects of a protobuf message through a principled API. This feature is called protobuf reflection. Just as how Go reflection provides an API for programmatically interacting with any arbitrary Go value, protobuf reflection provides an API for programmatically interacting with any arbitrary protobuf message.
Since an interface cannot be extended in a backwards compatible way, this
suggested the need for a new major version that defines a new proto.Message
interface:
type Message interface {
ProtoReflect() protoreflect.Message
}
The new
proto.Message
interface contains a single ProtoReflect
method that returns a
protoreflect.Message
,
which is a reflective view over a protobuf message. In addition to making a
breaking change to the proto.Message
interface, we took this opportunity to
cleanup the supporting functionality that operate on a proto.Message
, split up
complicated functionality apart into manageable packages, and to hide
implementation details away from the public API.
The goal for this major revision is to improve upon all the benefits of, while addressing all the shortcomings of the old API. We hope that it will serve the Go ecosystem well for the next 10 years and beyond.
Directories
¶
Path | Synopsis |
---|---|
cmd
|
|
protoc-gen-go
The protoc-gen-go binary is a protoc plugin to generate Go code for both proto2 and proto3 versions of the protocol buffer language.
|
The protoc-gen-go binary is a protoc plugin to generate Go code for both proto2 and proto3 versions of the protocol buffer language. |
protoc-gen-go/internal_gengo
Package internal_gengo is internal to the protobuf module.
|
Package internal_gengo is internal to the protobuf module. |
compiler
|
|
protogen
Package protogen provides support for writing protoc plugins.
|
Package protogen provides support for writing protoc plugins. |
encoding
|
|
protojson
Package protojson marshals and unmarshals protocol buffer messages as JSON format.
|
Package protojson marshals and unmarshals protocol buffer messages as JSON format. |
prototext
Package prototext marshals and unmarshals protocol buffer messages as the textproto format.
|
Package prototext marshals and unmarshals protocol buffer messages as the textproto format. |
protowire
Package protowire parses and formats the raw wire encoding.
|
Package protowire parses and formats the raw wire encoding. |
Package proto provides functions operating on protocol buffer messages.
|
Package proto provides functions operating on protocol buffer messages. |
reflect
|
|
protodesc
Package protodesc provides functionality for converting FileDescriptorProto messages to/from protoreflect.FileDescriptor values.
|
Package protodesc provides functionality for converting FileDescriptorProto messages to/from protoreflect.FileDescriptor values. |
protoreflect
Package protoreflect provides interfaces to dynamically manipulate messages.
|
Package protoreflect provides interfaces to dynamically manipulate messages. |
protoregistry
Package protoregistry provides data structures to register and lookup protobuf descriptor types.
|
Package protoregistry provides data structures to register and lookup protobuf descriptor types. |
runtime
|
|
protoiface
Package protoiface contains types referenced or implemented by messages.
|
Package protoiface contains types referenced or implemented by messages. |
protoimpl
Package protoimpl contains the default implementation for messages generated by protoc-gen-go.
|
Package protoimpl contains the default implementation for messages generated by protoc-gen-go. |
testing
|
|
protocmp
Package protocmp provides protobuf specific options for the "github.com/google/go-cmp/cmp" package.
|
Package protocmp provides protobuf specific options for the "github.com/google/go-cmp/cmp" package. |
protopack
Package protopack enables manual encoding and decoding of protobuf wire data.
|
Package protopack enables manual encoding and decoding of protobuf wire data. |
prototest
Package prototest exercises protobuf reflection.
|
Package prototest exercises protobuf reflection. |
types
|
|
dynamicpb
Package dynamicpb creates protocol buffer messages using runtime type information.
|
Package dynamicpb creates protocol buffer messages using runtime type information. |
known/anypb
Package anypb contains generated types for google/protobuf/any.proto.
|
Package anypb contains generated types for google/protobuf/any.proto. |
known/durationpb
Package durationpb contains generated types for google/protobuf/duration.proto.
|
Package durationpb contains generated types for google/protobuf/duration.proto. |
known/fieldmaskpb
Package fieldmaskpb contains generated types for google/protobuf/field_mask.proto.
|
Package fieldmaskpb contains generated types for google/protobuf/field_mask.proto. |
known/structpb
Package structpb contains generated types for google/protobuf/struct.proto.
|
Package structpb contains generated types for google/protobuf/struct.proto. |
known/timestamppb
Package timestamppb contains generated types for google/protobuf/timestamp.proto.
|
Package timestamppb contains generated types for google/protobuf/timestamp.proto. |
internal
|
|
cmd/generate-corpus
Program generate-corpus generates a seed corpus for the fuzzers.
|
Program generate-corpus generates a seed corpus for the fuzzers. |
cmd/pbdump
pbdump is a tool for decoding the wire format for protocol buffer messages.
|
pbdump is a tool for decoding the wire format for protocol buffer messages. |
descfmt
Package descfmt provides functionality to format descriptors.
|
Package descfmt provides functionality to format descriptors. |
descopts
Package descopts contains the nil pointers to concrete descriptor options.
|
Package descopts contains the nil pointers to concrete descriptor options. |
detrand
Package detrand provides deterministically random functionality.
|
Package detrand provides deterministically random functionality. |
encoding/defval
Package defval marshals and unmarshals textual forms of default values.
|
Package defval marshals and unmarshals textual forms of default values. |
encoding/messageset
Package messageset encodes and decodes the obsolete MessageSet wire format.
|
Package messageset encodes and decodes the obsolete MessageSet wire format. |
encoding/tag
Package tag marshals and unmarshals the legacy struct tags as generated by historical versions of protoc-gen-go.
|
Package tag marshals and unmarshals the legacy struct tags as generated by historical versions of protoc-gen-go. |
encoding/text
Package text implements the text format for protocol buffers.
|
Package text implements the text format for protocol buffers. |
errors
Package errors implements functions to manipulate errors.
|
Package errors implements functions to manipulate errors. |
filedesc
Package filedesc provides functionality for constructing descriptors.
|
Package filedesc provides functionality for constructing descriptors. |
filetype
Package filetype provides functionality for wrapping descriptors with Go type information.
|
Package filetype provides functionality for wrapping descriptors with Go type information. |
flags
Package flags provides a set of flags controlled by build tags.
|
Package flags provides a set of flags controlled by build tags. |
fuzz/jsonfuzz
Package jsonfuzz includes fuzzers for protojson.Marshal and protojson.Unmarshal.
|
Package jsonfuzz includes fuzzers for protojson.Marshal and protojson.Unmarshal. |
fuzz/textfuzz
Package textfuzz includes fuzzers for prototext.Marshal and prototext.Unmarshal.
|
Package textfuzz includes fuzzers for prototext.Marshal and prototext.Unmarshal. |
fuzz/wirefuzz
Package wirefuzz includes a fuzzer for the wire marshaler and unmarshaler.
|
Package wirefuzz includes a fuzzer for the wire marshaler and unmarshaler. |
fuzztest
Package fuzztest contains a common fuzzer test.
|
Package fuzztest contains a common fuzzer test. |
genid
Package genid contains constants for declarations in descriptor.proto and the well-known types.
|
Package genid contains constants for declarations in descriptor.proto and the well-known types. |
msgfmt
Package msgfmt implements a text marshaler combining the desirable features of both the JSON and proto text formats.
|
Package msgfmt implements a text marshaler combining the desirable features of both the JSON and proto text formats. |
order
Package order provides ordered access to messages and maps.
|
Package order provides ordered access to messages and maps. |
pragma
Package pragma provides types that can be embedded into a struct to statically enforce or prevent certain language properties.
|
Package pragma provides types that can be embedded into a struct to statically enforce or prevent certain language properties. |
protobuild
Package protobuild constructs messages.
|
Package protobuild constructs messages. |
protolegacy
Package protolegacy is a stub version of the v1 proto package to satisfy internal/testprotos/legacy dependencies.
|
Package protolegacy is a stub version of the v1 proto package to satisfy internal/testprotos/legacy dependencies. |
set
Package set provides simple set data structures for uint64s.
|
Package set provides simple set data structures for uint64s. |
strs
Package strs provides string manipulation functionality specific to protobuf.
|
Package strs provides string manipulation functionality specific to protobuf. |
testprotos/legacy/proto2_20160225_2fc053c5
Package proto2_20160225_2fc053c5 is a generated protocol buffer package.
|
Package proto2_20160225_2fc053c5 is a generated protocol buffer package. |
testprotos/legacy/proto2_20160519_a4ab9ec5
Package proto2_20160519_a4ab9ec5 is a generated protocol buffer package.
|
Package proto2_20160519_a4ab9ec5 is a generated protocol buffer package. |
testprotos/legacy/proto2_20180125_92554152
Package proto2_20180125_92554152 is a generated protocol buffer package.
|
Package proto2_20180125_92554152 is a generated protocol buffer package. |
testprotos/legacy/proto3_20160225_2fc053c5
Package proto3_20160225_2fc053c5 is a generated protocol buffer package.
|
Package proto3_20160225_2fc053c5 is a generated protocol buffer package. |
testprotos/legacy/proto3_20160519_a4ab9ec5
Package proto3_20160519_a4ab9ec5 is a generated protocol buffer package.
|
Package proto3_20160519_a4ab9ec5 is a generated protocol buffer package. |
testprotos/legacy/proto3_20180125_92554152
Package proto3_20180125_92554152 is a generated protocol buffer package.
|
Package proto3_20180125_92554152 is a generated protocol buffer package. |
version
Package version records versioning information about this module.
|
Package version records versioning information about this module. |
weakdeps
Package weakdeps exists to add weak module dependencies.
|
Package weakdeps exists to add weak module dependencies. |