pbparser

README

pbparser

Pbparser is a library for parsing protocol buffer (".proto") files.

Why?

Protocol buffers are a flexible and efficient mechanism for serializing structured data. The Protbuf compiler (protoc) is the source of truth when it comes to parsing proto files. However protoc can be challenging to use in some scenarios :-

• Protoc can be invoked by spawning a process from go code. If the caller now relies on the output of the compiler, they would have to parse the messages on stdout. This is fine for situations which need mere validations of proto files but does not work for usecases which require a standard defined parsed output structure to work with.
• Protoc can also be invoked with --descriptor_set_out option to write out the proto file as a FileDescriptorSet (a protocol buffer defined in descriptor.proto). Ideally, this should have been sufficient. However, this again requires one to write a text parser to parse it.

This parser library is meant to address the above mentioned challenges.

Installing

Using pbparser is easy. First, use go get to install the latest version of the library.

go get -u github.com/tallstoat/pbparser

Next, include pbparser in your application code.

import "github.com/tallstoat/pbparser"

APIs

This library exposes two apis. Both the apis return a ProtoFile datastructure and a non-nil Error if there is an issue in the parse operation itself or the subsequent validations.

func Parse(r io.Reader, p ImportModuleProvider) (ProtoFile, error)

The Parse() function expects the client code to provide a reader for the protobuf content and also a ImportModuleProvider which can be used to callback the client code for any imports in the protobuf content. If there are no imports, the client can choose to pass this as nil.

func ParseFile(file string) (ProtoFile, error)

The ParseFile() function is a utility function which expects the client code to provide only the path of the protobuf file. If there are any imports in the protobuf file, the parser will look for them in the same directory where the protobuf file resides.

Choosing an API

Clients should use the Parse() function if they are not comfortable with letting the pbparser library access the disk directly. This function should also be preferred if the imports in the protobuf file are accessible to the client code but the client code does not want to give pbparser direct access to them. In such cases, the client code has to construct a ImportModuleProvider instance and pass it to the library. This instance must know how to resolve a given "import" and provide a reader for it.

On the other hand, Clients should use the ParseFile() function if all the imported files as well as the protobuf file are on disk relative to the directory in which the protobuf file resides and they are comfortable with letting the pbparser library access the disk directly.

Usage

Please refer to the examples for API usage.

Issues

If you run into any issues or have enhancement suggestions, please create an issue here.

Contributing

1. Fork this repo.
2. Create your feature branch (git checkout -b my-new-feature).
3. Commit your changes (git commit -am 'Add some feature').
4. Push to the branch (git push origin my-new-feature).
5. Create new Pull Request.

License

Pbparser is released under the MIT license. See LICENSE

Documentation

Overview

Package pbparser is a library for parsing protocol buffer (".proto") files.

It exposes two apis for parsing protocol buffer files. Both the apis return a ProtoFile datastructure and a non-nil Error if there is an issue.

After the parsing operation, this library also validates any references to imported constructs i.e. any references to imported enums, messages etc in the file match the definitions in the imported modules.

API

Clients should invoke the following apis :-

func Parse(r io.Reader, p ImportModuleProvider) (ProtoFile, error)

The Parse() function expects the client code to provide a reader for the protobuf content and also a ImportModuleProvider which can be used to callback the client code for any imports in the protobuf content. If there are no imports, the client can choose to pass this as nil.

func ParseFile(file string) (ProtoFile, error)

The ParseFile() function is a utility function which expects the client code to provide only the path of the protobuf file. If there are any imports in the protobuf file, the parser will look for them in the same directory where the protobuf file resides.

Choosing an API

Clients should use the Parse() function if they are not comfortable with letting the pbparser library access the disk directly. This function should also be preferred if the imports in the protobuf file are accessible to the client code but the client code does not want to give pbparser direct access to them. In such cases, the client code has to construct a ImportModuleProvider instance and pass it to the library. This instance must know how to resolve a given "import" and provide a reader for it.

On the other hand, Clients should use the ParseFile() function if all the imported files as well as the protobuf file are on disk relative to the directory in which the protobuf file resides and they are comfortable with letting the pbparser library access the disk directly.

ProtoFile datastructure

This datastructure represents parsed model of the given protobuf file. It includes the following information :-

type ProtoFile struct {
	PackageName        string               // name of the package
	Syntax             string               // the protocol buffer syntax
	Dependencies       []string             // names of any imports
	PublicDependencies []string             // names of any public imports
	Options            []OptionElement      // any package level options
	Enums              []EnumElement        // any defined enums
	Messages           []MessageElement     // any defined messages
	Services           []ServiceElement     // any defined services
	ExtendDeclarations []ExtendElement      // any extends directives
}

Each attribute in turn has a defined structure, which is explained in the godoc of the corresponding elements.

Design Considerations

This library consciously chooses to log no information on it's own. Any failures are communicated back to client code via the returned Error.

In case of a parsing error, it returns an Error back to the client with a line and column number in the file on which the parsing error was encountered.

In case of a post-parsing validation error, it returns an Error with enough information to identify the erroneous protobuf construct.

Example (Parse)

Example code for the Parse() API

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"os"
	"path/filepath"
	"strings"

	"github.com/tallstoat/pbparser"
)

// Example code for the Parse() API
func main() {
	// read the proto file contents from disk & create a reader
	file := "./examples/mathservice.proto"
	raw, err := ioutil.ReadFile(file)
	if err != nil {
		fmt.Printf("Unable to read proto file: %v \n", err)
		os.Exit(-1)
	}
	r := strings.NewReader(string(raw[:]))

	// implement a dir based import module provider which reads
	// import modules from the same dir as the original proto file
	dir := filepath.Dir(file)
	pr := DirBasedImportModuleProvider{dir: dir}

	// invoke Parse() API to parse the file
	pf, err := pbparser.Parse(r, &pr)
	if err != nil {
		fmt.Printf("Unable to parse proto file: %v \n", err)
		os.Exit(-1)
	}

	// print attributes of the returned datastructure
	fmt.Printf("PackageName: %v, Syntax: %v\n", pf.PackageName, pf.Syntax)
}

// DirBasedImportModuleProvider is a import module provider which looks for import
// modules in the dir that it was initialized with.
type DirBasedImportModuleProvider struct {
	dir string
}

func (pi *DirBasedImportModuleProvider) Provide(module string) (io.Reader, error) {
	modulePath := pi.dir + string(filepath.Separator) + module

	// read the module file contents from dir & create a reader...
	raw, err := ioutil.ReadFile(modulePath)
	if err != nil {
		return nil, err
	}

	return strings.NewReader(string(raw[:])), nil
}

Example (ParseFile)

Example code for the ParseFile() API

package main

import (
	"fmt"
	"os"

	"github.com/tallstoat/pbparser"
)

func main() {
	file := "./examples/mathservice.proto"

	// invoke ParseFile() API to parse the file
	pf, err := pbparser.ParseFile(file)
	if err != nil {
		fmt.Printf("Unable to parse proto file: %v \n", err)
		os.Exit(-1)
	}

	// print attributes of the returned datastructure
	fmt.Printf("PackageName: %v, Syntax: %v\n", pf.PackageName, pf.Syntax)
}

Index

• type DataType
• type DataTypeCategory
• type EnumConstantElement
• type EnumElement
• type ExtendElement
• type ExtensionsElement
• type FieldElement
• type ImportModuleProvider
• type MapDataType
  • func (mdt MapDataType) Category() DataTypeCategory
  • func (mdt MapDataType) Name() string
• type MessageElement
• type NamedDataType
  • func (ndt NamedDataType) Category() DataTypeCategory
  • func (ndt NamedDataType) IsStream() bool
  • func (ndt NamedDataType) Name() string
• type OneOfElement
• type OptionElement
• type ProtoFile
  • func Parse(r io.Reader, p ImportModuleProvider) (ProtoFile, error)
  • func ParseFile(file string) (ProtoFile, error)
• type RPCElement
• type ReservedRangeElement
• type ScalarDataType
  • func NewScalarDataType(s string) (ScalarDataType, error)
  • func (sdt ScalarDataType) Category() DataTypeCategory
  • func (sdt ScalarDataType) Name() string
• type ScalarType
• type ServiceElement

Examples

• Package (Parse)
• Package (ParseFile)

Types

type DataType

type DataType interface {
	Name() string
	Category() DataTypeCategory
}

DataType is the interface which must be implemented by the field datatypes. Name() returns the name of the datatype and Category() returns the category of the datatype.

type DataTypeCategory

type DataTypeCategory int

DataTypeCategory is an enumeration which represents the possible kinds of field datatypes in message, oneof and extend declaration constructs.

const (
	ScalarDataTypeCategory DataTypeCategory = iota
	MapDataTypeCategory
	NamedDataTypeCategory
)

type EnumConstantElement

type EnumConstantElement struct {
	Name          string
	Documentation string
	Options       []OptionElement
	Tag           int
}

EnumConstantElement is a datastructure which models the fields within an enum construct. Enum constants can also have inline options specified.

type EnumElement

type EnumElement struct {
	Name          string
	QualifiedName string
	Documentation string
	Options       []OptionElement
	EnumConstants []EnumConstantElement
}

EnumElement is a datastructure which models the enum construct in a protobuf file. Enums are defined standalone or as nested entities within messages.

type ExtendElement

type ExtendElement struct {
	Name          string
	QualifiedName string
	Documentation string
	Fields        []FieldElement
}

ExtendElement is a datastructure which models the extend construct in a protobuf file which is used to add new fields to a previously declared message type.

type ExtensionsElement

type ExtensionsElement struct {
	Documentation string
	Start         int
	End           int
}

ExtensionsElement is a datastructure which models an extensions construct in a protobuf file. An extension is a placeholder for a field whose type is not defined by the original .proto file. This allows other .proto files to add to the original message definition by defining field ranges which can be used for extensions.

type FieldElement

type FieldElement struct {
	Name          string
	Documentation string
	Options       []OptionElement
	Label         string /* optional, required, repeated, oneof */
	Type          DataType
	Tag           int
}

FieldElement is a datastructure which models a field of a message, a field of a oneof element or an entry in the extend declaration in a protobuf file.

type ImportModuleProvider

type ImportModuleProvider interface {
	Provide(module string) (io.Reader, error)
}

ImportModuleProvider is the interface which given a protobuf import module returns a reader for it.

The import module could be on disk or elsewhere. In order for the pbparser library to not be tied in to a specific method of reading the import modules, it exposes this interface to the clients. The clients must provide a implementation of this interface which knows how to interpret the module string & returns a reader for the module. This is needed if the client is calling the Parse() function of the pbparser library.

If the client knows the import modules are on disk, they can instead call the ParseFile() function which internally creates a default import module reader which performs disk access to load the contents of the dependency modules.

type MapDataType

type MapDataType struct {
	// contains filtered or unexported fields
}

MapDataType is a construct which represents a protobuf map datatype.

func (MapDataType) Category

func (mdt MapDataType) Category() DataTypeCategory

Category function implementation of interface DataType for MapDataType

func (MapDataType) Name

func (mdt MapDataType) Name() string

Name function implementation of interface DataType for MapDataType

type MessageElement

type MessageElement struct {
	Name               string
	QualifiedName      string
	Documentation      string
	Options            []OptionElement
	Fields             []FieldElement
	Enums              []EnumElement
	Messages           []MessageElement
	OneOfs             []OneOfElement
	ExtendDeclarations []ExtendElement
	Extensions         []ExtensionsElement
	ReservedRanges     []ReservedRangeElement
	ReservedNames      []string
}

MessageElement is a datastructure which models the message construct in a protobuf file.

type NamedDataType

type NamedDataType struct {
	// contains filtered or unexported fields
}

NamedDataType is a construct which represents a message datatype as a RPC request or response and a message/enum datatype as a field in message, oneof or extend declarations.

func (NamedDataType) Category

func (ndt NamedDataType) Category() DataTypeCategory

Category function implementation of interface DataType for NamedDataType

func (NamedDataType) IsStream

func (ndt NamedDataType) IsStream() bool

IsStream returns true if the NamedDataType is being used in a rpc as a request or response and is preceded by a Stream keyword.

func (NamedDataType) Name

func (ndt NamedDataType) Name() string

Name function implementation of interface DataType for NamedDataType

type OneOfElement

type OneOfElement struct {
	Name          string
	Documentation string
	Options       []OptionElement
	Fields        []FieldElement
}

OneOfElement is a datastructure which models a oneoff construct in a protobuf file. All the fields in a oneof construct share memory, and at most one field can be set at any time.

type OptionElement

type OptionElement struct {
	Name            string
	Value           string
	IsParenthesized bool
}

OptionElement is a datastructure which models the option construct in a protobuf file. Option constructs exist at various levels/contexts like file, message etc.

type ProtoFile

type ProtoFile struct {
	PackageName        string
	Syntax             string
	Dependencies       []string
	PublicDependencies []string
	Options            []OptionElement
	Enums              []EnumElement
	Messages           []MessageElement
	Services           []ServiceElement
	ExtendDeclarations []ExtendElement
}

ProtoFile is a datastructure which represents the parsed model of the given protobuf file.

It includes the package name, the syntax, the import dependencies, any public import dependencies, any options, enums, messages, services, extension declarations etc.

This is populated by the parser & post-validation returned to the client code.

func Parse

func Parse(r io.Reader, p ImportModuleProvider) (ProtoFile, error)

Parse function parses the protobuf content passed to it by the the client code via the reader. It also uses the passed-in ImportModuleProvider to callback the client code for any imports in the protobuf content. If there are no imports, the client can choose to pass this as nil.

This function returns populated ProtoFile struct if parsing is successful. If the parsing or validation fails, it returns an Error.

func ParseFile

func ParseFile(file string) (ProtoFile, error)

ParseFile function reads and parses the content of the protobuf file whose path is provided as sole argument to the function. If there are any imports in the protobuf file, the parser will look for them in the same directory where the protobuf file resides.

This function returns populated ProtoFile struct if parsing is successful. If the parsing or validation fails, it returns an Error.

type RPCElement

type RPCElement struct {
	Name          string
	Documentation string
	Options       []OptionElement
	RequestType   NamedDataType
	ResponseType  NamedDataType
}

RPCElement is a datastructure which models the rpc construct in a protobuf file. RPCs are defined nested within ServiceElements.

type ReservedRangeElement

type ReservedRangeElement struct {
	Documentation string
	Start         int
	End           int
}

ReservedRangeElement is a datastructure which models a reserved construct in a protobuf message.

type ScalarDataType

type ScalarDataType struct {
	// contains filtered or unexported fields
}

ScalarDataType is a construct which represents all supported protobuf scalar datatypes.

func NewScalarDataType

func NewScalarDataType(s string) (ScalarDataType, error)

NewScalarDataType creates and returns a new ScalarDataType for the given string. If a scalar data type mapping does not exist for the given string, an Error is returned.

func (ScalarDataType) Category

func (sdt ScalarDataType) Category() DataTypeCategory

Category function implementation of interface DataType for ScalarDataType

func (ScalarDataType) Name

func (sdt ScalarDataType) Name() string

Name function implementation of interface DataType for ScalarDataType

type ScalarType

type ScalarType int

ScalarType is an enumeration which represents all known supported scalar field datatypes.

const (
	AnyScalar ScalarType = iota + 1
	BoolScalar
	BytesScalar
	DoubleScalar
	FloatScalar
	Fixed32Scalar
	Fixed64Scalar
	Int32Scalar
	Int64Scalar
	Sfixed32Scalar
	Sfixed64Scalar
	Sint32Scalar
	Sint64Scalar
	StringScalar
	Uint32Scalar
	Uint64Scalar
)

type ServiceElement

type ServiceElement struct {
	Name          string
	QualifiedName string
	Documentation string
	Options       []OptionElement
	RPCs          []RPCElement
}

ServiceElement is a datastructure which models the service construct in a protobuf file. Service construct defines the rpcs (apis) for the service.