protostructure

README

Fork of mitchellh/protostructure tuned to CloudQuery use-case. This is used to send struct defs over gRPC from CQ plugins to main process when SQLite is used as a backend so plugins will be CGO free.

protostructure

protostructure is a Go library for encoding and decoding a struct type over the wire.

This library is useful when you want to send arbitrary structures over protocol buffers for behavior such as configuration decoding (encoding/json, etc.), validation (using packages that use tags), etc. This works because we can reconstruct the struct type dynamically using reflect including any field tags.

This library only sends the structure of the struct, not the value. If you want to send the value, you should build your protocol buffer message in such a way that it encodes that somehow using something such as JSON.

Installation

Standard go get:

$ go get github.com/mitchellh/protostructure

Usage & Example

For usage and examples see the Godoc.

A quick code example is shown below using both the imaginary proto file and the Go code that uses it.

syntax = "proto3";
package myapp;
import "protostructure.proto";

// Response is an example response structure for an RPC endpoint.
message Response {
	protostructure.Struct config = 1;
}

type Config struct {
	Name string            `json:"name"`
	Meta map[string]string `json:"metadata"`
	Port []*Port           `json:"ports"`
}

type Port struct {
	Number uint `json:"number"`
	Desc   string `json:"desc"`
}

// You can encode the structure on one side:
message, err := protostructure.Encode(Config{})

// And you can use the structure on the other side. Imagine resp
// is populated using some protobuf RPC such as gRPC.
val, err := protostructure.New(resp.Config)
json.Unmarshal([]byte(`{
	"name": "example",
	"meta": { "env": "prod" },
	"ports": [
		{ "number": 8080 },
		{ "number": 8100, desc: "backup" },
	]
}`, val)

// val now holds the same structure dynamically. You can pair with other
// libraries such as https://github.com/go-playground/validator to also
// send validation using this library.

Limitations

There are several limitations on the structures that can be encoded. Some of these limitations are fixable but the effort hasn't been put in while others are fundamental due to the limitations of Go currently:

• Circular references are not allowed between any struct types.
• Embedded structs are not supported
• Methods are not preserved, and therefore interface implementation is not known. This is also an important detail because custom callbacks such as UnmarshalJSON may not work properly.
• Field types cannot be: interfaces, channels, functions
• Certain stdlib types such as time.Time currently do not encode well.

But... why?

The real world use case that led to the creation of this library was to facilitate decoding and validating configuration for plugins via go-plugin, a plugin system for Go that communicates using gRPC.

The plugins for this particular program have dynamic configuration structures that were decoded using an encoding/json-like interface (struct tags) and validated using go-playground/validator which also uses struct tags. Using protostructure, we can send the configuration structure across the wire, decode and validate the configuration in the host process, and report more rich errors that way.

Another reason we wanted to ship the config structure vs. ship the config is because the actual language we are using for configuration is HCL which supports things like function calls, logic, and more and shipping that runtime across is much, much more difficult.

This was extracted into a separate library because the ability to encode a Go structure (particulary to include tags) seemed more generally useful, although rare.

Documentation

Overview

Package protostructure provides a mechanism for encoding and decoding a struct _type_ using protocol buffers. To be clear: this encodes the _type_ and not the _value_.

Most importantly, this lets you do things such as transferring a struct that supports JSON decoding across a protobuf RPC, and then decoding a JSON value directly into it since you have access to things such as struct tags from the remote end.

For a pure JSON use case, it may make sense to instead send the JSON rather than send the struct type. There are other scenarios where sending the type is easier and this library facilitates those use cases.

The primary functions you want to look at are "Encode" and "New".

Index

• Variables
• func New(s *Struct) (result interface{}, err error)
• type Container
  • func (*Container) Descriptor() ([]byte, []int)
  • func (x *Container) GetCount() int32
  • func (x *Container) GetElem() *Type
  • func (x *Container) GetKind() uint32
  • func (*Container) ProtoMessage()
  • func (x *Container) ProtoReflect() protoreflect.Message
  • func (x *Container) Reset()
  • func (x *Container) String() string
• type InternalStruct
  • func (*InternalStruct) Descriptor() ([]byte, []int)
  • func (x *InternalStruct) GetName() string
  • func (*InternalStruct) ProtoMessage()
  • func (x *InternalStruct) ProtoReflect() protoreflect.Message
  • func (x *InternalStruct) Reset()
  • func (x *InternalStruct) String() string
• type Primitive
  • func (*Primitive) Descriptor() ([]byte, []int)
  • func (x *Primitive) GetKind() uint32
  • func (*Primitive) ProtoMessage()
  • func (x *Primitive) ProtoReflect() protoreflect.Message
  • func (x *Primitive) Reset()
  • func (x *Primitive) String() string
• type Struct
  • func Encode(s interface{}) (*Struct, error)
  • func (*Struct) Descriptor() ([]byte, []int)
  • func (x *Struct) GetFields() []*Struct_Field
  • func (x *Struct) GetStructName() string
  • func (*Struct) ProtoMessage()
  • func (x *Struct) ProtoReflect() protoreflect.Message
  • func (x *Struct) Reset()
  • func (x *Struct) String() string
• type Struct_Field
  • func (*Struct_Field) Descriptor() ([]byte, []int)
  • func (x *Struct_Field) GetName() string
  • func (x *Struct_Field) GetPkgPath() string
  • func (x *Struct_Field) GetTag() string
  • func (x *Struct_Field) GetType() *Type
  • func (*Struct_Field) ProtoMessage()
  • func (x *Struct_Field) ProtoReflect() protoreflect.Message
  • func (x *Struct_Field) Reset()
  • func (x *Struct_Field) String() string
• type Type
  • func (*Type) Descriptor() ([]byte, []int)
  • func (x *Type) GetContainer() *Container
  • func (x *Type) GetInternalStruct() *InternalStruct
  • func (x *Type) GetPrimitive() *Primitive
  • func (m *Type) GetType() isType_Type
  • func (*Type) ProtoMessage()
  • func (x *Type) ProtoReflect() protoreflect.Message
  • func (x *Type) Reset()
  • func (x *Type) String() string
• type Type_Container
• type Type_InternalStruct
• type Type_Primitive

Variables

var File_protostructure_proto protoreflect.FileDescriptor

Functions

func New

func New(s *Struct) (result interface{}, err error)

New returns a pointer to an allocated struct for the structure given or an error if there are any invalid fields.

This interface{} value can be used directly in functions such as json.Unmarshal, or it can be inspected further as necessary.

Types

type Container

type Container struct {

	// kind must be one of: array, ptr, slice
	Kind uint32 `protobuf:"varint,1,opt,name=kind,proto3" json:"kind,omitempty"`
	// elem is the type of the element of this container
	Elem *Type `protobuf:"bytes,2,opt,name=elem,proto3" json:"elem,omitempty"`
	// count is the number of elements, only if kind == array
	Count int32 `protobuf:"varint,4,opt,name=count,proto3" json:"count,omitempty"`
	// contains filtered or unexported fields
}

Container represents any "container" type such as a sliec, array, map, etc.

func (*Container) Descriptor

func (*Container) Descriptor() ([]byte, []int)

Deprecated: Use Container.ProtoReflect.Descriptor instead.

func (*Container) GetCount

func (x *Container) GetCount() int32

func (*Container) GetElem

func (x *Container) GetElem() *Type

func (*Container) GetKind

func (x *Container) GetKind() uint32

func (*Container) ProtoMessage

func (*Container) ProtoMessage()

func (*Container) ProtoReflect

func (x *Container) ProtoReflect() protoreflect.Message

func (*Container) Reset

func (x *Container) Reset()

func (*Container) String

func (x *Container) String() string

type InternalStruct

type InternalStruct struct {
	Name string `protobuf:"bytes,1,opt,name=Name,proto3" json:"Name,omitempty"`
	// contains filtered or unexported fields
}

func (*InternalStruct) Descriptor

func (*InternalStruct) Descriptor() ([]byte, []int)

Deprecated: Use InternalStruct.ProtoReflect.Descriptor instead.

func (*InternalStruct) GetName

func (x *InternalStruct) GetName() string

func (*InternalStruct) ProtoMessage

func (*InternalStruct) ProtoMessage()

func (*InternalStruct) ProtoReflect

func (x *InternalStruct) ProtoReflect() protoreflect.Message

func (*InternalStruct) Reset

func (x *InternalStruct) Reset()

func (*InternalStruct) String

func (x *InternalStruct) String() string

type Primitive

type Primitive struct {

	// kind is the reflect.Kind value for this primitive. This MUST be
	// a primitive value. For example, reflect.Ptr would be invalid here.
	Kind uint32 `protobuf:"varint,1,opt,name=kind,proto3" json:"kind,omitempty"`
	// contains filtered or unexported fields
}

Primitive is a primitive type such as int, bool, etc.

func (*Primitive) Descriptor

func (*Primitive) Descriptor() ([]byte, []int)

Deprecated: Use Primitive.ProtoReflect.Descriptor instead.

func (*Primitive) GetKind

func (x *Primitive) GetKind() uint32

func (*Primitive) ProtoMessage

func (*Primitive) ProtoMessage()

func (*Primitive) ProtoReflect

func (x *Primitive) ProtoReflect() protoreflect.Message

func (*Primitive) Reset

func (x *Primitive) Reset()

func (*Primitive) String

func (x *Primitive) String() string

type Struct

type Struct struct {
	StructName string `protobuf:"bytes,1,opt,name=StructName,proto3" json:"StructName,omitempty"`
	// fields is the list of fields in the struct
	Fields []*Struct_Field `protobuf:"bytes,2,rep,name=fields,proto3" json:"fields,omitempty"`
	// contains filtered or unexported fields
}

Struct represents a struct type.

This has the following limitations:

• Circular references are not allowed between any struct types
• Embedded structs are not supported
• Methods are not preserved

func Encode

func Encode(s interface{}) (*Struct, error)

Encode converts a struct to a *Struct which implements proto.Message and can therefore be sent over the wire. Note that only the _structure_ of the struct is encoded and NOT any fields values.

Encoding has a number of limitations:

• Circular references are not allowed between any struct types
• Embedded structs are not supported
• Methods are not preserved
• Field types cannot be: interfaces, channels, functions

func (*Struct) Descriptor

func (*Struct) Descriptor() ([]byte, []int)

Deprecated: Use Struct.ProtoReflect.Descriptor instead.

func (*Struct) GetFields

func (x *Struct) GetFields() []*Struct_Field

func (*Struct) GetStructName

func (x *Struct) GetStructName() string

func (*Struct) ProtoMessage

func (*Struct) ProtoMessage()

func (*Struct) ProtoReflect

func (x *Struct) ProtoReflect() protoreflect.Message

func (*Struct) Reset

func (x *Struct) Reset()

func (*Struct) String

func (x *Struct) String() string

type Struct_Field

type Struct_Field struct {
	Name    string `protobuf:"bytes,1,opt,name=Name,proto3" json:"Name,omitempty"`
	PkgPath string `protobuf:"bytes,2,opt,name=PkgPath,proto3" json:"PkgPath,omitempty"`
	Tag     string `protobuf:"bytes,3,opt,name=Tag,proto3" json:"Tag,omitempty"`
	Type    *Type  `protobuf:"bytes,4,opt,name=type,proto3" json:"type,omitempty"`
	// contains filtered or unexported fields
}

Field is a field type. See reflect.StructField in the Go stdlib since the fields in this message match that almost exactly.

func (*Struct_Field) Descriptor

func (*Struct_Field) Descriptor() ([]byte, []int)

Deprecated: Use Struct_Field.ProtoReflect.Descriptor instead.

func (*Struct_Field) GetName

func (x *Struct_Field) GetName() string

func (*Struct_Field) GetPkgPath

func (x *Struct_Field) GetPkgPath() string

func (*Struct_Field) GetTag

func (x *Struct_Field) GetTag() string

func (*Struct_Field) GetType

func (x *Struct_Field) GetType() *Type

func (*Struct_Field) ProtoMessage

func (*Struct_Field) ProtoMessage()

func (*Struct_Field) ProtoReflect

func (x *Struct_Field) ProtoReflect() protoreflect.Message

func (*Struct_Field) Reset

func (x *Struct_Field) Reset()

func (*Struct_Field) String

func (x *Struct_Field) String() string

type Type

type Type struct {

	// Types that are assignable to Type:
	//	*Type_Primitive
	//	*Type_Container
	//	*Type_InternalStruct
	Type isType_Type `protobuf_oneof:"type"`
	// contains filtered or unexported fields
}

Type represents a Go type.

func (*Type) Descriptor

func (*Type) Descriptor() ([]byte, []int)

Deprecated: Use Type.ProtoReflect.Descriptor instead.

func (*Type) GetContainer

func (x *Type) GetContainer() *Container

func (*Type) GetInternalStruct

func (x *Type) GetInternalStruct() *InternalStruct

func (*Type) GetPrimitive

func (x *Type) GetPrimitive() *Primitive

func (*Type) GetType

func (m *Type) GetType() isType_Type

func (*Type) ProtoMessage

func (*Type) ProtoMessage()

func (*Type) ProtoReflect

func (x *Type) ProtoReflect() protoreflect.Message

func (*Type) Reset

func (x *Type) Reset()

func (*Type) String

func (x *Type) String() string

type Type_Container

type Type_Container struct {
	Container *Container `protobuf:"bytes,2,opt,name=container,proto3,oneof"`
}

type Type_InternalStruct

type Type_InternalStruct struct {
	InternalStruct *InternalStruct `protobuf:"bytes,3,opt,name=internal_struct,json=internalStruct,proto3,oneof"`
}

type Type_Primitive

type Type_Primitive struct {
	Primitive *Primitive `protobuf:"bytes,1,opt,name=primitive,proto3,oneof"`
}