Documentation

Overview

Package starlarkproto exposes protobuf messages as Starlark types.

Uses a slightly modified vendored copy of "google.golang.org/protobuf" internally. It's expected that once "google.golang.org/protobuf" is officially released, starlarkproto will switch to using the released version.

Internally a message is stored as a tree of Starlark values, with some type checking done when manipulating fields, based on proto message descriptors loaded dynamically from a serialized FileDescriptorSet.

For example, reading or assigning to a field not defined in a message will cause a runtime error. Similarly, trying to assign a value of a wrong type to a non-repeated field will fail.

Instantiating messages and default field values

Each proto message in a loaded package is exposed via a constructor function that takes optional keyword arguments and produces a new object of *Message type.

All unassigned fields are implicitly set to their default zero values on first access, including message-typed fields, lists and maps. It means, for example, if a message 'a' has a singular field 'b', that has a field 'c', it is always fine to write 'a.b.c' to read or set 'c' value, without explicitly checking that 'b' is set.

To clear a field, assign None to it (regardless of its type).

References and aliasing

Messages are passed around everywhere by reference. In particular it is possible to have multiple fields pointing to the exact same message, e.g.

m1 = M()
m2 = M()
a = A()
m1.f = a
m2.f = a
a.i = 123  # changes both m1.f.i and m2.f.i

Note that 'm1.f = a' assignment checks the type of 'a', it should either match type of 'f' identically (no duck typing), or be a dict or None (which will be converted to messages, see below).

Working with repeated fields and maps

Values of repeated fields are represented by special sequence types that behave as strongly-typed lists. Think of them as list[T] types or as hypothetical 'message List<T>' messages.

When assigning a non-None value R to a repeated field F of type list[T], the following rules apply (sequentially):

1. If R is not a sequence => error.
2. If R has type list[T'], then
   a. If T == T', then F becomes an alias of R.
   b. If T != T' => error.
3. A new list[T] is instantiated from R and assigned to F.

Notice that rule 2 is exactly like the aliasing rule for messages. This is where "think of them as 'message List<T>'" point of view comes into play.

As a concrete example, consider this:

m1 = M()
m1.int64s = [1, 2]  # a list is implicitly converted (copied) to list[T]

m2 = M()
m2.int64s = m1.int64s  # points to the exact same list[T] object now
m2.int64s.append(3)    # updates both m1 and m2

m1.int32s = m1.int64s        # error! list[int64] is not list[int32]
m1.int32s = list(m1.int64s)  # works now (by making a copy of a list copy)

Maps behave in the similar way. Think of them as strongly-typed map<K,V> values or as 'message List<Pair<K,V>>' messages. They can be implicitly instantiated from iterable mappings (e.g. dicts).

Auto-conversion of dicts and None's into Messages

When assigning to a message-typed value (be it a field, an element of a list[T] or a value of map<K,V>) dicts are implicitly converted into messages as if via 'T(**d)' call. Similarly, None's are converted into empty messages, as if via 'T()' call.

Differences from starlarkproto (beside using different guts):

* Message types are instantiated through proto.new_loader().
* Text marshaller appends\removes trailing '\n' somewhat differently.
* Text marshaller marshals empty messages as '<>' (without line breaks).
* Bytes fields are represented as str, not as []uint8{...}.
* proto.to_jsonpb doesn't have 'emit_defaults' kwarg (it is always False).
* Better support for proto2 messages.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ProtoLib

func ProtoLib() starlark.StringDict

    ProtoLib returns a dict with single struct named "proto" that holds public Starlark API for working with proto messages.

    Exported functions:

    def new_descriptor_set(name=None, blob=None, deps=None):
      """Returns a new DescriptorSet.
    
      Args:
        name: name of this set for debug and error messages, default is '???'.
        blob: raw serialized FileDescriptorSet, if any.
        deps: an iterable of DescriptorSet's with dependencies, if any.
    
      Returns:
        New DescriptorSet.
      """
    
    def new_loader(*descriptor_sets):
      """Returns a new proto loader."""
    
    def default_loader():
      """Returns a loader used by default when registering descriptor sets."""
    
    def message_type(msg):
      """Returns proto.MessageType of the given message."""
    
    def to_textpb(msg):
      """Serializes a protobuf message to text proto.
    
      Args:
        msg: a *Message to serialize.
    
      Returns:
        A str representing msg in text format.
      """
    
    def to_jsonpb(msg):
      """Serializes a protobuf message to JSONPB string.
    
      Args:
        msg: a *Message to serialize.
    
      Returns:
        A str representing msg in JSONPB format.
      """
    
    def to_wirepb(msg):
      """Serializes a protobuf message to a string using binary wire encoding.
    
      Args:
        msg: a *Message to serialize.
    
      Returns:
        A str representing msg in binary wire format.
      """
    
    def from_textpb(ctor, body):
      """Deserializes a protobuf message given in text proto form.
    
      Unknown fields are not allowed.
    
      Args:
        ctor: a message constructor function.
        body: a string with serialized message.
    
      Returns:
        Deserialized message constructed via `ctor`.
      """
    
    def from_jsonpb(ctor, body):
      """Deserializes a protobuf message given as JBONPB string.
    
      Unknown fields are silently skipped.
    
      Args:
        ctor: a message constructor function.
        body: a string with serialized message.
    
      Returns:
        Deserialized message constructed via `ctor`.
      """
    
    def from_wirepb(ctor, body):
      """Deserializes a protobuf message given its wire serialization.
    
      Unknown fields are silently skipped.
    
      Args:
        ctor: a message constructor function.
        body: a string with serialized message.
    
      Returns:
        Deserialized message constructed via `ctor`.
      """
    
    def struct_to_textpb(s):
      """Converts a struct to a text proto string.
    
      Args:
        s: a struct object. May not contain dicts.
    
      Returns:
        A str containing a text format protocol buffer message.
      """
    
    def clone(msg):
      """Returns a deep copy of a given proto message.
    
      Args:
        msg: a proto message to make a copy of.
    
      Returns:
        A deep copy of the message
      """
    
    def has(msg, field):
      """Checks if a proto message has the given optional field set.
    
      Args:
        msg: a message to check.
        field: a string name of the field to check.
    
      Returns:
        True if the message has the field set.
      """
    

    func SetDefaultLoader

    func SetDefaultLoader(th *starlark.Thread, l *Loader)

      SetDefaultLoader installs the given loader as default in the thread.

      It can be obtained via DefaultLoader or proto.default_loader() from Starlark. Note that Starlark code has no way of changing the default loader. It's responsibility of the hosting environment to prepare the default loader (just like it prepares starlark.Thread itself).

      func ToJSONPB

      func ToJSONPB(msg *Message) ([]byte, error)

        ToJSONPB serializes a protobuf message to JSONPB string.

        func ToTextPB

        func ToTextPB(msg *Message) ([]byte, error)

          ToTextPB serializes a protobuf message to text proto.

          func ToWirePB

          func ToWirePB(msg *Message) ([]byte, error)

            ToWirePB serializes a protobuf message to binary wire format.

            Types

            type DescriptorSet

            type DescriptorSet struct {
            	// contains filtered or unexported fields
            }

              DescriptorSet contains FileDescriptorProto of zero or more *.proto files, along with pointers to DescriptorSets with their imports.

              A descriptor set can be registered in a Loader. Doing so transitively registers all its dependencies. See Loader.AddDescriptorSet for more details.

              Implements starlark.Value and starlark.HasAttrs interfaces. Usage from Starlark side may look like this:

              load(".../wellknown_descpb.star", "wellknown_descpb")
              myprotos_descpb = proto.new_descriptor_set(
                  name = "myprotos",
                  deps = [wellknown_descpb],
                  blob = io.read_file("myprotos.descpb"),
              )
              myprotos_descpb.register()
              

              By default register() registers the descriptor set in the default loader, i.e. ds.register() is same as ds.register(loader=proto.default_loader()). Also note that ds.register(loader=l) is a sugar for l.add_descriptor_set(ds), so ds.register() is same as proto.default_loader().add_descriptor_set(ds), just much shorter.

              func NewDescriptorSet

              func NewDescriptorSet(name string, fdps []*descriptorpb.FileDescriptorProto, deps []*DescriptorSet) (*DescriptorSet, error)

                NewDescriptorSet evaluates given file descriptors and their dependencies and produces new DescriptorSet if there are no unresolved imports and no duplicated files.

                'fdps' should be ordered topologically (i.e. if file A imports file B, then B should precede A in 'fdps' or be somewhere among 'deps'). This is always the case when generating sets via 'protoc --descriptor_set_out=...'.

                Note that dependencies can only be specified when creating the descriptor set and can't be changed later. Cycles thus are impossible.

                func (*DescriptorSet) Attr

                func (ds *DescriptorSet) Attr(name string) (starlark.Value, error)

                  Attr returns an attribute given its name (or nil if not present).

                  func (*DescriptorSet) AttrNames

                  func (ds *DescriptorSet) AttrNames() []string

                    AtrrNames lists available attributes.

                    func (*DescriptorSet) Freeze

                    func (ds *DescriptorSet) Freeze()

                      Freeze does nothing since DescriptorSet is already immutable.

                      func (*DescriptorSet) Hash

                      func (ds *DescriptorSet) Hash() (uint32, error)

                        Hash returns unique value associated with this set.

                        func (*DescriptorSet) String

                        func (ds *DescriptorSet) String() string

                          String returns str(...) representation of the set, for debug messages.

                          func (*DescriptorSet) Truth

                          func (ds *DescriptorSet) Truth() starlark.Bool

                            Truth returns the truth value of an object.

                            func (*DescriptorSet) Type

                            func (ds *DescriptorSet) Type() string

                              Type returns a short string describing the value's type.

                              type Loader

                              type Loader struct {
                              	// contains filtered or unexported fields
                              }

                                Loader can instantiate Starlark values that correspond to proto messages.

                                Holds a pool of descriptors that describe all available proto types. Use AddDescriptorSet to seed it. Once seeded, use Module to get a Starlark module with symbols defined in some registered `*.proto` file.

                                Loader is also a Starlark value itself, with the following methods:

                                * add_descriptor_set(ds) - see AddDescriptorSet.
                                * module(path) - see Module.
                                

                                Can be used concurrently. Non-freezable.

                                func DefaultLoader

                                func DefaultLoader(th *starlark.Thread) *Loader

                                  DefaultLoader returns a loader installed in the thread via SetDefaultLoader.

                                  Returns nil if there's no default loader.

                                  func NewLoader

                                  func NewLoader() *Loader

                                    NewLoader instantiates a new loader with empty proto registry.

                                    func (*Loader) AddDescriptorSet

                                    func (l *Loader) AddDescriptorSet(ds *DescriptorSet) error

                                      AddDescriptorSet makes all *.proto files defined in the given descriptor set and all its dependencies available for use from Starlark.

                                      AddDescriptorSet is idempotent in a sense that calling AddDescriptorSet(ds) multiple times with the exact same 'ds' is not an error. But trying to register a proto file through multiple different descriptor sets is an error.

                                      func (*Loader) Attr

                                      func (l *Loader) Attr(name string) (starlark.Value, error)

                                        Attr returns an attribute given its name (or nil if not present).

                                        func (*Loader) AttrNames

                                        func (l *Loader) AttrNames() []string

                                          AtrrNames lists available attributes.

                                          func (*Loader) Freeze

                                          func (l *Loader) Freeze()

                                            Freeze is noop for now.

                                            func (*Loader) Hash

                                            func (l *Loader) Hash() (uint32, error)

                                              Hash returns an integer assigned to this loader when it was created.

                                              func (*Loader) MessageType

                                              func (l *Loader) MessageType(desc protoreflect.MessageDescriptor) *MessageType

                                                MessageType creates new (or returns existing) MessageType.

                                                The return value can be used to instantiate Starlark values via Message() or MessageFromProto(m).

                                                func (*Loader) Module

                                                func (l *Loader) Module(path string) (*starlarkstruct.Module, error)

                                                  Module returns a module with top-level definitions from some *.proto file.

                                                  The descriptor of this proto file should be registered already via AddDescriptorSet. 'path' here is matched to what's in the descriptor, which is a path to *.proto EXACTLY as it was given to 'protoc'.

                                                  The name of the module matches the proto package name (per 'package ...' statement in the proto file).

                                                  func (*Loader) String

                                                  func (l *Loader) String() string

                                                    String returns str(...) representation of the loader.

                                                    func (*Loader) Truth

                                                    func (l *Loader) Truth() starlark.Bool

                                                      Truth returns True.

                                                      func (*Loader) Type

                                                      func (l *Loader) Type() string

                                                        Type returns "proto.Loader".

                                                        type Message

                                                        type Message struct {
                                                        	// contains filtered or unexported fields
                                                        }

                                                          Message is a Starlark value that implements a struct-like type structured like a protobuf message.

                                                          Implements starlark.Value, starlark.HasAttrs, starlark.HasSetField and starlark.Comparable interfaces.

                                                          Can be instantiated through Loader as loader.MessageType(...).Message() or loader.MessageType(...).MessageFromProto(p).

                                                          TODO(vadimsh): Currently not safe for a cross-goroutine use without external locking, even when frozen, due to lazy initialization of default fields on first access.

                                                          func FromJSONPB

                                                          func FromJSONPB(typ *MessageType, blob []byte) (*Message, error)

                                                            FromJSONPB deserializes a protobuf message given as JBONPB string.

                                                            func FromTextPB

                                                            func FromTextPB(typ *MessageType, blob []byte) (*Message, error)

                                                              FromTextPB deserializes a protobuf message given in text proto form.

                                                              func FromWirePB

                                                              func FromWirePB(typ *MessageType, blob []byte) (*Message, error)

                                                                FromWirePB deserializes a protobuf message given as a wire-encoded blob.

                                                                func (*Message) Attr

                                                                func (m *Message) Attr(name string) (starlark.Value, error)

                                                                  Attr is called when a field is read from Starlark code.

                                                                  func (*Message) AttrNames

                                                                  func (m *Message) AttrNames() []string

                                                                    AttrNames lists available attributes.

                                                                    func (*Message) CompareSameType

                                                                    func (m *Message) CompareSameType(op syntax.Token, y starlark.Value, depth int) (bool, error)

                                                                      CompareSameType does 'm <op> y' comparison.

                                                                      func (*Message) Freeze

                                                                      func (m *Message) Freeze()

                                                                        Freeze makes this message immutable.

                                                                        func (*Message) FromDict

                                                                        func (m *Message) FromDict(d starlark.IterableMapping) error

                                                                          FromDict populates fields of this message based on values in an iterable mapping (usually a starlark.Dict).

                                                                          Doesn't reset the message. Basically does this:

                                                                          for k in d:
                                                                            setattr(msg, k, d[k])
                                                                          

                                                                          Returns an error on type mismatch.

                                                                          func (*Message) HasProtoField

                                                                          func (m *Message) HasProtoField(name string) bool

                                                                            HasProtoField returns true if the message has the given field initialized.

                                                                            func (*Message) Hash

                                                                            func (m *Message) Hash() (uint32, error)

                                                                              Hash returns an error, indicating proto messages are not hashable.

                                                                              func (*Message) MessageType

                                                                              func (m *Message) MessageType() *MessageType

                                                                                MessageType returns type information about this message.

                                                                                func (*Message) SetField

                                                                                func (m *Message) SetField(name string, val starlark.Value) error

                                                                                  SetField is called when a field is assigned to from Starlark code.

                                                                                  func (*Message) String

                                                                                  func (m *Message) String() string

                                                                                    String returns compact text serialization of this message.

                                                                                    func (*Message) ToProto

                                                                                    func (m *Message) ToProto() proto.Message

                                                                                      ToProto returns a new populated proto message of an appropriate type.

                                                                                      func (*Message) Truth

                                                                                      func (m *Message) Truth() starlark.Bool

                                                                                        Truth always returns True.

                                                                                        func (*Message) Type

                                                                                        func (m *Message) Type() string

                                                                                          Type returns full proto message name.

                                                                                          type MessageType

                                                                                          type MessageType struct {
                                                                                          	*starlark.Builtin // the callable, initialize in Loader
                                                                                          	// contains filtered or unexported fields
                                                                                          }

                                                                                            MessageType represents a proto message type and acts as its constructor: it is a Starlark callable that produces instances of Message.

                                                                                            Implements starlark.HasAttrs interface. Attributes represent constructors for nested messages and int values of enums. Note that starlark.HasSetField is not implemented, making values of MessageType immutable.

                                                                                            Stringifying an instance of MessageType (e.g. with `str(...)` in Starlark) returns its full proto type name.

                                                                                            Given a MessageDescriptor, MessageType can be instantiated through Loader as loader.MessageType(...).

                                                                                            func (*MessageType) Attr

                                                                                            func (t *MessageType) Attr(name string) (starlark.Value, error)

                                                                                              Attr returns either a nested message or an enum value.

                                                                                              func (*MessageType) AttrNames

                                                                                              func (t *MessageType) AttrNames() []string

                                                                                                AttrNames return names of all nested messages and enum values.

                                                                                                func (*MessageType) Converter

                                                                                                func (t *MessageType) Converter() typed.Converter

                                                                                                  Converter returns an object that can convert Starlark dicts and Nones to values of this message type.

                                                                                                  Can be used by typed.List and typed.Dict.

                                                                                                  func (*MessageType) Descriptor

                                                                                                  func (t *MessageType) Descriptor() protoreflect.MessageDescriptor

                                                                                                    Descriptor returns protobuf type information for this message type.

                                                                                                    func (*MessageType) Message

                                                                                                    func (t *MessageType) Message() *Message

                                                                                                      Message instantiates a new empty message of this type.

                                                                                                      func (*MessageType) MessageFromProto

                                                                                                      func (t *MessageType) MessageFromProto(p proto.Message) *Message

                                                                                                        MessageFromProto instantiates a new message of this type and populates it based on values in the given proto.Message that should have a matching type.

                                                                                                        Here "matching type" means p.ProtoReflect().Descriptor() *is* t.Descriptor(). Panics otherwise.

                                                                                                        func (*MessageType) String

                                                                                                        func (t *MessageType) String() string

                                                                                                          String returns the full proto type name.

                                                                                                          func (*MessageType) Type

                                                                                                          func (t *MessageType) Type() string

                                                                                                            Type returns "proto.MessageType", it's the type of the message type itself.

                                                                                                            It is sort of like a meta-type, i.e. like "type" type in Python.

                                                                                                            Directories

                                                                                                            Path Synopsis