serde

package
v1.10.0-rc2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2020 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package serde contains Pachyderm-specific data structures for marshalling and unmarshalling Go structs and maps to structured text formats (currently just JSON and YAML).

Similar to https://github.com/ghodss/yaml, all implementations of the Format interface marshal and unmarshal data using the following pipeline: 

Go struct/map (fully structured)
  <-> JSON document
  <-> map[string]interface{}
  <-> target format document

Despite the redundant round of marshalling and unmarshalling, there are two main advantages to this approach:

  • YAML (and any future storage formats) can re-use existing `json:` struct tags
  • The intermediate map[string]interface{} can be manipulated, making it possible to have flexible converstions between structs and documents. For examples, PPS pipelines may include a full TFJob spec, which is converted to a string and stored in the 'TFJob' field of Pachyderm's CreatePipelineRequest struct.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DecodeJSON 

func DecodeJSON(jsonData []byte, v interface{}) error

DecodeJSON is a convenience function that decodes json data using a JSONDecoder, but can be called inline

func DecodeYAML 

func DecodeYAML(yamlData []byte, v interface{}) error

DecodeYAML is a convenience function that decodes yaml data using a YAMLDecoder, but can be called inline

func EncodeJSON 

func EncodeJSON(v interface{}, options ...EncoderOption) ([]byte, error)

EncodeJSON is a convenience function that encodes json data using a JSONEncoder, but can be called inline

func EncodeYAML 

func EncodeYAML(v interface{}, options ...EncoderOption) ([]byte, error)

EncodeYAML is a convenience function that encodes yaml data using a YAMLEncoder, but can be called inline

func WithIndent 

func WithIndent(numSpaces int) func(d Encoder)

WithIndent is an EncoderOption that causes the returned encoder to use 'numSpaces' spaces as the indentation prefix for embedded messages. If applied to a JSON encoder, it also changes the encoder output to be multi-line (instead of inline).

func WithOrigName 

func WithOrigName(origName bool) func(d Encoder)

WithOrigName is an EncoderOption that, if set, encodes proto messages using the name set in the original .proto message definition, rather than the munged version of the generated struct's field name

Types

type Decoder 

type Decoder interface {
	// Decode parses an underlying stream of text in the given format into 'v'
	// (a struct or Go map)
	Decode(interface{}) error

	// DecodeProto is similar to Decode, but instead of converting between
	// the canonicalized JSON and 'v' using 'encoding/json', it does so using
	// 'gogo/protobuf/jsonpb'.  This allows callers to take advantage of more
	// sophisticated timestamp parsing and such in the 'jsonpb' library.
	//
	// TODO(msteffen) same comment re: proto.Message as for
	// Encoder.EncodeProto()
	DecodeProto(proto.Message) error

	// DecodeTransform is similar to Decode, but users can manipulate the
	// intermediate map[string]interface generated by Format implementations by
	// passing a function. Note that 'Encode(v)' is equivalent to
	// 'EncodeTransform(v, nil)'
	DecodeTransform(interface{}, func(map[string]interface{}) error) error

	// DecodeProtoTransform is similar to DecodeTransform, but instead of
	// converting between the canonicalized JSON and 'v' using 'encoding/json', it
	// does so using 'gogo/protobuf/jsonpb'.  This allows callers to take
	// advantage of more sophisticated timestamp parsing and such in the 'jsonpb'
	// library.
	//
	// TODO(msteffen) same comment re: proto.Message as for
	// Encoder.EncodeProto()
	DecodeProtoTransform(proto.Message, func(map[string]interface{}) error) error
}

Decoder is the interface for decoding a particular serialization format. Currently, the only implementation of Decoder is in serde/yaml/decoder.go (the point of having a common interface is to make it possible to switch between decoding different serialization formats at runtime, but we have no need to switch between a YAML decoder and a JSON decoder, as our YAML decoder can decode JSON).

type Encoder 

type Encoder interface {
	// Marshall converts 'v' (a struct or Go map) to a structured-text document
	// and writes it to this Encoder's output stream
	Encode(v interface{}) error

	// EncodeProto is similar to Encode, but instead of converting between the
	// canonicalized JSON and 'v' using 'encoding/json', it does so using
	// 'gogo/protobuf/jsonpb'.  This allows callers to take advantage of more
	// sophisticated timestamp parsing and such in the 'jsonpb' library.
	//
	// TODO(msteffen): We can *almost* avoid the Encode/EncodeProto split by
	// checking if 'v' implements 'proto.Message', except for one case: the
	// kubernetes client library includes structs that are pseudo-protobufs.
	// Structs in the kubernetes go client implement the 'proto.Message()'
	// interface but are hand-generated and contain embedded structs, which
	// 'jsonpb' can't handle when parsing. If jsonpb is ever extended to be able
	// to parse JSON into embedded structs (even though the protobuf compiler
	// itself never generates such structs) then we could merge this into
	// Encode() and rely on:
	//
	//   if msg, ok := v.(proto.Message); ok {
	//     ... use jsonpb ...
	//   } else {
	//     ... use encoding/json ...
	//   }
	EncodeProto(proto.Message) error

	// EncodeTransform is similar to Encode, but users can manipulate the
	// intermediate map[string]interface generated by Format implementations by
	// passing a function. Note that 'Encode(v)' is equivalent to
	// 'EncodeTransform(v, nil)'
	EncodeTransform(interface{}, func(map[string]interface{}) error) error

	// EncodeProtoTransform is similar to EncodeTransform(), but instead of
	// converting between the canonicalized JSON and 'v' using 'encoding/json', it
	// does so using 'gogo/protobuf/jsonpb'.  This allows callers to take
	// advantage of more sophisticated timestamp parsing and such in the 'jsonpb'
	// library.
	//
	// TODO(msteffen) same comment re: proto.Message as for EncodeProto()
	EncodeProtoTransform(proto.Message, func(map[string]interface{}) error) error
}

Encoder is an interface for encoding data to an output stream (every implementation should provide the ability to construct an Encoder tied to an output stream, to which encoded text should be written)

func GetEncoder 

func GetEncoder(encoding string, w io.Writer, opts ...EncoderOption) (Encoder, error)

GetEncoder dynamically creates and returns an Encoder for the text format 'encoding' (currently, 'encoding' must be "yaml" or "json"). 'defaultEncoding' specifies the text format that should be used if 'encoding' is "". 'opts' are the list of options that should be applied to any result, if any are applicable. Typically EncoderOptions are encoder-specific (e.g. setting json indentation). If an option is passed to GetEncoder for e.g. a json encoder but a yaml encoder is requested, then the option will be ignored. This makes it possible to pass all options to GetEncoder, but defer the decision about what kind of encoding to use until runtime, like so:

enc, _ := GetEncoder(outputFlag, os.Stdout, 

...options to use if json...,
...options to use if yaml...,

) enc.Encode(obj)

Note: There is currently no corresponding GetDecoder, because the only implementation of the Decoder interface is YAMLDecoder

type EncoderOption 

type EncoderOption func(Encoder)

EncoderOption modifies the behavior of new encoders and can be passed to GetEncoder.

type JSONDecoder 

type JSONDecoder struct {
	// contains filtered or unexported fields
}

JSONDecoder is an implementation of serde.Decoder that operates on JSON data.

func NewJSONDecoder 

func NewJSONDecoder(r io.Reader) *JSONDecoder

NewJSONDecoder returns a new JSONDecoder that reads from 'r'

func (*JSONDecoder) Decode 

func (d *JSONDecoder) Decode(v interface{}) error

Decode implements the corresponding method of serde.Decoder

func (*JSONDecoder) DecodeProto 

func (d *JSONDecoder) DecodeProto(v proto.Message) error

DecodeProto implements the corresponding method of serde.Decoder

func (*JSONDecoder) DecodeProtoTransform 

func (d *JSONDecoder) DecodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

DecodeProtoTransform implements the corresponding method of serde.Decoder

func (*JSONDecoder) DecodeTransform 

func (d *JSONDecoder) DecodeTransform(v interface{}, f func(map[string]interface{}) error) error

DecodeTransform implements the corresponding method of serde.Decoder

type JSONEncoder 

type JSONEncoder struct {
	// contains filtered or unexported fields
}

JSONEncoder is an implementation of serde.Encoder that operates on JSON data

func NewJSONEncoder 

func NewJSONEncoder(w io.Writer, options ...EncoderOption) *JSONEncoder

NewJSONEncoder returns a new JSONEncoder that writes to 'w'

func (*JSONEncoder) Encode 

func (e *JSONEncoder) Encode(v interface{}) error

Encode implements the corresponding method of serde.Encoder

func (*JSONEncoder) EncodeProto 

func (e *JSONEncoder) EncodeProto(v proto.Message) error

EncodeProto implements the corresponding method of serde.Encoder

func (*JSONEncoder) EncodeProtoTransform 

func (e *JSONEncoder) EncodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

EncodeProtoTransform implements the corresponding method of serde.Encoder

func (*JSONEncoder) EncodeTransform 

func (e *JSONEncoder) EncodeTransform(v interface{}, f func(map[string]interface{}) error) error

EncodeTransform implements the corresponding method of serde.Encoder

type YAMLDecoder 

type YAMLDecoder struct {
	// contains filtered or unexported fields
}

YAMLDecoder is an implementation of serde.Decoder that operates on yaml data

func NewYAMLDecoder 

func NewYAMLDecoder(r io.Reader) *YAMLDecoder

NewYAMLDecoder returns a new YAMLDecoder that reads from 'r'

func (*YAMLDecoder) Decode 

func (d *YAMLDecoder) Decode(v interface{}) error

Decode implements the corresponding method of serde.Decoder

func (*YAMLDecoder) DecodeProto 

func (d *YAMLDecoder) DecodeProto(v proto.Message) error

DecodeProto implements the corresponding method of serde.Decoder

func (*YAMLDecoder) DecodeProtoTransform 

func (d *YAMLDecoder) DecodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

DecodeProtoTransform implements the corresponding method of serde.Decoder

func (*YAMLDecoder) DecodeTransform 

func (d *YAMLDecoder) DecodeTransform(v interface{}, f func(map[string]interface{}) error) error

DecodeTransform implements the corresponding method of serde.Decoder

type YAMLEncoder 

type YAMLEncoder struct {
	// contains filtered or unexported fields
}

YAMLEncoder is an implementation of serde.Encoder that operates on YAML data

func NewYAMLEncoder 

func NewYAMLEncoder(w io.Writer, options ...EncoderOption) *YAMLEncoder

NewYAMLEncoder returns a new YAMLEncoder that writes to 'w'

func (*YAMLEncoder) Encode 

func (e *YAMLEncoder) Encode(v interface{}) error

Encode implements the corresponding method of serde.Encoder

func (*YAMLEncoder) EncodeProto 

func (e *YAMLEncoder) EncodeProto(v proto.Message) error

EncodeProto implements the corresponding method of serde.Encoder

func (*YAMLEncoder) EncodeProtoTransform 

func (e *YAMLEncoder) EncodeProtoTransform(v proto.Message, f func(map[string]interface{}) error) error

EncodeProtoTransform implements the corresponding method of serde.Encoder

func (*YAMLEncoder) EncodeTransform 

func (e *YAMLEncoder) EncodeTransform(v interface{}, f func(map[string]interface{}) error) error

EncodeTransform implements the corresponding method of serde.Encoder

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.