avro

package module
v0.2.2-0...-3d81695 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 21, 2020 License: Apache-2.0 Imports: 13 Imported by: 0

README

Apache Avro for Golang

Build Status

Please note that this project is still an early alpha and is subject to change.

Installation is as easy as follows:

go get github.com/elodina/go-avro

Some usage examples are located in examples folder:

go-avro now also supports code generation from Avro schemas which is available in codegen folder

Documentation

Index

Constants

View Source
const (
	// Record schema type constant
	Record int = iota

	// Enum schema type constant
	Enum

	// Array schema type constant
	Array

	// Map schema type constant
	Map

	// Union schema type constant
	Union

	// Fixed schema type constant
	Fixed

	// String schema type constant
	String

	// Bytes schema type constant
	Bytes

	// Int schema type constant
	Int

	// Long schema type constant
	Long

	// Float schema type constant
	Float

	// Double schema type constant
	Double

	// Boolean schema type constant
	Boolean

	// Null schema type constant
	Null

	// Recursive schema type constant. Recursive is an artificial type that means a Record schema without its definition
	// that should be looked up in some registry.
	Recursive
)

Variables

View Source
var BlockNotFinished = errors.New("Block read is unfinished")

Happens when trying to read next block without finishing the previous one.

View Source
var EOF = errors.New("End of file reached")

Signals that an end of file or stream has been reached unexpectedly.

View Source
var FieldDoesNotExist = errors.New("Field does not exist")

FieldDoesNotExist happens when a struct does not have a necessary field.

View Source
var IntOverflow = errors.New("Overflowed an int value")

Happens when the given value to decode overflows maximum int32 value.

View Source
var InvalidBool = errors.New("Invalid bool value")

Happens when given value to decode as bool is neither 0x00 nor 0x01.

View Source
var InvalidFixedSize = errors.New("Invalid Fixed type size")

Happens when avro schema contains invalid value for fixed size.

View Source
var InvalidInt = errors.New("Invalid int value")

Happens when given value to decode as a int is invalid

View Source
var InvalidLong = errors.New("Invalid long value")

Happens when given value to decode as a long is invalid

View Source
var InvalidSchema = errors.New("Invalid schema")

Happens when avro schema is unparsable or is invalid in any other way.

View Source
var InvalidStringLength = errors.New("Invalid string length")

Happens when given value to decode as string has either negative or undecodable length.

View Source
var InvalidSync = errors.New("Invalid sync")

Happens when file header's sync and block's sync do not match - indicates corrupted data.

View Source
var InvalidValueType = errors.New("Invalid array or map value type")

Happens when avro schema contains invalid value for map value type or array item type.

View Source
var LongOverflow = errors.New("Overflowed a long value")

Happens when the given value to decode overflows maximum int64 value.

View Source
var NegativeBytesLength = errors.New("Negative bytes length")

Happens when given value to decode as bytes has negative length.

View Source
var NestedUnionsNotAllowed = errors.New("Nested unions are not allowed")

Happens when avro schema contains a union within union.

View Source
var NotAvroFile = errors.New("Not an Avro data file")

Indicates the given file to decode does not correspond to Avro data file format.

View Source
var SchemaNotSet = errors.New("Schema not set")

Happens when a datum reader has no set schema.

View Source
var UnionTypeOverflow = errors.New("Union type overflow")

UnionTypeOverflow happens when the numeric index of the union type is invalid.

Functions

func GetFullName

func GetFullName(schema Schema) string

GetFullName returns a fully-qualified name for a schema if possible. The format is namespace.name.

func LoadSchemas

func LoadSchemas(path string) map[string]Schema

LoadSchemas loads and parses a schema file or directory. Directory names MUST end with "/"

Types

type ArraySchema

type ArraySchema struct {
	Items      Schema
	Properties map[string]interface{}
}

ArraySchema implements Schema and represents Avro array type.

func (*ArraySchema) GetName

func (*ArraySchema) GetName() string

GetName returns a type name for this ArraySchema.

func (*ArraySchema) MarshalJSON

func (s *ArraySchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*ArraySchema) Prop

func (s *ArraySchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*ArraySchema) String

func (s *ArraySchema) String() string

String returns a JSON representation of ArraySchema.

func (*ArraySchema) Type

func (*ArraySchema) Type() int

Type returns a type constant for this ArraySchema.

func (*ArraySchema) Validate

func (s *ArraySchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type AvroRecord

type AvroRecord interface {
	// Schema returns an Avro schema for this AvroRecord.
	Schema() Schema
}

AvroRecord is an interface for anything that has an Avro schema and can be serialized/deserialized by this library.

type BinaryDecoder

type BinaryDecoder struct {
	// contains filtered or unexported fields
}

BinaryDecoder implements Decoder and provides low-level support for deserializing Avro values.

func NewBinaryDecoder

func NewBinaryDecoder(buf []byte) *BinaryDecoder

NewBinaryDecoder creates a new BinaryDecoder to read from a given buffer.

func (*BinaryDecoder) ArrayNext

func (bd *BinaryDecoder) ArrayNext() (int64, error)

ArrayNext processes the next block of an array and returns the number of items in the block. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) MapNext

func (bd *BinaryDecoder) MapNext() (int64, error)

MapNext processes the next block of map entries and returns the number of items in the block. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadArrayStart

func (bd *BinaryDecoder) ReadArrayStart() (int64, error)

ReadArrayStart reads and returns the size of the first block of an array. If call to this return non-zero, then the caller should read the indicated number of items and then call ArrayNext() to find out the number of items in the next block. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadBoolean

func (bd *BinaryDecoder) ReadBoolean() (bool, error)

ReadBoolean reads a boolean value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadBytes

func (bd *BinaryDecoder) ReadBytes() ([]byte, error)

ReadBytes reads a bytes value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadDouble

func (bd *BinaryDecoder) ReadDouble() (float64, error)

ReadDouble reads a double value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadEnum

func (bd *BinaryDecoder) ReadEnum() (int32, error)

ReadEnum reads an enum value (which is an Avro int value). Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadFixed

func (bd *BinaryDecoder) ReadFixed(bytes []byte) error

ReadFixed reads fixed sized binary object into the provided buffer. Returns an error if it occurs.

func (*BinaryDecoder) ReadFixedWithBounds

func (bd *BinaryDecoder) ReadFixedWithBounds(bytes []byte, start int, length int) error

ReadFixedWithBounds reads fixed sized binary object into the provided buffer. The second parameter is the position where the data needs to be written, the third is the size of binary object. Returns an error if it occurs.

func (*BinaryDecoder) ReadFloat

func (bd *BinaryDecoder) ReadFloat() (float32, error)

ReadFloat reads a float value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadInt

func (bd *BinaryDecoder) ReadInt() (int32, error)

ReadInt reads an int value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadLong

func (bd *BinaryDecoder) ReadLong() (int64, error)

ReadLong reads a long value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadMapStart

func (bd *BinaryDecoder) ReadMapStart() (int64, error)

ReadMapStart reads and returns the size of the first block of map entries. If call to this return non-zero, then the caller should read the indicated number of items and then call MapNext() to find out the number of items in the next block. Usage is similar to ReadArrayStart(). Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadNull

func (bd *BinaryDecoder) ReadNull() (interface{}, error)

ReadNull reads a null value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadString

func (bd *BinaryDecoder) ReadString() (string, error)

ReadString reads a string value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) Seek

func (bd *BinaryDecoder) Seek(pos int64)

Seek sets the reading position of this Decoder to a given value allowing to skip items etc.

func (*BinaryDecoder) SetBlock

func (bd *BinaryDecoder) SetBlock(block *DataBlock)

SetBlock is used for Avro Object Container Files where the data is split in blocks and sets a data block for this decoder and sets the position to the start of this block.

func (*BinaryDecoder) Tell

func (bd *BinaryDecoder) Tell() int64

Tell returns the current reading position of this Decoder.

type BinaryEncoder

type BinaryEncoder struct {
	// contains filtered or unexported fields
}

BinaryEncoder implements Encoder and provides low-level support for serializing Avro values.

func NewBinaryEncoder

func NewBinaryEncoder(buffer io.Writer) *BinaryEncoder

NewBinaryEncoder creates a new BinaryEncoder that will write to a given io.Writer.

func (*BinaryEncoder) WriteArrayNext

func (be *BinaryEncoder) WriteArrayNext(count int64)

WriteArrayNext should be called after finishing writing an array block either passing it the number of items in next block or 0 indicating the end of array.

func (*BinaryEncoder) WriteArrayStart

func (be *BinaryEncoder) WriteArrayStart(count int64)

WriteArrayStart should be called when starting to serialize an array providing it with a number of items in array block.

func (*BinaryEncoder) WriteBoolean

func (be *BinaryEncoder) WriteBoolean(x bool)

WriteBoolean writes a boolean value.

func (*BinaryEncoder) WriteBytes

func (be *BinaryEncoder) WriteBytes(x []byte)

WriteBytes writes a bytes value.

func (*BinaryEncoder) WriteDouble

func (be *BinaryEncoder) WriteDouble(x float64)

WriteDouble writes a double value.

func (*BinaryEncoder) WriteFloat

func (be *BinaryEncoder) WriteFloat(x float32)

WriteFloat writes a float value.

func (*BinaryEncoder) WriteInt

func (be *BinaryEncoder) WriteInt(x int32)

WriteInt writes an int value.

func (*BinaryEncoder) WriteLong

func (be *BinaryEncoder) WriteLong(x int64)

WriteLong writes a long value.

func (*BinaryEncoder) WriteMapNext

func (be *BinaryEncoder) WriteMapNext(count int64)

WriteMapNext should be called after finishing writing a map block either passing it the number of items in next block or 0 indicating the end of map.

func (*BinaryEncoder) WriteMapStart

func (be *BinaryEncoder) WriteMapStart(count int64)

WriteMapStart should be called when starting to serialize a map providing it with a number of items in map block.

func (*BinaryEncoder) WriteNull

func (be *BinaryEncoder) WriteNull(_ interface{})

WriteNull writes a null value. Doesn't actually do anything in this implementation.

func (*BinaryEncoder) WriteRaw

func (be *BinaryEncoder) WriteRaw(x []byte)

WriteRaw writes raw bytes to this Encoder.

func (*BinaryEncoder) WriteString

func (be *BinaryEncoder) WriteString(x string)

WriteString writes a string value.

type BooleanSchema

type BooleanSchema struct{}

BooleanSchema implements Schema and represents Avro boolean type.

func (*BooleanSchema) GetName

func (*BooleanSchema) GetName() string

GetName returns a type name for this BooleanSchema.

func (*BooleanSchema) MarshalJSON

func (*BooleanSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BooleanSchema) Prop

func (*BooleanSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BooleanSchema.

func (*BooleanSchema) String

func (*BooleanSchema) String() string

String returns a JSON representation of BooleanSchema.

func (*BooleanSchema) Type

func (*BooleanSchema) Type() int

Type returns a type constant for this BooleanSchema.

func (*BooleanSchema) Validate

func (*BooleanSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type BytesSchema

type BytesSchema struct{}

BytesSchema implements Schema and represents Avro bytes type.

func (*BytesSchema) GetName

func (*BytesSchema) GetName() string

GetName returns a type name for this BytesSchema.

func (*BytesSchema) MarshalJSON

func (*BytesSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BytesSchema) Prop

func (*BytesSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BytesSchema.

func (*BytesSchema) String

func (*BytesSchema) String() string

String returns a JSON representation of BytesSchema.

func (*BytesSchema) Type

func (*BytesSchema) Type() int

Type returns a type constant for this BytesSchema.

func (*BytesSchema) Validate

func (*BytesSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type CodeGenerator

type CodeGenerator struct {
	// contains filtered or unexported fields
}

CodeGenerator is a code generation tool for structs from given Avro schemas.

func NewCodeGenerator

func NewCodeGenerator(schemas []string) *CodeGenerator

NewCodeGenerator creates a new CodeGenerator for given Avro schemas.

func (*CodeGenerator) Generate

func (codegen *CodeGenerator) Generate() (string, error)

Generate generates source code for Avro schemas specified on creation. The ouput is Go formatted source code that contains struct definitions for all given schemas. May return an error if code generation fails, e.g. due to unparsable schema.

type DataBlock

type DataBlock struct {
	// Actual data
	Data []byte

	// Number of entries encoded in Data.
	NumEntries int64

	// Size of data buffer in bytes.
	BlockSize int

	// Number of unread entries in this DataBlock.
	BlockRemaining int64
}

DataBlock is a structure that holds a certain amount of entries and the actual buffer to read from.

type DataFileReader

type DataFileReader struct {
	// contains filtered or unexported fields
}

DataFileReader is a reader for Avro Object Container Files. More here: https://avro.apache.org/docs/current/spec.html#Object+Container+Files

func NewDataFileReader

func NewDataFileReader(filename string, datumReader DatumReader) (*DataFileReader, error)

NewDataFileReader creates a new DataFileReader for a given file and using the given DatumReader to read the data from that file. May return an error if the file contains invalid data or is just missing.

func (*DataFileReader) Close

func (reader *DataFileReader) Close() error

Close any resources opened by the DataFileReader

func (*DataFileReader) Next

func (reader *DataFileReader) Next(v interface{}) (bool, error)

Next reads the next value from file and fills the given value with data. First return value indicates whether the read was successful. Second return value indicates whether there was an error while reading data. Returns (false, nil) when no more data left to read.

func (*DataFileReader) NextBlock

func (reader *DataFileReader) NextBlock() error

NextBlock tells this DataFileReader to skip current block and move to next one. May return an error if the block is malformed or no more blocks left to read.

func (*DataFileReader) Seek

func (reader *DataFileReader) Seek(pos int64)

Seek switches the reading position in this DataFileReader to a provided value.

type DataFileWriter

type DataFileWriter struct {
	// contains filtered or unexported fields
}

DataFileWriter lets you write object container files.

func NewDataFileWriter

func NewDataFileWriter(output io.Writer, schema Schema, datumWriter DatumWriter) (writer *DataFileWriter, err error)

NewDataFileWriter creates a new DataFileWriter for given output and schema using the given DatumWriter to write the data to that Writer. May return an error if writing fails.

func (*DataFileWriter) Close

func (w *DataFileWriter) Close() error

Close this DataFileWriter. This is required to finish out the data file format. After Close() is called, this DataFileWriter cannot be used anymore.

func (*DataFileWriter) Flush

func (w *DataFileWriter) Flush() error

Flush out any previously written datums to our underlying io.Writer. Does nothing if no datums had previously been written.

It's up to the library user to decide how often to flush; doing it often will spend a lot of time on tiny I/O but save memory.

func (*DataFileWriter) Write

func (w *DataFileWriter) Write(v interface{}) error

Write out a single datum.

Encoded datums are buffered internally and will not be written to the underlying io.Writer until Flush() is called.

type DatumReader

type DatumReader interface {
	// Reads a single structured entry using this DatumReader according to provided Schema.
	// Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type.
	// May return an error indicating a read failure.
	Read(interface{}, Decoder) error

	// Sets the schema for this DatumReader to know the data structure.
	// Note that it must be called before calling Read.
	SetSchema(Schema)
}

DatumReader is an interface that is responsible for reading structured data according to schema from a decoder

type DatumWriter

type DatumWriter interface {
	// Write writes a single entry using this DatumWriter according to provided Schema.
	// Accepts a value to write and Encoder to write to.
	// May return an error indicating a write failure.
	Write(interface{}, Encoder) error

	// Sets the schema for this DatumWriter to know the data structure.
	// Note that it must be called before calling Write.
	SetSchema(Schema)
}

DatumWriter is an interface that is responsible for writing structured data according to schema to an encoder.

type Decoder

type Decoder interface {
	// Reads a null value. Returns a decoded value and an error if it occurs.
	ReadNull() (interface{}, error)

	// Reads a boolean value. Returns a decoded value and an error if it occurs.
	ReadBoolean() (bool, error)

	// Reads an in value. Returns a decoded value and an error if it occurs.
	ReadInt() (int32, error)

	// Reads a long value. Returns a decoded value and an error if it occurs.
	ReadLong() (int64, error)

	// Reads a float value. Returns a decoded value and an error if it occurs.
	ReadFloat() (float32, error)

	// Reads a double value. Returns a decoded value and an error if it occurs.
	ReadDouble() (float64, error)

	// Reads a bytes value. Returns a decoded value and an error if it occurs.
	ReadBytes() ([]byte, error)

	// Reads a string value. Returns a decoded value and an error if it occurs.
	ReadString() (string, error)

	// Reads an enum value (which is an Avro int value). Returns a decoded value and an error if it occurs.
	ReadEnum() (int32, error)

	// Reads and returns the size of the first block of an array. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call ArrayNext() to find out the number of items in the
	// next block. Returns a decoded value and an error if it occurs.
	ReadArrayStart() (int64, error)

	// Processes the next block of an array and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	ArrayNext() (int64, error)

	// Reads and returns the size of the first block of map entries. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call MapNext() to find out the number of items in the
	// next block. Usage is similar to ReadArrayStart(). Returns a decoded value and an error if it occurs.
	ReadMapStart() (int64, error)

	// Processes the next block of map entries and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	MapNext() (int64, error)

	// Reads fixed sized binary object into the provided buffer.
	// Returns an error if it occurs.
	ReadFixed([]byte) error

	// Reads fixed sized binary object into the provided buffer.
	// The second parameter is the position where the data needs to be written, the third is the size of binary object.
	// Returns an error if it occurs.
	ReadFixedWithBounds([]byte, int, int) error

	// SetBlock is used for Avro Object Container Files where the data is split in blocks and sets a data block
	// for this decoder and sets the position to the start of this block.
	SetBlock(*DataBlock)

	// Seek sets the reading position of this Decoder to a given value allowing to skip items etc.
	Seek(int64)

	// Tell returns the current reading position of this Decoder.
	Tell() int64
}

Decoder is an interface that provides low-level support for deserializing Avro values.

type DoubleSchema

type DoubleSchema struct{}

DoubleSchema implements Schema and represents Avro double type.

func (*DoubleSchema) GetName

func (*DoubleSchema) GetName() string

GetName returns a type name for this DoubleSchema.

func (*DoubleSchema) MarshalJSON

func (*DoubleSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*DoubleSchema) Prop

func (*DoubleSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for DoubleSchema.

func (*DoubleSchema) String

func (*DoubleSchema) String() string

Returns a JSON representation of DoubleSchema.

func (*DoubleSchema) Type

func (*DoubleSchema) Type() int

Type returns a type constant for this DoubleSchema.

func (*DoubleSchema) Validate

func (*DoubleSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Encoder

type Encoder interface {
	// Writes a null value. Doesn't actually do anything but may advance the state of Encoder implementation if it
	// is stateful.
	WriteNull(interface{})

	// Writes a boolean value.
	WriteBoolean(bool)

	// Writes an int value.
	WriteInt(int32)

	// Writes a long value.
	WriteLong(int64)

	// Writes a float value.
	WriteFloat(float32)

	// Writes a double value.
	WriteDouble(float64)

	// Writes a bytes value.
	WriteBytes([]byte)

	// Writes a string value.
	WriteString(string)

	// WriteArrayStart should be called when starting to serialize an array providing it with a number of items in
	// array block.
	WriteArrayStart(int64)

	// WriteArrayNext should be called after finishing writing an array block either passing it the number of items in
	// next block or 0 indicating the end of array.
	WriteArrayNext(int64)

	// WriteMapStart should be called when starting to serialize a map providing it with a number of items in
	// map block.
	WriteMapStart(int64)

	// WriteMapNext should be called after finishing writing a map block either passing it the number of items in
	// next block or 0 indicating the end of map.
	WriteMapNext(int64)

	// Writes raw bytes to this Encoder.
	WriteRaw([]byte)
}

Encoder is an interface that provides low-level support for serializing Avro values.

type EnumSchema

type EnumSchema struct {
	Name       string
	Namespace  string
	Aliases    []string
	Doc        string
	Symbols    []string
	Properties map[string]interface{}
}

EnumSchema implements Schema and represents Avro enum type.

func (*EnumSchema) GetName

func (s *EnumSchema) GetName() string

GetName returns an enum name for this EnumSchema.

func (*EnumSchema) MarshalJSON

func (s *EnumSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*EnumSchema) Prop

func (s *EnumSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*EnumSchema) String

func (s *EnumSchema) String() string

String returns a JSON representation of EnumSchema.

func (*EnumSchema) Type

func (*EnumSchema) Type() int

Type returns a type constant for this EnumSchema.

func (*EnumSchema) Validate

func (*EnumSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FixedSchema

type FixedSchema struct {
	Namespace  string
	Name       string
	Size       int
	Properties map[string]interface{}
}

FixedSchema implements Schema and represents Avro fixed type.

func (*FixedSchema) GetName

func (s *FixedSchema) GetName() string

GetName returns a fixed name for this FixedSchema.

func (*FixedSchema) MarshalJSON

func (s *FixedSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*FixedSchema) Prop

func (s *FixedSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*FixedSchema) String

func (s *FixedSchema) String() string

String returns a JSON representation of FixedSchema.

func (*FixedSchema) Type

func (*FixedSchema) Type() int

Type returns a type constant for this FixedSchema.

func (*FixedSchema) Validate

func (s *FixedSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FloatSchema

type FloatSchema struct{}

FloatSchema implements Schema and represents Avro float type.

func (*FloatSchema) GetName

func (*FloatSchema) GetName() string

GetName returns a type name for this FloatSchema.

func (*FloatSchema) MarshalJSON

func (*FloatSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*FloatSchema) Prop

func (*FloatSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for FloatSchema.

func (*FloatSchema) String

func (*FloatSchema) String() string

String returns a JSON representation of FloatSchema.

func (*FloatSchema) Type

func (*FloatSchema) Type() int

Type returns a type constant for this FloatSchema.

func (*FloatSchema) Validate

func (*FloatSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type GenericDatumReader

type GenericDatumReader struct {
	// contains filtered or unexported fields
}

GenericDatumReader implements DatumReader and is used for filling GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) with data. Each value passed to Read is expected to be a pointer.

func NewGenericDatumReader

func NewGenericDatumReader() *GenericDatumReader

NewGenericDatumReader creates a new GenericDatumReader.

func (*GenericDatumReader) Read

func (reader *GenericDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single entry using this GenericDatumReader. Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type. May return an error indicating a read failure.

func (*GenericDatumReader) SetSchema

func (reader *GenericDatumReader) SetSchema(schema Schema)

SetSchema sets the schema for this GenericDatumReader to know the data structure. Note that it must be called before calling Read.

type GenericDatumWriter

type GenericDatumWriter struct {
	// contains filtered or unexported fields
}

GenericDatumWriter implements DatumWriter and is used for writing GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) to a given Encoder.

func NewGenericDatumWriter

func NewGenericDatumWriter() *GenericDatumWriter

NewGenericDatumWriter creates a new GenericDatumWriter.

func (*GenericDatumWriter) SetSchema

func (writer *GenericDatumWriter) SetSchema(schema Schema)

SetSchema sets the provided schema for this GenericDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*GenericDatumWriter) Write

func (writer *GenericDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single entry using this GenericDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. May return an error indicating a write failure.

type GenericEnum

type GenericEnum struct {
	// Avro enum symbols.
	Symbols []string
	// contains filtered or unexported fields
}

GenericEnum is a generic Avro enum representation. This is still subject to change and may be rethought.

func NewGenericEnum

func NewGenericEnum(symbols []string) *GenericEnum

NewGenericEnum returns a new GenericEnum that uses provided enum symbols.

func (*GenericEnum) Get

func (enum *GenericEnum) Get() string

Get gets the string value for this enum (e.g. symbol).

func (*GenericEnum) GetIndex

func (enum *GenericEnum) GetIndex() int32

GetIndex gets the numeric value for this enum.

func (*GenericEnum) Set

func (enum *GenericEnum) Set(symbol string)

Set sets the string value for this enum (e.g. symbol). Panics if the given symbol does not exist in this enum.

func (*GenericEnum) SetIndex

func (enum *GenericEnum) SetIndex(index int32)

SetIndex sets the numeric value for this enum.

type GenericRecord

type GenericRecord struct {
	// contains filtered or unexported fields
}

GenericRecord is a generic instance of a record schema. Fields are accessible by their name.

func NewGenericRecord

func NewGenericRecord(schema Schema) *GenericRecord

NewGenericRecord creates a new GenericRecord.

func (*GenericRecord) Get

func (gr *GenericRecord) Get(name string) interface{}

Get gets a value by its name.

func (*GenericRecord) Map

func (gr *GenericRecord) Map() map[string]interface{}

Map returns a map representation of this GenericRecord.

func (*GenericRecord) Schema

func (gr *GenericRecord) Schema() Schema

Schema returns a schema for this GenericRecord.

func (*GenericRecord) Set

func (gr *GenericRecord) Set(name string, value interface{})

Set sets a value for a given name.

func (*GenericRecord) String

func (gr *GenericRecord) String() string

String returns a JSON representation of this GenericRecord.

type IntSchema

type IntSchema struct{}

IntSchema implements Schema and represents Avro int type.

func (*IntSchema) GetName

func (*IntSchema) GetName() string

GetName returns a type name for this IntSchema.

func (*IntSchema) MarshalJSON

func (*IntSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*IntSchema) Prop

func (*IntSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for IntSchema.

func (*IntSchema) String

func (*IntSchema) String() string

String returns a JSON representation of IntSchema.

func (*IntSchema) Type

func (*IntSchema) Type() int

Type returns a type constant for this IntSchema.

func (*IntSchema) Validate

func (*IntSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type LongSchema

type LongSchema struct{}

LongSchema implements Schema and represents Avro long type.

func (*LongSchema) GetName

func (*LongSchema) GetName() string

GetName returns a type name for this LongSchema.

func (*LongSchema) MarshalJSON

func (*LongSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*LongSchema) Prop

func (*LongSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for LongSchema.

func (*LongSchema) String

func (*LongSchema) String() string

Returns a JSON representation of LongSchema.

func (*LongSchema) Type

func (*LongSchema) Type() int

Type returns a type constant for this LongSchema.

func (*LongSchema) Validate

func (*LongSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type MapSchema

type MapSchema struct {
	Values     Schema
	Properties map[string]interface{}
}

MapSchema implements Schema and represents Avro map type.

func (*MapSchema) GetName

func (*MapSchema) GetName() string

GetName returns a type name for this MapSchema.

func (*MapSchema) MarshalJSON

func (s *MapSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*MapSchema) Prop

func (s *MapSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*MapSchema) String

func (s *MapSchema) String() string

String returns a JSON representation of MapSchema.

func (*MapSchema) Type

func (*MapSchema) Type() int

Type returns a type constant for this MapSchema.

func (*MapSchema) Validate

func (s *MapSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type NullSchema

type NullSchema struct{}

NullSchema implements Schema and represents Avro null type.

func (*NullSchema) GetName

func (*NullSchema) GetName() string

GetName returns a type name for this NullSchema.

func (*NullSchema) MarshalJSON

func (*NullSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*NullSchema) Prop

func (*NullSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for NullSchema.

func (*NullSchema) String

func (*NullSchema) String() string

String returns a JSON representation of NullSchema.

func (*NullSchema) Type

func (*NullSchema) Type() int

Type returns a type constant for this NullSchema.

func (*NullSchema) Validate

func (*NullSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Reader

type Reader interface {
	Read(dec Decoder) error
}

Reader is an interface that may be implemented to avoid using runtime reflection during deserialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

type RecordSchema

type RecordSchema struct {
	Name       string   `json:"name,omitempty"`
	Namespace  string   `json:"namespace,omitempty"`
	Doc        string   `json:"doc,omitempty"`
	Aliases    []string `json:"aliases,omitempty"`
	Properties map[string]interface{}
	Fields     []*SchemaField `json:"fields"`
}

RecordSchema implements Schema and represents Avro record type.

func (*RecordSchema) GetName

func (s *RecordSchema) GetName() string

GetName returns a record name for this RecordSchema.

func (*RecordSchema) MarshalJSON

func (s *RecordSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*RecordSchema) Prop

func (s *RecordSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*RecordSchema) String

func (s *RecordSchema) String() string

String returns a JSON representation of RecordSchema.

func (*RecordSchema) Type

func (*RecordSchema) Type() int

Type returns a type constant for this RecordSchema.

func (*RecordSchema) Validate

func (s *RecordSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type RecursiveSchema

type RecursiveSchema struct {
	Actual *RecordSchema
}

RecursiveSchema implements Schema and represents Avro record type without a definition (e.g. that should be looked up).

func (*RecursiveSchema) GetName

func (s *RecursiveSchema) GetName() string

GetName returns a record name for enclosed RecordSchema.

func (*RecursiveSchema) MarshalJSON

func (s *RecursiveSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*RecursiveSchema) Prop

func (*RecursiveSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for RecursiveSchema.

func (*RecursiveSchema) String

func (s *RecursiveSchema) String() string

String returns a JSON representation of RecursiveSchema.

func (*RecursiveSchema) Type

func (*RecursiveSchema) Type() int

Type returns a type constant for this RecursiveSchema.

func (*RecursiveSchema) Validate

func (s *RecursiveSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Schema

type Schema interface {
	// Returns an integer constant representing this schema type.
	Type() int

	// If this is a record, enum or fixed, returns its name, otherwise the name of the primitive type.
	GetName() string

	// Gets a custom non-reserved property from this schema and a bool representing if it exists.
	Prop(key string) (interface{}, bool)

	// Converts this schema to its JSON representation.
	String() string

	// Checks whether the given value is writeable to this schema.
	Validate(v reflect.Value) bool
}

Schema is an interface representing a single Avro schema (both primitive and complex).

func MustParseSchema

func MustParseSchema(rawSchema string) Schema

MustParseSchema is like ParseSchema, but panics if the given schema cannot be parsed.

func ParseSchema

func ParseSchema(rawSchema string) (Schema, error)

ParseSchema parses a given schema without provided schemas to reuse. Equivalent to call ParseSchemaWithResistry(rawSchema, make(map[string]Schema)) May return an error if schema is not parsable or has insufficient information about any type.

func ParseSchemaFile

func ParseSchemaFile(file string) (Schema, error)

ParseSchemaFile parses a given file. May return an error if schema is not parsable or file does not exist.

func ParseSchemaWithRegistry

func ParseSchemaWithRegistry(rawSchema string, schemas map[string]Schema) (Schema, error)

ParseSchemaWithRegistry parses a given schema using the provided registry for type lookup. Registry will be filled up during parsing. May return an error if schema is not parsable or has insufficient information about any type.

func Prepare

func Prepare(schema Schema) Schema

Prepare optimizes a schema for decoding/encoding.

It makes a recursive copy of the schema given and returns an immutable wrapper of the schema with some optimizations applied.

func PrepareResolving

func PrepareResolving(writer, reader Schema) Schema

PrepareResolving handles reading where the writer and reader schemas differ but are compatible. Support is currently limited to adding or removing record fields.

type SchemaField

type SchemaField struct {
	Name       string      `json:"name,omitempty"`
	Doc        string      `json:"doc,omitempty"`
	Default    interface{} `json:"default"`
	Type       Schema      `json:"type,omitempty"`
	Properties map[string]interface{}
}

SchemaField represents a schema field for Avro record.

func (*SchemaField) MarshalJSON

func (s *SchemaField) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema field as JSON.

func (*SchemaField) Prop

func (this *SchemaField) Prop(key string) (interface{}, bool)

Gets a custom non-reserved property from this schemafield and a bool representing if it exists.

func (*SchemaField) String

func (s *SchemaField) String() string

String returns a JSON representation of SchemaField.

type SpecificDatumReader

type SpecificDatumReader struct {
	// contains filtered or unexported fields
}

SpecificDatumReader implements DatumReader and is used for filling Go structs with data. Each value passed to Read is expected to be a pointer.

func NewSpecificDatumReader

func NewSpecificDatumReader() *SpecificDatumReader

NewSpecificDatumReader creates a new SpecificDatumReader.

func (*SpecificDatumReader) Read

func (reader *SpecificDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single structured entry using this SpecificDatumReader. Accepts a Go struct with exported fields to fill with data and a Decoder to read from. Given value MUST be of pointer type. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a read failure.

func (*SpecificDatumReader) SetSchema

func (reader *SpecificDatumReader) SetSchema(schema Schema)

SetSchema sets the schema for this SpecificDatumReader to know the data structure. Note that it must be called before calling Read.

type SpecificDatumWriter

type SpecificDatumWriter struct {
	// contains filtered or unexported fields
}

SpecificDatumWriter implements DatumWriter and is used for writing Go structs in Avro format.

func NewSpecificDatumWriter

func NewSpecificDatumWriter() *SpecificDatumWriter

NewSpecificDatumWriter creates a new SpecificDatumWriter.

func (*SpecificDatumWriter) SetSchema

func (writer *SpecificDatumWriter) SetSchema(schema Schema)

SetSchema sets the provided schema for this SpecificDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*SpecificDatumWriter) Write

func (writer *SpecificDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single Go struct using this SpecificDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a write failure.

type StringSchema

type StringSchema struct{}

StringSchema implements Schema and represents Avro string type.

func (*StringSchema) GetName

func (*StringSchema) GetName() string

GetName returns a type name for this StringSchema.

func (*StringSchema) MarshalJSON

func (*StringSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*StringSchema) Prop

func (*StringSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for StringSchema.

func (*StringSchema) String

func (*StringSchema) String() string

Returns a JSON representation of StringSchema.

func (*StringSchema) Type

func (*StringSchema) Type() int

Type returns a type constant for this StringSchema.

func (*StringSchema) Validate

func (*StringSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type UnionSchema

type UnionSchema struct {
	Types []Schema
}

UnionSchema implements Schema and represents Avro union type.

func (*UnionSchema) GetName

func (*UnionSchema) GetName() string

GetName returns a type name for this UnionSchema.

func (*UnionSchema) GetType

func (s *UnionSchema) GetType(v reflect.Value) int

GetType gets the index of actual union type for a given value.

func (*UnionSchema) MarshalJSON

func (s *UnionSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*UnionSchema) Prop

func (*UnionSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for UnionSchema.

func (*UnionSchema) String

func (s *UnionSchema) String() string

String returns a JSON representation of UnionSchema.

func (*UnionSchema) Type

func (*UnionSchema) Type() int

Type returns a type constant for this UnionSchema.

func (*UnionSchema) Validate

func (s *UnionSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Writer

type Writer interface {
	Write(enc Encoder) error
}

Writer is an interface that may be implemented to avoid using runtime reflection during serialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

Directories

Path Synopsis
examples

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL