avro

package module
v1.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 10, 2022 License: MIT Imports: 24 Imported by: 70

README

Logo

Go Report Card Build Status Coverage Status GoDoc GitHub release GitHub license

A fast Go avro codec

Overview

Install with:

go get github.com/hamba/avro

Usage

type SimpleRecord struct {
	A int64  `avro:"a"`
	B string `avro:"b"`
}

schema, err := avro.Parse(`{
    "type": "record",
    "name": "simple",
    "namespace": "org.hamba.avro",
    "fields" : [
        {"name": "a", "type": "long"},
        {"name": "b", "type": "string"}
    ]
}`)
if err != nil {
	log.Fatal(err)
}

in := SimpleRecord{A: 27, B: "foo"}

data, err := avro.Marshal(schema, in)
if err != nil {
	log.Fatal(err)
}

fmt.Println(data)
// Outputs: [54 6 102 111 111]

out := SimpleRecord{}
err = avro.Unmarshal(schema, data, &out)
if err != nil {
	log.Fatal(err)
}

fmt.Println(out)
// Outputs: {27 foo}

More examples in the godoc.

Types Conversions
Avro Go Struct Go Interface
null nil nil
boolean bool bool
bytes []byte []byte
float float32 float32
double float64 float64
long int64 int64
int int, int32, int16, int8 int
string string string
array []T []interface{}
enum string string
fixed [n]byte []byte
map map[string]T{} map[string]interface{}
record struct map[string]interface{}
union see below see below
int.date time.Time time.Time
int.time-millis time.Duration time.Duration
long.time-micros time.Duration time.Duration
long.timestamp-millis time.Time time.Time
long.timestamp-micros time.Time time.Time
bytes.decimal *big.Rat *big.Rat
fixed.decimal *big.Rat *big.Rat
Unions

The following union types are accepted: map[string]interface{}, *T and interface{}.

  • map[string]interface{}: If the union value is nil, a nil map will be en/decoded. When a non-nil union value is encountered, a single key is en/decoded. The key is the avro type name, or scheam full name in the case of a named schema (enum, fixed or record).
  • *T: This is allowed in a "nullable" union. A nullable union is defined as a two schema union, with one of the types being null (ie. ["null", "string"] or ["string", "null"]), in this case a *T is allowed, with T matching the conversion table above.
  • interface{}: An interface can be provided and the type or name resolved. Primitive types are pre-registered, but named types, maps and slices will need to be registered with the Register function. In the case of arrays and maps the enclosed schema type or name is postfix to the type with a : separator, e.g "map:string". If any type cannot be resolved the map type above is used unless Config.UnionResolutionError is set to true in which case an error is returned.
TextMarshaler and TextUnmarshaler

The interfaces TextMarshaler and TextUnmarshaler are supported for a string schema type. The object will be tested first for implementation of these interfaces, in the case of a string schema, before trying regular encoding and decoding.

Recursive Structs

At this moment recursive structs are not supported. It is planned for the future.

Benchmark

Benchmark source code can be found at: https://github.com/nrwiersma/avro-benchmarks

BenchmarkGoAvroDecode-10       	  495176	      2413 ns/op	     418 B/op	      27 allocs/op
BenchmarkGoAvroEncode-10       	  420168	      2917 ns/op	     948 B/op	      63 allocs/op
BenchmarkGoGenAvroDecode-10    	  757150	      1552 ns/op	     728 B/op	      45 allocs/op
BenchmarkGoGenAvroEncode-10    	 1882940	       639.0 ns/op	     256 B/op	       3 allocs/op
BenchmarkHambaDecode-10        	 3138063	       383.0 ns/op	      64 B/op	       4 allocs/op
BenchmarkHambaEncode-10        	 4377513	       273.3 ns/op	     112 B/op	       1 allocs/op
BenchmarkLinkedinDecode-10     	 1000000	      1109 ns/op	    1688 B/op	      35 allocs/op
BenchmarkLinkedinEncode-10     	 2641016	       456.0 ns/op	     248 B/op	       5 allocs/op

Always benchmark with your own workload. The result depends heavily on the data input.

Documentation

Overview

Package avro implements encoding and decoding of Avro as defined by the Avro specification.

See the Avro specification for an understanding of Avro: http://avro.apache.org/docs/current/

Example (Usage)
package main

import (
	"fmt"
	"log"

	"github.com/hamba/avro"
)

var Schema = `{
		"type": "record",
		"name": "simple",
		"namespace": "org.hamba.avro",
		"fields" : [
			{"name": "a", "type": "long"},
			{"name": "b", "type": "string"}
		]
	}`

type SimpleRecord struct {
	A int64  `avro:"a"`
	B string `avro:"b"`
}

func main() {
	schema, err := avro.Parse(Schema)
	if err != nil {
		log.Fatal(err)
	}

	in := SimpleRecord{A: 27, B: "foo"}

	data, err := avro.Marshal(schema, in)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%+v\n", data)

	out := SimpleRecord{}
	err = avro.Unmarshal(schema, data, &out)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%+v\n", out)

}
Output:

[54 6 102 111 111]
{A:27 B:foo}

Index

Examples

Constants

This section is empty.

Variables

View Source
var DefaultConfig = Config{}.Freeze()

DefaultConfig is the default API.

View Source
var DefaultSchemaCache = &SchemaCache{}

DefaultSchemaCache is the default cache for schemas.

View Source
var NoDefault = noDef{}

NoDefault is used when no default exists for a field.

Functions

func Marshal

func Marshal(schema Schema, v interface{}) ([]byte, error)

Marshal returns the Avro encoding of v.

Example
package main

import (
	"fmt"

	"github.com/hamba/avro"
)

func main() {
	schema := avro.MustParse(`{
	    "type": "record",
	    "name": "simple",
	    "namespace": "org.hamba.avro",
	    "fields" : [
	        {"name": "a", "type": "long"},
	        {"name": "b", "type": "string"}
	    ]
	}`)

	type SimpleRecord struct {
		A int64  `avro:"a"`
		B string `avro:"b"`
	}

	simple := SimpleRecord{A: 27, B: "foo"}
	b, err := avro.Marshal(schema, simple)
	if err != nil {
		fmt.Println("error:", err)
	}

	fmt.Println(b)

}
Output:

[54 6 102 111 111]

func Register added in v0.5.0

func Register(name string, obj interface{})

Register registers names to their types for resolution. All primitive types are pre-registered.

Example
package main

import (
	"fmt"
	"log"

	"github.com/hamba/avro"
)

func main() {
	data := []byte{0x02, 0x02} // Your Avro data here
	schema := avro.MustParse(`["null", {"type":"enum", "name": "test", "symbols": ["A", "B"]}]`)

	avro.Register("test", "") // Register the name test as a string type

	var result interface{}
	err := avro.Unmarshal(schema, data, &result)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(result)

}
Output:

B

func Unmarshal

func Unmarshal(schema Schema, data []byte, v interface{}) error

Unmarshal parses the Avro encoded data and stores the result in the value pointed to by v. If v is nil or not a pointer, Unmarshal returns an error.

Example
package main

import (
	"fmt"

	"github.com/hamba/avro"
)

func main() {
	schema := avro.MustParse(`{
	    "type": "record",
	    "name": "simple",
	    "namespace": "org.hamba.avro",
	    "fields" : [
	        {"name": "a", "type": "long"},
	        {"name": "b", "type": "string"}
	    ]
	}`)

	type SimpleRecord struct {
		A int64  `avro:"a"`
		B string `avro:"b"`
	}

	data := []byte{0x36, 0x06, 0x66, 0x6F, 0x6F} // Your Avro data here
	simple := SimpleRecord{}
	if err := avro.Unmarshal(schema, data, &simple); err != nil {
		fmt.Println("error:", err)
	}

	fmt.Printf("%+v", simple)

}
Output:

{A:27 B:foo}

Types

type API

type API interface {
	// Marshal returns the Avro encoding of v.
	Marshal(schema Schema, v interface{}) ([]byte, error)

	// Unmarshal parses the Avro encoded data and stores the result in the value pointed to by v.
	// If v is nil or not a pointer, Unmarshal returns an error.
	Unmarshal(schema Schema, data []byte, v interface{}) error

	// NewEncoder returns a new encoder that writes to w using schema.
	NewEncoder(schema Schema, w io.Writer) *Encoder

	// NewDecoder returns a new decoder that reads from reader r using schema.
	NewDecoder(schema Schema, r io.Reader) *Decoder

	// DecoderOf returns the value decoder for a given schema and type.
	DecoderOf(schema Schema, typ reflect2.Type) ValDecoder

	// EncoderOf returns the value encoder for a given schema and type.
	EncoderOf(schema Schema, tpy reflect2.Type) ValEncoder

	// Register registers names to their types for resolution. All primitive types are pre-registered.
	Register(name string, obj interface{})
}

API represents a frozen Config.

type ArraySchema

type ArraySchema struct {
	// contains filtered or unexported fields
}

ArraySchema is an Avro array type schema.

func NewArraySchema added in v0.2.0

func NewArraySchema(items Schema) *ArraySchema

NewArraySchema creates an array schema instance.

func (*ArraySchema) AddProp added in v0.3.0

func (p *ArraySchema) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*ArraySchema) Fingerprint

func (s *ArraySchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*ArraySchema) FingerprintUsing added in v0.6.0

func (s *ArraySchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (*ArraySchema) Items

func (s *ArraySchema) Items() Schema

Items returns the items schema of an array.

func (*ArraySchema) MarshalJSON added in v1.6.0

func (s *ArraySchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (*ArraySchema) Prop added in v0.3.0

func (p *ArraySchema) Prop(name string) interface{}

Prop gets a property from the schema.

func (*ArraySchema) String

func (s *ArraySchema) String() string

String returns the canonical form of the schema.

func (*ArraySchema) Type

func (s *ArraySchema) Type() Type

Type returns the type of the schema.

type Config

type Config struct {
	// TagKey is the struct tag key used when en/decoding structs.
	// This defaults to "avro".
	TagKey string

	// BlockLength is the length of blocks for maps and arrays.
	// This defaults to 100.
	BlockLength int

	// UnionResolutionError determines if an error will be returned
	// when a type cannot be resolved while decoding a union.
	UnionResolutionError bool
}

Config customises how the codec should behave.

func (Config) Freeze

func (c Config) Freeze() API

Freeze makes the configuration immutable.

type DecimalLogicalSchema added in v1.0.0

type DecimalLogicalSchema struct {
	// contains filtered or unexported fields
}

DecimalLogicalSchema is a decimal logical type.

func NewDecimalLogicalSchema added in v1.0.0

func NewDecimalLogicalSchema(prec, scale int) *DecimalLogicalSchema

NewDecimalLogicalSchema creates a new decimal logical schema instance.

func (*DecimalLogicalSchema) Precision added in v1.0.0

func (s *DecimalLogicalSchema) Precision() int

Precision returns the precision of the decimal logical schema.

func (*DecimalLogicalSchema) Scale added in v1.0.0

func (s *DecimalLogicalSchema) Scale() int

Scale returns the scale of the decimal logical schema.

func (*DecimalLogicalSchema) String added in v1.0.0

func (s *DecimalLogicalSchema) String() string

String returns the canonical form of the logical schema.

func (*DecimalLogicalSchema) Type added in v1.0.0

Type returns the type of the logical schema.

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder reads and decodes Avro values from an input stream.

func NewDecoder

func NewDecoder(s string, r io.Reader) (*Decoder, error)

NewDecoder returns a new decoder that reads from reader r using schema s.

Example
package main

import (
	"bytes"
	"fmt"

	"github.com/hamba/avro"
)

func main() {
	schema := `{
	    "type": "record",
	    "name": "simple",
	    "namespace": "org.hamba.avro",
	    "fields" : [
	        {"name": "a", "type": "long"},
	        {"name": "b", "type": "string"}
	    ]
	}`

	type SimpleRecord struct {
		A int64  `avro:"a"`
		B string `avro:"b"`
	}

	r := bytes.NewReader([]byte{}) // Your reader goes here
	decoder, err := avro.NewDecoder(schema, r)
	if err != nil {
		fmt.Println("error:", err)
	}

	simple := SimpleRecord{}
	if err := decoder.Decode(&simple); err != nil {
		fmt.Println("error:", err)
	}

	fmt.Printf("%+v", simple)
}
Output:

func NewDecoderForSchema

func NewDecoderForSchema(schema Schema, reader io.Reader) *Decoder

NewDecoderForSchema returns a new decoder that reads from r using schema.

Example
package main

import (
	"bytes"
	"fmt"

	"github.com/hamba/avro"
)

func main() {
	schema := avro.MustParse(`{
	    "type": "record",
	    "name": "simple",
	    "namespace": "org.hamba.avro",
	    "fields" : [
	        {"name": "a", "type": "long"},
	        {"name": "b", "type": "string"}
	    ]
	}`)

	type SimpleRecord struct {
		A int64  `avro:"a"`
		B string `avro:"b"`
	}

	r := bytes.NewReader([]byte{0x36, 0x06, 0x66, 0x6F, 0x6F}) // Your reader goes here
	decoder := avro.NewDecoderForSchema(schema, r)

	simple := SimpleRecord{}
	if err := decoder.Decode(&simple); err != nil {
		fmt.Println("error:", err)
	}

	fmt.Printf("%+v", simple)

}
Output:

{A:27 B:foo}

func (*Decoder) Decode

func (d *Decoder) Decode(obj interface{}) error

Decode reads the next Avro encoded value from its input and stores it in the value pointed to by v.

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder writes Avro values to an output stream.

func NewEncoder

func NewEncoder(s string, w io.Writer) (*Encoder, error)

NewEncoder returns a new encoder that writes to w using schema s.

Example
package main

import (
	"bytes"
	"fmt"

	"github.com/hamba/avro"
)

func main() {
	schema := `{
	    "type": "record",
	    "name": "simple",
	    "namespace": "org.hamba.avro",
	    "fields" : [
	        {"name": "a", "type": "long"},
	        {"name": "b", "type": "string"}
	    ]
	}`

	type SimpleRecord struct {
		A int64  `avro:"a"`
		B string `avro:"b"`
	}

	w := &bytes.Buffer{}
	encoder, err := avro.NewEncoder(schema, w)
	if err != nil {
		fmt.Println("error:", err)
	}

	simple := SimpleRecord{A: 27, B: "foo"}
	if err := encoder.Encode(simple); err != nil {
		fmt.Println("error:", err)
	}

	fmt.Println(w.Bytes())

}
Output:

[54 6 102 111 111]

func NewEncoderForSchema

func NewEncoderForSchema(schema Schema, w io.Writer) *Encoder

NewEncoderForSchema returns a new encoder that writes to w using schema.

Example
package main

import (
	"bytes"
	"fmt"

	"github.com/hamba/avro"
)

func main() {
	schema := avro.MustParse(`{
	    "type": "record",
	    "name": "simple",
	    "namespace": "org.hamba.avro",
	    "fields" : [
	        {"name": "a", "type": "long"},
	        {"name": "b", "type": "string"}
	    ]
	}`)

	type SimpleRecord struct {
		A int64  `avro:"a"`
		B string `avro:"b"`
	}

	w := &bytes.Buffer{}
	encoder := avro.NewEncoderForSchema(schema, w)

	simple := SimpleRecord{A: 27, B: "foo"}
	if err := encoder.Encode(simple); err != nil {
		fmt.Println("error:", err)
	}

	fmt.Println(w.Bytes())

}
Output:

[54 6 102 111 111]

func (*Encoder) Encode

func (e *Encoder) Encode(v interface{}) error

Encode writes the Avro encoding of v to the stream.

type EnumSchema

type EnumSchema struct {
	// contains filtered or unexported fields
}

EnumSchema is an Avro enum type schema.

func NewEnumSchema added in v0.2.0

func NewEnumSchema(name, namespace string, symbols []string) (*EnumSchema, error)

NewEnumSchema creates a new enum schema instance.

func (*EnumSchema) AddProp added in v0.3.0

func (p *EnumSchema) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*EnumSchema) Fingerprint

func (s *EnumSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*EnumSchema) FingerprintUsing added in v0.6.0

func (s *EnumSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (EnumSchema) FullName added in v0.2.0

func (n EnumSchema) FullName() string

FullName returns the full qualified name of a schema.

func (*EnumSchema) MarshalJSON added in v1.6.0

func (s *EnumSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (EnumSchema) Name

func (n EnumSchema) Name() string

Name returns the name of a schema.

func (EnumSchema) Namespace added in v0.2.0

func (n EnumSchema) Namespace() string

Namespace returns the namespace of a schema.

func (*EnumSchema) Prop added in v0.3.0

func (p *EnumSchema) Prop(name string) interface{}

Prop gets a property from the schema.

func (*EnumSchema) String

func (s *EnumSchema) String() string

String returns the canonical form of the schema.

func (*EnumSchema) Symbols

func (s *EnumSchema) Symbols() []string

Symbols returns the symbols of an enum.

func (*EnumSchema) Type

func (s *EnumSchema) Type() Type

Type returns the type of the schema.

type Field

type Field struct {
	// contains filtered or unexported fields
}

Field is an Avro record type field.

func NewField added in v0.2.0

func NewField(name string, typ Schema, def interface{}) (*Field, error)

NewField creates a new field instance.

func (*Field) AddDoc added in v1.6.3

func (f *Field) AddDoc(doc string)

AddDoc add documentation to the field.

func (*Field) AddProp added in v0.3.0

func (p *Field) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*Field) Default

func (f *Field) Default() interface{}

Default returns the default of a field or nil.

The only time a nil default is valid is for a Null Type.

func (*Field) Doc added in v1.6.3

func (f *Field) Doc() string

Doc returns the documentation of a field.

func (*Field) HasDefault added in v0.5.0

func (f *Field) HasDefault() bool

HasDefault determines if the field has a default value.

func (*Field) MarshalJSON added in v1.6.0

func (f *Field) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (*Field) Name

func (f *Field) Name() string

Name returns the name of a field.

func (*Field) Prop added in v0.3.0

func (p *Field) Prop(name string) interface{}

Prop gets a property from the schema.

func (*Field) String

func (f *Field) String() string

String returns the canonical form of a field.

func (*Field) Type

func (f *Field) Type() Schema

Type returns the schema of a field.

type FingerprintType added in v0.6.0

type FingerprintType string

FingerprintType is a fingerprinting algorithm.

const (
	CRC64Avro FingerprintType = "CRC64-AVRO"
	MD5       FingerprintType = "MD5"
	SHA256    FingerprintType = "SHA256"
)

Fingerprint type constants.

type FixedSchema

type FixedSchema struct {
	// contains filtered or unexported fields
}

FixedSchema is an Avro fixed type schema.

func NewFixedSchema added in v0.2.0

func NewFixedSchema(name, namespace string, size int, logical LogicalSchema) (*FixedSchema, error)

NewFixedSchema creates a new fixed schema instance.

func (*FixedSchema) AddProp added in v0.3.0

func (p *FixedSchema) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*FixedSchema) Fingerprint

func (s *FixedSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*FixedSchema) FingerprintUsing added in v0.6.0

func (s *FixedSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (FixedSchema) FullName added in v0.2.0

func (n FixedSchema) FullName() string

FullName returns the full qualified name of a schema.

func (*FixedSchema) Logical added in v1.0.0

func (s *FixedSchema) Logical() LogicalSchema

Logical returns the logical schema or nil.

func (*FixedSchema) MarshalJSON added in v1.6.0

func (s *FixedSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (FixedSchema) Name

func (n FixedSchema) Name() string

Name returns the name of a schema.

func (FixedSchema) Namespace added in v0.2.0

func (n FixedSchema) Namespace() string

Namespace returns the namespace of a schema.

func (*FixedSchema) Prop added in v0.3.0

func (p *FixedSchema) Prop(name string) interface{}

Prop gets a property from the schema.

func (*FixedSchema) Size

func (s *FixedSchema) Size() int

Size returns the number of bytes of the fixed schema.

func (*FixedSchema) String

func (s *FixedSchema) String() string

String returns the canonical form of the schema.

func (*FixedSchema) Type

func (s *FixedSchema) Type() Type

Type returns the type of the schema.

type LogicalSchema added in v1.0.0

type LogicalSchema interface {
	// Type returns the type of the logical schema.
	Type() LogicalType

	// String returns the canonical form of the logical schema.
	String() string
}

LogicalSchema represents an Avro schema with a logical type.

type LogicalType added in v1.0.0

type LogicalType string

LogicalType is a schema logical type.

const (
	Decimal         LogicalType = "decimal"
	UUID            LogicalType = "uuid"
	Date            LogicalType = "date"
	TimeMillis      LogicalType = "time-millis"
	TimeMicros      LogicalType = "time-micros"
	TimestampMillis LogicalType = "timestamp-millis"
	TimestampMicros LogicalType = "timestamp-micros"
	Duration        LogicalType = "duration"
)

Schema logical type constants.

type LogicalTypeSchema added in v1.0.0

type LogicalTypeSchema interface {
	// Logical returns the logical schema or nil.
	Logical() LogicalSchema
}

LogicalTypeSchema represents a schema that can contain a logical type.

type MapSchema

type MapSchema struct {
	// contains filtered or unexported fields
}

MapSchema is an Avro map type schema.

func NewMapSchema added in v0.2.0

func NewMapSchema(values Schema) *MapSchema

NewMapSchema creates a map schema instance.

func (*MapSchema) AddProp added in v0.3.0

func (p *MapSchema) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*MapSchema) Fingerprint

func (s *MapSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*MapSchema) FingerprintUsing added in v0.6.0

func (s *MapSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (*MapSchema) MarshalJSON added in v1.6.0

func (s *MapSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (*MapSchema) Prop added in v0.3.0

func (p *MapSchema) Prop(name string) interface{}

Prop gets a property from the schema.

func (*MapSchema) String

func (s *MapSchema) String() string

String returns the canonical form of the schema.

func (*MapSchema) Type

func (s *MapSchema) Type() Type

Type returns the type of the schema.

func (*MapSchema) Values

func (s *MapSchema) Values() Schema

Values returns the values schema of a map.

type Message added in v0.5.0

type Message struct {
	// contains filtered or unexported fields
}

Message is an Avro protocol message.

func NewMessage added in v0.5.0

func NewMessage(req *RecordSchema, resp Schema, errors *UnionSchema, oneWay bool) *Message

NewMessage creates a protocol message instance.

func (*Message) AddProp added in v0.5.0

func (p *Message) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*Message) Errors added in v0.5.0

func (m *Message) Errors() *UnionSchema

Errors returns the message errors union schema.

func (*Message) OneWay added in v0.5.0

func (m *Message) OneWay() bool

OneWay determines of the message is a one way message.

func (*Message) Prop added in v0.5.0

func (p *Message) Prop(name string) interface{}

Prop gets a property from the schema.

func (*Message) Request added in v0.5.0

func (m *Message) Request() *RecordSchema

Request returns the message request schema.

func (*Message) Response added in v0.5.0

func (m *Message) Response() Schema

Response returns the message response schema.

func (*Message) String added in v0.5.0

func (m *Message) String() string

String returns the canonical form of the message.

type NamedSchema

type NamedSchema interface {
	Schema
	PropertySchema

	// Name returns the name of the schema.
	Name() string

	// Namespace returns the namespace of a schema.
	Namespace() string

	// FullName returns the full qualified name of a schema.
	FullName() string
}

NamedSchema represents a schema with a name.

type NullSchema

type NullSchema struct {
	// contains filtered or unexported fields
}

NullSchema is an Avro null type schema.

func (*NullSchema) Fingerprint

func (s *NullSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*NullSchema) FingerprintUsing added in v0.6.0

func (s *NullSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (*NullSchema) MarshalJSON added in v1.6.0

func (s *NullSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (*NullSchema) String

func (s *NullSchema) String() string

String returns the canonical form of the schema.

func (*NullSchema) Type

func (s *NullSchema) Type() Type

Type returns the type of the schema.

type PrimitiveLogicalSchema added in v1.0.0

type PrimitiveLogicalSchema struct {
	// contains filtered or unexported fields
}

PrimitiveLogicalSchema is a logical type with no properties.

func NewPrimitiveLogicalSchema added in v1.0.0

func NewPrimitiveLogicalSchema(typ LogicalType) *PrimitiveLogicalSchema

NewPrimitiveLogicalSchema creates a new primitive logical schema instance.

func (*PrimitiveLogicalSchema) String added in v1.0.0

func (s *PrimitiveLogicalSchema) String() string

String returns the canonical form of the logical schema.

func (*PrimitiveLogicalSchema) Type added in v1.0.0

Type returns the type of the logical schema.

type PrimitiveSchema

type PrimitiveSchema struct {
	// contains filtered or unexported fields
}

PrimitiveSchema is an Avro primitive type schema.

func NewPrimitiveSchema

func NewPrimitiveSchema(t Type, l LogicalSchema) *PrimitiveSchema

NewPrimitiveSchema creates a new PrimitiveSchema.

func (*PrimitiveSchema) AddProp added in v1.8.0

func (p *PrimitiveSchema) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*PrimitiveSchema) Fingerprint

func (s *PrimitiveSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*PrimitiveSchema) FingerprintUsing added in v0.6.0

func (s *PrimitiveSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (*PrimitiveSchema) Logical added in v1.0.0

func (s *PrimitiveSchema) Logical() LogicalSchema

Logical returns the logical schema or nil.

func (*PrimitiveSchema) MarshalJSON added in v1.6.0

func (s *PrimitiveSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (*PrimitiveSchema) Prop added in v1.8.0

func (p *PrimitiveSchema) Prop(name string) interface{}

Prop gets a property from the schema.

func (*PrimitiveSchema) String

func (s *PrimitiveSchema) String() string

String returns the canonical form of the schema.

func (*PrimitiveSchema) Type

func (s *PrimitiveSchema) Type() Type

Type returns the type of the schema.

type PropertySchema added in v0.3.0

type PropertySchema interface {
	// AddProp adds a property to the schema.
	//
	// AddProp will not overwrite existing properties.
	AddProp(name string, value interface{})

	// Prop gets a property from the schema.
	Prop(string) interface{}
}

PropertySchema represents a schema with properties.

type Protocol added in v0.5.0

type Protocol struct {
	// contains filtered or unexported fields
}

Protocol is an Avro protocol.

func MustParseProtocol added in v0.5.0

func MustParseProtocol(protocol string) *Protocol

MustParseProtocol parses an Avro protocol, panicing if there is an error.

func NewProtocol added in v0.5.0

func NewProtocol(name, space string, types []NamedSchema, messages map[string]*Message) (*Protocol, error)

NewProtocol creates a protocol instance.

func ParseProtocol added in v0.5.0

func ParseProtocol(protocol string) (*Protocol, error)

ParseProtocol parses an Avro protocol.

func ParseProtocolFile added in v0.5.0

func ParseProtocolFile(path string) (*Protocol, error)

ParseProtocolFile parses an Avro protocol from a file.

func (*Protocol) AddProp added in v0.5.0

func (p *Protocol) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (Protocol) FullName added in v0.5.0

func (n Protocol) FullName() string

FullName returns the full qualified name of a schema.

func (*Protocol) Hash added in v0.5.0

func (p *Protocol) Hash() string

Hash returns the MD5 hash of the protocol.

func (*Protocol) Message added in v0.5.0

func (p *Protocol) Message(name string) *Message

Message returns a message with the given name or nil.

func (Protocol) Name added in v0.5.0

func (n Protocol) Name() string

Name returns the name of a schema.

func (Protocol) Namespace added in v0.5.0

func (n Protocol) Namespace() string

Namespace returns the namespace of a schema.

func (*Protocol) Prop added in v0.5.0

func (p *Protocol) Prop(name string) interface{}

Prop gets a property from the schema.

func (*Protocol) String added in v0.5.0

func (p *Protocol) String() string

String returns the canonical form of the protocol.

type Reader

type Reader struct {
	Error error
	// contains filtered or unexported fields
}

Reader is an Avro specific io.Reader.

func NewReader

func NewReader(r io.Reader, bufSize int, opts ...ReaderFunc) *Reader

NewReader creates a new Reader.

func (*Reader) Read

func (r *Reader) Read(b []byte)

Read reads data into the given bytes.

func (*Reader) ReadArrayCB

func (r *Reader) ReadArrayCB(callback func(*Reader) bool)

ReadArrayCB reads an array with a callback per item.

func (*Reader) ReadBlockHeader

func (r *Reader) ReadBlockHeader() (int64, int64)

ReadBlockHeader reads a Block Header from the Reader.

func (*Reader) ReadBool

func (r *Reader) ReadBool() bool

ReadBool reads a Bool from the Reader.

func (*Reader) ReadBytes

func (r *Reader) ReadBytes() []byte

ReadBytes reads Bytes from the Reader.

func (*Reader) ReadDouble

func (r *Reader) ReadDouble() float64

ReadDouble reads a Double from the Reader.

func (*Reader) ReadFloat

func (r *Reader) ReadFloat() float32

ReadFloat reads a Float from the Reader.

func (*Reader) ReadInt

func (r *Reader) ReadInt() int32

ReadInt reads an Int from the Reader.

func (*Reader) ReadLong

func (r *Reader) ReadLong() int64

ReadLong reads a Long from the Reader.

func (*Reader) ReadMapCB

func (r *Reader) ReadMapCB(callback func(*Reader, string) bool)

ReadMapCB reads an array with a callback per item.

func (*Reader) ReadNext

func (r *Reader) ReadNext(schema Schema) interface{}

ReadNext reads the next Avro element as a generic interface.

func (*Reader) ReadString

func (r *Reader) ReadString() string

ReadString reads a String from the Reader.

func (*Reader) ReadVal

func (r *Reader) ReadVal(schema Schema, obj interface{})

ReadVal parses Avro value and stores the result in the value pointed to by obj.

func (*Reader) ReportError

func (r *Reader) ReportError(operation, msg string)

ReportError record a error in iterator instance with current position.

func (*Reader) Reset

func (r *Reader) Reset(b []byte) *Reader

Reset resets a Reader with a new byte array attached.

func (*Reader) SkipBool

func (r *Reader) SkipBool()

SkipBool skips a Bool in the reader.

func (*Reader) SkipBytes

func (r *Reader) SkipBytes()

SkipBytes skips Bytes in the reader.

func (*Reader) SkipDouble

func (r *Reader) SkipDouble()

SkipDouble skips a Double in the reader.

func (*Reader) SkipFloat

func (r *Reader) SkipFloat()

SkipFloat skips a Float in the reader.

func (*Reader) SkipInt

func (r *Reader) SkipInt()

SkipInt skips an Int in the reader.

func (*Reader) SkipLong

func (r *Reader) SkipLong()

SkipLong skips a Long in the reader.

func (*Reader) SkipNBytes

func (r *Reader) SkipNBytes(n int)

SkipNBytes skips the given number of bytes in the reader.

func (*Reader) SkipString

func (r *Reader) SkipString()

SkipString skips a String in the reader.

type ReaderFunc

type ReaderFunc func(r *Reader)

ReaderFunc is a function used to customize the Reader.

func WithReaderConfig

func WithReaderConfig(cfg API) ReaderFunc

WithReaderConfig specifies the configuration to use with a reader.

type RecordSchema

type RecordSchema struct {
	// contains filtered or unexported fields
}

RecordSchema is an Avro record type schema.

func NewErrorRecordSchema added in v0.5.0

func NewErrorRecordSchema(name, space string, fields []*Field) (*RecordSchema, error)

NewErrorRecordSchema creates a new error record schema instance.

func NewRecordSchema added in v0.2.0

func NewRecordSchema(name, space string, fields []*Field) (*RecordSchema, error)

NewRecordSchema creates a new record schema instance.

func (*RecordSchema) AddDoc added in v1.6.3

func (s *RecordSchema) AddDoc(doc string)

AddDoc add documentation to the record.

func (*RecordSchema) AddProp added in v0.3.0

func (p *RecordSchema) AddProp(name string, value interface{})

AddProp adds a property to the schema.

AddProp will not overwrite existing properties.

func (*RecordSchema) Doc added in v1.6.3

func (s *RecordSchema) Doc() string

Doc returns the documentation of a record.

func (*RecordSchema) Fields

func (s *RecordSchema) Fields() []*Field

Fields returns the fields of a record.

func (*RecordSchema) Fingerprint

func (s *RecordSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*RecordSchema) FingerprintUsing added in v0.6.0

func (s *RecordSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (RecordSchema) FullName added in v0.2.0

func (n RecordSchema) FullName() string

FullName returns the full qualified name of a schema.

func (*RecordSchema) IsError added in v0.5.0

func (s *RecordSchema) IsError() bool

IsError determines is this is an error record.

func (*RecordSchema) MarshalJSON added in v1.6.0

func (s *RecordSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (RecordSchema) Name

func (n RecordSchema) Name() string

Name returns the name of a schema.

func (RecordSchema) Namespace added in v0.2.0

func (n RecordSchema) Namespace() string

Namespace returns the namespace of a schema.

func (*RecordSchema) Prop added in v0.3.0

func (p *RecordSchema) Prop(name string) interface{}

Prop gets a property from the schema.

func (*RecordSchema) String

func (s *RecordSchema) String() string

String returns the canonical form of the schema.

func (*RecordSchema) Type

func (s *RecordSchema) Type() Type

Type returns the type of the schema.

type RefSchema

type RefSchema struct {
	// contains filtered or unexported fields
}

RefSchema is a reference to a named Avro schema.

func NewRefSchema added in v0.2.0

func NewRefSchema(schema NamedSchema) *RefSchema

NewRefSchema creates a ref schema instance.

func (*RefSchema) Fingerprint

func (s *RefSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*RefSchema) FingerprintUsing added in v0.6.0

func (s *RefSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (*RefSchema) MarshalJSON added in v1.6.0

func (s *RefSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (*RefSchema) Schema

func (s *RefSchema) Schema() Schema

Schema returns the schema being referenced.

func (*RefSchema) String

func (s *RefSchema) String() string

String returns the canonical form of the schema.

func (*RefSchema) Type

func (s *RefSchema) Type() Type

Type returns the type of the schema.

type Schema

type Schema interface {
	// Type returns the type of the schema.
	Type() Type

	// String returns the canonical form of the schema.
	String() string

	// Fingerprint returns the SHA256 fingerprint of the schema.
	Fingerprint() [32]byte

	// FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.
	FingerprintUsing(FingerprintType) ([]byte, error)
}

Schema represents an Avro schema.

func MustParse

func MustParse(schema string) Schema

MustParse parses a schema string, panicing if there is an error.

func Parse

func Parse(schema string) (Schema, error)

Parse parses a schema string.

Example
package main

import (
	"fmt"
	"log"

	"github.com/hamba/avro"
)

func main() {
	schema, err := avro.Parse(`{
	    "type": "record",
	    "name": "simple",
	    "namespace": "org.hamba.avro",
	    "fields" : [
	        {"name": "a", "type": "long"},
	        {"name": "b", "type": "string"}
	    ]
	}`)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(schema.Type())
	// Outputs: record
}
Output:

func ParseFiles

func ParseFiles(paths ...string) (Schema, error)

ParseFiles parses the schemas in the files, in the order they appear, returning the last schema.

This is useful when your schemas rely on other schemas.

func ParseWithCache added in v0.5.0

func ParseWithCache(schema, namespace string, cache *SchemaCache) (Schema, error)

ParseWithCache parses a schema string using the given namespace and schema cache.

type SchemaCache added in v0.5.0

type SchemaCache struct {
	// contains filtered or unexported fields
}

SchemaCache is a cache of schemas.

func (*SchemaCache) Add added in v0.5.0

func (c *SchemaCache) Add(name string, schema Schema)

Add adds a schema to the cache with the given name.

func (*SchemaCache) Get added in v0.5.0

func (c *SchemaCache) Get(name string) Schema

Get returns the Schema if it exists.

type SchemaCompatibility added in v0.5.0

type SchemaCompatibility struct {
	// contains filtered or unexported fields
}

SchemaCompatibility determines the compatibility of schemas.

func NewSchemaCompatibility added in v0.5.0

func NewSchemaCompatibility() *SchemaCompatibility

NewSchemaCompatibility creates a new schema compatibility instance.

func (*SchemaCompatibility) Compatible added in v0.5.0

func (c *SchemaCompatibility) Compatible(reader, writer Schema) error

Compatible determines the compatibility if the reader and writer schemas.

type Schemas

type Schemas []Schema

Schemas is a slice of Schemas.

func (Schemas) Get

func (s Schemas) Get(name string) (Schema, int)

Get gets a schema and position by type or name if it is a named schema.

type Type

type Type string

Type is a schema type.

const (
	Record  Type = "record"
	Error   Type = "error"
	Ref     Type = "<ref>"
	Enum    Type = "enum"
	Array   Type = "array"
	Map     Type = "map"
	Union   Type = "union"
	Fixed   Type = "fixed"
	String  Type = "string"
	Bytes   Type = "bytes"
	Int     Type = "int"
	Long    Type = "long"
	Float   Type = "float"
	Double  Type = "double"
	Boolean Type = "boolean"
	Null    Type = "null"
)

Schema type constants.

type TypeResolver added in v0.5.0

type TypeResolver struct {
	// contains filtered or unexported fields
}

TypeResolver resolves types by name.

func NewTypeResolver added in v0.5.0

func NewTypeResolver() *TypeResolver

NewTypeResolver creates a new type resolver with all primitive types registered.

func (*TypeResolver) Name added in v0.5.0

func (r *TypeResolver) Name(typ reflect2.Type) ([]string, error)

Name gets the name for a type, or an error.

func (*TypeResolver) Register added in v0.5.0

func (r *TypeResolver) Register(name string, obj interface{})

Register registers names to their types for resolution.

func (*TypeResolver) Type added in v0.5.0

func (r *TypeResolver) Type(name string) (reflect2.Type, error)

Type gets the type for a name, or an error.

type UnionSchema

type UnionSchema struct {
	// contains filtered or unexported fields
}

UnionSchema is an Avro union type schema.

func NewUnionSchema added in v0.2.0

func NewUnionSchema(types []Schema) (*UnionSchema, error)

NewUnionSchema creates a union schema instance.

func (*UnionSchema) Fingerprint

func (s *UnionSchema) Fingerprint() [32]byte

Fingerprint returns the SHA256 fingerprint of the schema.

func (*UnionSchema) FingerprintUsing added in v0.6.0

func (s *UnionSchema) FingerprintUsing(typ FingerprintType) ([]byte, error)

FingerprintUsing returns the fingerprint of the schema using the given algorithm or an error.

func (*UnionSchema) Indices added in v1.1.0

func (s *UnionSchema) Indices() (null, typ int)

Indices returns the index of the null and type schemas for a nullable schema. For non-nullable schemas 0 is returned for both.

func (*UnionSchema) MarshalJSON added in v1.6.0

func (s *UnionSchema) MarshalJSON() ([]byte, error)

MarshalJSON marshals the schema to json.

func (*UnionSchema) Nullable

func (s *UnionSchema) Nullable() bool

Nullable returns the Schema if the union is nullable, otherwise nil.

func (*UnionSchema) String

func (s *UnionSchema) String() string

String returns the canonical form of the schema.

func (*UnionSchema) Type

func (s *UnionSchema) Type() Type

Type returns the type of the schema.

func (*UnionSchema) Types

func (s *UnionSchema) Types() Schemas

Types returns the types of a union.

type ValDecoder

type ValDecoder interface {
	Decode(ptr unsafe.Pointer, r *Reader)
}

ValDecoder represents an internal value decoder.

You should never use ValDecoder directly.

type ValEncoder

type ValEncoder interface {
	Encode(ptr unsafe.Pointer, w *Writer)
}

ValEncoder represents an internal value encoder.

You should never use ValEncoder directly.

type Writer

type Writer struct {
	Error error
	// contains filtered or unexported fields
}

Writer is an Avro specific io.Writer.

func NewWriter

func NewWriter(out io.Writer, bufSize int, opts ...WriterFunc) *Writer

NewWriter creates a new Writer.

func (*Writer) Buffer

func (w *Writer) Buffer() []byte

Buffer gets the Writer buffer.

func (*Writer) Buffered

func (w *Writer) Buffered() int

Buffered returns the number of buffered bytes.

func (*Writer) Flush

func (w *Writer) Flush() error

Flush writes any buffered data to the underlying io.Writer.

func (*Writer) Reset

func (w *Writer) Reset(out io.Writer)

Reset resets the Writer with a new io.Writer attached.

func (*Writer) Write

func (w *Writer) Write(b []byte)

Write writes raw bytes to the Writer.

func (*Writer) WriteBlockCB

func (w *Writer) WriteBlockCB(callback func(w *Writer) int64) int64

WriteBlockCB writes a block using the callback.

func (*Writer) WriteBlockHeader

func (w *Writer) WriteBlockHeader(l, s int64)

WriteBlockHeader writes a Block Header to the Writer.

func (*Writer) WriteBool

func (w *Writer) WriteBool(b bool)

WriteBool writes a Bool to the Writer.

func (*Writer) WriteBytes

func (w *Writer) WriteBytes(b []byte)

WriteBytes writes Bytes to the Writer.

func (*Writer) WriteDouble

func (w *Writer) WriteDouble(f float64)

WriteDouble writes a Double to the Writer.

func (*Writer) WriteFloat

func (w *Writer) WriteFloat(f float32)

WriteFloat writes a Float to the Writer.

func (*Writer) WriteInt

func (w *Writer) WriteInt(i int32)

WriteInt writes an Int to the Writer.

func (*Writer) WriteLong

func (w *Writer) WriteLong(i int64)

WriteLong writes a Long to the Writer.

func (*Writer) WriteString

func (w *Writer) WriteString(s string)

WriteString reads a String to the Writer.

func (*Writer) WriteVal

func (w *Writer) WriteVal(schema Schema, val interface{})

WriteVal writes the Avro encoding of obj.

type WriterFunc

type WriterFunc func(w *Writer)

WriterFunc is a function used to customize the Writer.

func WithWriterConfig

func WithWriterConfig(cfg API) WriterFunc

WithWriterConfig specifies the configuration to use with a writer.

Directories

Path Synopsis
internal
bytesx
Package bytesx implements bytes extensions.
Package bytesx implements bytes extensions.
Package ocf implements encoding and decoding of Avro Object Container Files as defined by the Avro specification.
Package ocf implements encoding and decoding of Avro Object Container Files as defined by the Avro specification.
pkg
crc64
Package crc64 implements the Avro CRC-64 checksum.
Package crc64 implements the Avro CRC-64 checksum.
Package registry implements a Confluent Schema Registry compliant client.
Package registry implements a Confluent Schema Registry compliant client.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL