container

package
v6.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 26, 2020 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Container provides a Reader and Writer which serialize and deserialize gogen-avro structs to the Avro Object Container File (OCF) format. These are low-level primitives you probably don't need to use directly.

Index

Constants

This section is empty.

Variables

View Source
var (
	LoggingEnabled = false
)

Functions

This section is empty.

Types

type AvroRecord

type AvroRecord interface {
	Serialize(io.Writer) error
	Schema() string
}

AvroRecord is an interface fulfilled by the structs generated by gogen-avro for Avro records.

type CloseableResettableWriter

type CloseableResettableWriter interface {
	Close() error
	Reset(io.Writer)
}

type Codec

type Codec string

A Codec specifies how the blocks within a container file should be compressed.

const (
	// No compression
	Null Codec = "null"

	// Deflate compression
	Deflate Codec = "deflate"

	// Snappy compression
	Snappy Codec = "snappy"
)

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is a low-level primitive for reading the OCF framing of a file. Generally you can create a Reader using the `New<RecordType>Reader` method generate for every record type.

func NewReader

func NewReader(r io.Reader) (*Reader, error)

func (*Reader) AvroContainerSchema

func (r *Reader) AvroContainerSchema() []byte

func (*Reader) Read

func (r *Reader) Read(b []byte) (n int, err error)

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer wraps an io.Writer and writes the file and block-level framing required for an OCF file. You can create a Writer for a given struct by calling the generated method `New<RecordType>Writer`.

func NewWriter

func NewWriter(writer io.Writer, codec Codec, recordsPerBlock int64, schema string) (*Writer, error)

Create a new Writer wrapping the provided io.Writer with the given Codec and number of records per block. The Writer will lazily write the container file header when WriteRecord is called the first time. You must call Flush on the Writer before closing the underlying io.Writer, to ensure the final block is written. A schema string must be passed to ensure that a correct header is written even if no records are written. This is required to produce valid empty Avro container files.

func (*Writer) BlockBufferSize added in v6.3.0

func (avroWriter *Writer) BlockBufferSize() int

Get the current block buffer size. caller might trigger an early Flush if current block size gets huge.

func (*Writer) Flush

func (avroWriter *Writer) Flush() error

Write the current block to the file if it has been filled. It is best-practise to always call this before the underlying io.Writer is closed.

func (*Writer) WriteRecord

func (avroWriter *Writer) WriteRecord(record AvroRecord) error

Write an AvroRecord to the container file. All gogen-avro generated structs fulfill the AvroRecord interface. Note that all records in a given container file must be of the same Avro type.

Directories

Path Synopsis
Code generated by github.com/actgardner/gogen-avro.
Code generated by github.com/actgardner/gogen-avro.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL