ipc

package
v0.0.0-...-bc21918 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2021 License: Apache-2.0, BSD-2-Clause, BSD-3-Clause, + 8 more Imports: 20 Imported by: 23

Documentation

Index

Constants

View Source
const (
	MetadataV1 = MetadataVersion(flatbuf.MetadataVersionV1) // version for Arrow-0.1.0
	MetadataV2 = MetadataVersion(flatbuf.MetadataVersionV2) // version for Arrow-0.2.0
	MetadataV3 = MetadataVersion(flatbuf.MetadataVersionV3) // version for Arrow-0.3.0 to 0.7.1
	MetadataV4 = MetadataVersion(flatbuf.MetadataVersionV4) // version for >= Arrow-0.8.0
	MetadataV5 = MetadataVersion(flatbuf.MetadataVersionV5) // version for >= Arrow-1.0.0, backward compatible with v4
)
View Source
const (

	// constants for the extension type metadata keys for the type name and
	// any extension metadata to be passed to deserialize.
	ExtensionTypeKeyName     = "ARROW:extension:name"
	ExtensionMetadataKeyName = "ARROW:extension:metadata"
)

Variables

View Source
var Magic = []byte("ARROW1")

Magic string identifying an Apache Arrow file.

Functions

This section is empty.

Types

type FileReader

type FileReader struct {
	// contains filtered or unexported fields
}

FileReader is an Arrow file reader.

func NewFileReader

func NewFileReader(r ReadAtSeeker, opts ...Option) (*FileReader, error)

NewFileReader opens an Arrow file using the provided reader r.

func (*FileReader) Close

func (f *FileReader) Close() error

Close cleans up resources used by the File. Close does not close the underlying reader.

func (*FileReader) NumDictionaries

func (f *FileReader) NumDictionaries() int

func (*FileReader) NumRecords

func (f *FileReader) NumRecords() int

func (*FileReader) Read

func (f *FileReader) Read() (rec array.Record, err error)

Read reads the current record from the underlying stream and an error, if any. When the Reader reaches the end of the underlying stream, it returns (nil, io.EOF).

The returned record value is valid until the next call to Read. Users need to call Retain on that Record to keep it valid for longer.

func (*FileReader) ReadAt

func (f *FileReader) ReadAt(i int64) (array.Record, error)

ReadAt reads the i-th record from the underlying stream and an error, if any.

func (*FileReader) Record

func (f *FileReader) Record(i int) (array.Record, error)

Record returns the i-th record from the file. The returned value is valid until the next call to Record. Users need to call Retain on that Record to keep it valid for longer.

func (*FileReader) RecordAt

func (f *FileReader) RecordAt(i int) (array.Record, error)

Record returns the i-th record from the file. Ownership is transferred to the caller and must call Release() to free the memory. This method is safe to call concurrently.

func (*FileReader) Schema

func (f *FileReader) Schema() *arrow.Schema

func (*FileReader) Version

func (f *FileReader) Version() MetadataVersion

type FileWriter

type FileWriter struct {
	// contains filtered or unexported fields
}

FileWriter is an Arrow file writer.

func NewFileWriter

func NewFileWriter(w io.WriteSeeker, opts ...Option) (*FileWriter, error)

NewFileWriter opens an Arrow file using the provided writer w.

func (*FileWriter) Close

func (f *FileWriter) Close() error

func (*FileWriter) Write

func (f *FileWriter) Write(rec array.Record) error

type Message

type Message struct {
	// contains filtered or unexported fields
}

Message is an IPC message, including metadata and body.

func NewMessage

func NewMessage(meta, body *memory.Buffer) *Message

NewMessage creates a new message from the metadata and body buffers. NewMessage panics if any of these buffers is nil.

func (*Message) BodyLen

func (msg *Message) BodyLen() int64

func (*Message) Release

func (msg *Message) Release()

Release decreases the reference count by 1. Release may be called simultaneously from multiple goroutines. When the reference count goes to zero, the memory is freed.

func (*Message) Retain

func (msg *Message) Retain()

Retain increases the reference count by 1. Retain may be called simultaneously from multiple goroutines.

func (*Message) Type

func (msg *Message) Type() MessageType

func (*Message) Version

func (msg *Message) Version() MetadataVersion

type MessageReader

type MessageReader interface {
	Message() (*Message, error)
	Release()
	Retain()
}

func NewMessageReader

func NewMessageReader(r io.Reader) MessageReader

NewMessageReader returns a reader that reads messages from an input stream.

type MessageType

type MessageType flatbuf.MessageHeader

MessageType represents the type of Message in an Arrow format.

func (MessageType) String

func (m MessageType) String() string

type MetadataVersion

type MetadataVersion flatbuf.MetadataVersion

MetadataVersion represents the Arrow metadata version.

func (MetadataVersion) String

func (m MetadataVersion) String() string

type Option

type Option func(*config)

Option is a functional option to configure opening or creating Arrow files and streams.

func WithAllocator

func WithAllocator(mem memory.Allocator) Option

WithAllocator specifies the Arrow memory allocator used while building records.

func WithCompressConcurrency

func WithCompressConcurrency(n int) Option

WithCompressConcurrency specifies a number of goroutines to spin up for concurrent compression of the body buffers when writing compress IPC records. If n <= 1 then compression will be done serially without goroutine parallelization. Default is 0.

func WithFooterOffset

func WithFooterOffset(offset int64) Option

WithFooterOffset specifies the Arrow footer position in bytes.

func WithLZ4

func WithLZ4() Option

WithLZ4 tells the writer to use LZ4 Frame compression on the data buffers before writing. Requires >= Arrow 1.0.0 to read/decompress

func WithSchema

func WithSchema(schema *arrow.Schema) Option

WithSchema specifies the Arrow schema to be used for reading or writing.

func WithZstd

func WithZstd() Option

WithZstd tells the writer to use ZSTD compression on the data buffers before writing. Requires >= Arrow 1.0.0 to read/decompress

type Payload

type Payload struct {
	// contains filtered or unexported fields
}

Payload is the underlying message object which is passed to the payload writer for actually writing out ipc messages

func (*Payload) Meta

func (p *Payload) Meta() *memory.Buffer

Meta returns the buffer containing the metadata for this payload, callers must call Release on the buffer

func (*Payload) Release

func (p *Payload) Release()

func (*Payload) SerializeBody

func (p *Payload) SerializeBody(w io.Writer) error

SerializeBody serializes the body buffers and writes them to the provided writer.

type PayloadWriter

type PayloadWriter interface {
	Start() error
	WritePayload(Payload) error
	Close() error
}

PayloadWriter is an interface for injecting a different payloadwriter allowing more reusability with the Writer object with other scenarios, such as with Flight data

type ReadAtSeeker

type ReadAtSeeker interface {
	io.Reader
	io.Seeker
	io.ReaderAt
}

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads records from an io.Reader. Reader expects a schema (plus any dictionaries) as the first messages in the stream, followed by records.

func NewReader

func NewReader(r io.Reader, opts ...Option) (*Reader, error)

NewReader returns a reader that reads records from an input stream.

func NewReaderFromMessageReader

func NewReaderFromMessageReader(r MessageReader, opts ...Option) (*Reader, error)

NewReaderFromMessageReader allows constructing a new reader object with the provided MessageReader allowing injection of reading messages other than by simple streaming bytes such as Arrow Flight which receives a protobuf message

func (*Reader) Err

func (r *Reader) Err() error

Err returns the last error encountered during the iteration over the underlying stream.

func (*Reader) Next

func (r *Reader) Next() bool

Next returns whether a Record could be extracted from the underlying stream.

func (*Reader) Read

func (r *Reader) Read() (array.Record, error)

Read reads the current record from the underlying stream and an error, if any. When the Reader reaches the end of the underlying stream, it returns (nil, io.EOF).

func (*Reader) Record

func (r *Reader) Record() array.Record

Record returns the current record that has been extracted from the underlying stream. It is valid until the next call to Next.

func (*Reader) Release

func (r *Reader) Release()

Release decreases the reference count by 1. When the reference count goes to zero, the memory is freed. Release may be called simultaneously from multiple goroutines.

func (*Reader) Retain

func (r *Reader) Retain()

Retain increases the reference count by 1. Retain may be called simultaneously from multiple goroutines.

func (*Reader) Schema

func (r *Reader) Schema() *arrow.Schema

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is an Arrow stream writer.

func NewWriter

func NewWriter(w io.Writer, opts ...Option) *Writer

NewWriter returns a writer that writes records to the provided output stream.

func NewWriterWithPayloadWriter

func NewWriterWithPayloadWriter(pw PayloadWriter, opts ...Option) *Writer

NewWriterWithPayloadWriter constructs a writer with the provided payload writer instead of the default stream payload writer. This makes the writer more reusable such as by the Arrow Flight writer.

func (*Writer) Close

func (w *Writer) Close() error

func (*Writer) Write

func (w *Writer) Write(rec array.Record) error

Directories

Path Synopsis
cmd
arrow-cat
Command arrow-cat displays the content of an Arrow stream or file.
Command arrow-cat displays the content of an Arrow stream or file.
arrow-ls
Command arrow-ls displays the listing of an Arrow file.
Command arrow-ls displays the listing of an Arrow file.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL