Back to godoc.org
github.com/grailbio/bigslice/sliceio

package sliceio

v0.0.0 (8e22b3d)
Latest Go to latest
Published: May 28, 2020 | License: Apache-2.0 | Module: github.com/grailbio/bigslice

Overview

Package sliceio provides utilities for managing I/O for Bigslice operations.

Index

Variables

var EOF = errors.New("EOF")

EOF is the error returned by Reader.Read when no more data is available. EOF is intended as a sentinel error: it signals a graceful end of output. If output terminates unexpectedly, a different error should be returned.

var SpillBatchSize = defaultChunksize

SpillBatchSize determines the amount of batching used in each spill file. A single read of a spill file produces this many rows. SpillBatchSize then trades off memory footprint for encoding size.

func ReadAll

func ReadAll(ctx context.Context, r Reader, columns ...interface{}) error

ReadAll copies all elements from reader r into the provided column pointers. ReadAll is not tuned for performance and is intended for testing purposes.

func ReadFull

func ReadFull(ctx context.Context, r Reader, f frame.Frame) (n int, err error)

ReadFull reads the full length of the frame. ReadFull reads short frames only on EOF.

type ClosingReader

type ClosingReader struct {
	// contains filtered or unexported fields
}

ClosingReader closes the wrapped ReadCloser when Read returns any error.

func NewClosingReader

func NewClosingReader(r ReadCloser) *ClosingReader

NewClosingReader returns a new ClosingReader for r.

func (*ClosingReader) Read

func (c *ClosingReader) Read(ctx context.Context, out frame.Frame) (int, error)

Read implements sliceio.Reader.

type EmptyReader

type EmptyReader struct{}

EmptyReader returns an EOF.

func (EmptyReader) Read

func (EmptyReader) Read(ctx context.Context, f frame.Frame) (int, error)

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

An Encoder manages transmission of slices through an underlying io.Writer. The stream of slice values represented by batches of rows stored in column-major order. Streams can be read by a Decoder.

func NewEncodingWriter

func NewEncodingWriter(w io.Writer) *Encoder

NewEncodingWriter returns a Writer that streams slices into the provided writer.

func (*Encoder) Write

func (e *Encoder) Write(_ context.Context, f frame.Frame) error

Encode encodes a batch of rows and writes the encoded output into the encoder's writer.

type PprofReader

type PprofReader struct {
	Reader
	Label string
}

PprofReader executes Read in a labeled Context.

func (*PprofReader) Read

func (r *PprofReader) Read(ctx context.Context, frame frame.Frame) (n int, err error)

type ReadCloser

type ReadCloser interface {
	Reader
	io.Closer
}

ReadCloser groups the Read and Close methods.

func MultiReader

func MultiReader(readers ...ReadCloser) ReadCloser

MultiReader returns a ReadCloser that's the logical concatenation of the provided input readers. Once every underlying ReadCloser has returned EOF, Read will return EOF, too. Non-EOF errors are returned immediately.

func NopCloser

func NopCloser(r Reader) ReadCloser

type Reader

type Reader interface {
	// Read reads a vector of records from the underlying Slice. Each
	// passed-in column should be a value containing a slice of column
	// values. The number of columns should match the number of columns
	// in the slice; their types should match the corresponding column
	// types of the slice. Each column should have the same slice
	// length.
	//
	// Read returns the total number of records read, or an error. When
	// no more records are available, Read returns EOF. Read may return
	// EOF when n > 0. In this case, n records were read, but no more
	// are available.
	//
	// Read should never reuse any allocated memory in the frame;
	// its callers should not mutate the data returned.
	//
	// Read should not be called concurrently.
	Read(ctx context.Context, frame frame.Frame) (int, error)
}

A Reader represents a stateful stream of records. Each call to Read reads the next set of available records.

func ErrReader

func ErrReader(err error) Reader

ErrReader returns a reader that returns the provided error on every call to read. ErrReader panics if err is nil.

func FrameReader

func FrameReader(frame frame.Frame) Reader

FrameReader returns a Reader that reads the provided Frame to completion.

func NewDecodingReader

func NewDecodingReader(r io.Reader) Reader

NewDecodingReader returns a new Reader that decodes values from the provided stream. Since values are streamed in vectors, decoding reader must buffer values until they are read by the consumer.

type ReaderWithCloseFunc

type ReaderWithCloseFunc struct {
	Reader
	CloseFunc func() error
}

ReaderWithCloseFunc is a ReadCloser that wraps an existing Reader and uses a provided function for its Close.

func (ReaderWithCloseFunc) Close

func (r ReaderWithCloseFunc) Close() error

Close implements io.Closer.

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

A Scanner provides a convenient interface for reading records (e.g. from a Slice or a shard of a Slice). Successive calls to Scan (or Scanv) returns the next record (batch of records). Scanning stops when no more data are available or if an error is encountered. Scan returns true while it's safe to continue scanning. When scanning is complete, the user should inspect the scanner's error to see if scanning stopped because of an EOF or because another error occurred.

Callers should not mix calls to Scan and Scanv.

func NewScanner

func NewScanner(typ slicetype.Type, r ReadCloser) *Scanner

NewScanner returns a new scanner of records of type typ from reader r.

func (*Scanner) Close

func (s *Scanner) Close() error

Close releases resources used by the scanner. This must be called exactly once on the scanner returned by NewScanner.

func (*Scanner) Err

func (s *Scanner) Err() error

Err returns any error that occurred while scanning.

func (*Scanner) Scan

func (s *Scanner) Scan(ctx context.Context, out ...interface{}) bool

Scan the next record into the provided columns. Scanning fails if the columns do not match arity and type with the underlying data set. Scan returns true while no errors are encountered and there remains data to be scanned.

func (*Scanner) Scanv

func (s *Scanner) Scanv(ctx context.Context, out ...interface{}) (int, bool)

Scanv scans a batch of elements into the provided column vectors. Each column should be a slice of the correct type. Scanv fails when the type or arity of the column vectors do not match the underlying dataset. The number of records scanned is returned together with a boolean indicating whether scanning should continue, as in Scan.

type Spiller

type Spiller string

A Spiller manages a set of spill files.

func NewSpiller

func NewSpiller(name string) (Spiller, error)

NewSpiller creates and returns a new spiller backed by a temporary directory. Spillers do not guarantee that the order of spillers returned matches the order of spills.

func (Spiller) Cleanup

func (dir Spiller) Cleanup() error

Cleanup removes the spiller's temporary files. It is safe to call Cleanup after Readers(), but before reading is done.

func (Spiller) ClosingReaders

func (dir Spiller) ClosingReaders() ([]Reader, error)

ClosingReaders returns a reader for each spiller file. The readers close the underlying file when Read returns a non-nil error (otherwise the underlying file resource will leak).

func (Spiller) Readers

func (dir Spiller) Readers() ([]ReadCloser, error)

Readers returns a ReadCloser for each spiller file.

func (Spiller) Spill

func (dir Spiller) Spill(frame frame.Frame) (int, error)

Spill spills the provided frame to a new file in the spiller. Spill returns the file's encoded size, or an error. The frame is encoded in batches of SpillBatchSize.

type Writer

type Writer interface {
	// Write writes f to an underlying data stream. It returns a non-nil error
	// if there is a problem writing, and f may have been partially written.
	Write(ctx context.Context, f frame.Frame) error
}

Writer can write a frame to an underlying data stream.

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
f or F : Jump to identifier