Documentation
¶
Overview ¶
Package sliceio provides utilities for managing I/O for Bigslice operations.
Index ¶
- Variables
- func ReadAll(ctx context.Context, r Reader, columns ...interface{}) error
- func ReadFull(ctx context.Context, r Reader, f frame.Frame) (n int, err error)
- type ClosingReader
- type EmptyReader
- type Encoder
- type PprofReader
- type ReadCloser
- type Reader
- type ReaderWithCloseFunc
- type Scanner
- type Spiller
- type Writer
Constants ¶
This section is empty.
Variables ¶
var EOF = errors.New("EOF")
EOF is the error returned by Reader.Read when no more data is available. EOF is intended as a sentinel error: it signals a graceful end of output. If output terminates unexpectedly, a different error should be returned.
var SpillBatchSize = defaultChunksize
SpillBatchSize determines the amount of batching used in each spill file. A single read of a spill file produces this many rows. SpillBatchSize then trades off memory footprint for encoding size.
Functions ¶
Types ¶
type ClosingReader ¶
type ClosingReader struct {
// contains filtered or unexported fields
}
ClosingReader closes the wrapped ReadCloser when Read returns any error.
func NewClosingReader ¶
func NewClosingReader(r ReadCloser) *ClosingReader
NewClosingReader returns a new ClosingReader for r.
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
An Encoder manages transmission of slices through an underlying io.Writer. The stream of slice values represented by batches of rows stored in column-major order. Streams can be read by a Decoder.
func NewEncodingWriter ¶
NewEncodingWriter returns a Writer that streams slices into the provided writer.
type PprofReader ¶
PprofReader executes Read in a labeled Context.
type ReadCloser ¶
ReadCloser groups the Read and Close methods.
func MultiReader ¶
func MultiReader(readers ...ReadCloser) ReadCloser
MultiReader returns a ReadCloser that's the logical concatenation of the provided input readers. Once every underlying ReadCloser has returned EOF, Read will return EOF, too. Non-EOF errors are returned immediately.
func NopCloser ¶
func NopCloser(r Reader) ReadCloser
type Reader ¶
type Reader interface {
// Read reads a vector of records from the underlying Slice. Each
// passed-in column should be a value containing a slice of column
// values. The number of columns should match the number of columns
// in the slice; their types should match the corresponding column
// types of the slice. Each column should have the same slice
// length.
//
// Read returns the total number of records read, or an error. When
// no more records are available, Read returns EOF. Read may return
// EOF when n > 0. In this case, n records were read, but no more
// are available.
//
// Read should never reuse any allocated memory in the frame;
// its callers should not mutate the data returned.
//
// Read should not be called concurrently.
Read(ctx context.Context, frame frame.Frame) (int, error)
}
A Reader represents a stateful stream of records. Each call to Read reads the next set of available records.
func ErrReader ¶
ErrReader returns a reader that returns the provided error on every call to read. ErrReader panics if err is nil.
func FrameReader ¶
FrameReader returns a Reader that reads the provided Frame to completion.
func NewDecodingReader ¶
NewDecodingReader returns a new Reader that decodes values from the provided stream. Since values are streamed in vectors, decoding reader must buffer values until they are read by the consumer.
type ReaderWithCloseFunc ¶
ReaderWithCloseFunc is a ReadCloser that wraps an existing Reader and uses a provided function for its Close.
func (ReaderWithCloseFunc) Close ¶
func (r ReaderWithCloseFunc) Close() error
Close implements io.Closer.
type Scanner ¶
type Scanner struct {
// contains filtered or unexported fields
}
A Scanner provides a convenient interface for reading records (e.g. from a Slice or a shard of a Slice). Successive calls to Scan (or Scanv) returns the next record (batch of records). Scanning stops when no more data are available or if an error is encountered. Scan returns true while it's safe to continue scanning. When scanning is complete, the user should inspect the scanner's error to see if scanning stopped because of an EOF or because another error occurred.
Callers should not mix calls to Scan and Scanv.
func NewScanner ¶
func NewScanner(typ slicetype.Type, r ReadCloser) *Scanner
NewScanner returns a new scanner of records of type typ from reader r.
func (*Scanner) Close ¶
Close releases resources used by the scanner. This must be called exactly once on the scanner returned by NewScanner.
func (*Scanner) Scan ¶
Scan the next record into the provided columns. Scanning fails if the columns do not match arity and type with the underlying data set. Scan returns true while no errors are encountered and there remains data to be scanned. Once Scan returns false, call Err to check for errors.
func (*Scanner) Scanv ¶
Scanv scans a batch of elements into the provided column vectors. Each column should be a slice of the correct type. Scanv fails when the type or arity of the column vectors do not match the underlying dataset. The number of records scanned is returned together with a boolean indicating whether scanning should continue, as in Scan. Once Scan returns false, call Err to check for errors.
type Spiller ¶
type Spiller string
A Spiller manages a set of spill files.
func NewSpiller ¶
NewSpiller creates and returns a new spiller backed by a temporary directory. Spillers do not guarantee that the order of spillers returned matches the order of spills.
func (Spiller) Cleanup ¶
Cleanup removes the spiller's temporary files. It is safe to call Cleanup after Readers(), but before reading is done.
func (Spiller) ClosingReaders ¶
ClosingReaders returns a reader for each spiller file. The readers close the underlying file when Read returns a non-nil error (otherwise the underlying file resource will leak).
func (Spiller) Readers ¶
func (dir Spiller) Readers() ([]ReadCloser, error)
Readers returns a ReadCloser for each spiller file.
Source Files