Package sliceio provides utilities for managing I/O for Bigslice operations.



    This section is empty.


    View Source
    var EOF = errors.New("EOF")

      EOF is the error returned by Reader.Read when no more data is available. EOF is intended as a sentinel error: it signals a graceful end of output. If output terminates unexpectedly, a different error should be returned.

      View Source
      var SpillBatchSize = defaultChunksize

        SpillBatchSize determines the amount of batching used in each spill file. A single read of a spill file produces this many rows. SpillBatchSize then trades off memory footprint for encoding size.


        func ReadAll

        func ReadAll(ctx context.Context, r Reader, columns ...interface{}) error

          ReadAll copies all elements from reader r into the provided column pointers. ReadAll is not tuned for performance and is intended for testing purposes.

          func ReadFull

          func ReadFull(ctx context.Context, r Reader, f frame.Frame) (n int, err error)

            ReadFull reads the full length of the frame. ReadFull reads short frames only on EOF.


            type ClosingReader

            type ClosingReader struct {
            	// contains filtered or unexported fields

              ClosingReader closes the wrapped ReadCloser when Read returns any error.

              func NewClosingReader

              func NewClosingReader(r ReadCloser) *ClosingReader

                NewClosingReader returns a new ClosingReader for r.

                func (*ClosingReader) Read

                func (c *ClosingReader) Read(ctx context.Context, out frame.Frame) (int, error)

                  Read implements sliceio.Reader.

                  type EmptyReader

                  type EmptyReader struct{}

                    EmptyReader returns an EOF.

                    func (EmptyReader) Read

                    func (EmptyReader) Read(ctx context.Context, f frame.Frame) (int, error)

                    type Encoder

                    type Encoder struct {
                    	// contains filtered or unexported fields

                      An Encoder manages transmission of slices through an underlying io.Writer. The stream of slice values represented by batches of rows stored in column-major order. Streams can be read by a Decoder.

                      func NewEncodingWriter

                      func NewEncodingWriter(w io.Writer) *Encoder

                        NewEncodingWriter returns a Writer that streams slices into the provided writer.

                        func (*Encoder) Write

                        func (e *Encoder) Write(_ context.Context, f frame.Frame) error

                          Encode encodes a batch of rows and writes the encoded output into the encoder's writer.

                          type PprofReader

                          type PprofReader struct {
                          	Label string

                            PprofReader executes Read in a labeled Context.

                            func (*PprofReader) Read

                            func (r *PprofReader) Read(ctx context.Context, frame frame.Frame) (n int, err error)

                            type ReadCloser

                            type ReadCloser interface {

                              ReadCloser groups the Read and Close methods.

                              func MultiReader

                              func MultiReader(readers ...ReadCloser) ReadCloser

                                MultiReader returns a ReadCloser that's the logical concatenation of the provided input readers. Once every underlying ReadCloser has returned EOF, Read will return EOF, too. Non-EOF errors are returned immediately.

                                func NopCloser

                                func NopCloser(r Reader) ReadCloser

                                type Reader

                                type Reader interface {
                                	// Read reads a vector of records from the underlying Slice. Each
                                	// passed-in column should be a value containing a slice of column
                                	// values. The number of columns should match the number of columns
                                	// in the slice; their types should match the corresponding column
                                	// types of the slice. Each column should have the same slice
                                	// length.
                                	// Read returns the total number of records read, or an error. When
                                	// no more records are available, Read returns EOF. Read may return
                                	// EOF when n > 0. In this case, n records were read, but no more
                                	// are available.
                                	// Read should never reuse any allocated memory in the frame;
                                	// its callers should not mutate the data returned.
                                	// Read should not be called concurrently.
                                	Read(ctx context.Context, frame frame.Frame) (int, error)

                                  A Reader represents a stateful stream of records. Each call to Read reads the next set of available records.

                                  func ErrReader

                                  func ErrReader(err error) Reader

                                    ErrReader returns a reader that returns the provided error on every call to read. ErrReader panics if err is nil.

                                    func FrameReader

                                    func FrameReader(frame frame.Frame) Reader

                                      FrameReader returns a Reader that reads the provided Frame to completion.

                                      func NewDecodingReader

                                      func NewDecodingReader(r io.Reader) Reader

                                        NewDecodingReader returns a new Reader that decodes values from the provided stream. Since values are streamed in vectors, decoding reader must buffer values until they are read by the consumer.

                                        type ReaderWithCloseFunc

                                        type ReaderWithCloseFunc struct {
                                        	CloseFunc func() error

                                          ReaderWithCloseFunc is a ReadCloser that wraps an existing Reader and uses a provided function for its Close.

                                          func (ReaderWithCloseFunc) Close

                                          func (r ReaderWithCloseFunc) Close() error

                                            Close implements io.Closer.

                                            type Scanner

                                            type Scanner struct {
                                            	// contains filtered or unexported fields

                                              A Scanner provides a convenient interface for reading records (e.g. from a Slice or a shard of a Slice). Successive calls to Scan (or Scanv) returns the next record (batch of records). Scanning stops when no more data are available or if an error is encountered. Scan returns true while it's safe to continue scanning. When scanning is complete, the user should inspect the scanner's error to see if scanning stopped because of an EOF or because another error occurred.

                                              Callers should not mix calls to Scan and Scanv.

                                              func NewScanner

                                              func NewScanner(typ slicetype.Type, r ReadCloser) *Scanner

                                                NewScanner returns a new scanner of records of type typ from reader r.

                                                func (*Scanner) Close

                                                func (s *Scanner) Close() error

                                                  Close releases resources used by the scanner. This must be called exactly once on the scanner returned by NewScanner.

                                                  func (*Scanner) Err

                                                  func (s *Scanner) Err() error

                                                    Err returns any error that occurred while scanning.

                                                    func (*Scanner) Scan

                                                    func (s *Scanner) Scan(ctx context.Context, out ...interface{}) bool

                                                      Scan the next record into the provided columns. Scanning fails if the columns do not match arity and type with the underlying data set. Scan returns true while no errors are encountered and there remains data to be scanned. Once Scan returns false, call Err to check for errors.

                                                      func (*Scanner) Scanv

                                                      func (s *Scanner) Scanv(ctx context.Context, out ...interface{}) (int, bool)

                                                        Scanv scans a batch of elements into the provided column vectors. Each column should be a slice of the correct type. Scanv fails when the type or arity of the column vectors do not match the underlying dataset. The number of records scanned is returned together with a boolean indicating whether scanning should continue, as in Scan. Once Scan returns false, call Err to check for errors.

                                                        type Spiller

                                                        type Spiller string

                                                          A Spiller manages a set of spill files.

                                                          func NewSpiller

                                                          func NewSpiller(name string) (Spiller, error)

                                                            NewSpiller creates and returns a new spiller backed by a temporary directory. Spillers do not guarantee that the order of spillers returned matches the order of spills.

                                                            func (Spiller) Cleanup

                                                            func (dir Spiller) Cleanup() error

                                                              Cleanup removes the spiller's temporary files. It is safe to call Cleanup after Readers(), but before reading is done.

                                                              func (Spiller) ClosingReaders

                                                              func (dir Spiller) ClosingReaders() ([]Reader, error)

                                                                ClosingReaders returns a reader for each spiller file. The readers close the underlying file when Read returns a non-nil error (otherwise the underlying file resource will leak).

                                                                func (Spiller) Readers

                                                                func (dir Spiller) Readers() ([]ReadCloser, error)

                                                                  Readers returns a ReadCloser for each spiller file.

                                                                  func (Spiller) Spill

                                                                  func (dir Spiller) Spill(frame frame.Frame) (int, error)

                                                                    Spill spills the provided frame to a new file in the spiller. Spill returns the file's encoded size, or an error. The frame is encoded in batches of SpillBatchSize.

                                                                    type Writer

                                                                    type Writer interface {
                                                                    	// Write writes f to an underlying data stream. It returns a non-nil error
                                                                    	// if there is a problem writing, and f may have been partially written.
                                                                    	Write(ctx context.Context, f frame.Frame) error

                                                                      Writer can write a frame to an underlying data stream.