Documentation

Overview

    Package sortio provides facilities for sorting slice outputs and merging and reducing sorted record streams.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    func NewMergeReader

    func NewMergeReader(ctx context.Context, typ slicetype.Type, readers []sliceio.Reader) (sliceio.Reader, error)

      NewMergeReader returns a new Reader that is sorted by its prefix columns. The readers to be merged must already be sorted.

      func Reduce

      func Reduce(typ slicetype.Type, name string, readers []sliceio.Reader, combiner slicefunc.Func) sliceio.Reader

        Reduce returns a Reader that merges and reduces a set of sorted (and possibly combined) readers. Reduce panics if the provided type is not reducable.

        func SortReader

        func SortReader(ctx context.Context, spillTarget int, typ slicetype.Type, r sliceio.Reader) (sliceio.Reader, error)

          SortReader sorts a Reader by its prefix columns. SortReader may spill to disk, in which case it targets spill file sizes of spillTarget (in bytes). Because the encoded size of objects is not known in advance, sortReader uses a "canary" batch size of ~16k rows in order to estimate the size of future reads. The estimate is revisited on every subsequent fill and adjusted if it is violated by more than 5%.

          Types

          type FrameBuffer

          type FrameBuffer struct {
          	// Frame is the buffer into which new data are read. The buffer is
          	// always allocated externally and must be nonempty.
          	frame.Frame
          	// Reader is the slice reader from which the buffer is filled.
          	sliceio.Reader
          	// Index, Len is current index and length of the frame.
          	Index, Len int
          
          	// Off is the global offset of this framebuffer. It is added to
          	// the index to compute the buffer's current position.
          	Off int
          }

            A FrameBuffer is a buffered frame. The frame is filled from a reader, and maintains a current index and length.

            func (*FrameBuffer) Fill

            func (f *FrameBuffer) Fill(ctx context.Context) error

              Fill (re-) fills the FrameBuffer when it's empty. An error is returned if the underlying reader returns an error. EOF is returned if no more data are available.

              func (FrameBuffer) Pos

              func (f FrameBuffer) Pos() int

                Pos returns this frame buffer's current position.

                type FrameBufferHeap

                type FrameBufferHeap struct {
                	Buffers []*FrameBuffer
                	// Less compares the current index of buffers i and j.
                	LessFunc func(i, j int) bool
                }

                  FrameBufferHeap implements a heap of FrameBuffers, ordered by the provided sorter.

                  func (*FrameBufferHeap) Len

                  func (f *FrameBufferHeap) Len() int

                  func (*FrameBufferHeap) Less

                  func (f *FrameBufferHeap) Less(i, j int) bool

                  func (*FrameBufferHeap) Pop

                  func (f *FrameBufferHeap) Pop() interface{}

                    Pop removes the FrameBuffer with the smallest priority from the heap.

                    func (*FrameBufferHeap) Push

                    func (f *FrameBufferHeap) Push(x interface{})

                      Push pushes a FrameBuffer onto the heap.

                      func (*FrameBufferHeap) Swap

                      func (f *FrameBufferHeap) Swap(i, j int)

                      Source Files