sortio

package
Version: v0.0.0-...-a873046 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 5, 2021 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package sortio provides facilities for sorting slice outputs and merging and reducing sorted record streams.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewMergeReader

func NewMergeReader(ctx context.Context, typ slicetype.Type, readers []sliceio.Reader) (sliceio.Reader, error)

NewMergeReader returns a new Reader that is sorted by its prefix columns. The readers to be merged must already be sorted.

func Reduce

func Reduce(typ slicetype.Type, name string, readers []sliceio.Reader, combiner slicefunc.Func) sliceio.Reader

Reduce returns a Reader that merges and reduces a set of sorted (and possibly combined) readers. Reduce panics if the provided type is not reducable.

func SortReader

func SortReader(ctx context.Context, spillTarget int, typ slicetype.Type, r sliceio.Reader) (sliceio.Reader, error)

SortReader sorts a Reader by its prefix columns. SortReader may spill to disk, in which case it targets spill file sizes of spillTarget (in bytes). Because the encoded size of objects is not known in advance, sortReader uses a "canary" batch size of ~16k rows in order to estimate the size of future reads. The estimate is revisited on every subsequent fill and adjusted if it is violated by more than 5%.

Types

type FrameBuffer

type FrameBuffer struct {
	// Frame is the buffer into which new data are read. The buffer is
	// always allocated externally and must be nonempty.
	frame.Frame
	// Reader is the slice reader from which the buffer is filled.
	sliceio.Reader
	// Index, Len is current index and length of the frame.
	Index, Len int

	// Off is the global offset of this framebuffer. It is added to
	// the index to compute the buffer's current position.
	Off int
}

A FrameBuffer is a buffered frame. The frame is filled from a reader, and maintains a current index and length.

func (*FrameBuffer) Fill

func (f *FrameBuffer) Fill(ctx context.Context) error

Fill (re-) fills the FrameBuffer when it's empty. An error is returned if the underlying reader returns an error. EOF is returned if no more data are available.

func (FrameBuffer) Pos

func (f FrameBuffer) Pos() int

Pos returns this frame buffer's current position.

type FrameBufferHeap

type FrameBufferHeap struct {
	Buffers []*FrameBuffer
	// Less compares the current index of buffers i and j.
	LessFunc func(i, j int) bool
}

FrameBufferHeap implements a heap of FrameBuffers, ordered by the provided sorter.

func (*FrameBufferHeap) Len

func (f *FrameBufferHeap) Len() int

func (*FrameBufferHeap) Less

func (f *FrameBufferHeap) Less(i, j int) bool

func (*FrameBufferHeap) Pop

func (f *FrameBufferHeap) Pop() interface{}

Pop removes the FrameBuffer with the smallest priority from the heap.

func (*FrameBufferHeap) Push

func (f *FrameBufferHeap) Push(x interface{})

Push pushes a FrameBuffer onto the heap.

func (*FrameBufferHeap) Swap

func (f *FrameBufferHeap) Swap(i, j int)

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto