idbatcher

Documentation

Overview

Package idbatcher defines a pipeline of fixed size in which the elements are batches of ids.

Index

• Variables
• type Batch
• type Batcher
  • func New(numBatches, newBatchesInitialCapacity, batchChannelSize uint64) (Batcher, error)
• type ID

Variables

var (
	// ErrInvalidNumBatches occurs when an invalid number of batches is specified.
	ErrInvalidNumBatches = errors.New("invalid number of batches, it must be greater than zero")
	// ErrInvalidBatchChannelSize occurs when an invalid batch channel size is specified.
	ErrInvalidBatchChannelSize = errors.New("invalid batch channel size, it must be greater than zero")
)

Types

type Batch

type Batch []ID

Batch is the type of batches held by the Batcher.

type Batcher

type Batcher interface {
	// AddToCurrentBatch puts the given id on the batch being currently built. The client is in charge
	// of limiting the growth of the current batch if appropriate for its scenario. It can
	// either call CloseCurrentAndTakeFirstBatch earlier or stop adding new items depending on what is
	// required by the scenario.
	AddToCurrentBatch(id ID)
	// CloseCurrentAndTakeFirstBatch takes the batch at the front of the pipe, and moves the current
	// batch to the end of the pipe, creating a new batch to receive new items. This operation should
	// be atomic.
	// It returns the batch that was in front of the pipe and a boolean that if true indicates that
	// there are more batches to be retrieved.
	CloseCurrentAndTakeFirstBatch() (Batch, bool)
	// Stop informs that no more items are going to be batched and the pipeline can be read until it
	// is empty. After this method is called attempts to enqueue new items will panic.
	Stop()
}

Batcher behaves like a pipeline of batches that has a fixed number of batches in the pipe and a new batch being built outside of the pipe. Items can be concurrently added to the batch currently being built. When the batch being built is closed, the oldest batch in the pipe is pushed out so the one just closed can be put on the end of the pipe (this is done as an atomic operation). The caller is in control of when a batch is completed and a new one should be started.

func New

func New(numBatches, newBatchesInitialCapacity, batchChannelSize uint64) (Batcher, error)

New creates a Batcher that will hold numBatches in its pipeline, having a channel with batchChannelSize to receive new items. New batches will be created with capacity set to newBatchesInitialCapacity.

type ID

type ID []byte

ID is the type of each element in the batch.