iterator

Documentation

Index

• func Reduce[T any, V any](ctx context.Context, it Iterator[T], reducer func(V, T) (V, error)) (ret V, err error)
• func ToSlice[T any](ctx context.Context, it Iterator[T]) (ret []T, err error)
• type Iterator
  • func Aggregate[T any](its ...Iterator[T]) Iterator[T]
  • func Empty[T any]() Iterator[T]
  • func Error[T any](err error) Iterator[T]
  • func Filter[T any](it Iterator[T], fn func(T) bool) Iterator[T]
  • func FromDocumentIterator[T any](it storageapi.Iterator) Iterator[T]
  • func FromFunc[T any](fn func(context.Context) (T, bool, error), close func() error) Iterator[T]
  • func FromSlice[T any](els []T) Iterator[T]
  • func IsEmpty[T any](ctx context.Context, i Iterator[T]) (Iterator[T], bool)
  • func Map[T any, V any](it Iterator[T], fn func(T) V) Iterator[V]
  • func Unslice[T any](it Iterator[[]T]) Iterator[T]

Functions

func Reduce

func Reduce[T any, V any](
	ctx context.Context,
	it Iterator[T],
	reducer func(V, T) (V, error),
) (ret V, err error)

Reduce combines all the elements in an iterator using a binary operation to produce a single value.

Reduce is a terminal operation: it always closes the input iterator before returning, so callers must not close it themselves. Close errors are joined with any reducer or iteration error using multierror so callers do not silently lose failures from the underlying resource (goroutines, file descriptors, gRPC streams).

See the Iterator type for the wrapper/terminal contract that governs which helpers own the Close call.

func ToSlice

func ToSlice[T any](ctx context.Context, it Iterator[T]) (ret []T, err error)

ToSlice consumes the given iterator and returns a slice with all of the elements produced.

ToSlice is a terminal operation: it always closes the input iterator before returning, so callers must not close it themselves. Close errors are joined with any iteration error using multierror.

See the Iterator type for the wrapper/terminal contract that governs which helpers own the Close call.

Types

type Iterator

type Iterator[T any] interface {
	// Next returns the next element and true or nil and false
	// if there's no more elements in this Iterator. Err should be
	// checked for any errors incurred during the lifecycle of this Iterator.
	// Note that if an error is found, it's up to the implementation as to
	// whether to return false and stop iteration or aggregate errors
	// and return at the end.
	//
	// This method blocks until data is available. Implementations should
	// use the given context's Done channel to know when data is no longer
	// required and so the call should return.
	Next(context.Context) (T, bool)
	// Err returns the first error or an aggreation of the errors
	// encountered by the Iterator.
	Err() error

	io.Closer
}

Iterator provides a convenient interface for iterating over chunks of structured or unstructured data such as a file of newline-delimited lines of text or a set of datastore documents.

Ownership and Close

Iterator extends io.Closer because implementations commonly own resources that outlive a single Next call: goroutines feeding an unbuffered channel, open files, gRPC streams, network connections. Forgetting to Close such an iterator leaks those resources. Helpers in this package follow a single convention so that ownership is always unambiguous at the call site:

• Wrappers (Map, Filter, Aggregate, Unslice, IsEmpty) return a new Iterator whose Close delegates to the inner iterator's Close. The caller still owns the returned Iterator and must Close it.

• Terminal helpers (Reduce, ToSlice) consume the iterator to completion (or to context cancellation / first error) and return a non-iterator value. They always Close the input iterator before returning, joining any Close error with the returned error using multierror. Callers must not Close the iterator themselves after handing it to a terminal helper.

When implementing Iterator, Close should be safe to call multiple times: Aggregate calls a child's Close as that child is drained, and defensive callers may also defer Close at higher levels even when handing the iterator to a terminal helper.

func Aggregate

func Aggregate[T any](its ...Iterator[T]) Iterator[T]

Aggregate combines multiple iterators of T into one single iterator of T. If its is nil or empty, this method safely returns an empty iterator. The returned iterator is a wrapper: its Close delegates to each input iterator's Close, and inputs already drained during iteration are also closed at that point. The caller is responsible for closing the returned iterator. See the Iterator type for the wrapper/terminal contract.

func Empty

func Empty[T any]() Iterator[T]

Empty returns an empty iterator.

func Error

func Error[T any](err error) Iterator[T]

Error returns an Iterator of T that never returns a value and always returns the given error.

func Filter

func Filter[T any](it Iterator[T], fn func(T) bool) Iterator[T]

Filter wraps an iterator of type T and returns another iterator that applies fn to each of the elements produced and filters out any elements for which fn returns false. The returned iterator is a wrapper: its Close delegates to it. The caller is responsible for closing the returned iterator. See the Iterator type for the wrapper/terminal contract.

func FromDocumentIterator

func FromDocumentIterator[T any](it storageapi.Iterator) Iterator[T]

FromDocumentIterator maps a storageapi.Iterator to an Iterator of type T.

func FromFunc

func FromFunc[T any](
	fn func(context.Context) (T, bool, error),
	close func() error,
) Iterator[T]

FromFunc returns an Iterator backed by the provided function.

func FromSlice

func FromSlice[T any](els []T) Iterator[T]

FromSlice returns an iterator backed by the given slice of elements.

func IsEmpty

func IsEmpty[T any](ctx context.Context, i Iterator[T]) (Iterator[T], bool)

IsEmpty consumes the first element in i and returns true if it is empty or false if not and returns a new iterator that should be used instead of i.

The given iterator is consumed in either case, so its Close method is wrapped with the returned iterator, or if the given iterator is empty, its Close method is called for the caller, so it's safe to override the variable holding the passed iterator with the return value of this function.

func Map

func Map[T any, V any](it Iterator[T], fn func(T) V) Iterator[V]

Map wraps an iterator of type T and returns another iterator that applies fn to each of the elements produced. The returned iterator is a wrapper: its Close delegates to it. The caller is responsible for closing the returned iterator. See the Iterator type for the wrapper/terminal contract.

func Unslice

func Unslice[T any](it Iterator[[]T]) Iterator[T]

Unslice converts an iterator of slices of T into an iterator of T. The returned iterator is a wrapper: its Close delegates to it. The caller is responsible for closing the returned iterator. See the Iterator type for the wrapper/terminal contract.