iter

Documentation

Overview

Generic forward-only iterator that is safe and leak-free.

This package is intended to support forward-only iteration in a variety of use cases while avoiding the normal errors and leaks that can happen with iterators in Go. It provides mechanisms for map/select filtering, background iteration through a goroutine, and error handling throughout.

The type of the iterator is interface{}, so it can store anything, at the cost that you have to cast it back out when you use it. This package can be used as is, or used as an example for creating your own forward-only iterators of more specific types.

sum := 0
iterator.Each(func(item interface{}) {
  sum = sum + item.(int)
})

Motivation

With the lack of generics and a builtin iterator pattern, iterators have been the topic of much discussion in Go. Here are the discussions that inspired this:

http://ewencp.org/blog/golang-iterators/: Ewan Cheslack-Postava's discussion of the major iteration patterns. Herein we have chosen the closure pattern for iterator implementation, and given the choice of callback and channel patterns for iteration callers.

http://blog.golang.org/pipelines: A March 2014 discussion of pipelines on the go blog presents some of the pitfalls of channel iteration, suggesting the "done" channel implementation to compensate.

Creating Iterators

Simple error- and cleanup-free iterators can be easily created:

// Create a simple iterator from a function
val := 1
iterator := iter.NewSimple(func() interface{} {
  val = val * 2;
  return val > 2000 ? val : nil // nil terminates iteration
})

Typically you will create iterators in packages ("please iterate over this complicated thing"). You will often handle errors and have cleanup to do. iter supports both of these. You can create a fully-functional iterator thusly:

// Create a normal iterator parsing a file, close when done
func ParseStream(reader io.ReadCloser) iter.Iterator {
  return iter.Iterator{
    Next: func() (iterator{}, error) {
      item, err := Parse()
      if item == nil && err == nil {
        return nil, iter.FINISHED
      }
      return item, err
    },
    Close: func() {
      reader.Close()
    }
  }
}

Iterating

Callback iteration looks like this and handles any bookkeeping automatically:

// Iterate over all values
err := iterator.Each(func(item interface{}) {
  fmt.Println(item)
})

Sometimes you need to handle errors:

// Iterate over all values, terminating if processing has a problem
var files []*File
err := iterator.EachWithError(func(item interface{}) error {
  file, err = os.Open(item.(string))
  if err == nil {
    files = append(files, file)
  }
  return err
})

Raw iteration looks like this:

defer iterator.Close() // allow the iterator to clean itself up
item, err := iterator.Next()
for err == nil {
  ... // do stuff with value
  item, err = iterator.Next()
}
if err != iter.FINISHED {
  ... // handle error
}

Background goroutine iteration (using channels) deserves special mention:

// Produce the values in a goroutine, cleaning up safely afterwards.
// This allows background iteration to continue at its own pace while we
// perform blocking operations in the foreground.
var responses []http.Response
err := iterator.BackgroundEach(1000, func(item interface{}) error {
  response, err := http.Get(item.(string))
  if err == nil {
    responses = append(list, response)
  }
  return err
})

Utilities

There are several useful functions provided to work with iterators:

// Square the ints
squaredIterator, err := iterator.Map(func(item interface{}) interface{} { item.int() * 2 })

// Select non-nil values
nonNilIterator, err := iterator.Select(func(item interface{}) bool) { item != nil })

// Produce a list
list, err := iterator.ToList()

Index

• Variables
• func EachFromChannel(items <-chan ChannelItem, done chan<- bool, processor func(interface{}) error) error
• type ChannelItem
• type Iterator
  • func Concat(iterators ...Iterator) Iterator
  • func NewSimple(f func() interface{}) Iterator
  • func (iterator Iterator) BackgroundEach(bufferSize int, processor func(interface{}) error) error
  • func (iterator Iterator) Each(processor func(interface{})) error
  • func (iterator Iterator) EachWithError(processor func(interface{}) error) error
  • func (iterator Iterator) Go() (items <-chan ChannelItem, done chan<- bool)
  • func (iterator Iterator) GoSimple() (values <-chan interface{})
  • func (iterator Iterator) IterateToChannel(items chan<- ChannelItem, done <-chan bool)
  • func (iterator Iterator) IterateToChannelSimple(values chan<- interface{})
  • func (iterator Iterator) Map(mapper func(interface{}) interface{}) Iterator
  • func (iterator Iterator) Select(selector func(interface{}) bool) Iterator
  • func (iterator Iterator) ToSlice() (list []interface{}, err error)

Variables

var FINISHED = errors.New("FINISHED")

Error returned from Iterator.Next() when iteration is done.

Functions

func EachFromChannel

func EachFromChannel(items <-chan ChannelItem, done chan<- bool, processor func(interface{}) error) error

Iterate over the channels from a Go(), calling a user-defined function for each value. This function handles all anomalous conditions including errors, early termination and safe cleanup of the goroutine and channels.

Types

type ChannelItem

type ChannelItem struct {
	Value interface{}
	Error error
}

A result from a iterator.Go() channel.

type Iterator

type Iterator struct {
	// Get the next value (or error).
	// nil, iter.FINISHED indicates iteration is complete.
	Next func() (interface{}, error)
	// Close out resources in case iteration terminates early. Callers are *not*
	// required to call this if iteration completes cleanly (but they may).
	Close func()
}

Iterator letting you specify just two functions defining iteration. All helper functions utilize this.

Iterator is designed to avoid common pitfalls and to allow common Go patterns: - errors can be yielded all the way up the chain to the caller - safe iteration in background channels - iterators are nil-clean (you can send nil values through the iterator) - allows for cleanup when iteration ends early

func Concat

func Concat(iterators ...Iterator) Iterator

Concatenate multiple iterators together in one.

d := iter.Concat(a, b, c)

func NewSimple

func NewSimple(f func() interface{}) Iterator

Use to create an iterator from a single closure. retun nil to stop iteration.

iter.NewSimple(func() interface{} { return x*2 })

func (Iterator) BackgroundEach

func (iterator Iterator) BackgroundEach(bufferSize int, processor func(interface{}) error) error

Perform the iteration in the background concurrently with the Each() statement. Useful when the iterator or iteratee will be doing blocking work.

The bufferSize parameter lets you set how far ahead the background goroutine can get.

iterator.BackgroundEach(100, func(item interface{}) { ... })

func (Iterator) Each

func (iterator Iterator) Each(processor func(interface{})) error

Iterate over all values, calling a user-defined function for each one.

iterator.Each(func(item interface{}) { fmt.Println(item) })

func (Iterator) EachWithError

func (iterator Iterator) EachWithError(processor func(interface{}) error) error

Iterate over all values, calling a user-defined function for each one. If an error is returned from the function, iteration terminates early and the EachWithError function returns the error.

iterator.EachWithError(func(item interface{}) error { return http.Get(item.(string)) })

func (Iterator) Go

func (iterator Iterator) Go() (items <-chan ChannelItem, done chan<- bool)

Runs the iterator in a goroutine, sending to a channel.

The returned items channel will send ChannelItems to you, which can indicate either a value or an error. If item.Error is set, iteration is ready to terminate. Otherwise item.Value is the next value of the iteration.

When you are done iterating, you must close the done channel. Even if iteration was terminated early, this will ensure that the goroutine and channel are properly cleaned up.

channel, done := iter.Go(iterator)
defer close(done) // if early return or panic happens, this will clean up the goroutine
for item := range channel {
  if item.Error != nil {
    // Iteration failed; handle the error and exit the loop
    ...
  }
  value := item.Value
  ...
}

func (Iterator) GoSimple

func (iterator Iterator) GoSimple() (values <-chan interface{})

Iterate to a channel in the background.

for value := range iter.GoSimple(iterator) {
  ...
}

With this method, two undesirable things can happen:

• if the iteration stops early due to an error, you will not be able to handle it (the goroutine will log and panic, and the program will exit).
• if callers panic or exit early without retrieving all values from the channel, the goroutine is blocked forever and leaks.

The Go() routine allows you to handle both of these issues, at a small cost to caller complexity. BackgroundEach() provides a simple way to use Go(), as well.

That said, if you can make guarantees about no panics or don't care, this method can make calling code easier to read.

func (Iterator) IterateToChannel

func (iterator Iterator) IterateToChannel(items chan<- ChannelItem, done <-chan bool)

Iterates all items and sends them to the given channel. Runs on the current goroutine (call go iterator.IterateToChannel to set it up on a new goroutine). This will close the items channel when done. If the done channel is closed, iteration will terminate.

func (Iterator) IterateToChannelSimple

func (iterator Iterator) IterateToChannelSimple(values chan<- interface{})

Iterates all items and sends them to the given channel. Runs on the current goroutine (call go iterator.IterateToChannelSimple() to set it up on a new goroutine). This will close the values channel when done. See warnings about GoSimple() vs. Go() in the GoSimple() method.

func (Iterator) Map

func (iterator Iterator) Map(mapper func(interface{}) interface{}) Iterator

Transform values in the input.

iterator.Map(func(item interface{}) interface{} { item.(int) * 2 })

func (Iterator) Select

func (iterator Iterator) Select(selector func(interface{}) bool) Iterator

Filter values from the input.

iterator.Select(func(item interface{}) bool { item.(int) > 10 })

func (Iterator) ToSlice

func (iterator Iterator) ToSlice() (list []interface{}, err error)

Convert the iteration directly to a slice.

var list []interface{}
list, err := iterator.ToSlice()