ioseq

README

Proof-of-concept implementation of io.Seq

This module holds experimental code implementing the functions described in this Go proposal.

See the package documentation for API details.

Documentation

Index

• Variables
• func CopySeq(w io.Writer, r Seq) (int64, error)
• func PipeThrough[W io.WriteCloser](r io.Reader, f func(io.Writer) W, bufSize int) io.ReadCloser
• func ReaderFromSeq(seq Seq) io.ReadCloser
• func ReaderWithContent(generate func(w io.Writer) error) io.ReadCloser
• func SeqWriter(yield func([]byte, error) bool, active *bool) io.Writer
• type Seq
  • func PipeSeqThrough[W io.WriteCloser](seq Seq, f func(w io.Writer) W) Seq
  • func SeqFromReader(r io.Reader, bufSize int) Seq

Examples

• PipeSeqThrough
• ReaderFromSeq

Variables

var ErrSequenceTerminated = errors.New("sequence terminated")

Functions

func CopySeq

func CopySeq(w io.Writer, r Seq) (int64, error)

CopySeq is like io.Copy but reads over r writing all the data to w. It returns the total number of bytes read.

func PipeThrough

func PipeThrough[W io.WriteCloser](r io.Reader, f func(io.Writer) W, bufSize int) io.ReadCloser

PipeThrough calls f; all data written by f to its argument writer will be made available on the returned ReadCloser; all data read from f will be written to the writer implementation returned by f.

In other words, it returns a reader that "pipes" the content from r through f.

func ReaderFromSeq

func ReaderFromSeq(seq Seq) io.ReadCloser

ReaderFromSeq converts an iterator into an io.ReadCloser. Close must be called after the caller is done with the reader.

Example

package main

import (
	"encoding/base64"
	"fmt"
	"io"
	"os"

	"github.com/rogpeppe/ioseq"
)

func main() {
	// The base64.NewEncoder API returns a io.Writer,
	// making it inconvenient to use on APIs that require a Reader.
	// Demonstrate how we'd use ReaderFromSeq to work around
	// that limitation without using io.Pipe.
	seq := func(yield func([]byte, error) bool) {
		w := base64.NewEncoder(base64.StdEncoding, ioseq.SeqWriter(yield, nil))
		defer w.Close()
		fmt.Fprintf(w, "hello, world\n")
	}
	r := ioseq.ReaderFromSeq(seq)
	defer r.Close()
	io.Copy(os.Stdout, r)
	r.Close()

}

Output:

aGVsbG8sIHdvcmxkCg==

func ReaderWithContent

func ReaderWithContent(generate func(w io.Writer) error) io.ReadCloser

ReaderWithContent returns a [Reader] that calls the given function to generate the data to be read. If the function returns an error, that error will be returned from the reader.

func SeqWriter

func SeqWriter(yield func([]byte, error) bool, active *bool) io.Writer

SeqWriter returns a [Writer] that operates on the yield function passed into a Seq iterator. Writes will succeed until the iteration is terminated, upon which Write will return ErrSequenceTerminated.

The returned Writer should not be used outside the scope of the iterator function, following the same rules as any yield function.

If active is non-nil, it reflects the "active" status of the generator and its value should be (but does not have to be) true initially. yield will not be called when *active is false. If yield returns false, *active will be set to false.

The caller can use the value of *active to find out whether the iterator is still active.

Types

type Seq

type Seq = iter.Seq2[[]byte, error]

Seq represents a sequence of byte slices. It's somewhat equivalent to [Reader], although simpler in some respects. See SeqFromReader and ReaderFromSeq for a way to convert between Seq and [Reader].

Each element in the sequence must have either a non-nil byte slice or a non-nil error; a producer should never produce either (nil, nil) or a non-nil slice and a non-nil error.

The sequence always ends at the first error: if there are temporary errors, it's up to the producer to deal with them.

The code ranging over the sequence must not use the slice outside of the loop or across iterations; that is, the receiver owns a slice until that particular iteration ends.

Callers must not mutate the slice. [TODO perhaps it might be OK to allow callers to mutate, but not append to, the slice].

func PipeSeqThrough

func PipeSeqThrough[W io.WriteCloser](seq Seq, f func(w io.Writer) W) Seq

PipeSeqThrough returns a Seq that iterates over the data written by the function f to its argument Writer. The Writer implementation that it returns will be written with the data read from seq.

In other words, data read from seq will be "piped through" f, resulting in a new Seq.

Example

package main

import (
	"compress/gzip"
	"fmt"

	"github.com/rogpeppe/ioseq"
)

func main() {
	zeros := func(yield func([]byte, error) bool) {
		buf := make([]byte, 8192)
		for i := 0; i < 1000; i++ {
			if !yield(buf, nil) {
				return
			}
		}
	}
	n := 0
	for data, err := range ioseq.PipeSeqThrough(zeros, gzip.NewWriter) {
		if err != nil {
			panic(err)
		}
		n += len(data)
	}
	fmt.Println(n)
}

Output:

7988

func SeqFromReader

func SeqFromReader(r io.Reader, bufSize int) Seq

SeqFromReader returns a Seq that reads from r, allocating one buffer of the given size to do so unless r implements [WriterTo], in which case no buffer is needed.