readahead

README

readahead

Asynchronous read-ahead for Go readers

This package will allow you to add readhead to any reader. This means a separate goroutine will perform reads from your upstream reader, so you can request from this reader without delay.

This is helpful for splitting an input stream into concurrent processing, and also helps smooth out bursts of input or output.

This should be fully transparent, except that once an error has been returned from the Reader, it will not recover. A panic will be caught and returned as an error.

The readahead object also fulfills the io.WriterTo interface, which is likely to speed up io.Copy and other code that use the interface.

See an introduction: An Async Read-ahead Package for Go

usage

To get the package use go get -u github.com/klauspost/readahead.

Here is a simple example that does file copy. Error handling has been omitted for brevity.

input, _ := os.Open("input.txt")
output, _ := os.Create("output.txt")
defer input.Close()
defer output.Close()

// Create a read-ahead Reader with default settings
ra := readahead.NewReader(input)
defer ra.Close()

// Copy the content to our output
_, _ = io.Copy(output, ra)

settings

You can finetune the read-ahead for your specific use case, and adjust the number of buffers and the size of each buffer.

The default the size of each buffer is 1MB, and there are 4 buffers. Do not make your buffers too small since there is a small overhead for passing buffers between goroutines. Other than that you are free to experiment with buffer sizes.

contributions

On this project contributions in terms of new features is limited to:

• Features that are widely usable and
• Features that have extensive tests

This package is meant to be simple and stable, so therefore these strict requirements.

license

This package is released under the MIT license. See the supplied LICENSE file for more info.

Documentation

Overview

Package readahead will do asynchronous read-ahead from an input io.Reader and make the data available as an io.Reader.

This should be fully transparent, except that once an error has been returned from the Reader, it will not recover.

The readahead object also fulfills the io.WriterTo interface, which is likely to speed up copies.

Package home: https://github.com/klauspost/readahead

Example

package main

import (
	"bytes"
	"fmt"
	"io"

	"github.com/klauspost/readahead"
)

func main() {
	// Buf is our input.
	buf := bytes.NewBufferString("Example data")

	// Create a Reader with default settings
	reader := readahead.NewReader(buf)
	defer reader.Close()

	// Copy the content to dst
	var dst = &bytes.Buffer{}
	_, err := io.Copy(dst, reader)
	if err != nil {
		panic("error when reading: " + err.Error())
	}

	fmt.Println(dst.String())
}

Output:

Example data

Index

• Constants
• func NewReadCloser(rd io.ReadCloser) io.ReadCloser
• func NewReadCloserBuffer(rc io.ReadCloser, buffers [][]byte) (res io.ReadCloser, err error)
• func NewReadCloserSize(rc io.ReadCloser, buffers, size int) (res io.ReadCloser, err error)
• func NewReader(rd io.Reader) io.ReadCloser
• func NewReaderBuffer(rd io.Reader, buffers [][]byte) (res io.ReadCloser, err error)
• func NewReaderSize(rd io.Reader, buffers, size int) (res io.ReadCloser, err error)
• type ReadSeekCloser
  • func NewReadSeekCloser(rd ReadSeekCloser) ReadSeekCloser
  • func NewReadSeekCloserBuffer(rd ReadSeekCloser, buffers [][]byte) (res ReadSeekCloser, err error)
  • func NewReadSeekCloserSize(rd ReadSeekCloser, buffers, size int) (res ReadSeekCloser, err error)
  • func NewReadSeeker(rd io.ReadSeeker) ReadSeekCloser
  • func NewReadSeekerSize(rd io.ReadSeeker, buffers, size int) (res ReadSeekCloser, err error)

Examples

• Package

Constants

const (
	// DefaultBuffers is the default number of buffers used.
	DefaultBuffers = 4

	// DefaultBufferSize is the default buffer size, 1 MB.
	DefaultBufferSize = 1 << 20
)

Functions

func NewReadCloser

func NewReadCloser(rd io.ReadCloser) io.ReadCloser

NewReadCloser returns a reader that will asynchronously read from the supplied reader into 4 buffers of 1MB each.

It will start reading from the input at once, maybe even before this function has returned.

The input can be read from the returned reader. When done use Close() to release the buffers, which will also close the supplied closer. If a reader supporting the io.Seeker is given, the returned reader will also support it.

func NewReadCloserBuffer

func NewReadCloserBuffer(rc io.ReadCloser, buffers [][]byte) (res io.ReadCloser, err error)

NewReadCloserBuffer returns a reader with a custom number of buffers and size. All buffers must be the same size. Buffers can be reused after Close has been called.

func NewReadCloserSize

func NewReadCloserSize(rc io.ReadCloser, buffers, size int) (res io.ReadCloser, err error)

NewReadCloserSize returns a reader with a custom number of buffers and size. buffers is the number of queued buffers and size is the size of each buffer in bytes.

func NewReader

func NewReader(rd io.Reader) io.ReadCloser

NewReader returns a reader that will asynchronously read from the supplied reader into 4 buffers of 1MB each.

It will start reading from the input at once, maybe even before this function has returned.

The input can be read from the returned reader. When done use Close() to release the buffers. If a reader supporting the io.Seeker is given, the returned reader will also support it.

func NewReaderBuffer

func NewReaderBuffer(rd io.Reader, buffers [][]byte) (res io.ReadCloser, err error)

NewReaderBuffer returns a reader with a custom number of buffers and size. All buffers must be the same size. Buffers can be reused after Close has been called.

func NewReaderSize

func NewReaderSize(rd io.Reader, buffers, size int) (res io.ReadCloser, err error)

NewReaderSize returns a reader with a custom number of buffers and size. buffers is the number of queued buffers and size is the size of each buffer in bytes.

Types

type ReadSeekCloser

type ReadSeekCloser interface {
	io.ReadCloser
	io.Seeker
}

func NewReadSeekCloser

func NewReadSeekCloser(rd ReadSeekCloser) ReadSeekCloser

NewReadSeekCloser returns a reader that will asynchronously read from the supplied reader into 4 buffers of 1MB each.

It will start reading from the input at once, maybe even before this function has returned.

The input can be read and seeked from the returned reader. When done use Close() to release the buffers, which will also close the supplied closer.

func NewReadSeekCloserBuffer

func NewReadSeekCloserBuffer(rd ReadSeekCloser, buffers [][]byte) (res ReadSeekCloser, err error)

NewReadSeekCloserBuffer returns a reader with a custom number of buffers and size. All buffers must be the same size. Buffers can be reused after Close has been called.

func NewReadSeekCloserSize

func NewReadSeekCloserSize(rd ReadSeekCloser, buffers, size int) (res ReadSeekCloser, err error)

NewReadSeekCloserSize returns a reader with a custom number of buffers and size. buffers is the number of queued buffers and size is the size of each buffer in bytes.

func NewReadSeeker

func NewReadSeeker(rd io.ReadSeeker) ReadSeekCloser

NewReadSeeker returns a reader that will asynchronously read from the supplied reader into 4 buffers of 1MB each.

It will start reading from the input at once, maybe even before this function has returned.

The input can be read and seeked from the returned reader. When done use Close() to release the buffers.

func NewReadSeekerSize

func NewReadSeekerSize(rd io.ReadSeeker, buffers, size int) (res ReadSeekCloser, err error)

NewReadSeekerSize returns a reader with a custom number of buffers and size. buffers is the number of queued buffers and size is the size of each buffer in bytes.