throughputbuffer

README

throughputbuffer

Package throughputbuffer provides a high throughput indefinitely growing io.ReadWriter, like bytes.Buffer, but optimized for minimal copies. It does the minimum amount of copies (1 per read + 1 per write) (or fewer through ReadFrom and WriteTo) and never has to move bytes in the buffer.

Byte slices are shared in a BufferPool. All byte slices are of the same length: the size given to New().

Buffers automatically shrink as data is read from them (they return their data to the pool).

package main

import (
	"crypto/rand"

	"github.com/Jille/throughputbuffer"
)

func main() {
	pool := throughputbuffer.New(64 * 1024 * 1024)

	buf1 := pool.Get()
	io.CopyN(buf1, rand.Reader, 1024 * 1024 * 1024)
	buf2 := pool.Get()
	io.Copy(buf2, buf1) // As data is read from buf, the byteslices are freed and reused by buf2.
}

Documentation

Overview

Package throughputbuffer provides a high throughput indefinitely growing io.ReadWriter. It does the minimum amount of copies (1 per read + 1 per write) and never has to move bytes in the buffer.

Memory is freed once read.

Index

• type Buffer
  • func (b *Buffer) Bytes() []byte
  • func (b *Buffer) Len() int
  • func (b *Buffer) Read(p []byte) (int, error)
  • func (b *Buffer) ReadFrom(r io.Reader) (int64, error)
  • func (b *Buffer) Write(p []byte) (int, error)
  • func (b *Buffer) WriteTo(w io.Writer) (int64, error)
• type BufferPool
  • func New(blocksize int) *BufferPool
  • func (p *BufferPool) Get() *Buffer

Types

type Buffer

type Buffer struct {
	// contains filtered or unexported fields
}

Buffer is a io.ReadWriter that can grow infinitely and does the minimum amount of copies (1 per read + 1 per write) and never has to move bytes.

func (*Buffer) Bytes

func (b *Buffer) Bytes() []byte

Bytes consumes all the data and returns it as a byte slice.

func (*Buffer) Len

func (b *Buffer) Len() int

func (*Buffer) Read

func (b *Buffer) Read(p []byte) (int, error)

func (*Buffer) ReadFrom

func (b *Buffer) ReadFrom(r io.Reader) (int64, error)

func (*Buffer) Write

func (b *Buffer) Write(p []byte) (int, error)

func (*Buffer) WriteTo

func (b *Buffer) WriteTo(w io.Writer) (int64, error)

type BufferPool

type BufferPool struct {
	// contains filtered or unexported fields
}

BufferPool holds a sync.Pool of byte slices and can be used to create new Buffers.

func New

func New(blocksize int) *BufferPool

New creates a new BufferPool. The blockSize is the size of the []byte slices internally. The blocksize should be within a few orders of magnitude of the expected size of your buffers. Using a larger blocksize results in more memory being held but unused, a smaller blocksize takes a bit more CPU cycles.

func (*BufferPool) Get

func (p *BufferPool) Get() *Buffer

Get creates a new Buffer using byte slices from this pool.