Documentation

Overview

    Package iobuf performs explicit memory management for data buffers used to perform network IO. The intent is that it is more efficient to perform manual allocation than to rely on the Go garbage collector to manage large chunks of frequently recycled memory.

    In this model, a Pool is a collection of contiguous memory area (of type <buf>) used for memory allocation. The bufs are subdivided into slices of type Slice.

    Pool: a source of memory areas.
    Slice: a contiguous region of allocated memory.
    Allocator: a Slice allocator.
    Reader: an IO reader that reads into Slices.
    

    There is an analogy with sbrk/malloc: the Pool is the source of memory (like sbrk), and the Allocator hands out small areas (like malloc). Allocations are mostly sequential within a buf, allowing sequentially-allocated Slices to be coalesced at some later point.

    For efficiency, Slice values hold reference counts to the underlying buf. When all references are to a buf released, the buf is recycled into its Pool. This does not happen automatically. The caller is responsible for calling slice.Release() when finished using a slice.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    This section is empty.

    Types

    type Allocator

    type Allocator struct {
    	// contains filtered or unexported fields
    }

      Allocator is an allocator for Slices that tries to allocate contiguously. That is, sequential allocations will tend to be contiguous, which means that Coalesce() will usually be able to perform coalescing (without copying the data).

      calloc := iobuf.Allocator(...)
      slice1 := calloc.Alloc(10)
      slice2 := calloc.Alloc(20)
      slices := iobuf.Coalesce([]*iobuf.Slice{slice1, slice2})
      // slices should contain 1 element with length 30.
      

      func NewAllocator

      func NewAllocator(pool *Pool, reserve uint) *Allocator

        NewAllocator returns a new Slice allocator.

        <reserve> is the number of spare bytes to reserve at the beginning of each allocated Slice. This can be used to reserve space for a header, for example.

        NOTE: It's a bit weird to set the number of reserve bytes in the NewAllocator call; it seems more natural in the Alloc call. But it's convenient to set it in NewAllocator, because in our use-cases, the code that calls Alloc doesn't know the number of reserve bytes.

        func (*Allocator) Alloc

        func (a *Allocator) Alloc(bytes uint) *Slice

          Alloc allocates a new Slice.

          func (*Allocator) Copy

          func (a *Allocator) Copy(buf []byte) *Slice

            Copy allocates a Slice and copies the buf into it.

            func (*Allocator) Release

            func (a *Allocator) Release()

              Release releases the allocator.

              type Pool

              type Pool struct {
              	// contains filtered or unexported fields
              }

                Pool manages a pool of iobufs. The size of the pool is not fixed, it can grow without bound.

                The implementation here allocates a new iobuf whenever there is an allocation request and the pool is empty. For iobufs to be recycled, explicit Release() calls are required. However, if these Release() calls are missing, the program will continue to function, recycling the buffers through the gc. Therefore, if you forget Release() calls, you will be putting pressure on gc to recycle the iobufs. You can examine the <allocated> field to check how many iobufs have been allocated during the lifetime of the Pool.

                func NewPool

                func NewPool(minSize uint) *Pool

                  NewPool creates a new pool. The pool will allocate iobufs in multiples of minSize. If minSize is zero, the default value (4K) will be used.

                  func (*Pool) Close

                  func (pool *Pool) Close()

                    Close shuts down the Pool.

                    type Reader

                    type Reader struct {
                    	// contains filtered or unexported fields
                    }

                      Reader wraps an io.Reader to provide a buffered Read() operation.

                      func NewReader

                      func NewReader(pool *Pool, conn io.Reader) *Reader

                        NewReader returns a new io reader.

                        func (*Reader) Close

                        func (r *Reader) Close()

                          Close closes the Reader. Do not call this concurrently with Read. Instead, close r.conn, wait until Read() has completed, then perform the Close().

                          func (*Reader) Read

                          func (r *Reader) Read(n int) (*Slice, error)

                            Read returns the next <n> bytes of input as a Slice. Returns an error if the read was short, even if some input was read.

                            type Slice

                            type Slice struct {
                            	Contents []byte // iobuf.Contents[base:bound]
                            	// contains filtered or unexported fields
                            }

                              Slice refers to an iobuf and the byte slice for the actual data.

                              func Coalesce

                              func Coalesce(slices []*Slice, maxSize uint) []*Slice

                                Coalesce a sequence of slices. If two slices are adjacent, they are combined. Takes ownership of the slices, caller takes ownership of the result.

                                func NewSlice

                                func NewSlice(buf []byte) *Slice

                                  NewSlice creates a Slice from a byte array. The value is not copied into an iobuf, it continues to refer to the buffer that was passed in.

                                  func (*Slice) ExpandFront

                                  func (slice *Slice) ExpandFront(bytes uint) bool

                                    ExpandFront tries to expand the Slice by <bytes> before the front of the Slice. Returns true if the Slice was expanded.

                                    func (*Slice) FreeEntirePrefix

                                    func (slice *Slice) FreeEntirePrefix()

                                      FreeEntirePrefix sets the free index to zero. Be careful when using this, you should ensure that no Slices are using the free region.

                                      func (*Slice) Release

                                      func (slice *Slice) Release()

                                        Release releases the slice, decrementing the reference count on the iobuf and destroying the slice.

                                        func (*Slice) ReleasePrevious

                                        func (slice *Slice) ReleasePrevious(prev *Slice)

                                          ReleasePrevious releases the <prev> slice, extending the free prefix of the target slice if possible.

                                          func (*Slice) Size

                                          func (slice *Slice) Size() int

                                            Size returns the number of bytes in the Slice.

                                            func (*Slice) TruncateFront

                                            func (slice *Slice) TruncateFront(bytes uint)

                                              TruncateFront removes <bytes> from the front of the Slice.