bufpool

README

bufpool

A small Go package for pooling and reusing byte buffers, reducing allocations and garbage-collector pressure in code that handles many short-lived buffers.

A Buffer implements io.Reader, io.ByteReader, io.Writer, io.ByteWriter, io.StringWriter, io.ReaderFrom, io.WriterTo, io.Closer and fmt.Stringer, so it drops into most code that already speaks the standard streaming interfaces. Buffers obtained from a Pool are returned to it for reuse, and an adaptive strike heuristic discards backing arrays that have grown large but are repeatedly under-utilized, so a single large write does not pin memory indefinitely. See COMPARISON.md for how this stacks up against sync.Pool idioms and other buffer pools.

Adapted from https://github.com/golang/go/issues/27735#issuecomment-739169121.

Install

go get github.com/JohanLindvall/bufpool

import "github.com/JohanLindvall/bufpool"

Quick start

var pool bufpool.Pool // the zero value is ready to use

// Get a buffer from the pool.
buf := pool.Get()

buf.WriteString("hello ")
buf.Write([]byte("world"))

// Use it as an io.Reader.
data, _ := bufpool.ReadAllBytes(buf) // []byte("hello world"), zero-copy for *Buffer
process(data)                        // use (or copy) the bytes *before* releasing:
                                     // data aliases the buffer's backing array

// Return the buffer to the pool for reuse. This invalidates data;
// do not use buf or data afterwards.
buf.Release()

Because a Buffer is an io.Closer, it also works with defer, and can be handed off as an io.ReadCloser that returns itself to the pool when closed:

func payload(pool *bufpool.Pool) io.ReadCloser {
    buf := pool.Get()
    buf.WriteString("body")
    return buf // the consumer's Close returns the buffer to the pool
}

The consumer now decides when the buffer is released, possibly on another goroutine, so hold no slice from Bytes/Next/ReadAllBytes across the handoff. net/http is the case to watch. Since *Buffer already satisfies io.ReadCloser, http.NewRequest adopts it as req.Body verbatim instead of wrapping it, and the transport closes — and therefore releases — it before Do returns. Its type switch also special-cases only *bytes.Buffer, *bytes.Reader and *strings.Reader, so the request goes out with ContentLength -1 and Transfer-Encoding: chunked, and with a nil GetBody that stops a 307/308 redirect from replaying the body.

These are two separate problems and neither remedy fixes both — Detach only stops the release, and the fields only fix the wire format — so do both:

buf.Detach() // the transport's Close must not return it to the pool
req, _ := http.NewRequest("POST", url, buf)
req.ContentLength = int64(buf.Len())                       // identity, not chunked
req.GetBody = func() (io.ReadCloser, error) {              // replay on 307/308
    buf.Rewind()
    return buf, nil
}

Returning buf itself from GetBody is only safe because Detach has already made its Close a no-op. Drop the GetBody if you do not need redirects followed; without one the client returns the 307/308 response rather than following it.

Usage

Pooling

var pool bufpool.Pool

buf := pool.Get()

Calling Release (or Close) returns the buffer to the pool and resets it to the zero value, so it must not be used afterwards. Re-acquire one with Get. A Buffer must not be copied after first use (go vet reports such copies).

Reading and writing

A Buffer separates writes (which append) from reads (which consume from the front, tracked by an internal read position):

buf := pool.Get()
buf.WriteString("abcdef")

p := make([]byte, 3)
buf.Read(p)            // p = "abc", read position now at 3
buf.Bytes()            // []byte("def") — the unread remainder (aliases the buffer)
buf.Next(2)            // []byte("de") — consume 2 bytes zero-copy (aliases the buffer)
buf.String()           // "f" — a copy, safe to keep after release
buf.Len()              // 1 — unread bytes, like bytes.Buffer.Len
buf.Size()             // 6 — total written length, including consumed bytes
buf.Cap()              // capacity of the backing array

buf.Rewind()           // reset the read position to re-read from the start

ReadByte and WriteByte round out the byte-at-a-time interfaces (io.ByteReader, io.ByteWriter), so callers like binary.ReadUvarint work directly on a Buffer without a bufio wrapper.

WriteTo streams the unread portion to any io.Writer, and ReadFrom fills the buffer from any io.Reader, so io.Copy in either direction avoids intermediate copy buffers:

n, err := io.Copy(buf, resp.Body) // uses buf.ReadFrom, no 32 KiB scratch buffer

Streaming

Reads never reclaim memory: the consumed prefix stays in place — that is what lets Rewind replay the full contents — and writes always append after it. Rewind's replay is therefore complete: it always reaches back to the last point the buffer was empty (its creation, or the most recent Reset or SetBytes), no matter how reads and writes were interleaved in between. Used as a long-lived FIFO on a single buffer (write a chunk, read it, repeat), the buffer therefore grows with the total bytes streamed through it, not with the working set. Bound it by calling Reset at natural message boundaries (it rewinds, truncates, and applies the same keep-or-discard heuristic as the pool).

A Release/Get round-trip does not bound it: Get re-slices the same array to [:0], so the round trip resets Size but not Cap. The growth also outlives the buffer — a FIFO buffer that grew to N bytes hands an N-byte array to the pool, where it scores as well utilized and is handed on to unrelated callers regardless of how little they asked for. Reset is what bounds pool-wide memory, not just one buffer's.

When the output size is known in advance, Grow pre-allocates capacity so subsequent writes do not reallocate:

buf.Grow(len(payload))
buf.Write(payload)

Reserved capacity is not exempt from the strike heuristic below, which measures utilization as written length against capacity. Capacity above 64 KiB that you reserve but leave unfilled counts as under-utilized, so a buffer that reserves far more than it writes loses the reservation every fifth cycle. Reserve close to what you will actually write.

Ownership and aliasing

The zero-copy calls trade safety for speed; their rules are:

• Bytes, Next, ReadAllBytes and Scratch return slices that alias the buffer. Release, Close and Reset invalidate them — the backing array re-enters the pool and the next Get may overwrite it. Copy the bytes (or use String) if they must outlive the buffer.

• The Next and ReadAllBytes slices are capped to their length (a three-index slice), so appending to one allocates a fresh array rather than writing into the buffer. This diverges from bytes.Buffer, where appending to a Next slice silently overwrites the bytes not yet read.

• The Bytes slice is not capped: as with bytes.Buffer.Bytes, its capacity runs to the end of the backing array, so appending to it fills the buffer's spare capacity without allocating. This exists for one idiom — handing the slack to an appending encoder and adopting the result (p := enc.MarshalAppend(buf.Bytes(), msg)). It also makes the slice a writable window onto the buffer: do not modify the buffer through it in any other pattern. Appended bytes lie beyond the buffer's length — the buffer's own next write lands directly over them, and Release hands the array, appended bytes included, to an unrelated caller. Finish with (or copy) the result before writing to, releasing or resetting the buffer.

• Scratch is the destination-slice counterpart of the Bytes append idiom: it returns the spare capacity as a full-length slice, sized for APIs that fill a caller-provided dst when it is big enough and allocate otherwise — some key that decision on len(dst) (golang/snappy), some on cap(dst) (klauspost's s2); the full-length slice satisfies both. Fill wraps the whole sequence — scratch, decode, adopt — in one call, and its closure keeps every aliasing slice out of the caller's scope:

  buf := pool.Get()
  err := buf.Fill(func(dst []byte) ([]byte, error) {
      return snappy.Decode(dst, packed)
  })
  

  Either way the decode goes, the buffer ends up owning the result: in place with no allocation when the scratch space sufficed, or adopting the decoder's fresh exact-size array — which warms the pool for the next round trip. Use it on an empty buffer (adoption discards existing contents), and note that on a pooled buffer the scratch space initially holds a previous user's bytes (see Secrets).

• NewBuffer and SetBytes adopt the given slice as the backing array without copying. Ownership transfers to the buffer (and, once released, to the pool): the caller must not use the slice afterwards. Because the pool sizes a buffer by cap() alone, adopting a capacity-capped sub-slice of a much larger array (arena[:n:n]) hands the pool the whole array while the heuristic classifies it by the small capacity, and it is never evicted — pass a full-capacity slice or bytes.Clone it instead.

• Handing a buffer to a consumer that calls Close transfers release timing to that consumer, possibly on another goroutine, so no aliased slice may be held across the call. net/http needs particular care — see below.

Secrets

Nothing on the Release/Get path clears a backing array, so a released buffer's bytes stay resident in the pool and are handed to the next, unrelated caller — readable through that buffer's spare capacity, and passed to any io.Reader that ReadFrom gives the spare capacity to. Wipe zeroes the whole array (unread bytes, consumed prefix and spare capacity alike) and then resets the buffer:

buf := pool.Get()
defer func() { buf.Wipe(); buf.Release() }()

Unlike Reset, Wipe does not apply the keep-or-discard heuristic, so the following Release or Reset is still the only application — wiping does not charge the array twice. It does leave the buffer empty for that application to score, so an array above 64 KiB takes a strike where releasing it unwiped would have kept it, and is dropped after five such cycles. That is the intended trade for not leaving secrets in the pool.

It costs a memclr of the full capacity, which is why it is opt-in rather than part of Release. It reaches from the current backing slice's start through its capacity, so for a slice adopted via NewBuffer/SetBytes it does not touch bytes before that start or beyond a capped capacity — and it cannot reach an array the buffer has already outgrown or otherwise replaced. Wipe before the buffer grows, not only at the end.

Detached buffers

A Buffer can be used standalone, without a pool, via NewBuffer:

buf := bufpool.NewBuffer([]byte("seed")) // adopts the slice; don't reuse it
buf.WriteString(" more")

A detached buffer's Release/Close are no-ops, so defer buf.Close() is safe regardless of a buffer's origin. Detach turns a pooled buffer into a detached one — useful when its contents must outlive a consumer that closes it.

In-place reuse

Reset rewinds the read position and truncates the buffer for reuse while keeping it attached to its pool — useful in a tight loop where you want to reuse the same Buffer without round-tripping through Get/Release:

buf := pool.Get()
for _, item := range items {
    buf.Reset()
    buf.WriteString(item)
    // ... use buf ...
}
buf.Release()

The strike heuristic

To avoid keeping unnecessarily large backing arrays alive, both Release and Reset apply the same heuristic when deciding whether to keep a buffer's backing array:

• Buffers with capacity ≤ 64 KiB are always kept (strike counter cleared).
• Buffers that are at least 50% utilized are always kept (strike counter cleared).
• An oversized, under-utilized buffer is given up to four consecutive strikes; on the fifth it is discarded — the buffer is left with no backing array at all (Cap() returns 0), and the next write allocates one from scratch.

This means a single large usage is not kept alive forever by a continuous stream of small ones, while transient large usages are still tolerated.

Two details worth knowing. The four-strike budget is spent only by consecutive under-utilized applications, so a large but well-utilized release recurring more often than once per five small ones clears the counter every time and keeps its array resident indefinitely. And each Reset and each Release counts as one application, so a Reset immediately before a Release charges the array two strikes for one use — it is redundant anyway, since Release resets the handle. The package documentation is the authoritative description of the policy.

API overview

Symbol	Description
Pool	Buffer pool; the zero value is ready to use.
(*Pool) Get() *Buffer	Get an empty buffer attached to the pool.
NewBuffer(data []byte) *Buffer	Create a detached buffer adopting data (no copy).
(*Buffer) Write / WriteString / WriteByte	Append bytes / a string / one byte.
(*Buffer) Read / ReadByte / WriteTo	Consume the unread portion.
(*Buffer) Next(n int) []byte	Consume the next n bytes zero-copy (aliasing).
(*Buffer) ReadFrom	Fill from an io.Reader until EOF.
(*Buffer) Bytes / String	Unread bytes (aliasing) / unread string (copy).
(*Buffer) Len / Size / Cap	Unread length / total length / capacity.
(*Buffer) Grow(n int)	Pre-allocate space for n more bytes.
(*Buffer) Rewind / Reset	Rewind read position / truncate for reuse.
(*Buffer) Wipe()	Zero the whole backing array, then reset.
ErrTooLarge	Panic value when a buffer cannot grow further.
(*Buffer) SetBytes(p []byte)	Replace contents, adopting p (no copy), and rewind.
(*Buffer) Scratch() []byte	Spare capacity as a full-length slice (aliasing) — the dst for decode-into APIs.
(*Buffer) Fill(fn func([]byte) ([]byte, error)) error	Decode into Scratch, adopt the result, return fn's error.
(*Buffer) Release() / Close() error	Release into the pool.
(*Buffer) Detach()	Detach from the pool; Release/Close become no-ops.
ReadAllBytes(r io.Reader) ([]byte, error)	Read all bytes, zero-copy for *Buffer.

See the Go doc comments for the full details of each call.

Performance

Pooling costs a single small allocation per Get/Release cycle (the buffer handle itself); the backing arrays and pool bookkeeping are fully reused, and the write path is allocation-free once capacity is established:

BenchmarkGetRelease-16            38.95 ns/op    64 B/op    1 allocs/op
BenchmarkGetReleaseParallel-16    23.21 ns/op    64 B/op    1 allocs/op
BenchmarkWrite-16                 34.81 ns/op     0 B/op    0 allocs/op

Run them with go test -bench=. -benchmem. For comparisons against bytes.Buffer+sync.Pool, valyala/bytebufferpool and oxtoacart/bpool — including the memory-retention behavior the strike heuristic exists for — see COMPARISON.md; the harness lives in _bench/.

License

MIT

Documentation

Overview

Package bufpool provides pooled, reusable byte buffers that reduce allocations and garbage-collector pressure in code handling many short-lived buffers.

A Buffer implements io.Reader, io.ByteReader, io.Writer, io.ByteWriter, io.StringWriter, io.ReaderFrom, io.WriterTo, io.Closer and fmt.Stringer. Writes append to the buffer; reads consume it from the front, tracked by an internal read position. Buffers obtained from a Pool (whose zero value is ready to use) are returned to it with Release (or Close), after which they must not be used. A Pool is safe for concurrent use; an individual Buffer is not.

Reads never reclaim memory: the consumed prefix stays in place so Rewind can replay the full contents, and writes append after it. A buffer used as a long-lived FIFO (write, read, repeat) therefore grows with the total bytes streamed through it, not the working set. Bound it by calling Reset at natural message boundaries. A Release and Get round trip does not bound it: Get re-slices the same array to zero length, so the round trip resets Size but not Cap.

That growth outlives the buffer. A FIFO buffer that grew to N bytes hands an N-byte array to the pool on Release, where it scores as well utilized and is handed on to unrelated callers regardless of how little they asked for, so Reset bounds pool-wide memory rather than just one buffer's.

Releasing transfers the backing array back to the pool, so slices obtained through Bytes, Next, ReadAllBytes or Scratch are invalidated by Release, Close and Reset; conversely, slices handed to NewBuffer or SetBytes are adopted as the buffer's backing array (and follow it into the pool when it is released), so the caller must not use them afterwards. Scratch exposes the spare capacity as a full-length slice for APIs that decode into a caller-provided destination, and Fill wraps the whole idiom — scratch, decode, adopt — in one call whose closure keeps every aliasing slice out of the caller's scope (see Scratch and Fill). Slices returned by Next and ReadAllBytes are capped to their length, unlike the equivalents on bytes.Buffer, so appending to one allocates instead of overwriting the bytes that follow. Bytes is not capped: as on bytes.Buffer, its slice carries the array's spare capacity so an appending encoder can fill the buffer's slack — but that makes it a writable window onto the buffer, and the buffer must not otherwise be modified through it (see Bytes).

Nothing on that path clears an array, so a released buffer's bytes stay readable to whichever unrelated caller receives it next — including through the spare capacity that Scratch exposes and ReadFrom hands to an io.Reader. Buffer.Wipe zeroes the whole array and is the opt-in for buffers that held secrets.

To keep pooled memory bounded, an adaptive strike heuristic decides on each Release or Reset whether a buffer's backing array is worth keeping: buffers whose capacity is at most 64 KiB, or which are at least 50% utilized, are always kept and have their strike counter cleared; an oversized, under-utilized buffer survives up to four consecutive strikes before it is discarded, meaning it is left with no backing array at all rather than a fresh one. This prevents a single large usage from pinning memory through a continuous stream of small ones. Two consequences follow from the details. The budget is spent only by consecutive under-utilized applications, so a large but well-utilized release recurring more often than once per five small ones clears the counter every time and keeps its array resident indefinitely. And each Reset and each Release is one application, so a Reset immediately before a Release charges two strikes for one use.

Utilization is measured by capacity, which Go reports per slice rather than per array, and against the written length rather than the unread length. Both matter in practice: adopting a capacity-capped sub-slice of a large array (see SetBytes) hides the array's real size from the heuristic, and capacity reserved by Grow but never filled counts against utilization.

Adapted from https://github.com/golang/go/issues/27735#issuecomment-739169121.

Index

• Variables
• func ReadAllBytes(r io.Reader) ([]byte, error)
• type Buffer
  • func NewBuffer(data []byte) *Buffer
  • func (b *Buffer) Bytes() []byte
  • func (b *Buffer) Cap() int
  • func (b *Buffer) Close() error
  • func (b *Buffer) Detach()
  • func (b *Buffer) Fill(fn func(scratch []byte) ([]byte, error)) error
  • func (b *Buffer) Grow(n int)
  • func (b *Buffer) Len() int
  • func (b *Buffer) Next(n int) []byte
  • func (b *Buffer) Read(p []byte) (int, error)
  • func (b *Buffer) ReadByte() (byte, error)
  • func (b *Buffer) ReadFrom(r io.Reader) (int64, error)
  • func (b *Buffer) Release()
  • func (b *Buffer) Reset()
  • func (b *Buffer) Rewind()
  • func (b *Buffer) Scratch() []byte
  • func (b *Buffer) SetBytes(p []byte)
  • func (b *Buffer) Size() int
  • func (b *Buffer) String() string
  • func (b *Buffer) Wipe()
  • func (b *Buffer) Write(p []byte) (int, error)
  • func (b *Buffer) WriteByte(c byte) error
  • func (b *Buffer) WriteString(s string) (int, error)
  • func (b *Buffer) WriteTo(w io.Writer) (int64, error)
• type Pool
  • func (p *Pool) Get() *Buffer

Variables

var ErrTooLarge = errors.New("bufpool.Buffer: too large")

ErrTooLarge is the value passed to panic when a buffer cannot grow to hold the requested data, mirroring bytes.ErrTooLarge. It is an error rather than a plain string so that the recover-and-classify idiom works:

defer func() {
	if r := recover(); r != nil {
		err, ok := r.(error)
		if !ok || !errors.Is(err, bufpool.ErrTooLarge) {
			panic(r) // not ours; let it go
		}
		// handle
	}
}()

The panics that report programmer errors rather than resource exhaustion — a negative count, or an io.Writer or io.Reader violating its contract — stay plain strings, as they are in the bytes package.

Functions

func ReadAllBytes

func ReadAllBytes(r io.Reader) ([]byte, error)

ReadAllBytes reads all remaining bytes from r. If r is a *Buffer, it returns the buffer's unread bytes directly without copying and advances the buffer to EOF; otherwise it falls back to io.ReadAll. The error is nil on success, mirroring io.ReadAll.

Regardless of r's dynamic type, treat the returned slice as aliasing r's internal storage: it is only valid until r is next written to, reset, released or closed. Copy it if it must outlive r. For the *Buffer path the slice's capacity is limited to its length, so appending to it allocates rather than writing into the buffer; that is not guaranteed of the io.ReadAll fallback.

Types

type Buffer

type Buffer struct {
	// contains filtered or unexported fields
}

Buffer is a byte buffer that may be attached to a Pool. It implements io.Reader, io.ByteReader, io.Writer, io.ByteWriter, io.StringWriter, io.ReaderFrom, io.WriterTo, io.Closer and fmt.Stringer. Writes append to the buffer; reads consume it from the front, tracked by an internal read position. The zero value is a usable, detached buffer.

Reads never reclaim memory: the consumed prefix stays in place so Rewind can replay it, and writes append after it. A buffer used as a long-lived FIFO (write, read, repeat) therefore grows with the total bytes streamed through it, not the working set; bound it by calling Reset at natural message boundaries. A Release and Get round trip does not bound it, because Get re-slices the same array to zero length: see the package documentation, which is the authoritative description of retention and the keep-or-discard policy.

A Buffer must not be copied after first use (go vet reports such copies), and must not be used after Release or Close. Unlike a Pool, a Buffer is not safe for concurrent use by multiple goroutines.

func NewBuffer

func NewBuffer(data []byte) *Buffer

NewBuffer creates a new detached buffer whose initial contents are data. The slice becomes the buffer's backing array; it is not copied, and the caller should not use data after this call. NewBuffer(nil) creates an empty buffer.

func (*Buffer) Bytes

func (b *Buffer) Bytes() []byte

Bytes returns the unread portion of the buffer. The slice aliases the buffer's backing array and is only valid until the next mutating call; Release, Close and Reset invalidate it. Copy the bytes (or use String) if they must outlive the buffer.

Like bytes.Buffer.Bytes, the slice's capacity runs to the end of the backing array, so appending to it writes into the buffer's spare capacity without allocating. That exists for one idiom — hand the spare capacity to an appending encoder and adopt the result:

p := enc.MarshalAppend(buf.Bytes(), msg) // fills the buffer's slack

It also makes the slice a writable window onto the buffer itself, so do not modify the buffer through it in any other pattern. Appended bytes lie beyond the buffer's length: the buffer neither sees nor preserves them, its next write lands directly over them, and Release hands the array — appended bytes included — to whichever unrelated caller gets it from the pool next. Finish with, or copy, an appended result before writing to, releasing or resetting the buffer. Next and ReadAllBytes still cap their slices to their length.

func (*Buffer) Cap

func (b *Buffer) Cap() int

Cap returns the capacity of the buffer's backing array: the total space, including the already-written portion, that can be used before another allocation.

func (*Buffer) Close

func (b *Buffer) Close() error

Close returns the buffer to its pool and always returns a nil error. It implements io.Closer and is equivalent to Release, so a pooled *Buffer can be handed off as an io.ReadCloser and is returned to the pool at the release site without a pool reference in scope.

Handing a buffer to a consumer that closes it transfers release timing to that consumer, possibly on another goroutine: the caller's own Release becomes a no-op, and no slice from Bytes, Next, ReadAllBytes or Scratch may be held across the call. net/http is the common case and needs care. Because *Buffer already satisfies io.ReadCloser, http.NewRequest adopts it as req.Body verbatim rather than wrapping it, and the transport closes it before Do returns; and because the type switch there special-cases only *bytes.Buffer, *bytes.Reader and *strings.Reader, the request is sent with ContentLength -1 and Transfer-Encoding: chunked, with a nil GetBody that prevents a 307/308 redirect from replaying the body. These are separate problems and neither remedy covers both: Detach stops the transport's Close from returning the buffer to the pool, and setting req.ContentLength (plus a req.GetBody, if a redirect must replay the body) fixes the wire format. Do both. See the README for a worked example.

func (*Buffer) Detach

func (b *Buffer) Detach()

Detach detaches the buffer from its pool, making Release and Close no-ops. Use it to let a buffer's contents safely outlive a consumer that closes it.

func (*Buffer) Fill

func (b *Buffer) Fill(fn func(scratch []byte) ([]byte, error)) error

Fill passes the buffer's spare capacity (Scratch) to fn, adopts the slice fn returns as the buffer's new contents (SetBytes), and returns fn's error unchanged. It packages the decode-into-dst idiom in one call:

buf := pool.Get()
err := buf.Fill(func(dst []byte) ([]byte, error) {
	return snappy.Decode(dst, packed)
})

fn should treat dst as scratch space of arbitrary length and content: use it when it is large enough, allocate otherwise, and return the slice holding the result — the contract of snappy.Decode and kin. Either way the buffer ends up owning the result: in place with no allocation when dst sufficed, or adopting fn's fresh exact-size array, which warms the pool for the next round trip. Call Grow first to guarantee the in-place path.

Neither dst nor the returned slice may be retained or used once fn returns — both alias the buffer's backing array; keeping everything inside fn is the point of Fill over calling Scratch and SetBytes directly. The result is adopted even when fn returns an error (decoders return a nil or partial result alongside their error, leaving the buffer empty or holding the partial result), so release or reset the buffer on error as usual. Use Fill on an empty buffer: adoption discards existing contents.

func (*Buffer) Grow

func (b *Buffer) Grow(n int)

Grow grows the buffer's capacity, if necessary, to guarantee space for another n bytes: after Grow(n), at least n bytes can be written without another allocation. When it does allocate, Grow over-allocates (at least doubling the capacity) so that repeated grow-and-fill cycles stay amortized O(n) rather than reallocating on every round. Grow panics with "bufpool.Buffer.Grow: negative count" if n is negative, and with ErrTooLarge if the buffer would grow beyond the maximum slice length.

Reserved capacity is not exempt from the keep-or-discard heuristic, which measures utilization as written length against capacity: capacity above 64 KiB that is reserved but left unfilled counts as under-utilized, so a buffer that reserves far more than it writes loses the reservation on the fifth consecutive Release or Reset. Reserve close to what will be written, or expect to pay for the reservation again every fifth cycle.

func (*Buffer) Len

func (b *Buffer) Len() int

Len returns the number of unread bytes in the buffer, matching the semantics of bytes.Buffer.Len. Use Size for the total written length.

func (*Buffer) Next

func (b *Buffer) Next(n int) []byte

Next consumes the next n unread bytes and returns them as a slice, as if read by Read; if fewer than n bytes are unread, Next returns all of them. Like Bytes, the slice aliases the buffer's backing array and is only valid until the next mutating call. Next panics if n is negative.

Unlike Bytes, the returned slice's capacity is limited to its length, so appending to it allocates rather than overwriting the bytes that follow — which, for Next, would be the unread remainder.

func (*Buffer) Read

func (b *Buffer) Read(p []byte) (int, error)

Read consumes up to len(p) unread bytes into p, advancing the read position. It returns io.EOF once the buffer is fully consumed. Read implements io.Reader.

func (*Buffer) ReadByte

func (b *Buffer) ReadByte() (byte, error)

ReadByte consumes and returns the next unread byte, advancing the read position. It returns io.EOF once the buffer is fully consumed. ReadByte implements io.ByteReader.

func (*Buffer) ReadFrom

func (b *Buffer) ReadFrom(r io.Reader) (int64, error)

ReadFrom reads from r until EOF, appending to the buffer and growing it as needed. It returns the number of bytes read and any error except io.EOF encountered during the read. ReadFrom panics with ErrTooLarge if the buffer can no longer grow. ReadFrom implements io.ReaderFrom, so io.Copy into a Buffer needs no intermediate copy buffer.

r is handed the buffer's spare capacity to read into. On a buffer that came from a pool that space may still hold bytes written by a previous, unrelated user of the pool, so a Reader that inspects or retains more of the slice than the n bytes it reports can observe them. Wipe clears an array before it re-enters the pool.

func (*Buffer) Release

func (b *Buffer) Release()

Release returns the buffer to its pool and resets it to the zero value. Releasing invalidates all slices previously returned by Bytes, Next, ReadAllBytes or Scratch: the backing array re-enters the pool and the next Get may overwrite it, so copy such slices first if they must outlive the buffer. After Release the buffer is a detached zero buffer — further calls operate on that empty buffer instead of panicking, but are programming errors. If the buffer is detached, Release is a no-op. The cost of a Get/Release round-trip is the single small allocation of the Buffer handle in Get.

func (*Buffer) Reset

func (b *Buffer) Reset()

Reset rewinds the read position and truncates the buffer for in-place reuse, applying the same keep-or-discard heuristic the pool uses on Release: an oversized, repeatedly under-utilized backing array is dropped (replaced with a fresh nil buffer) instead of kept, so a single large use does not pin memory across resets. Unlike Release, the buffer stays usable and attached to its pool.

Reset invalidates slices previously returned by Bytes, Next, ReadAllBytes or Scratch. It also counts as one application of the heuristic, as Release does, so a Reset immediately before a Release charges the array two strikes for one use; that is redundant, since Release resets the handle anyway.

func (*Buffer) Rewind

func (b *Buffer) Rewind()

Rewind resets the read position to zero so the buffer's full contents can be read again. It does not modify the contents. The replay is complete: reads and writes never discard the consumed prefix, so Rewind always reaches back to the last point the buffer was empty — its creation, or the most recent Reset or SetBytes. (Supporting this replay is why reads retain the consumed prefix; see the package documentation on streaming.)

func (*Buffer) Scratch

func (b *Buffer) Scratch() []byte

Scratch returns the buffer's spare capacity — the region from its length to its capacity — as a full-length slice. It is the destination-slice counterpart of the Bytes append idiom: pass it as the dst of an API that fills a caller-provided slice when it is long enough and otherwise allocates, such as snappy.Decode, then adopt the result with SetBytes — or use Fill, which wraps the whole sequence. Such APIs key the reuse decision on len(dst) (github.com/golang/snappy) or on cap(dst) (klauspost/compress's s2); the full-length slice satisfies both, where a fresh pooled buffer's Bytes — empty, however large its capacity — is silently ignored by the len-based ones. Both outcomes of the decode adopt correctly: an in-place result re-slices the buffer's own array with no allocation, and a fresh exact-size array replaces it and warms the pool for the next round trip. Call Grow first to guarantee the in-place path.

Scratch is not a write path: bytes written into the slice are invisible to the buffer — its length does not change — until the result is adopted. Use it on an empty buffer: adoption discards existing contents, and adopting an in-place result on a partly-written buffer strands the written prefix in the array. The slice aliases the backing array, so it is only valid until the next mutating call, and on a buffer from a pool it initially holds bytes left by a previous, unrelated user of the pool (Wipe clears an array before it re-enters the pool).

func (*Buffer) SetBytes

func (b *Buffer) SetBytes(p []byte)

SetBytes replaces the buffer's contents with p and rewinds the read position. The slice becomes the new backing array; it is not copied, so ownership of p transfers to the buffer (and, once released, to its pool) and the caller must not use p after this call.

The keep-or-discard heuristic sizes a buffer by cap(p) alone, because Go cannot recover an array's size from a slice. Adopting a capacity-capped sub-slice of a much larger array (arena[:n:n]) therefore hands the pool the whole array while the heuristic classifies it by the small capacity, and it is never evicted. Pass a full-capacity slice, or bytes.Clone it, when the underlying array is substantially larger than the contents.

Strikes belong to the backing array, so adopting a slice of the current array — as Fill does when the decoder used the scratch space — preserves the array's strike history, while adopting a foreign array resets it. A capacity-capped sub-slice of the current array cannot be told apart from a foreign array and conservatively resets it too.

func (*Buffer) Size

func (b *Buffer) Size() int

Size returns the total length of the buffer, including any portion already consumed by Read.

func (*Buffer) String

func (b *Buffer) String() string

String returns a copy of the unread portion of the buffer as a string, implementing fmt.Stringer. If b is nil, it returns "<nil>".

func (*Buffer) Wipe

func (b *Buffer) Wipe()

Wipe zeroes the buffer's whole capacity — the consumed prefix, the unread bytes, and the spare capacity beyond them — then rewinds and truncates it.

Nothing on the Release and Get path clears an array, so without Wipe a buffer's contents stay resident in the pool and are handed to the next, unrelated caller: readable through that buffer's spare capacity, and passed to any io.Reader that ReadFrom hands the spare capacity to. Call Wipe before Release on any buffer that held secrets:

buf := pool.Get()
defer func() { buf.Wipe(); buf.Release() }()

Unlike Reset, Wipe does not apply the keep-or-discard heuristic, so the Release or Reset that follows it is still the single application — wiping does not charge the array twice. It does mean the buffer is empty by the time that application runs, so it scores 0% utilization: an array above 64 KiB accrues a strike where releasing it unwiped would have kept it, and is dropped after five such cycles. That is the intended trade for not leaving secrets in the pool.

Two limits are worth stating. Wipe costs a memclr of the whole capacity, which is why it is opt-in rather than part of Release. And it reaches only from the current backing slice's start through its capacity: for an array bufpool allocated that is the whole array, but for one adopted through NewBuffer or SetBytes it excludes anything before the slice's start or beyond a capped capacity (arena[:n:n]) — and it cannot reach an array the buffer has already outgrown or otherwise replaced, since Grow and SetBytes abandon arrays without clearing them. Nothing the buffer itself wrote can lie outside that region; wipe before the buffer grows, not only at the end.

func (*Buffer) Write

func (b *Buffer) Write(p []byte) (int, error)

Write appends p to the buffer, growing the backing array as needed. It always returns len(p) and a nil error, but panics with ErrTooLarge if the buffer can no longer grow. Write implements io.Writer.

func (*Buffer) WriteByte

func (b *Buffer) WriteByte(c byte) error

WriteByte appends c to the buffer, growing the backing array as needed. It always returns a nil error, but panics with ErrTooLarge if the buffer can no longer grow. WriteByte implements io.ByteWriter.

func (*Buffer) WriteString

func (b *Buffer) WriteString(s string) (int, error)

WriteString appends s to the buffer without copying it into a temporary []byte first. It always returns len(s) and a nil error, but panics with ErrTooLarge if the buffer can no longer grow. WriteString implements io.StringWriter.

func (*Buffer) WriteTo

func (b *Buffer) WriteTo(w io.Writer) (int64, error)

WriteTo writes the unread portion of the buffer to w, advancing the read position by the number of bytes accepted by w. If w accepts fewer bytes than offered without returning an error, WriteTo returns io.ErrShortWrite. WriteTo implements io.WriterTo.

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool is a pool of reusable byte buffers, backed by a sync.Pool. The zero value is ready to use. A Pool is safe for concurrent use by multiple goroutines and must not be copied after first use.

func (*Pool) Get

func (p *Pool) Get() *Buffer

Get returns an empty Buffer drawn from the pool. The buffer is attached to p, so calling Release or Close on it puts it back into the pool.