circular

README

circular

Package circular implements an efficient thread-safe circular byte buffer to keep in-memory logs. It implements both io.Writer and io.WriterTo.

Features

• No allocation during Write.
• Full concurrent streaming (reading) support via multiple readers.
• Automatic lock-less flushes for readers supporting http.Flusher.
• Full test coverage.

Example

• Safely writes log to disk synchronously in addition to keeping a circular buffer.
• Serves the circular log buffer over HTTP asychronously.
• Writes log to stderr asynchronously.

This permits keeping all the output in case of a panic() on disk. Note that panic() output itself is only written to stderr since it uses print() builtin.

import (
  "log"
  "net/http"
  "sync"

  "github.com/maruel/circular"
)

func main() {
  logBuffer := circular.New(10 * 1024 * 1024)
  defer func() {
    // Flush ensures all readers have caught up.
    logBuffer.Flush()
    // Close gracefully closes the readers.
    logBuffer.Close()
  }()
  f, err := os.OpenFile("app.log", os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0666)
  if err != nil {
      panic(err)
  }
  defer f.Close()

  // Send to both circular buffer and file.
  log.SetOutput(io.MultiWriter(logBuffer, f))

  // Asynchronously write to stderr.
  go logBuffer.WriteTo(os.Stderr)

  log.Printf("This line is served over HTTP; file and stderr")

  http.HandleFunc("/",
      func(w http.ResponseWriter, r *http.Request) {
          w.Header().Set("Content-Type", "text/plain; charset=utf-8")
          // Streams the log buffer over HTTP until Close() is called.
          // AutoFlush ensures the log is not buffered locally indefinitely.
          logBuffer.WriteTo(circular.AutoFlush(w, time.Second))
      })
  http.ListenAndServe(":6060", nil)
}

Documentation

Overview

Package circular implements an efficient thread-safe circular byte buffer to keep in-memory logs.

Index

• type Buffer
  • func New(size int) Buffer
• type WriteFlusher
  • func AutoFlush(w io.Writer, delay time.Duration) WriteFlusher

Examples

• New

Types

type Buffer

type Buffer interface {
	// Close() aborts all WriteTo() streamers synchronously and blocks until they
	// all quit. It also aborts in-progress Write() calls. Close() can safely be
	// called in a reentrant context, as it doesn't use any lock.
	//
	// If the caller wants the data to be flushed to all readers, Flush() must be
	// called first.
	io.Closer

	// Write(p) writes the data to the circular buffer and wakes up any WriteTo()
	// reader. If p is longer or equal to the internal buffer, this call will
	// block until all readers have kept up with the data. To not get into this
	// condition and keep Write() performant, ensure the internal buffer is
	// significantly larger than the largest writes.
	io.Writer

	// WriteTo() streams a buffer to a io.Writer until the buffer is closed or a
	// write error occurs. It forcibly flushes the output if w supports
	// http.Flusher so it is sent to the underlying TCP connection as data is
	// appended.
	io.WriterTo

	// Flush implements http.Flusher. It forcibly blocks until all readers are up
	// to date.
	Flush()
}

Buffer is the interface to a circular buffer. It can be written to and read from concurrently.

It is expected that objects implementing this interface are thread-safe.

func New

func New(size int) Buffer

New returns an initialized Buffer.

It is designed to keep recent logs in-memory efficiently and in a thread-safe manner. No heap allocation is done within Write(). Independent readers each have their read position and are synchronized with the writer to not have any data loss.

Example

// Normally, use a larger buffer.
logBuffer := New(1024)

// Normally, use an actual file.
f, err := ioutil.TempFile("", "circular")
if err != nil {
	panic(err)
}
defer func() {
	f.Close()
	if err := os.Remove(f.Name()); err != nil {
		panic(err)
	}
}()

// Sends to both circular buffer and file.
log.SetOutput(io.MultiWriter(logBuffer, f))
// Normally remove this call. Used here so output can be verified below.
log.SetFlags(0)

var wgDone sync.WaitGroup
wgDone.Add(1)

var wgReady sync.WaitGroup
wgReady.Add(1)
go func() {
	defer wgDone.Done()
	// This has to be done otherwise this goroutine may not even have the
	// chance of getting scheduled before the original function exits.
	wgReady.Done()
	// Asynchronously write to stdout. Normally, use os.Stderr.
	_, _ = logBuffer.WriteTo(os.Stdout)
}()

log.Printf("This line is served over HTTP; file and stdout")

server := &http.Server{
	Addr: ":",
	Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
		// Streams the log buffer over HTTP until Close() is called. Data is
		// automatically flushed after a second. This is a trade-off about small
		// TCP packets causing lots of I/O overhead vs delay.
		_, _ = logBuffer.WriteTo(AutoFlush(w, time.Second))
	}),
}
go func() {
	_ = server.ListenAndServe()
}()

wgReady.Wait()

// <DO ACTUAL WORK>

// Flush ensures all readers have caught up before quitting.
logBuffer.Flush()
// Close gracefully closes the readers. This will properly TCP close the
// connections.
logBuffer.Close()

wgDone.Wait()

Output:

This line is served over HTTP; file and stdout

type WriteFlusher

type WriteFlusher interface {
	io.Writer

	Flush()
}

WriteFlusher is a io.Writer that can also be flushed. It is compatible with http.Flusher.

func AutoFlush

func AutoFlush(w io.Writer, delay time.Duration) WriteFlusher

AutoFlush converts an io.Writer supporting http.Flusher to call Flush() automatically after each write after a small delay.

To flush after each Write() call, pass 0 as delay.

The main use case is http connection when piping a circular buffer to it.