events

README

events

Go package for routing, formatting and publishing events produced by a program.

Motivations

While Go's standard log package is handy it definitely lacks crucial features, like the ability to control the output format of the events for example. There are many packages that provides logger implementations for Go but they often expose complex APIs and were not designed to be efficient in terms of CPU and memory usage.

The events package attempts to address these problems by providing high level abstractions with highly efficient implementations. But it also goes further, offering a new way to think about what logging is in a program, starting with the package name, events, which expresses what this problem is about. During its execution, a program produces events, and these events need to be captured, routed, formatted and published to a persitence system in order to be later analyzed.

The package was inspired by this post from Dave Cheney. It borrowed a lot of the ideas but tried to find the sweet spot between Dave's idealistic view of what logging is supposed to be, and production constraints that we have here at Segment.

Events

At the core of the package is the Event type. Instances of this type carry the context in which the event was generated and the inforation related to the event.

Events are passed from the sources that trigger them to handlers, which are types implementing the Handler interface:

type Handler interface {
    HandleEvent(*Event)
}

The sub-packages provide implementations of handlers that publish events to various formats and locations.

Logging

The Logger type is a source of events, the program uses loggers to generate events with an API that helps the developer express its intent. Unlike a lot of logging libraries, the logger doesn't support levels of messages, instead it exposes a Log and Debug methods. Events generated by the Log method are always produced by the logger, while those generated by Debug may be turned on or off if necessary.

The package also exposes a default logger via top-level functions which cover the needs of most programs. The Log and Debug functions support fmt-style formatting but augment the syntax with features that make it simpler to generate meaningful events. Refer to the package's documentation to learn more about it.

Log message formatting

The events package supports a superset of the fmt formatting language. The percent-base notation for placeholders is enhanced to automatically generated event arguments from values passed to the call to Log or Debug functions. This works by inserting an argument name wrapped in braces ({}) between the % sign and the verb of the format.

For example, this piece of code generates an event that has an argument named "name" and a value named "Luke":

package main

import (
    "github.com/segmentio/events"
)

func main() {
    events.Log("Hello %{name}s!", "Luke")
}

Note that using the extended syntax is optional and the regular fmt format is supported as well.

Compatibility with the standard library

The standard log package doesn't give much flexibility when it comes to its logger type. It is a concrete type and there is no Logger interface which would make it easy to plugin different implementations in packages that need to log events. Unfortunately many of these packages have hard dependencies on the standard logger, making it hard to capture their events and produce them in different formats.
However, the events/log package is a shim between the standard log package, and a stream of events. It exposes an API compatible with the standard library, and automatically configures the log package to reroute the messages it emits as events to the default logger.

Handlers

Event handlers are the abstraction layer that allows to connect event sources to arbitrary processing pipelines.
The sub-packages provides pre-defiend implementations of handlers.

text

The events/text package provides the implementation of an event handler which formats the event it receives in a human-readable format.

ecs-logs

The events/ecslogs package provides the implementation of an event handler which formats the events it receives in a format that is understood by ecs-logs.

We said the logger doesn't support log levels, however these levels have proven useful to get a signal on a program misbehaving when it starts emitting tons of ERROR level messages.
However, the program doesn't have to express what the severity level is in order to get the right behavior. The events/ecslogs package analyzes the events it receives and guess what the level should be, here are the rules:

• By default events are set to the INFO level.
• If an event was generated from a Debug call then handler sets the event level to DEBUG.
• If the event's arguments contains at least one value that satisfies the error interface then the level is set to ERROR. These rules allow for the best of both worlds, giving the program a small and expressive API to produce events while maintaining compatibility with our existing tools.

DEBUG/INFO/ERROR

The events package has two main log levels (events.Log and events.Debug), but the ecslogs subpackage will automatically extract error values in the event arguments, generate ERROR level messages, and put the error and stack trace (if any is available) into the event data.

For example, this code will output a structured log message with ERROR level.

package main

import (
    "errors"
    "os"

    "github.com/segmentio/events"
    "github.com/segmentio/events/ecslogs"
)

func main() {
    events.DefaultHandler = ecslogs.NewHandler(os.Stdout)
    events.Log("something went wrong: %{error}v", errors.New("oops!"))
}

Otherwise, events generated by a call to Log will be shown as INFO messages and events generated by a call to Debug will be shown as DEBUG messages.

Automatic Configuration

The sub-packages have side-effects when they are importaed, both events/text and events/ecslogs set the default logger's handler. The events/text package sets a handler if the program's output is a terminal, while events/ecslogs does the opposite.
This approach mimics what we've achieved in many parts of our software stack and has proven to be good defaults, doing the right thing whether the program is dealing with a production or development environment.

Here's a code example that is commonly used to configure the events package:

package main

import (
    "github.com/segmentio/events"
    _ "github.com/segmentio/events/ecslogs"
    _ "github.com/segmentio/events/text"
)

func main() {
    events.Log("enjoy!")
}

Documentation

Index

• Variables
• func Debug(format string, args ...interface{})
• func IsInterruption(err error) bool
• func IsSignal(err error, signals ...os.Signal) bool
• func IsTerminal(fd int) bool
• func IsTermination(err error) bool
• func Log(format string, args ...interface{})
• func Signal(sigchan <-chan os.Signal) <-chan os.Signal
• func SignalWith(handler Handler, sigchan <-chan os.Signal) <-chan os.Signal
• func SortArgs(args Args)
• func SourceForPC(pc uintptr) (file string, line int)
• func WithSignals(ctx context.Context, signals ...os.Signal) (context.Context, context.CancelFunc)
• type Arg
• type Args
  • func A(m map[string]interface{}) Args
  • func (args Args) Get(name string) (v interface{}, ok bool)
  • func (args Args) Map() map[string]interface{}
• type Event
  • func (e *Event) Clone() *Event
• type Handler
  • func MultiHandler(handlers ...Handler) Handler
• type HandlerFunc
  • func (f HandlerFunc) HandleEvent(e *Event)
• type Logger
  • func NewLogger(handler Handler) *Logger
  • func (l *Logger) Debug(format string, args ...interface{})
  • func (l *Logger) Log(format string, args ...interface{})
  • func (l *Logger) With(args Args) *Logger
• type SignalError
  • func (s *SignalError) Error() string

Variables

var DefaultLogger = NewLogger(nil)

DefaultLogger is the default logger used by the Log function. This may be overwritten by the program to change the default route for log events.

Functions

func Debug

func Debug(format string, args ...interface{})

Debug emits a debug event to the default logger.

func IsInterruption

func IsInterruption(err error) bool

IsInterruption returns true if the given error was caused by receiving an interruption signal.

func IsSignal

func IsSignal(err error, signals ...os.Signal) bool

IsSignal returns true if the given error is a *SignalError that was generated upon receipt of one of the given signals. If no signal is passed, the function only tests for err to be of type *SginalError.

func IsTerminal

func IsTerminal(fd int) bool

IsTerminal returns true if the given file descriptor is a terminal.

func IsTermination

func IsTermination(err error) bool

IsTermination returns true if the given error was caused by receiving a termination signal.

func Log

func Log(format string, args ...interface{})

Log emits a log event to the default logger.

func Signal

func Signal(sigchan <-chan os.Signal) <-chan os.Signal

Signal triggers events on the default handler for every signal that arrives on sigchan. The function returns a channel on which signals are forwarded, the program should use this channel instead of sigchan to receive signals.

func SignalWith

func SignalWith(handler Handler, sigchan <-chan os.Signal) <-chan os.Signal

SignalWith triggers events on handler for every signal that arrives on sigchan. The function returns a channel on which signals are forwarded, the program should use this channel instead of sigchan to receive signals.

func SortArgs

func SortArgs(args Args)

SortArgs sorts a list of argument by their argument names.

This is not a stable sorting operation, elements with equal values may not be in the same order they were originally after the function returns.

func SourceForPC

func SourceForPC(pc uintptr) (file string, line int)

SourceForPC returns the file and line given a program counter address. The file path is in the canonical form for Go programs, starting with the package path.

func WithSignals

func WithSignals(ctx context.Context, signals ...os.Signal) (context.Context, context.CancelFunc)

WithSignals returns a copy of the given context which may be canceled if any of the given signals is received by the program.

Types

type Arg

type Arg struct {
	Name  string
	Value interface{}
}

Arg represents a single event argument.

type Args

type Args []Arg

Args reprsents a list of event arguments.

func A

func A(m map[string]interface{}) Args

A constructs an argument list from a map.

func (Args) Get

func (args Args) Get(name string) (v interface{}, ok bool)

Get returns the value of the argument with name within args.

func (Args) Map

func (args Args) Map() map[string]interface{}

Map converts an argument list to a map representation. In cases where the list contains multiple arguments with the same name the value of the last one will be seen in the map.

type Event

type Event struct {
	// Message carries information about the event in a human-readable format.
	Message string

	// Source represents the location where this event was generated from.
	Source string

	// Args is the list of arguments of the event, it is intended to give
	// context about the information carried by the even in a format that can
	// be processed by a program.
	Args Args

	// Time is the time at which the event was generated.
	Time time.Time

	// Debug is set to true if this is a debugging event.
	Debug bool
}

The Event type represents unique events generated by the program. They carry context about how they were triggered and information to pass to handlers.

func (*Event) Clone

func (e *Event) Clone() *Event

Clone makes a deep copy of the event, the returned value doesn't shared any pointer with the original.

type Handler

type Handler interface {
	// HandleEvent is called by event producers on their event handler, passing
	// one event object as argument to the function.
	//
	// The handler MUST NOT retain any references to the event or its fields. If
	// the handler needs to capture the event is has to create a copy by calling
	// e.Clone.
	HandleEvent(e *Event)
}

The Handler interface is implemented by types that intend to be event routers or apply transformations to an event before forwarding it to another handler.

var (
	// Discard is a handler that does nothing with the events it receives.
	Discard Handler = HandlerFunc(func(e *Event) {})

	// DefaultHandler is the default handler used when non is specified.
	DefaultHandler Handler = Discard
)

func MultiHandler

func MultiHandler(handlers ...Handler) Handler

MultiHandler returns a new Handler which broadcasts the events it receives to its list of handlers.

type HandlerFunc

type HandlerFunc func(*Event)

HandlerFunc makes it possible for simple function types to be used as event handlers.

func (HandlerFunc) HandleEvent

func (f HandlerFunc) HandleEvent(e *Event)

HandleEvent calls f.

type Logger

type Logger struct {
	// Handler is the event handler receiving events from the logger.
	Handler Handler

	// Args is a list of arguments that get automatically injected into every
	// events produced by the logger.
	Args Args

	// CallDepth is used to adjust which caller the logger should report its
	// events are coming from.
	// Leaving to zero means reporting the direct caller of the logger's
	// methods.
	CallDepth int

	// EnableSource controls whether the logger should report the program counter
	// address of its caller on the events it produces.
	// This has a significant impact on the performance of the logger's Log and
	// Debug method but also provides very important insights, be mindful about
	// turning it on or off.
	EnableSource bool

	// EnableDebug controls whether calls to Debug produces events.
	EnableDebug bool
}

A Logger is a wrapper around an event handler which exposes a Log method for formatting event messages before sending them to the handler.

The format supported by the Log method is a superset of the fmt-style format, where the 'verbs' may include a column-surrounded value representing the name of the matching argument.

The Log method also makes a special case when it gets an events.Args as last argument, it doesn't use it to format the message and instead simply append it to the event's argument list.

Here's an example with the defalut logger:

events.Log("Hello %{name}s!", "Luke", events.Args{
	{"from", "Han"},
})

Which produces an event that looks like this:

events.Event{
	Message: "Hello Luke!",
	Args:    events.Args{{"name", "Luke"}, {"from", "Han"}},
	...
}

Logger instances are safe to use concurrently from multiple goroutines.

func NewLogger

func NewLogger(handler Handler) *Logger

NewLogger allocates and returns a new logger which sends events to handler.

func (*Logger) Debug

func (l *Logger) Debug(format string, args ...interface{})

Debug is like Log but only produces events if the logger has debugging enabled.

func (*Logger) Log

func (l *Logger) Log(format string, args ...interface{})

Log formats an event and sends it to the logger's handler.

func (*Logger) With

func (l *Logger) With(args Args) *Logger

With returns a new Logger which is a copy of l augmented with args.

type SignalError

type SignalError struct {
	os.Signal
}

SignalError is a wrapper for the os.Signal type which also implements the error interface so it can be reported by the Err method of a context.

func (*SignalError) Error

func (s *SignalError) Error() string

Error satisfies the error interface.