Back to

Package flume

Latest Go to latest
Published: Oct 9, 2019 | License: MIT | Module:


Package flume is a logging package, build on top of zap. It's structured and leveled logs, like zap/logrus/etc. It adds global, runtime re-configuration of all loggers, via an internal logger registry.

There are two interaction points with flume: code that generates logs, and code that configures logging output. Code which generates logs needs to create named logger instances, and call log functions on it, like Info() and Debug(). But by default, all these logs will be silently discarded. Flume does not output log entries unless explicitly told to do so. This ensures libraries can freely use flume internally, without polluting the stdout of the programs importing the library.

The Logger type is a small interface. Libraries should allow replacement of their Logger instances so importers can entirely replace flume if they wish. Alternately, importers can use flume to configure the library's log output, and/or redirect it into the overall program's log stream.


This package does not offer package level log functions, so you need to create a logger instance first: A common pattern is to create a single, package-wide logger, named after the package:

var log = flume.New("mypkg")

Then, write some logs:

log.Debug("created user", "username", "frank", "role", "admin")

Logs have a message, then matched pairs of key/value properties. Child loggers can be created and pre-seeded with a set of properties:

reqLogger := log.With("remoteAddr", req.RemoteAddr)

Expensive log events can be avoid by explicitly checking level:

if log.IsDebug() {
    log.Debug("created resource", "resource", resource.ExpensiveToString())

Loggers can be bound to context.Context, which is convenient for carrying per-transaction loggers (pre-seeded with transaction specific context) through layers of request processing code:

ctx = flume.WithLogger(ctx, log.With("transactionID", tid))
// ...later...
flume.FromContext(ctx).Info("Request handled.")

The standard Logger interface only supports 3 levels of log, DBG, INF, and ERR. This is inspired by this article: However, you can create instances of DeprecatedLogger instead, which support more levels.


There are several package level functions which reconfigure logging output. They control which levels are discarded, which fields are included in each log entry, and how those fields are rendered, and how the overall log entry is rendered (JSON, LTSV, colorized, etc).

To configure logging settings from environment variables, call the configuration function from main():


This reads the log configuration from the environment variable "FLUME" (the default, which can be overridden). The value is JSON, e.g.:


The properties of the config string:

    - "level": ERR, INF, or DBG.  The default level for all loggers.
    - "levels": A string configuring log levels for specific loggers, overriding the default level.
      See note below for syntax.
    - "development": true or false.  In development mode, the defaults for the other
      settings change to be more suitable for developers at a terminal (colorized, multiline, human
      readable, etc).  See note below for exact defaults.
    - "addCaller": true or false.  Adds call site information to log entries (file and line).
    - "encoding": json, ltsv, term, or term-color.  Configures how log entries are encoded in the output.
      "term" and "term-color" are multi-line, human-friendly
      formats, intended for terminal output.
    - "encoderConfig": a JSON object which configures advanced encoding settings, like how timestamps
      are formatted.  See docs for

        - "messageKey": the label of the message property of the log entry.  If empty, message is omitted.
        - "levelKey": the label of the level property of the log entry.  If empty, level is omitted.
        - "timeKey": the label of the timestamp of the log entry.  If empty, timestamp is omitted.
        - "nameKey": the label of the logger name in the log entry.  If empty, logger name is omitted.
        - "callerKey": the label of the logger name in the log entry.  If empty, logger name is omitted.
        - "lineEnding": the end of each log output line.
        - "levelEncoder": capital, capitalColor, color, lower, or abbr.  Controls how the log entry level
          is rendered.  "abbr" renders 3-letter abbreviations, like ERR and INF.
        - "timeEncoder": iso8601, millis, nanos, unix, or justtime.  Controls how timestamps are rendered.
			 "millis", "nanos", and "unix" are since UNIX epoch.  "unix" is in floating point seconds.
          "justtime" omits the date, and just prints the time in the format "15:04:05.000".
        - "durationEncoder": string, nanos, or seconds.  Controls how time.Duration values are rendered.
        - "callerEncoder": full or short.  Controls how the call site is rendered.
          "full" includes the entire package path, "short" only includes the last folder of the package.



These defaults are only applied if one of the configuration functions is called, like ConfigFromEnv(), ConfigString(), Configure(), or LevelsString(). Initially, all loggers are configured to discard everything, following flume's opinion that log packages should be silent unless spoken too. Ancillary to this: library packages should *not* call these functions, or configure logging levels or output in anyway. Only program entry points, like main() or test code, should configure logging. Libraries should just create loggers and log to them.

Development mode: if "development"=true, the defaults for the rest of the settings change, equivalent to:

  "encodingConfig": {

The "levels" value is a list of key=value pairs, configuring the level of individual named loggers. If the key is "*", it sets the default level. If "level" and "levels" both configure the default level, "levels" wins. Examples:

*            // set the default level to ALL, equivalent to {"level"="ALL"}
*=INF		// same, but set default level to INF
*,sql=WRN	// set default to ALL, set "sql" logger to WRN
*=INF,http=ALL	// set default to INF, set "http" to ALL
*=INF,http	// same as above.  If name has no level, level is set to ALL
*=INF,-http	// set default to INF, set "http" to OFF
http=INF		// leave default setting unchanged.


Most usages of flume will use its package functions. The package functions delegate to an internal instance of Factory, which a the logger registry. You can create and manage your own instance of Factory, which will be an isolated set of Loggers.


The implementation is a wrapper around zap. zap does levels, structured logs, and is very fast. zap doesn't do centralized, global configuration, so this package adds that by maintaining an internal registry of all loggers, and using the sync.atomic stuff to swap out levels and writers in a thread safe way.



err := flume.Configure(flume.Config{
	Development:  true,
	DefaultLevel: flume.DebugLevel,
	Encoding:     "ltsv",
if err != nil {
	fmt.Println("logging config failed:", err.Error())

log := flume.New("root")

log.Info("Hello World!")
log.Info("This entry has properties", "color", "red")
log.Debug("This is a debug message")
log.Error("This is an error message")
log.Info("This message has a multiline value", "essay", `Four score and seven years ago
our fathers brought forth on this continent, a new nation, 
conceived in Liberty, and dedicated to the proposition that all men are created equal.`)



Package Files


const (
	// OffLevel disables all logs
	OffLevel = Level(127)
	// DebugLevel should be used for low-level, non-production logs.  Typically intended only for developers.
	DebugLevel = Level(zapcore.DebugLevel)
	// InfoLevel should be used for production level logs.  Typically intended for end-users and developers.
	InfoLevel = Level(zapcore.InfoLevel)
	// ErrorLevel should be used for errors.  Generally, this should be reserved for events which truly
	// need to be looked at by an admin, and might be reported to an error-tracking system.
	ErrorLevel = Level(zapcore.ErrorLevel)


var Bright = ansi.ColorCode("default+b")

Bright is the color used for the message

var DefaultColors = Colors{
	Debug: ansi.ColorCode("cyan"),
	Info:  ansi.ColorCode("green+h"),
	Warn:  ansi.ColorCode("yellow+bh"),
	Error: ansi.ColorCode("red+bh"),

DefaultColors is the default instance of Colors, used as the default colors if a nil Colorizer is passed to NewColorizedConsoleEncoder.

var DefaultConfigEnvVars = []string{"FLUME"}

DefaultConfigEnvVars is a list of the environment variables that ConfigFromEnv will search by default.

var DefaultLogger = New("")

DefaultLogger is returned by FromContext if no other logger has been injected into the context.

var Dim = ansi.ColorCode("240")

Dim is the color used for context keys, time, and caller information

func AbbrLevelEncoder

func AbbrLevelEncoder(l zapcore.Level, enc zapcore.PrimitiveArrayEncoder)

AbbrLevelEncoder encodes logging levels to the strings in the log entries. Encodes levels as 3-char abbreviations in upper case.

encConfig := flume.EncoderConfig{}
encConfig.EncodeTime = flume.AbbrLevelEncoder

func ConfigFromEnv

func ConfigFromEnv(envvars ...string) error

ConfigFromEnv configures flume from environment variables. It should be called from main():

func main() {

It searches envvars for the first environment variable that is set, and attempts to parse the value.

If no environment variable is set, it silently does nothing.

If an environment variable with a value is found, but parsing fails, an error is printed to stdout, and the error is returned.

If envvars is empty, it defaults to DefaultConfigEnvVars.

func ConfigString

func ConfigString(s string) error

ConfigString configures the package level Factory. The string can either be a JSON-serialized Config object, or just a LevelsString (see Factory.LevelsString for format).

Note: this will reconfigure the logging levels for all loggers.

func Configure

func Configure(cfg Config) error

Configure configures the package level Factory from the settings in the Config object. See Config for details.

Note: this will reconfigure the logging levels for all loggers.

func JustTimeEncoder

func JustTimeEncoder(t time.Time, enc zapcore.PrimitiveArrayEncoder)

JustTimeEncoder is a timestamp encoder function which encodes time as a simple time of day, without a date. Intended for development and testing. Not good in a production system, where you probably need to know the date.

encConfig := flume.EncoderConfig{}
encConfig.EncodeTime = flume.JustTimeEncoder

func LogFuncWriter

func LogFuncWriter(l func(args ...interface{}), trimSpace bool) io.Writer

LogFuncWriter is a writer which writes to a logging function signature like that of testing.T.Log() and fmt/log.Println(). It can be used to redirect flumes *output* to some other logger.

SetOut(LogFuncWriter(fmt.Println, true))
SetOut(LogFuncWriter(t.Log, true))

func LoggerFuncWriter

func LoggerFuncWriter(l func(msg string, kvpairs ...interface{})) io.Writer

LoggerFuncWriter is a writer which writes lines to a logging function with a signature like that of flume.Logger's functions, like Info(), Debug(), and Error().

    ErrorLog: log.New(LoggerFuncWriter(flume.New("http").Error), "", 0),

func SetAddCaller

func SetAddCaller(b bool)

SetAddCaller enables/disables call site logging on the package-level Factory

func SetDefaultLevel

func SetDefaultLevel(l Level)

SetDefaultLevel sets the default log level on the package-level Factory.

func SetDevelopmentDefaults

func SetDevelopmentDefaults() error

SetDevelopmentDefaults sets useful default settings on the package-level Factory which are appropriate for a development setting. Default log level is set to INF, all loggers are reset to the default level, call site information is logged, and the encoder is a colorized, multi-line friendly console encoder with a simplified time stamp format.

func SetEncoder

func SetEncoder(e Encoder)

SetEncoder sets the encoder for the package-level Factory

func SetLevel

func SetLevel(name string, l Level)

SetLevel sets a log level for a named logger on the package-level Factory.

func SetOut

func SetOut(w io.Writer) func()

SetOut sets the output writer for all logs produced by the default factory. Returns a function which sets the output writer back to the prior setting.

func WithLogger

func WithLogger(ctx context.Context, l Logger) context.Context

WithLogger returns a new context with the specified logger injected into it.

type Colorizer

type Colorizer interface {
	Level(l Level) string

Colorizer returns ansi escape sequences for the colors for each log level. See Colors for a default implementation.

type Colors

type Colors struct {
	Debug, Info, Warn, Error string

Colors is an implementation of the Colorizer interface, which assigns colors to the default log levels.

func (*Colors) Level

func (c *Colors) Level(l Level) string

Level implements Colorizer

type Config

type Config struct {
	// DefaultLevel is the default log level for all loggers not
	// otherwise configured by Levels.  Defaults to Info.
	DefaultLevel Level `json:"level" yaml:"level"`
	// Levels configures log levels for particular named loggers.  See
	// LevelsString for format.
	Levels string `json:"levels" yaml:"levels"`
	// AddCaller annotates logs with the calling function's file
	// name and line number. Defaults to true when the Development
	// flag is set, false otherwise.
	AddCaller *bool `json:"addCaller" yaml:"addCaller"`
	// Encoding sets the logger's encoding. Valid values are "json",
	// "console", "ltsv", "term", and "term-color".
	// Defaults to "term-color" if development is true, else
	// "ltsv"
	Encoding string `json:"encoding" yaml:"encoding"`
	// Development toggles the defaults used for the other
	// settings.  Defaults to false.
	Development bool `json:"development" yaml:"development"`
	// EncoderConfig sets options for the chosen encoder. See
	// EncoderConfig for details.  Defaults to NewEncoderConfig() if
	// Development is false, otherwise defaults to NewDevelopmentEncoderConfig().
	EncoderConfig *EncoderConfig `json:"encoderConfig" yaml:"encoderConfig"`

Config offers a declarative way to configure a Factory.

The same things can be done by calling Factory methods, but Configs can be unmarshaled from JSON, making it a convenient way to configure most logging options from env vars or files, i.e.:

err := flume.ConfigString(os.Getenv("flume"))

Configs can be created and applied programmatically:

err := flume.Configure(flume.Config{})

Defaults are appropriate for a JSON encoded production logger:

- LTSV encoder - full timestamps - default log level set to INFO - call sites are not logged

An alternate set of defaults, more appropriate for development environments, can be configured with `Config{Development:true}`:

err := flume.Configure(flume.Config{Development:true})

- colorized terminal encoder - short timestamps - call sites are logged

err := flume.Configure(flume.Config{Development:true})

Any of the other configuration options can be specified to override the defaults.

Note: If configuring the EncoderConfig setting, if any of the *Key properties are omitted, that entire field will be omitted.

func (*Config) SetAddCaller

func (c *Config) SetAddCaller(b bool)

SetAddCaller sets the Config's AddCaller flag.

func (*Config) UnsetAddCaller

func (c *Config) UnsetAddCaller()

UnsetAddCaller unsets the Config's AddCaller flag (reverting to defaults).

type Core

type Core struct {
	// contains filtered or unexported fields

Core is the concrete implementation of Logger. It has some additional lower-level methods which can be used by other logging packages which wrap flume, to build alternate logging interfaces.

func NewCore

func NewCore(name string, options ...CoreOption) *Core

NewCore returns a new Core

func (*Core) Debug

func (l *Core) Debug(msg string, args ...interface{})

Debug logs at DBG level. args should be alternative keys and values. keys should be strings.

func (*Core) Error

func (l *Core) Error(msg string, args ...interface{})

Error logs at ERR level. args should be alternative keys and values. keys should be strings.

func (*Core) Info

func (l *Core) Info(msg string, args ...interface{})

Info logs at INF level. args should be alternative keys and values. keys should be strings.

func (*Core) IsDebug

func (l *Core) IsDebug() bool

IsDebug returns true if DBG level is enabled.

func (*Core) IsEnabled

func (l *Core) IsEnabled(lvl Level) bool

IsEnabled returns true if the specified level is enabled.

func (*Core) IsInfo

func (l *Core) IsInfo() bool

IsDebug returns true if INF level is enabled

func (*Core) Log

func (l *Core) Log(lvl Level, template string, fmtArgs, context []interface{}) bool

Log is the core logging method, used by the convenience methods Debug(), Info(), and Error().

Returns true if the log was actually logged.

AddCaller option will report the caller of this method. If wrapping this, be sure to use the AddCallerSkip option.

func (*Core) With

func (l *Core) With(args ...interface{}) Logger

With returns a new Logger with some context baked in. All entries logged with the new logger will include this context.

args should be alternative keys and values. keys should be strings.

reqLogger := l.With("requestID", reqID)

func (*Core) WithArgs

func (l *Core) WithArgs(args ...interface{}) *Core

WithArgs is the same as With() but returns the concrete type. Useful for other logging packages which wrap this one.

type CoreOption

type CoreOption interface {
	// contains filtered or unexported methods

An CoreOption configures a Core.

func AddCallerSkip

func AddCallerSkip(skip int) CoreOption

AddCallerSkip increases the number of callers skipped by caller annotation (as enabled by the AddCaller option). When building wrappers around a Core, supplying this CoreOption prevents Core from always reporting the wrapper code as the caller.

type Encoder

type Encoder zapcore.Encoder

Encoder serializes log entries. Re-exported from zap for now to avoid exporting zap.

func NewColorizedConsoleEncoder

func NewColorizedConsoleEncoder(cfg *EncoderConfig, colorizer Colorizer) Encoder

NewColorizedConsoleEncoder creates a console encoder, like NewConsoleEncoder, but colors the text with ansi escape codes. `colorize` configures which colors to use for each level.

If `colorizer` is nil, it will default to DefaultColors.

`` is a convenient package for getting color codes, e.g.:


func NewConsoleEncoder

func NewConsoleEncoder(cfg *EncoderConfig) Encoder

NewConsoleEncoder creates an encoder whose output is designed for human - rather than machine - consumption. It serializes the core log entry data (message, level, timestamp, etc.) in a plain-text format. The context is encoded in LTSV.

Note that although the console encoder doesn't use the keys specified in the encoder configuration, it will omit any element whose key is set to the empty string.

func NewJSONEncoder

func NewJSONEncoder(cfg *EncoderConfig) Encoder

NewJSONEncoder just hides the zap json encoder, to avoid exporting zap

func NewLTSVEncoder

func NewLTSVEncoder(cfg *EncoderConfig) Encoder

NewLTSVEncoder creates a fast, low-allocation LTSV encoder.

type EncoderConfig

type EncoderConfig zapcore.EncoderConfig

EncoderConfig captures the options for encoders. Type alias to avoid exporting zap.

func NewDevelopmentEncoderConfig

func NewDevelopmentEncoderConfig() *EncoderConfig

NewDevelopmentEncoderConfig returns an EncoderConfig which is intended for local development.

func NewEncoderConfig

func NewEncoderConfig() *EncoderConfig

NewEncoderConfig returns an EncoderConfig with default settings.

func (*EncoderConfig) UnmarshalJSON

func (enc *EncoderConfig) UnmarshalJSON(b []byte) error

UnmarshalJSON implements json.Marshaler

type Factory

type Factory struct {
	// contains filtered or unexported fields

Factory is a log management core. It spawns loggers. The Factory has methods for dynamically reconfiguring all the loggers spawned from Factory.

The flume package has mirrors of most of the functions which delegate to a default, package-level factory.

func NewFactory

func NewFactory() *Factory

NewFactory returns a factory. The default level is set to OFF (all logs disabled)

func (*Factory) Configure

func (r *Factory) Configure(cfg Config) error

Configure uses a serializable struct to configure most of the options. This is useful when fully configuring the logging from an env var or file.

The zero value for Config will set defaults for a standard, production logger:

See the Config docs for details on settings.

func (*Factory) LevelsString

func (r *Factory) LevelsString(s string) error

LevelsString reconfigures the log level for all loggers. Calling it with an empty string will reset the default level to info, and reset all loggers to use the default level.

The string can contain a list of directives, separated by commas. Directives can set the default log level, and can explicitly set the log level for individual loggers.


- Default level: Use the `*` directive to set the default log level. Examples:

    * 	// set the default log level to debug
    -* // set the default log level to off

If the `*` directive is omitted, the default log level will be set to info.

- Logger level: Use the name of the logger to set the log level for a specific

logger.  Examples:

    http		// set the http logger to debug
    -http		// set the http logger to off
    http=INF	// set the http logger to info

Multiple directives can be included, separated by commas. Examples:

http         	// set http logger to debug
http,sql     	// set http and sql logger to debug
*,-http,sql=INF	// set the default level to debug, disable the http logger,
                 // and set the sql logger to info

func (*Factory) NewCore

func (r *Factory) NewCore(name string, options ...CoreOption) *Core

NewCore returns a new Core.

func (*Factory) NewLogger

func (r *Factory) NewLogger(name string) Logger

NewLogger returns a new Logger

func (*Factory) SetAddCaller

func (r *Factory) SetAddCaller(b bool)

SetAddCaller enables adding the logging callsite (file and line number) to the log entries.

func (*Factory) SetDefaultLevel

func (r *Factory) SetDefaultLevel(l Level)

SetDefaultLevel sets the default log level for all loggers which don't have a specific level assigned to them

func (*Factory) SetEncoder

func (r *Factory) SetEncoder(e Encoder)

SetEncoder sets the encoder for all loggers created by (in the past or future) this factory.

func (*Factory) SetLevel

func (r *Factory) SetLevel(name string, l Level)

SetLevel sets the log level for a particular named logger. All loggers with this same are affected, in the past or future.

func (*Factory) SetOut

func (r *Factory) SetOut(w io.Writer) func()

SetOut sets the output writer for all logs produced by this factory. Returns a function which sets the output writer back to the prior setting.

type Level

type Level zapcore.Level

Level is a log level

func (*Level) Get

func (l *Level) Get() interface{}

Get implements flag.Getter

func (Level) MarshalText

func (l Level) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler

func (*Level) Set

func (l *Level) Set(s string) error

Set implements flags.Value

func (Level) String

func (l Level) String() string

String implements stringer and a few other interfaces.

func (*Level) UnmarshalText

func (l *Level) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler

type Logger

type Logger interface {
	Debug(msg string, args ...interface{})
	Info(msg string, args ...interface{})
	Error(msg string, args ...interface{})

	IsDebug() bool
	IsInfo() bool

	// With creates a new Logger with some context already attached.  All
	// entries logged with the child logger will include this context.
	With(args ...interface{}) Logger

Logger is the basic logging interface. Construct instances of Logger with a Factory, or with the package functions (which use a package level Factory).

func FromContext

func FromContext(ctx context.Context) Logger

FromContext returns a logger from the context. If the context doesn't contain a logger, the DefaultLogger will be returned.

func New

func New(name string) Logger

New creates a new Logger

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to identifier