logging

package
v0.0.0-...-8cba18c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 1, 2021 License: MIT Imports: 18 Imported by: 6

README

go-common/logging

This repository contains the go-common/logging library.

To install:

go get github.com/ugorji/go-common/logging

Package Documentation

Package logging provides a precise logging framework.

LogRecord

A LogRecord can be sent by the application. It contains a level, message, timestamp, target (which subsystem the message came from) and PC information (file, line number).

Filter

A Filter can determine whether a LogRecord should be accepted or not.

Handler

A Handler can persist a LogRecord as it deems fit. Each Handler can have a (set of) Filters configured. The Handler will log a record iff all Filters accept it. The lifecycle of a Handler is: init, open, log OR flush ..., close.

Multiple Handles can be configured on a running system.

All Handles will write to an in-memory buffer which can typically hold up to 20 log records. When the buffer cannot add another log record, it is flushed.

Once a Handler has been created for a given name, it cannot be replaced.

This allows you to have different Handles who can log based on different criteria e.g. stackdriver only logs error and severe messages from web container at night.

NOTE: The framework tries to ensure that only one handler is writing to standard error or output streams. This allows us to handle backtraces, as we can ensure that the Handler writing to the standard error stream is flushed first.

Formatter

A Formatter can take a LogRecord and convert it to a string for easy persisting.

A Handler may have a Formatter determine how the LogRecord should be persisted.

Logger

A Logger can be retrieved for any subsystem (target). This can be retrieved explicitly by name or implicitly (where we use the package path to infer the subsystem).

Note that a Logger is only lazily initialized on first use, not on declaration/assignment. This ensures that the logging framework is initialized before the first log message is called.

A Logger can also have a Filter attached to it, which determines whether to send the LogRecord to the Handlers to persist.

To allow short-circuiting creating a LogRecord, a Logger has a minimum log level defined, so it can bypass logging quickly.

There is no hierachy of Loggers.

Once a Logger has been retrieved for a given subsystem, it cannot be replaced.

Customizing Target Subsystem name in Record

By default, the target subsystem is named according to the name of the package, or whatever name is explicitly specified by calling NamedLogger.

However, it is possible and sometimes necessary to customize it. For example, you have a proxy that handles many different domain names, and you want each to have their own grouping of log records. You can specify a function that will take the package name and give a richer Target name in the Record.

For simplicity, this "extra" information can be passed into the context by the running application using the key: logging.SubsystemExtraContextKey. The value retrieved from the key, by default, will be a string, or a func() string, or a func(string) string, or a value which can be coerced to a string via fmt.Sprintf.

Backtraces

A Logger can be configured to write a backtrace to standard error stream at the point that a Log is being generated on a given line in a given go file, and at a Level >= the Level for populating the file/line PC info in the Record.

File and Line PC information in logs

Each log Record will contain File and Line PC information if the Level == DEBUG or if Level >= populatePCLevel (which is WARNING by default but can be configured).

Flushing

The framework has a timer that will flush each Handler when triggered. By default, the timer is triggered every 5 seconds. This can be configured.

Each Handle is expected to use a buffer to cache its output. When the buffer is full, it is flushed. Also, the framework timer will flush so that you always see log persisted within the timer schedule duration.

Whenever a Record with level NOTICE or higher is logged, the system flushes immediately.

Framework Initialization

The logging framework is typically initialized by the running application ie early in its main method.

To make this easier, the logging framework provides an initialization function taking Config objects that can be configured via json.

The first time that a LogRecord is to be published, it will ensure that the logging framework is initialized. If it was not initialized apriori, then it is initialized to have a single Handler that writes a single human readable line for each log record with minimum level of NOTICE (can be configured).

Note that this affect the whole process.

Framework Runtime

At startup, logging is closed.

It must be started explicitly by calling logging.Open().

Default Configuration

By default, the logging framework is configured with:

  • FlushInterval: 5 seconds - BufferSize: 32KB (max of 1MB) -

PopulatePCLevel: WARNING - MinLevel: NOTICE - SubsystemFunc: func(context.Context, string)

Levels based on syslog

The levels are roughly model'ed after syslog. We however omit ALERT and EMERGENCY, as we expect these are just finer forms of a SEVERE issue, and developers typically cannot separate the two.

We include ALWAYS and OFF for configuration of MinimumLogLevel only. This way, it is easy to configure that ALL records should be logged, or no records should be logged.

Context

All the logging methods (.Trace, .Debug, .Info, etc) take a Context as the first parameter.

This allows us grab information from the context where appropriate e.g. App Engine, HTTP Request, etc.

Debugging

By default, we include the program counter file/line info when logging at DEBUG level and at >= WARNING level. The WARNING level can be configured.

Typical Usage

Initialization:

    logging.Addhandler(...)...
    logging.Addhandler("", ...)
    logging.AddLogger("", ...)
    logging.Open(...)
    // ...
    logging.Close()

Usage:

    // make log a package-level variable
    var log = logging.PkgLogger()
    // Use these within your function calls
    log.Info(ctx, formatString, params...)
    log.Info(nil, formatString, params...)

Exported Package API

var FilterRejectedErr = errorutil.String("logging: log level lower than logger threshold") ...
var AppContextKey = new(int)
var CorrelationIDContextKey = new(int)
var ErrorContextKey = new(int)
var HTTPRequestContextKey = new(int)
var SubsystemExtraContextKey = new(int)
func AddHandler(name string, f Handler) (err error)
func AddLogger(name string, minLevel Level, backtraces []Backtrace, handlerNames []string)
func BasicInit(names []string, c Config) (err error)
func Close() error
func Flush() error
func Open(c Config) error
func Reopen() error
func NewHandlerFile(fname string, fmter Formatter, ff Filter) (h *handlerWriter)
func NewHandlerWriter(w io.Writer, fmter Formatter, ff Filter) (h *handlerWriter)
type Backtrace struct{ ... }
type CSVFormatter struct{}
type Config struct{ ... }
type Filter interface{ ... }
type FilterFunc func(ctx context.Context, r *Record) error
    func FilterByLevel(level Level) FilterFunc
type Flags struct{ ... }
type Formatter interface{ ... }
type Handler interface{ ... }
type HandlerFunc func(ctx context.Context, r *Record) error
type HumanFormatter struct{ ... }
type JSONFormatter struct{}
type Level uint8
    const DEBUG Level = 100 + iota ...
    func ParseLevel(s string) (l Level)
type Logger struct{ ... }
    func NamedLogger(name string) *Logger
    func PkgLogger() *Logger
type Noop struct{}
type Record struct{ ... }

Documentation

Overview

Package logging provides a precise logging framework.

LogRecord

A LogRecord can be sent by the application. It contains a level, message, timestamp, target (which subsystem the message came from) and PC information (file, line number).

Filter

A Filter can determine whether a LogRecord should be accepted or not.

Handler

A Handler can persist a LogRecord as it deems fit. Each Handler can have a (set of) Filters configured. The Handler will log a record iff all Filters accept it. The lifecycle of a Handler is: init, open, log OR flush ..., close.

Multiple Handles can be configured on a running system.

All Handles will write to an in-memory buffer which can typically hold up to 20 log records. When the buffer cannot add another log record, it is flushed.

Once a Handler has been created for a given name, it cannot be replaced.

This allows you to have different Handles who can log based on different criteria e.g. stackdriver only logs error and severe messages from web container at night.

NOTE: The framework *tries* to ensure that only one handler is writing to standard error or output streams. This allows us to handle backtraces, as we can ensure that the Handler writing to the standard error stream is flushed first.

Formatter

A Formatter can take a LogRecord and convert it to a string for easy persisting.

A Handler *may* have a Formatter determine how the LogRecord should be persisted.

Logger

A Logger can be retrieved for any subsystem (target). This can be retrieved explicitly by name or implicitly (where we use the package path to infer the subsystem).

Note that a Logger is only lazily initialized on first use, not on declaration/assignment. This ensures that the logging framework is initialized before the first log message is called.

A Logger can also have a Filter attached to it, which determines whether to send the LogRecord to the Handlers to persist.

To allow short-circuiting creating a LogRecord, a Logger has a minimum log level defined, so it can bypass logging quickly.

There is no hierachy of Loggers.

Once a Logger has been retrieved for a given subsystem, it cannot be replaced.

Customizing Target Subsystem name in Record

By default, the target subsystem is named according to the name of the package, or whatever name is explicitly specified by calling NamedLogger.

However, it is possible and sometimes necessary to customize it. For example, you have a proxy that handles many different domain names, and you want each to have their own grouping of log records. You can specify a function that will take the package name and give a richer Target name in the Record.

For simplicity, this "extra" information can be passed into the context by the running application using the key: `logging.SubsystemExtraContextKey`. The value retrieved from the key, by default, will be a string, or a func() string, or a func(string) string, or a value which can be coerced to a string via fmt.Sprintf.

Backtraces

A Logger can be configured to write a backtrace to standard error stream at the point that a Log is being generated on a given line in a given go file, and at a Level >= the Level for populating the file/line PC info in the Record.

File and Line PC information in logs

Each log Record will contain File and Line PC information if the Level == DEBUG or if Level >= populatePCLevel (which is WARNING by default but can be configured).

Flushing

The framework has a timer that will flush each Handler when triggered. By default, the timer is triggered every 5 seconds. This can be configured.

Each Handle is expected to use a buffer to cache its output. When the buffer is full, it is flushed. Also, the framework timer will flush so that you always see log persisted within the timer schedule duration.

Whenever a Record with level NOTICE or higher is logged, the system flushes immediately.

Framework Initialization

The logging framework is typically initialized by the running application ie early in its main method.

To make this easier, the logging framework provides an initialization function taking Config objects that can be configured via json.

The first time that a LogRecord is to be published, it will ensure that the logging framework is initialized. If it was not initialized apriori, then it is initialized to have a single Handler that writes a single human readable line for each log record with minimum level of NOTICE (can be configured).

Note that this affect the whole process.

Framework Runtime

At startup, logging is closed.

It must be started explicitly by calling logging.Open().

Default Configuration

By default, the logging framework is configured with:

- FlushInterval: 5 seconds - BufferSize: 32KB (max of 1MB) - PopulatePCLevel: WARNING - MinLevel: NOTICE - SubsystemFunc: func(context.Context, string)

Levels based on syslog

The levels are roughly model'ed after syslog. We however omit ALERT and EMERGENCY, as we expect these are just finer forms of a SEVERE issue, and developers typically cannot separate the two.

We include ALWAYS and OFF for configuration of MinimumLogLevel only. This way, it is easy to configure that ALL records should be logged, or no records should be logged.

Context

All the logging methods (.Trace, .Debug, .Info, etc) take a Context as the first parameter.

This allows us grab information from the context where appropriate e.g. App Engine, HTTP Request, etc.

Debugging

By default, we include the program counter file/line info when logging at DEBUG level and at >= WARNING level. The WARNING level can be configured.

Typical Usage

Initialization: 

logging.Addhandler(...)...
logging.Addhandler("", ...)
logging.AddLogger("", ...)
logging.Open(...)
// ...
logging.Close()

Usage: 

// make log a package-level variable
var log = logging.PkgLogger()
// Use these within your function calls
log.Info(ctx, formatString, params...)
log.Info(nil, formatString, params...)

Index

Constants

This section is empty.

Variables

View Source 
var (
	FilterRejectedErr     = errorutil.String("logging: log level lower than logger threshold")
	EmptyMessageErr       = errorutil.String("logging: empty message")
	NoWriterForHandlerErr = errorutil.String("logging: no writer for handler")

	OnlyOneStderrHandlerErr = errorutil.String("logging: only one stderr handler can exist")
	OnlyOneStdoutHandlerErr = errorutil.String("logging: only one stdout handler can exist")
)
View Source 
var AppContextKey = new(int)

AppContextKey is the context.Context key used to store an app.Context

View Source 
var CorrelationIDContextKey = new(int)

CorrelationIDContextKey is the context.Context key used to track multiple records as part of a request flow

View Source 
var ErrorContextKey = new(int)

ErrorContextKey is the context.Context key used to store an error

View Source 
var HTTPRequestContextKey = new(int)

HTTPRequestKey is the context.Context key used to store a http.Request (clone)

View Source 
var SubsystemExtraContextKey = new(int)

SubsystemExtraContextKey is the context.Context key used to obtain a subsystem prefix.

Different SubsystemFunc can handle it differently. By default, the value retrieved is a value which can be coerced to a string. e.g. func() string, func(string) string, a string, or any value passed to fmt.Sprintf("%v", v)

Functions

func AddHandler 

func AddHandler(name string, f Handler) (err error)

AddHandler will bind a handler to a given name, iff no handler is bound to that name.

Note that a Handler is bound one time only.

func AddLogger 

func AddLogger(name string, minLevel Level, backtraces []Backtrace, handlerNames []string)

AddLogger will register a Logger for a given name if not existing.

If name="" and it is not bound to any logger, it is created and will serve as default for minLevel and handlerNames for loggers not explicitly added.

func BasicInit 

func BasicInit(names []string, c Config) (err error)

BasicInit is used to simply initialize the logging subsystem.

It creates a Handler for each name, logging using the HumanFormatter.

  • If the name is "" or <stderr>, then it logs to standard error stream
  • Else If the name is <stdout>, then it logs to standard output stream
  • Else it logs to a file with the name given

func Close 

func Close() error

func Flush 

func Flush() error

func NewHandlerFile 

func NewHandlerFile(fname string, fmter Formatter, ff Filter) (h *handlerWriter)

NewHandlerFile returns an un-opened handler.

func NewHandlerWriter 

func NewHandlerWriter(w io.Writer, fmter Formatter, ff Filter) (h *handlerWriter)

NewHandlerWriter returns an un-opened handler.

func Open 

func Open(c Config) error

func Open(flush time.Duration, buffer uint16, minLevel, populatePCLevel Level) error {

func Reopen 

func Reopen() error

Reopen will close the system if opened, an then Open it using the last configured values (which may be the defaults).

Types

type Backtrace 

type Backtrace struct {
	File string
	Line uint32
}

type CSVFormatter 

type CSVFormatter struct{}

func (CSVFormatter) Format 

func (h CSVFormatter) Format(ctx context.Context, r *Record, seqId string, w io.Writer) (err error)

type Config 

type Config struct {
	FlushInterval   time.Duration
	BufferSize      int
	MinLevel        Level
	PopulatePCLevel Level
	SubsystemFunc   func(ctx context.Context, name string) string
}

func (*Config) CopySanitize 

func (y *Config) CopySanitize(c Config)

type Filter 

type Filter interface {
	Accept(ctx context.Context, r *Record) error
}

type FilterFunc 

type FilterFunc func(ctx context.Context, r *Record) error

func FilterByLevel 

func FilterByLevel(level Level) FilterFunc

func (FilterFunc) Accept 

func (f FilterFunc) Accept(ctx context.Context, r *Record) error

type Flags 

type Flags struct {
	Files string
	Config
}

func (*Flags) Flags 

func (p *Flags) Flags(flags *flag.FlagSet)

type Formatter 

type Formatter interface {
	// Format will write the record into the writer.
	//
	// It ensures that a new record can be written right after.
	// Consequently, where appropriate, new lines or other record separators may be written
	// after each record.
	Format(ctx context.Context, r *Record, seqId string, w io.Writer) error
}

type Handler 

type Handler interface {
	// Name() string
	Handle(ctx context.Context, r *Record) error
	Filter() Filter
	Flush() error
	Close() error
	Open(buffer uint16) error
}

type HandlerFunc 

type HandlerFunc func(ctx context.Context, r *Record) error

func (HandlerFunc) Close 

func (f HandlerFunc) Close() error

func (HandlerFunc) Filter 

func (f HandlerFunc) Filter() Filter

func (HandlerFunc) Flush 

func (f HandlerFunc) Flush() error

func (HandlerFunc) Handle 

func (f HandlerFunc) Handle(ctx context.Context, r *Record) error

func (HandlerFunc) Open 

func (f HandlerFunc) Open(buffer uint16) error

type HumanFormatter 

type HumanFormatter struct {
	ANSIColor bool
}

func (HumanFormatter) Format 

func (h HumanFormatter) Format(ctx context.Context, r *Record, seqId string, w io.Writer) (err error)

type JSONFormatter 

type JSONFormatter struct{}

func (JSONFormatter) Format 

func (h JSONFormatter) Format(ctx context.Context, r *Record, seqId string, w io.Writer) error

type Level 

type Level uint8

Level is an int representing the log levels.

The levels are roughly model'ed around syslog. 

const (
	DEBUG   Level = 100 + iota // Debug or trace information.
	INFO                       // Routine information, such as ongoing status or performance
	NOTICE                     // Normal but significant events, such as start up, shut down, config
	WARNING                    // Warning events might cause problems
	ERROR                      // Error events are likely to cause problems
	SEVERE                     // Critical events cause more severe problems or brief outages

	ALWAYS Level = 1             // Config only: specify that ALL messages are logged
	OFF    Level = math.MaxUint8 // Config only: specify that NO messages are logged
)

func ParseLevel 

func ParseLevel(s string) (l Level)

func (*Level) CodecDecodeSelf 

func (l *Level) CodecDecodeSelf(d *codec.Decoder)

func (Level) CodecEncodeSelf 

func (l Level) CodecEncodeSelf(e *codec.Encoder)

func (Level) ShortString 

func (l Level) ShortString() byte

func (Level) String 

func (l Level) String() string

type Logger 

type Logger struct {
	// contains filtered or unexported fields
}

func NamedLogger 

func NamedLogger(name string) *Logger

func PkgLogger 

func PkgLogger() *Logger

func (*Logger) Debug 

func (l *Logger) Debug(ctx context.Context, message string, params ...interface{})

func (*Logger) Error 

func (l *Logger) Error(ctx context.Context, message string, params ...interface{})

Error will log a message at ERROR level.

If the last parameter is a non-nil error value, it will be presented specially to the Handlers for possible special consideration.

func (*Logger) IfError 

func (l *Logger) IfError(ctx context.Context, err error, message string, params ...interface{}) error

IfError logs at ERROR level iff err IS NOT nil. It is a no-op if err is nil.

func (*Logger) Info 

func (l *Logger) Info(ctx context.Context, message string, params ...interface{})

func (*Logger) Log 

func (l *Logger) Log(ctx context.Context, calldepth uint8, level Level, message string, params ...interface{}) error

Log is the all-encompassing function that can be used by helper log functions in packages without losing caller positon.

A nil *Logger does nothing - equivalent to a no-op.

Example: 

func logT(message string, params ...interface{}) {
  logging.Log(nil, 1, level.TRACE, message, params...)
}

func (*Logger) Notice 

func (l *Logger) Notice(ctx context.Context, message string, params ...interface{})

func (*Logger) Severe 

func (l *Logger) Severe(ctx context.Context, message string, params ...interface{})

func (*Logger) Warning 

func (l *Logger) Warning(ctx context.Context, message string, params ...interface{})

type Noop 

type Noop struct{}

Noop Handler and Filter.

func (Noop) Accept 

func (n Noop) Accept(ctx context.Context, r *Record) error

func (Noop) Handle 

func (n Noop) Handle(ctx context.Context, r *Record) error

type Record 

type Record struct {
	Target      string    `codec:"s"`
	ProgramFile string    `codec:"f"`
	ProgramFunc string    `codec:"c"`
	Message     string    `codec:"m"`
	Time        time.Time `codec:"t"`
	ProgramLine uint32    `codec:"n"`
	Level       Level     `codec:"l"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.