tlog

package module
v0.21.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 26, 2023 License: MIT Imports: 29 Imported by: 0

README

Documentation Go workflow CircleCI codecov Go Report Card GitHub tag (latest SemVer)

tlog

At least it is a logger, but it is much more than that. It is an observability events system. Event is a log or tracing message, tracing span start or finish, metric value, or anything you need. Tons of work has been done to make it effective yet comfortable to use. The events are encoded in a machine-readable format to be processed in any way, instant or later. Events could be printed as logs, combined to build distributed traces, filtered and sent to an alerting service, processed and analyzed, and more.

tlog is a new way of instrumentation. Log once use smart.

Explore examples and extensions.

Status

The logging API is pretty solid. Now I'm working mostly on backend parts, web interface, integrations.

Quick Start

Logger

tlog.Printf("just like log.Printf")

tlog.Printw("but structured is", "much", "better")

type Req struct {
	Start time.Time
	Path  string
}

tlog.Printw("any value type is", "supported", &Req{Start: time.Now(), Path: "/resource/path"})

l := tlog.New(ioWriter)
l.Printw("yet another logger, seriously?")

Debug Topics Instead of Log Levels

No need to choose between tons of unrelated Debug logs and scant Info logs. Each event can be filtered precisely and filter can be changed at runtime.

tlog.SetVerbosity("rawdb,dump_request")

tlog.V("rawdb").Printw("make db query", "query", query) // V is inspired by glog.V

if tlog.If("dump_request") {
	// some heavy calculations could also be here
	tlog.Printw("full request data", "request", request)
}

if tlog.If("full_token") {
	tlog.Printw("db token", "token", token)
} else {
	tlog.Printw("db token", "token", token.ID)
}

Filtering is very flexible. You can select topics, functions, types, files, packages, topics in locations. You can select all in the file and then unselect some functions, etc.

Traces

Traces are vital if you have simultaneous requests or distributed request propagation. So they integrated into the logger to have the best experience.

func ServeRequest(req *Request) {
	span := tlog.Start("request_root", "client", req.RemoteAddr, "path", req.URL.Path)
	defer span.Finish()

	ctx := tlog.ContextWithSpan(req.Context(), span)

	doc, err := loadFromDB(ctx, req.URL.Path)
	// if err ...

	_ = doc
}

func loadFromDB(ctx context.Context, doc string) (err error) {
	parent := tlog.SpanFromContext(ctx)
	span := parent.V("dbops").Spawn("load_from_db", "doc", doc)
	defer func() {
		span.Finish("err", err) // record result error
	}()

	span.Printw("prepare query")
	// ...

	if dirtyPages > tooMuch {
		// record event to the local span or to the parent if the local was not selected
		span.Or(parent).Printw("too much of dirty pages", "durty_pages", dirtyPages,
			tlog.KeyLogLevel, tlog.Warn)
	}
}

Trace events are the same to log events, except they have IDs. You do not need to add the same data to trace attributes and write them to logs. It's the same!

Data Format

Events are just key-value associative arrays. All keys are optional, any can be added. Some keys have special meaning, like event timestamp or log level. But it's only a convention; representational parts primarily use it: console pretty text formatter moves time to the first column, for example.

The default format is a machine readable CBOR-like binary format. And the logger backend is just io.Writer. Text, JSON, Logfmt converters are provided. Any other can be implemented.

There is also a special compression format: as fast and efficient as snappy yet safe in a sense that each event (or batch write) emits single Write to the file (io.Writer actually).

Performance

Performance was in mind from the very beginning. The idea is to emit as many events as you want and not to pay for that by performance. In a typical efficient application CPU profile, the logger takes only 1-3% of CPU usage with no events economy. Almost all allocations were eliminated. That means less work is done, no garbage collector pressure, and lower memory usage.

Documentation

Index

Constants

View Source 
const (
	Ldate = 1 << iota
	Ltime
	Lseconds
	Lmilliseconds
	Lmicroseconds
	Lshortfile
	Llongfile
	Ltypefunc // pkg.(*Type).Func
	Lfuncname // Func
	LUTC
	Lloglevel // log level

	LstdFlags = Ldate | Ltime
	LdetFlags = Ldate | Ltime | Lmicroseconds | Lshortfile | Lloglevel

	Lnone = 0
)
View Source 
const (
	WireLabel = tlwire.SemanticTlogBase + iota
	WireID
	WireMessage
	WireEventKind
	WireLogLevel

	SemanticUserBase
)
View Source 
const KeyAuto = ""

Variables

View Source 
var (
	ResetColor = Color(0)

	DefaultColorScheme = ColorScheme{
		TimeColor:       Color(90),
		TimeChangeColor: Color(38, 5, 244, 1),
		FileColor:       Color(90),
		FuncColor:       Color(90),
		KeyColor:        Color(36),
		LevelColor: struct {
			Info  []byte
			Warn  []byte
			Error []byte
			Fatal []byte
			Debug []byte
		}{
			Info:  Color(90),
			Warn:  Color(31),
			Error: Color(31, 1),
			Fatal: Color(31, 1),
			Debug: Color(90),
		},
	}
)
View Source 
var (
	Stdout = os.Stdout
	Stderr = os.Stderr
)
View Source 
var (
	KeySpan      = "_s"
	KeyParent    = "_p"
	KeyTimestamp = "_t"
	KeyElapsed   = "_e"
	KeyCaller    = "_c"
	KeyMessage   = "_m"
	KeyEventKind = "_k"
	KeyLogLevel  = "_l"
)

Predefined keys.

View Source 
var (
	None          = RawMessage{tlwire.Special | tlwire.None}
	Nil           = RawMessage{tlwire.Special | tlwire.Nil}
	Break         = RawMessage{tlwire.Special | tlwire.Break}
	NextAsHex     = Modify{tlwire.Semantic | tlwire.Hex}
	NextAsMessage = Modify{tlwire.Semantic | WireMessage}
	NextAsType    = FormatNext("%T")
)
View Source 
var AutoLabels = map[string]func() interface{}{
	"_hostname":   func() interface{} { return Hostname() },
	"_user":       func() interface{} { return User() },
	"_os":         func() interface{} { return runtime.GOOS },
	"_arch":       func() interface{} { return runtime.GOARCH },
	"_numcpu":     func() interface{} { return fmt.Sprintf("%v", runtime.NumCPU()) },
	"_gomaxprocs": func() interface{} { return fmt.Sprintf("%v", runtime.GOMAXPROCS(0)) },
	"_goversion":  func() interface{} { return runtime.Version },
	"_pid": func() interface{} {
		return os.Getpid()
	},
	"_timezone": func() interface{} {
		n, _ := time.Now().Zone()
		return n
	},
	"_execmd5":  func() interface{} { return ExecutableMD5() },
	"_execsha1": func() interface{} { return ExecutableSHA1() },
	"_execname": func() interface{} {
		return filepath.Base(os.Args[0])
	},
	"_randid": func() interface{} {
		return MathRandID().StringFull()
	},
}

AutoLabels is a list of automatically filled labels 

_hostname - local hostname
_user - current user
_pid - process pid
_timezone - local timezone code (UTC, MSK)
_goversion - go version
_execmd5 - this binary md5 hash
_execsha1 - this binary sha1 hash
_execname - executable base name (project name)
_randid - random id. May be used to distinguish different runs.
View Source 
var DefaultLogger = New(NewConsoleWriter(os.Stderr, LdetFlags))

Functions

func AppendKVs added in v0.11.0 

func AppendKVs(b []byte, kvs []interface{}) []byte

func AppendLabels added in v0.20.0 

func AppendLabels(b []byte, kvs []interface{}) []byte

func Color added in v0.10.0 

func Color(c ...int) (r []byte)

func ContextWithSpan added in v0.3.0 

func ContextWithSpan(ctx context.Context, s Span) context.Context

ContextWithSpan creates new context with Span ID context.Value. It returns the same context if id is zero.

func ExecutableMD5 added in v0.2.0 

func ExecutableMD5() string

ExecutableMD5 returns current process executable md5 hash. May be useful to find exact executable later.

func ExecutableSHA1 added in v0.2.0 

func ExecutableSHA1() string

ExecutableSHA1 returns current process executable sha1 hash. May be useful to find exact executable later.

func Hostname added in v0.2.0 

func Hostname() string

Hostname returns hostname or err.Error().

func If added in v0.6.0 

func If(topics string) bool

func IfDepth added in v0.20.0 

func IfDepth(d int, topics string) bool

func LoggerSetCallers added in v0.21.0 

func LoggerSetCallers(l *Logger, skip int, callers func(skip int, pc []uintptr) int)

func LoggerSetTimeNow added in v0.20.0 

func LoggerSetTimeNow(l *Logger, now func() time.Time, nano func() int64)

func ParseLabels 

func ParseLabels(s string) []interface{}

ParseLabels parses comma separated list of labels and fills them with values (See FillLabelsWithDefaults).

func Printf 

func Printf(fmt string, args ...interface{})

func Printw added in v0.7.0 

func Printw(msg string, kvs ...interface{})

func RandIDFromReader added in v0.20.0 

func RandIDFromReader(read func(p []byte) (int, error)) func() ID

func SetLabels 

func SetLabels(kvs ...interface{})

func SetVerbosity added in v0.20.0 

func SetVerbosity(vfilter string)

func UUID added in v0.9.0 

func UUID(read func(p []byte) (int, error)) func() ID

UUID creates ID generation function. read is a random Read method. Function panics on Read error. read must be safe for concurrent use.

It's got from github.com/google/uuid.

func User added in v0.2.0 

func User() string

User returns current username or err.Error().

Types

type ColorScheme added in v0.21.2 

type ColorScheme struct {
	TimeColor       []byte
	TimeChangeColor []byte // if different from TimeColor
	FileColor       []byte
	FuncColor       []byte
	MessageColor    []byte
	KeyColor        []byte
	ValColor        []byte
	LevelColor      struct {
		Info  []byte
		Warn  []byte
		Error []byte
		Fatal []byte
		Debug []byte
	}
}

type ConsoleWriter 

type ConsoleWriter struct {
	io.Writer
	Flags int

	Colorize        bool
	PadEmptyMessage bool
	AllLabels       bool
	AllCallers      bool

	LevelWidth   int
	MessageWidth int
	IDWidth      int
	Shortfile    int
	Funcname     int
	MaxValPad    int

	TimeFormat     string
	TimeLocation   *time.Location
	DurationFormat string
	DurationDiv    time.Duration
	FloatFormat    string
	FloatChar      byte
	FloatPrecision int
	CallerFormat   string
	BytesFormat    string

	StringOnNewLineMinLen int

	PairSeparator string
	KVSeparator   string

	QuoteChars      string
	QuoteAnyValue   bool
	QuoteEmptyValue bool

	ColorScheme
	// contains filtered or unexported fields
}

func NewConsoleWriter 

func NewConsoleWriter(w io.Writer, f int) *ConsoleWriter

func (*ConsoleWriter) AppendDuration added in v0.21.2 

func (w *ConsoleWriter) AppendDuration(b []byte, d time.Duration) []byte

func (*ConsoleWriter) ConvertValue added in v0.21.2 

func (w *ConsoleWriter) ConvertValue(b, p []byte, st, ff int) (_ []byte, i int)

func (*ConsoleWriter) Write added in v0.10.0 

func (w *ConsoleWriter) Write(p []byte) (i int, err error)

type EventKind added in v0.13.0 

type EventKind rune
const (
	EventSpanStart  EventKind = 's'
	EventSpanFinish EventKind = 'f'
	EventMetric     EventKind = 'm'
)

Event kinds.

func (EventKind) String added in v0.21.4 

func (ek EventKind) String() string

func (EventKind) TlogAppend added in v0.13.0 

func (ek EventKind) TlogAppend(b []byte) []byte

func (*EventKind) TlogParse added in v0.13.0 

func (ek *EventKind) TlogParse(p []byte, i int) int

type FormatNext added in v0.10.0 

type FormatNext string

type ID 

type ID [16]byte

func IDFromBytes added in v0.3.0 

func IDFromBytes(b []byte) (id ID, err error)

IDFromBytes decodes ID from bytes slice.

If byte slice is shorter than type length result is returned as is and ShortIDError as error value. You may use result if you expected short ID prefix.

func IDFromString added in v0.3.0 

func IDFromString(s string) (id ID, err error)

IDFromString parses ID from string.

If parsed string is shorter than type length result is returned as is and ShortIDError as error value. You may use result if you expected short ID prefix (profuced by ID.String, for example).

func IDFromStringAsBytes added in v0.7.0 

func IDFromStringAsBytes(s []byte) (id ID, err error)

IDFromStringAsBytes is the same as IDFromString. It avoids alloc in IDFromString(string(b)).

func MathRandID added in v0.9.0 

func MathRandID() (id ID)

func MustID added in v0.7.0 

func MustID(id ID, err error) ID

MustID wraps IDFrom* call and panics if error occurred.

func ShouldID added in v0.7.0 

func ShouldID(id ID, err error) ID

ShouldID wraps IDFrom* call and skips error if any.

func (ID) Format added in v0.3.0 

func (id ID) Format(s fmt.State, c rune)

Format is fmt.Formatter interface implementation. It supports width. '+' flag sets width to full ID length.

func (ID) FormatTo added in v0.3.0 

func (id ID) FormatTo(b []byte, f rune)

FormatTo is alloc free Format alternative.

func (ID) MarshalJSON added in v0.6.0 

func (id ID) MarshalJSON() ([]byte, error)

func (ID) String 

func (id ID) String() string

String returns short string representation.

It's not supposed to be able to recover it back to the same value as it was.

func (ID) StringFull added in v0.11.0 

func (id ID) StringFull() string

StringFull returns full id represented as string.

func (ID) TlogAppend added in v0.11.0 

func (id ID) TlogAppend(b []byte) []byte

func (*ID) TlogParse added in v0.11.0 

func (id *ID) TlogParse(p []byte, i int) int

func (*ID) UnmarshalJSON added in v0.6.0 

func (id *ID) UnmarshalJSON(b []byte) error

type LogLevel added in v0.10.0 

type LogLevel int
const (
	Info LogLevel = iota
	Warn
	Error
	Fatal

	Debug LogLevel = -1
)

Log levels.

func (LogLevel) TlogAppend added in v0.11.0 

func (l LogLevel) TlogAppend(b []byte) []byte

func (*LogLevel) TlogParse added in v0.11.0 

func (l *LogLevel) TlogParse(p []byte, i int) int

type Logger 

type Logger struct {
	io.Writer // protected by Mutex below

	tlwire.Encoder

	NewID func() ID `deep:"compare=pointer"` // must be threadsafe

	sync.Mutex
	// contains filtered or unexported fields
}

func New 

func New(w io.Writer) *Logger

func V 

func V(topics string) *Logger

func (*Logger) Copy added in v0.21.2 

func (l *Logger) Copy() *Logger

func (*Logger) CopyWriter added in v0.21.2 

func (l *Logger) CopyWriter(w io.Writer) *Logger

func (*Logger) Event added in v0.10.0 

func (l *Logger) Event(kvs ...interface{}) (err error)

func (*Logger) IOWriter added in v0.8.0 

func (l *Logger) IOWriter(d int) io.Writer

func (*Logger) If added in v0.5.0 

func (l *Logger) If(topics string) bool

func (*Logger) IfDepth added in v0.11.0 

func (l *Logger) IfDepth(d int, topics string) bool

func (*Logger) Labels added in v0.4.0 

func (l *Logger) Labels() RawMessage

func (*Logger) NewMessage added in v0.11.0 

func (l *Logger) NewMessage(d int, id ID, msg interface{}, kvs ...interface{})

func (*Logger) NewSpan added in v0.10.0 

func (l *Logger) NewSpan(d int, par ID, name string, kvs ...interface{}) Span

func (*Logger) Or added in v0.14.0 

func (l *Logger) Or(l2 *Logger) *Logger

func (*Logger) Printf 

func (l *Logger) Printf(fmt string, args ...interface{})

func (*Logger) Printw added in v0.7.0 

func (l *Logger) Printw(msg string, kvs ...interface{})

func (*Logger) Root added in v0.20.0 

func (l *Logger) Root() Span

func (*Logger) SetLabels added in v0.5.0 

func (l *Logger) SetLabels(kvs ...interface{})

func (*Logger) SetVerbosity added in v0.20.0 

func (l *Logger) SetVerbosity(vfilter string)

func (*Logger) Start 

func (l *Logger) Start(name string, kvs ...interface{}) Span

func (*Logger) V 

func (l *Logger) V(topics string) *Logger

func (*Logger) Write added in v0.4.0 

func (l *Logger) Write(p []byte) (int, error)

type Modify added in v0.20.0 

type Modify []byte

func NextIs added in v0.20.0 

func NextIs(semantic int) Modify

type RawMessage added in v0.11.0 

type RawMessage []byte

func RawTag added in v0.20.0 

func RawTag(tag byte, sub int) RawMessage

func Special added in v0.10.0 

func Special(value int) RawMessage

func (RawMessage) TlogAppend added in v0.11.0 

func (r RawMessage) TlogAppend(b []byte) []byte

func (*RawMessage) TlogParse added in v0.11.0 

func (r *RawMessage) TlogParse(p []byte, st int) (i int)

type ShortIDError added in v0.10.0 

type ShortIDError struct {
	Bytes int // Bytes successfully parsed
}

ShortIDError is an ID parsing error.

func (ShortIDError) Error added in v0.10.0 

func (e ShortIDError) Error() string

Error is an error interface implementation.

type Span 

type Span struct {
	*Logger
	ID        ID
	StartedAt time.Time
}

func Root added in v0.13.0 

func Root() Span

func SpanFromContext added in v0.3.0 

func SpanFromContext(ctx context.Context) (s Span)

SpanFromContext loads saved by ContextWithSpan Span from Context. It returns valid empty (no-op) Span if none was found.

func SpawnFromContext 

func SpawnFromContext(ctx context.Context, name string, kvs ...interface{}) Span

SpawnFromContext spawns new Span derived form Span or ID from Context. It returns empty (no-op) Span if no ID found.

func SpawnFromContextAndWrap added in v0.21.2 

func SpawnFromContextAndWrap(ctx context.Context, name string, kvs ...interface{}) (Span, context.Context)

func SpawnFromContextOrStart added in v0.4.0 

func SpawnFromContextOrStart(ctx context.Context, name string, kvs ...interface{}) Span

func Start 

func Start(name string, kvs ...interface{}) Span

func (Span) Copy added in v0.21.2 

func (s Span) Copy() Span

func (Span) CopyWriter added in v0.21.2 

func (s Span) CopyWriter(w io.Writer) Span

func (Span) Event added in v0.10.0 

func (s Span) Event(kvs ...interface{}) (err error)

func (Span) Finish 

func (s Span) Finish(kvs ...interface{})

func (Span) IOWriter added in v0.8.0 

func (s Span) IOWriter(d int) io.Writer

func (Span) If added in v0.5.0 

func (s Span) If(topics string) bool

func (Span) IfDepth added in v0.11.0 

func (s Span) IfDepth(d int, topics string) bool

func (Span) NewMessage added in v0.11.0 

func (s Span) NewMessage(d int, msg interface{}, kvs ...interface{})

func (Span) Or added in v0.14.0 

func (s Span) Or(s2 Span) Span

func (Span) Printf 

func (s Span) Printf(fmt string, args ...interface{})

func (Span) Printw added in v0.7.0 

func (s Span) Printw(msg string, kvs ...interface{})

func (Span) Spawn added in v0.7.0 

func (s Span) Spawn(name string, kvs ...interface{}) Span

func (Span) V 

func (s Span) V(topics string) Span

type Timestamp added in v0.10.0 

type Timestamp int64

Source Files

View all Source files

Directories

Path Synopsis
cmd
tlog command
tlog/tlogcmd
examples
charmlog command
dumper command
loglevel command
raw command
simplest command
simplest/sub
ext
tlclick
tlflag
tlgin
tlhttp
tlslog

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.