xop

package module
v0.0.0-...-79cac07 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 27, 2022 License: MIT Imports: 12 Imported by: 0

README

xop - Golang structured log generation combined with tracing (Cross Obserability Platform)

GoDoc

Development status

In development, not ready for use.

Context

The problem with the existing model

The industry model of tracing as documented in the W3C spec requires that spans have full identifiers. If you give each part of dealing with a request inside a single server, lots of different spans, then how can you quickly reference the request-level span from one the sub-spans or one of the other requests that is a child of the main request. There is no standard way to distinguish a span that is simply a separate thread of execution or one that is a related request on a different server.

The format of logs isn't easy to extend because there are is no meta-level or standard for what log fields mean. The closest for this is the naming semantics that are included in the Open Telementry project.

Once the logs are generated, good logging systems tag each line with trance and span identifiers, but the logs are still stored, searched, and dispalyed as lines. Most of the value comes from the context of the log so recording them as lines removes misses the point.

Another issue with most structured loggers is that they over-collect details that don't matter and un-invest in how the logs are presented. The extra details are sometimes useful, but they increase the cost to process and store the logs. More importantly, they can clutter the display so that it has lots of data but does not present much information.

The standard model does not lend itself to experimentation with what kinds of things are logged and how they're presented. At BlueOwl, we discovered that logging tables was very valuable. We had support for displaying tables in our log viewer. For some very complicated bugs, displaying tables in the logs was instrumental in finding the problem.

Alternatives

other takes

Software Tracing With Go

Architecture

A complete logging solution has:

Generating Logs

Logging API for writing code. There are a couple of flavors that are commonly found: Printf-style; Key/Value style; Structure with functional args style; and Structured with methods style.

Tracing APIs have less variation. Opentracing is a fine example.

Writing logs

Logs can be sent directly to a server; they can be written to STDOUT. Most systems use JSON for structured logging, but Open Telementry uses Protobufs.

Gathering logs

Most frameworks include an intermediary program that scrapes up logs, especially STDOUT logs, and sends them on to servers relibably. This has an advantage of lowering the likelihood of backpressure.

Should backpressure exist at all? Maybe. Some logs may be required for auditing purposes and such logs may need to be sent reliably.

Indexing logs

ElasticSearch vs OpenSearch vs Solr? Elastic loses on the licensing front.

It makes sense to do full-text indexing and also trace-level indexing separately so that the trace-level indexing can be retained for a longer duration.

Storing logs

S3 and other bulk storage

Generating tracelogs API

Request-level

At the request-level (HTTP request) or program being run, etc, tag the request with: fields and indexes. The fields are descriptive and the indexes are searchable. Both have the same format.

The request model and response model are good examples of fields.

The URL is a good example of an index.

One required indexed key is 'type'. In an organization, each type can have different required fields and associated value types.

For example, the 'http' type can have: 'endpoint', 'path', 'method', 'response-code' indexes and 'request-data', 'response-data' fields.

Should this be enforced? XXX how?

Span-level

Spans have a text description

Spans have arbitray key/value pairs that are added as users create the span.

Spans have a W3C-compatible id

Spans have a .1 .A prefix to show how they relate to the other spans in the same request

Spans are either included in the request indexes or not included. If included, they have type=span.

Spans have a full W3C trace context

Log-line level

Log lines do not have any inherited data except their span reference

Log lines can have key/value pairs

If the key is "span" then the value should be full span reference

The key of "type" should be displayed prominately by the front-end and the following types are suggested:

  • "reference": requires a "span" and referes to that span

  • "request": a request to another system, if that system replied with a "tracereference" then record that here.

Generating Tracelogs - Data format

Log lines

Open Telemetry Log Specification

Every log line gets a span id and trace id.

In the key/values attached to log lines, a key of "reference" to another trace/span

Spans

Span records are only emitted after the span is complete or N minutes have passed.

Must reference their parent span, if possilbe

May have a "thread.context" like "B.3." to indicate it's the third thing in the 2nd parallel execution. If there is no "thread.context", then attributes are searchable.

May have an http.status_code even if it's not an http request

An "error.count" attribute is automatically added

Spans must have names. For parallel forks, this is the same as the thread context.

Requests to other services get their own span.

Open Telementy's "SpanKind" isn't rich enough. It's missing CLI_COMMAND, TASK

Span data can continue to be updated even after the span is first sent.

Forwarding tracelogs

No changes are needed for log forwarding

LogStash introduction Open Telementry data collection Open Telementry protocol

Reforming tracelogs

Accepts both logs and traces.

Step 1: save the raw data. Data is written to persistent disk in the "inbox". Logs that don't have trace ids and span ids are dropped.

Step 2: If there are multiple servers, forward raw data to the appropriate server based on sharding of the trace id.

Step 3: Write data to appropriate trace-collection buckets (files, one per trace). Mark buckets as not-indexed.

Step 4: Once written to buckets, inbox files can be removed.

Step 5: Add/update Elastic indexes.

NOTES

Terminology

A "trace" is the the entire set of spans relating to one starting request or action. It can span multiple servers.

A "request" is a single request or action being handled by one program. It does not span multiple servers. There can be multiple requests in a trace.

A "span" is a linear portion of the processing required to handle a request. A single span should not include multiple threads of execution. Span should represent a logical component to of the work being done. Breaking the work into spans is an exercise for the programmer.

A "logger" is something that is used throughout code to generate log lines and spans.

A "base logger" is the layer below that the "logger" uses to send output to different systems.

A "bytes logger" is an optional layer below "base logger" that works with logs that have already become []bytes.

Naming

Name registry

Arbitrary names are supported for tagging log lines. For attributes to be displayed specially in front-ends, they need to follow standards. Standard attribute groups are pre-registered as structs. These can be shared between organizations by contributing them to the Xop repository.

The following names are reserved. What happens if they're used is undefined and up to the individual base loggers.

  • xop. Used to indicate the kind of item begin emitted in a stream of objects. Empty for lines, span for spans. enum to establish enum -> string mappings. chunk for things broken up because they're too big. template for lines that need template expansion.
  • msg. Used for the text of a log line.
  • time. Used for the timestamp of the log event, if included.
  • stack. Used for stacktraces when errors or alerts are logged.
  • span. Used for the span-id of log lines for some base loggers.
  • caller. Used to indicate the immediate caller (file & line) when that's desired.
  • level. The log level (debug, trace, info, warn, error, alert)

The data associated with spans, traces, and requests must come from pre-registered keys.

Philosphy

Xop is opinionated. It gently nudges in certain directions. Perhaps the biggest nudge is that there is no support for generating logs outside of a span. There is no default logger. There is no global logger.

Log less

Do not log details that don't materialy add to the value of the log

Log more

Use logs as a narritive of what's going on in the program so that when you look at the logs, you can follow along with what's going on.

Always log in context

Logs are best viewed in context: without without needing to search and correlate, you should know how you go to the point of the log line you're looking at. This means the line itself needs less detail and it contributes to the context of the lines around it.

No log.Fatal

Panic should be caught and logged. If panic is caught, log.Fatal() is not needed and is even redundant as it would problaby panic itself causing multiple log.Alert() for the same event.

Defer work

Most logs won't be looked at. Ever. When possilbe defer the work of assembling the log to when it viewed.

Other systems

This logger is inspired by a proprietary logger at BlueOwl; onelog; phuslog; zap; zerolog; Open Telementry; and Jaeger.

Open Telementry

They say to use dots (.) to separate namespaces in attribute names and underscores (_) to separate words within a name. Do not use a namespace as an attribute.

They have lots of examples for:

Open Tracing

The Open Tracing project has been "archived" in favor of Open Telementry. That said, they have a much shorter set of semantic conventions.

Zipkin

While lacking a full set of semantic conventions, Zipkin has some sage advice around how to instrument spans

OpenCensus

OpenCensus lacks a full set of semantic conventions, but it does having suggestions for how to name spans. In OpenCensus, tags names need to be registered.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var Default = NewSeed().Request("discard")

TODO: have a default log that prints

View Source 
var DefaultConfig = Config{
	FlushDelay: time.Minute * 5,
	ErrorReporter: func(err error) {
		golog.Print("Error from zop", err)
	},
}

Functions

This section is empty.

Types

type Config 

type Config struct {
	UseB3      bool // Zipkin
	FlushDelay time.Duration

	// ErrorReporter provides a way to choose the behavior
	// for when underlying log functions throw an error.
	// Generally speaking, needing to check errors when
	// generating logs is a non-starter because the cost is
	// too high.  It would discourage logging.  That said,
	// there is a an error, we don't want to completely
	// ignore it.
	//
	// TODO: If ErrorReporter is called too frequently,
	// it will automatically be throttled
	ErrorReporter func(error)
}

type ConfigModifier 

type ConfigModifier func(*Config)

type Log 

type Log struct {
	// contains filtered or unexported fields
}

func FromContext 

func FromContext(ctx context.Context) (*Log, bool)

func FromContextOrDefault 

func FromContextOrDefault(ctx context.Context) *Log

func FromContextOrPanic 

func FromContextOrPanic(ctx context.Context) *Log

func (*Log) Alert 

func (l *Log) Alert() *LogLine

func (*Log) Boring 

func (l *Log) Boring()

Marks this request as boring. Any log at the Alert or Error level will mark this request as not boring.

func (*Log) Config 

func (l *Log) Config() Config

func (*Log) Debug 

func (l *Log) Debug() *LogLine

func (*Log) Done 

func (l *Log) Done()

Done is used to indicate that a seed.Reqeust(), log.Fork().Wait(), or log.Step().Wait() is done. When all of the parts of a request are finished, the log is automatically flushed.

func (*Log) Error 

func (l *Log) Error() *LogLine

func (*Log) Flush 

func (l *Log) Flush()

func (*Log) Fork 

func (l *Log) Fork(msg string, mods ...SeedModifier) *Log

Fork creates a new Log that does not need to be terminated because it is assumed to be done with the current log is finished.

func (*Log) Info 

func (l *Log) Info() *LogLine

func (*Log) IntoContext 

func (l *Log) IntoContext(ctx context.Context) context.Context

func (*Log) LogLine 

func (l *Log) LogLine(level xopconst.Level) *LogLine

func (*Log) Request 

func (l *Log) Request() *Span

Request provides access to the span that describes the overall request. Metadata may be added at the request level.

func (*Log) Span 

func (l *Log) Span() *Span

Request provides access to the current span Metadata may be added at the span level.

func (*Log) Step 

func (l *Log) Step(msg string, mods ...SeedModifier) *Log

Step creates a new log that does not need to be terminated -- it represents the continued execution of the current log bug doing something that is different and should be in a fresh span. The expectation is that there is a parent log that is creating various sub-logs using Step over and over as it does different things.

func (*Log) Trace 

func (l *Log) Trace() *LogLine

func (*Log) Wait 

func (l *Log) Wait() *Log

Wait modifies (and returns) a Log to indicate that the overall request is not finished until an additional Done() is called.

func (*Log) Warn 

func (l *Log) Warn() *LogLine

type LogLine 

type LogLine struct {
	// contains filtered or unexported fields
}

func (*LogLine) Any 

func (ll *LogLine) Any(k string, v interface{}) *LogLine

Any can be used to log something that might be modified after this call. If any base logger does not immediately serialize, then the object will be copied using https://github.com/mohae/deepcopy 's Copy().

func (*LogLine) AnyImmutable 

func (ll *LogLine) AnyImmutable(k string, v interface{}) *LogLine

AnyImmutable can be used to log something that is not going to be further modified after this call.

func (*LogLine) Bool 

func (ll *LogLine) Bool(k string, v bool) *LogLine

func (*LogLine) Duration 

func (ll *LogLine) Duration(k string, v time.Duration) *LogLine

func (*LogLine) EmbeddedEnum 

func (ll *LogLine) EmbeddedEnum(k xopconst.EmbeddedEnum) *LogLine

func (*LogLine) Enum 

func (ll *LogLine) Enum(k *xopconst.EnumAttribute, v xopconst.Enum) *LogLine

func (*LogLine) Error 

func (ll *LogLine) Error(k string, v error) *LogLine

func (*LogLine) Int 

func (ll *LogLine) Int(k string, v int) *LogLine

func (*LogLine) Int8 

func (ll *LogLine) Int8(k string, v int8) *LogLine

func (*LogLine) Int16 

func (ll *LogLine) Int16(k string, v int16) *LogLine

func (*LogLine) Int32 

func (ll *LogLine) Int32(k string, v int32) *LogLine

func (*LogLine) Int64 

func (ll *LogLine) Int64(k string, v int64) *LogLine
func (ll *LogLine) Link(k string, v trace.Trace) *LogLine

func (*LogLine) Msg 

func (ll *LogLine) Msg(msg string)

func (*LogLine) Msgf 

func (ll *LogLine) Msgf(msg string, v ...interface{})

TODO: generate these TODO: the rest of the set

func (*LogLine) Msgs 

func (ll *LogLine) Msgs(v ...interface{})

func (*LogLine) Str 

func (ll *LogLine) Str(k string, v string) *LogLine

func (*LogLine) Template 

func (ll *LogLine) Template(template string)

Template is an alternative to Msg() sends a log line. Template is a string that uses "{name}" substitutions from the data already sent with the line to format that data for human consumption. Template is expected to be more expensive than Msg so it should be used somewhat sparingly. Data elements do not have to be consumed by the template.

The names used for "{name}" substitutions are restricted: they may not include any characters that would be escapsed in a JSON string. No double quote. No linefeed. No backslash. Etc.

func (*LogLine) Time 

func (ll *LogLine) Time(k string, v time.Time) *LogLine

func (*LogLine) Uint 

func (ll *LogLine) Uint(k string, v uint) *LogLine

func (*LogLine) Uint8 

func (ll *LogLine) Uint8(k string, v uint8) *LogLine

func (*LogLine) Uint16 

func (ll *LogLine) Uint16(k string, v uint16) *LogLine

func (*LogLine) Uint32 

func (ll *LogLine) Uint32(k string, v uint32) *LogLine

func (*LogLine) Uint64 

func (ll *LogLine) Uint64(k string, v uint64) *LogLine

type Seed 

type Seed struct {
	// contains filtered or unexported fields
}

Seed is used to create a Log.

func NewSeed 

func NewSeed(mods ...SeedModifier) Seed

func (Seed) Copy 

func (s Seed) Copy() Seed

func (Seed) Request 

func (s Seed) Request(descriptionOrName string) *Log

func (Seed) SubSpan 

func (s Seed) SubSpan() Seed

func (Seed) Trace 

func (s Seed) Trace() trace.Bundle

type SeedModifier 

type SeedModifier func(*Seed)

func WithAnyPrefill 

func WithAnyPrefill(k string, v interface{}) SeedModifier

WithAnyPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Any() to add a span-level key/value pair that is logged once (with the span).

func WithB3 

func WithB3(b bool) SeedModifier

func WithBaseLogger 

func WithBaseLogger(name string, logger xopbase.Logger) SeedModifier

func WithBoolPrefill 

func WithBoolPrefill(k string, v bool) SeedModifier

WithBoolPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Bool() to add a span-level key/value pair that is logged once (with the span).

func WithBundle 

func WithBundle(bundle trace.Bundle) SeedModifier

func WithConfig 

func WithConfig(config Config) SeedModifier

func WithConfigChanges 

func WithConfigChanges(mods ...ConfigModifier) SeedModifier

func WithDurationPrefill 

func WithDurationPrefill(k string, v time.Duration) SeedModifier

WithDurationPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Duration() to add a span-level key/value pair that is logged once (with the span).

func WithErrorPrefill 

func WithErrorPrefill(k string, v error) SeedModifier

WithErrorPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Error() to add a span-level key/value pair that is logged once (with the span).

func WithFlushDelay 

func WithFlushDelay(d time.Duration) SeedModifier

func WithIntPrefill 

func WithIntPrefill(k string, v int64) SeedModifier

WithIntPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Int() to add a span-level key/value pair that is logged once (with the span).

func WithLinkPrefill 

func WithLinkPrefill(k string, v trace.Trace) SeedModifier

WithLinkPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Link() to add a span-level key/value pair that is logged once (with the span).

func WithNoPrefill 

func WithNoPrefill() SeedModifier

func WithPrefillText 

func WithPrefillText(m string) SeedModifier

func WithStrPrefill 

func WithStrPrefill(k string, v string) SeedModifier

WithStrPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Str() to add a span-level key/value pair that is logged once (with the span).

func WithTimePrefill 

func WithTimePrefill(k string, v time.Time) SeedModifier

WithTimePrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Time() to add a span-level key/value pair that is logged once (with the span).

func WithTrace 

func WithTrace(trace trace.Trace) SeedModifier

func WithUintPrefill 

func WithUintPrefill(k string, v uint64) SeedModifier

WithUintPrefill adds a key/value pair that will be included as part of every log line in this span. If there are no log lines in the span then this data will not be logged at all. Use Span.Uint() to add a span-level key/value pair that is logged once (with the span).

func WithoutBaseLogger 

func WithoutBaseLogger(name string) SeedModifier

type Span 

type Span struct {
	// contains filtered or unexported fields
}

func (*Span) Any 

func (s *Span) Any(k *xopconst.AnyAttribute, v interface{}) *Span

Any adds a key/value attribute to the current Span. The provided value may be copied using github.com/mohae/deepcopy if any of the base loggers hold the value instead of immediately serializing it. While the AnyAttribute has an expectation for the type of the value, that type may or may not be checked depending on the base logger being used.

func (*Span) AnyImmutable 

func (s *Span) AnyImmutable(k *xopconst.AnyAttribute, v interface{}) *Span

AnyImmutable adds a key/value attribute to the current Span. The provided value must be immutable. If it is not, then there could be race conditions or the value that ends up logged could be different from the value at the time when AnyImmutible was called.

While the AnyAttribute has an expectation for the type of the value, that type may or may not be checked depending on the base logger being used.

func (*Span) Bool 

func (s *Span) Bool(k *xopconst.BoolAttribute, v bool) *Span

Bool adds a bool key/value attribute to the current Span

func (*Span) Bundle 

func (s *Span) Bundle() trace.Bundle

func (*Span) Duration 

func (s *Span) Duration(k *xopconst.DurationAttribute, v time.Duration) *Span

should skip Int64 Duration adds a time.Duration key/value attribute to the current Span

func (*Span) EmbeddedEnum 

func (s *Span) EmbeddedEnum(kv xopconst.EmbeddedEnum) *Span

EmbeddedEnum adds a kev/value attribute to the Span. The key and the value are bundled together: the key is derrived from the type of the Enum. Alternatively, use xopconst.KeyedEnumAttribute() to create functions to add enum key/value pairs where the key and value are specified separately.

func (*Span) Enum 

func (s *Span) Enum(k *xopconst.EnumAttribute, v xopconst.Enum) *Span

Enum adds a xopconst.Enum key/value attribute to the current Span

func (*Span) Int 

func (s *Span) Int(k *xopconst.IntAttribute, v int) *Span

Int adds a int key/value attribute to the current Span

func (*Span) Int8 

func (s *Span) Int8(k *xopconst.Int8Attribute, v int8) *Span

Int8 adds a int8 key/value attribute to the current Span

func (*Span) Int16 

func (s *Span) Int16(k *xopconst.Int16Attribute, v int16) *Span

Int16 adds a int16 key/value attribute to the current Span

func (*Span) Int32 

func (s *Span) Int32(k *xopconst.Int32Attribute, v int32) *Span

Int32 adds a int32 key/value attribute to the current Span

func (*Span) Int64 

func (s *Span) Int64(k *xopconst.Int64Attribute, v int64) *Span
func (s *Span) Link(k *xopconst.LinkAttribute, v trace.Trace) *Span

Link adds a trace.Trace key/value attribute to the current Span

func (*Span) Number 

func (s *Span) Number(k *xopconst.NumberAttribute, v float64) *Span

Number adds a float64 key/value attribute to the current Span

func (*Span) Seed 

func (s *Span) Seed(mods ...SeedModifier) Seed

func (*Span) Str 

func (s *Span) Str(k *xopconst.StrAttribute, v string) *Span

Str adds a string key/value attribute to the current Span

func (*Span) Time 

func (s *Span) Time(k *xopconst.TimeAttribute, v time.Time) *Span

Time adds a time.Time key/value attribute to the current Span

func (*Span) Trace 

func (s *Span) Trace() trace.Trace

func (*Span) TraceBaggage 

func (s *Span) TraceBaggage() trace.Baggage

func (*Span) TraceParent 

func (s *Span) TraceParent() trace.Trace

func (*Span) TraceState 

func (s *Span) TraceState() trace.State

Source Files

View all Source files

Directories

Path Synopsis
tools
xopzzz command
xopjson
xoptest

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.