telemetry

package
v0.48.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 11, 2024 License: Apache-2.0 Imports: 20 Imported by: 0

README

Telemetry

This package acts to wire up the various components of OpenTelemetry transparently to Moov developers. As such, basic understanding of how OpenTelemetry project works is critical to understanding of this library. The documentation can be found here: https://opentelemetry.io/docs/instrumentation/go/

OpenTelemetry Purpose

To allow users, especially of distributed systems, to track execution through the stack via a single Trace that is composed of a series of Spans. The Trace will persist from ingress in the system until processing is complete. Traces are seperated by spans, tht are typically started and stopped at microservice boundaries but can also be started/stopped anywhere the client chooses. Spans started within other spans have a praent child relationship to the span than they were derived from.

OpenTelemetry not only lets you trace execution, allowing you to delineate functional locks of the overarching execution via spans, but also allows you to include meta-data such as attibutes and baggage (both covered below) to ensure the metrics being generated contain adequate information to properly query from.

OpenTelemetry Components

Attributes: Key/value pairs associated with a span, can be added at any time during execution. Attaching attributes to span is one of the most common ways we as Moov developers will add tracing information.

Context: While technically not an OpenTelemetry construct, it is very important to understand that this is how spans are propagated (they are embedded inside the context, and passed across microservice boundaries via baggage).

Baggage: A bucket for metadata that is passed along the entire Trace. Library warns that any data you pass in this you should expect to be visible and to be careful. Can somewhat be thought of Attributes associated with a Trace instead of a Span. Baggage is the means that transfers span information allowing it to continue through all the microservices. Baggage is added to all internal API requests and to the producing of events.

Exporter: Sends spans (that have been sent to it via a batch processor) to the system to be recorded. Two most common variants are the stdoutexporter and the gRPC exporter. Only used by this code base's boilerplate.

Instrumentation Libraries: Pre-made libraries that provide quality of life instrumentation, these typically cover auxiliary libraries such as kafka or net/http.

Propagators: Establishes the technique to pass Spans across microservice boundaries. Example below allows the Baggage to be included in the context that is marshalled/unmarshalled across boundaries. Only used by this code base's boilerplate.

Resource: Represents metadata about the physical device the code is executing on. Only used by this code base's boilerplate. Example below:

Sampler: Determines what percent of spans are recorded. You can always send, never send, or any ratio between. Only used by this code base's boilerplate.

Span: The main unit of work in OpenTelemetry, and the API Moov developers need to be most familiar with, spans delineates a functional block of execution. Creating a span from a tracer accepts a ctx, if that context already contains a span the new span becomes the child of that span. Contains an API that allows adding attributes, span status, and error details.

oneotel.Tracer(packageName).Start(ctx, “SpanName”)

SpanProcessor: Receives span information and is the pipeline that provides it to the exporter. It is configured within the TraceProvider, multiple SpanProcessors can be configured. BatchSpanProcessor which sends telemetry in batches and should be used for production applications (use NewBatchSpanProcessor). SimpleSpanProcessor is a non-production debugging processor (use NewSimpleSpanProcessor)

Tracer: Created from its factory object TracerProvider and provided a name. By OpenTelemetry standards this name should match the package name, but we do not follow this explicitly. Tracers are factories for Spans, and the main purpose of the Tracer is to associate the spans it spawns with the correct package name.

TracerProvider: The "glue" of OpenTelemetry, you define a factory and provide it with Sampler, Batcher, A factory of Tracers, defined only one time in an application. Accessed globally via the following: otel.SetTracerProvider(tp) otel.GetTracerProvider()

OpenTelemetry "Putting it all Together"

To configure a system to generate and export spans the following steps must be done:

  1. Define your exporter, where do you want your data to go? In this package, honey.go, stdout.go and collector.go all store methods to create Exporters. The beginning blocks of logic in SetupTelemetry determines which one should be used based on configuration values. Honeycomb exporter shown as example:
opts := []otlptracegrpc.Option{
    otlptracegrpc.WithCompressor("gzip"),
    otlptracegrpc.WithEndpoint(config.URL),
    otlptracegrpc.WithHeaders(map[string]string{
        "x-honeycomb-team": config.Team,
    }),
    otlptracegrpc.WithTLSCredentials(credentials.NewClientTLSFromCert(nil, "")),
}

client := otlptracegrpc.NewClient(opts...)
return otlptrace.New(ctx, client)
  1. Define your TracerProvider, typically this will require you to define the other constructs TracerProvider uses at the same time:
resource := resource.NewWithAttributes(
    semconv.SchemaURL,
    semconv.ServiceNameKey.String(config.ServiceName),
    semconv.ServiceVersionKey.String(version),
)

return trace.NewTracerProvider(
    trace.WithSampler(trace.AlwaysSample()),
    trace.WithBatcher(exp,
        trace.WithMaxQueueSize(3*trace.DefaultMaxQueueSize),
        trace.WithMaxExportBatchSize(3*trace.DefaultMaxExportBatchSize),
        trace.WithBatchTimeout(5*time.Second),
    ),
    trace.WithResource(resource),
)
  1. Globally set the defined TracerProvider:
otel.SetTracerProvider(tp)
  1. Allow Trace information to be propagated across microservice boundaries:
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
))
  1. Acquire a Tracer from the TracerProvider, and start a Span wrapping your desired block of execution* [IMPORTANT NOTE]: Our libraries (usually) automatically start spans for you:
    1. Kakfa Consumers: start a span for processing each consumed event.
    2. Kafka Producers: start a span for producing each event
    3. HTTPS Endpoints: start a Span for each request and are protected by the go-zero-trust-middleware.
  2. Add attributes, state code, or error messages at any location during a span, acquiring the span from the context if needed:
span := telemetry.SpanFromContext(r.Context())
attributes := []attribute.KeyValue{
    attribute.String("account_id", accountID),
    attribute.String("mode", claims.CallingAccountMode.String()),
}
if claims.CallingAccountID != nil {
    attributes = append(attributes, attribute.String("mode", *claims.CallingAccountID))
}
span.SetAttributes(attributes...)
Example of how HTTPS endpoints gain tracing via Zero-Trust library
    if env.ZeroTrustMiddleware == nil {
        gatewayMiddleware, err := middleware.NewServerFromConfig(env.Logger, env.TimeService, env.Config.Gateway)
        if err != nil {
        return nil, env.Logger.Fatal().LogErrorf("failed to startup Gateway middleware: %w", err).Err()
        }
        env.ZeroTrustMiddleware = gatewayMiddleware.Handler
    }

followed by attaching it to your router of choice:

    env.PublicRouter.Use(env.ZeroTrustMiddleware)

Documentation

Index

Constants

View Source 
const (
	InstrumentationName = "moov.io"
	AttributeTag        = "otel"
	MaxArrayAttributes  = 10
)
View Source 
const DropSpanKey = "span.drop"
View Source 
const MoovKnownIssueKey = "moov.known_issue"

Variables

This section is empty.

Functions

func AddEvent 

func AddEvent(ctx context.Context, name string, options ...trace.EventOption)

AddEvent adds an event the Span in `ctx` with the provided name and options.

func AttributeMoovKnownIssue 

func AttributeMoovKnownIssue() attribute.KeyValue

AttributeMoovKnownIssue is an attribute to mark a trace as a previously observed issue. IMPORTANT: if a trace has this attribute it will NOT fire a critical PD alert defined in https://github.com/moovfinancial/infra/blob/master/terraform-modules/apps/go-service/honeycomb.tf#L42

func DropSpan 

func DropSpan() attribute.KeyValue

DropSpan informs the sampler to skip this event if theirs no links tied to it.

func GetTracer 

func GetTracer(opts ...trace.TracerOption) trace.Tracer

GetTracer returns a unique Tracer scoped to be used by instrumentation code to trace computational workflows.

func HasSpanDrop 

func HasSpanDrop(s tracesdk.ReadOnlySpan) bool

Allows for services to just flag a span to be dropped.

func IsEmptyConsume 

func IsEmptyConsume(s tracesdk.ReadOnlySpan) bool

Detects if its an event that was consumed but ignored. These can cause a lot of cluttering in the traces and we want to filter them out.

func NewFilteredExporter 

func NewFilteredExporter(inner tracesdk.SpanExporter) tracesdk.SpanExporter

func RecordError 

func RecordError(ctx context.Context, err error, options ...trace.EventOption) error

RecordError will record err as an exception span event for this span. It will also return the err passed in.

func SetAttributes 

func SetAttributes(ctx context.Context, kv ...attribute.KeyValue)

SetAttributes sets kv as attributes of the Span. If a key from kv already exists for an attribute of the Span it will be overwritten with the value contained in kv.

func SpanFromContext 

func SpanFromContext(ctx context.Context) trace.Span

SpanFromContext returns the current Span from ctx.

If no Span is currently set in ctx an implementation of a Span that performs no operations is returned.

func StartSpan 

func StartSpan(ctx context.Context, spanName string, opts ...trace.SpanStartOption) (context.Context, trace.Span)

StartSpan will create a Span and a context containing the newly created Span.

If the context.Context provided contains a Span then the new span will be a child span, otherwise the new span will be a root span.

OTEL recommends creating all attributes via `WithAttributes()` SpanOption when the span is created.

Created spans MUST be ended with `.End()` and is the responsibility of callers.

func StructAttributes 

func StructAttributes(s interface{}) (kv []attribute.KeyValue)

StructAttributes creates an attribute.KeyValue for each field in the struct that has an "otel" tag defined. Nested structs will also be included, with attribute names formatted as "parent_attribute.nested_field_attribute".

Types

type Config 

type Config struct {
	ServiceName            string
	ServiceNamespace       *string
	Stdout                 bool
	OpenTelemetryCollector *OtelConfig
	Honeycomb              *HoneycombConfig
	// contains filtered or unexported fields
}

func TestConfig 

func TestConfig(w io.Writer) Config

Allows for testing where the output of the traces are sent to a io.Writer instance.

type HoneycombConfig 

type HoneycombConfig struct {
	URL  string
	Team string
}

type LinkedSpan 

type LinkedSpan struct {
	// contains filtered or unexported fields
}

func StartLinkedRootSpan 

func StartLinkedRootSpan(ctx context.Context, name string, options ...trace.SpanStartOption) *LinkedSpan

StartLinkedRootSpan starts a new root span where the parent and child spans share links to each other. This is particularly useful in batch processing applications where separate spans are wanted for each subprocess in the batch, but without cluttering the parent span.

func (*LinkedSpan) AddEvent 

func (l *LinkedSpan) AddEvent(name string, options ...trace.EventOption)

func (*LinkedSpan) ChildContext 

func (l *LinkedSpan) ChildContext() context.Context

func (*LinkedSpan) ChildSpan 

func (l *LinkedSpan) ChildSpan() trace.Span

func (*LinkedSpan) End 

func (l *LinkedSpan) End(options ...trace.SpanEndOption)

func (*LinkedSpan) ParentContext 

func (l *LinkedSpan) ParentContext() context.Context

func (*LinkedSpan) ParentSpan 

func (l *LinkedSpan) ParentSpan() trace.Span

func (*LinkedSpan) RecordError 

func (l *LinkedSpan) RecordError(err error, options ...trace.EventOption)

func (*LinkedSpan) SetAttributes 

func (l *LinkedSpan) SetAttributes(kv ...attribute.KeyValue)

func (*LinkedSpan) SetName 

func (l *LinkedSpan) SetName(name string)

func (*LinkedSpan) SetStatus 

func (l *LinkedSpan) SetStatus(code codes.Code, description string)

type OtelConfig 

type OtelConfig struct {
	Host string
	TLS  bool
}

type ShutdownFunc 

type ShutdownFunc func() error
var NoopShutdown ShutdownFunc = func() error {
	return nil
}

func SetupTelemetry 

func SetupTelemetry(ctx context.Context, config Config, version string) (ShutdownFunc, error)

type TracerProvider 

type TracerProvider interface {
	trace.TracerProvider

	ForceFlush(ctx context.Context) error
	Shutdown(ctx context.Context) error
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.