obsreport

Documentation

Overview

Package obsreport provides unified and consistent observability signals ( metrics, tracing, etc) for components of the OpenTelemetry collector.

The function Configure is used to control which signals are going to be generated. It provides functions for the typical operations of receivers, processors, and exporters.

Receivers should use the respective start and end according to the data type being received, ie.:

• TraceData receive operations should use the pair: StartTraceDataReceiveOp/EndTraceDataReceiveOp

• Metrics receive operations should use the pair: StartMetricsReceiveOp/EndMetricsReceiveOp

Similar for exporters:

• TraceData export operations should use the pair: StartTraceDataExportOp/EndTraceDataExportOp

• Metrics export operations should use the pair: StartMetricsExportOp/EndMetricsExportOp

The package is capable of generating legacy metrics by using the observability package allowing a controlled transition from legacy to the new metrics. The goal is to eventually remove the legacy metrics and use only the new metrics.

The main differences regarding the legacy metrics are:

1. "Amount of metric data" is measured as metric points (ie.: a single value in time), contrast it with number of time series used legacy. Number of metric data points is a more general concept regarding various metric formats.

2. Exporters measure the number of items, ie.: number of spans or metric points, that were sent and the ones for which the attempt to send failed. For more information about this see Notes below about reporting data loss.

3. All measurements of "amount of data" used in the new metrics for receivers and exporters should reflect their native formats, not the internal format used in the Collector. This is to facilitate reconciliation between Collector, client and backend. For instance: certain metric formats do not provide direct support for histograms and have predefined conventions to represent those, this conversion may end with a different number of time series and data points than the internal Collector format.

Notes:

* Data loss should be recorded only when the component itself remove the data from the pipeline. Legacy metrics for receivers used "dropped" in their names but these could be non-zero under normal operations and reflected no actual data loss when components like the "queued_retry" are used. New metrics were renamed to avoid this misunderstanding. Here are the general recommendations to report data loss:

• Receivers reporting errors to clients typically result in the client re-sending the same data so it is more correct to report "receive errors", not actual data loss.

• Exporters need to report individual failures to send data, but on typical production pipelines processors usually take care of retries, so these should be reported as "send errors".

• Data "filtered out" should have its own metrics and not be confused with dropped data.

Naming Convention for New Metrics

Common Metrics: Metrics shared by different components should follow the convention below:

`<component>/<metric_name>`

As a label the metric should have at least `{<component>="<name>"}` where `<name>` is the name used in the configuration for the instance of the component, eg.:

`receiver/accepted_spans{receiver="opencensus",...}` `exporter/sent_spans{exporter="jaeger/prod",...}`

Component Specific Metrics: These metrics are implemented by specific components, eg.: batch processor. The general pattern is the same as the common metrics but with the addition of the component type (as it appears in the configuration) before the actual metric:

`<component>/<type>/<metric_name>`

Even metrics exclusive to a single type should follow the conventions above and also include the type (as written in the configuration) as part of the metric name since there could be multiple instances of the same type in different pipelines, eg.:

`processor/batch/batch_size_trigger_send{processor="batch/dev",...}`

Index

• Constants
• func BuildProcessorCustomMetricName(configType, metric string) string
• func Configure(generateLegacy, generateNew bool) (views []*view.View)
• func CountMetricPoints(md consumerdata.MetricsData) (numTimeSeries int, numPoints int)
• func EndMetricsExportOp(exporterCtx context.Context, numExportedPoints int, numExportedTimeSeries int, ...)
• func EndMetricsReceiveOp(receiverCtx context.Context, format string, numReceivedPoints int, ...)
• func EndTraceDataExportOp(exporterCtx context.Context, numExportedSpans int, numDroppedSpans int, ...)
• func EndTraceDataReceiveOp(receiverCtx context.Context, format string, numReceivedSpans int, err error)
• func ExporterContext(ctx context.Context, exporter string) context.Context
• func ProcessorContext(ctx context.Context, processor string) context.Context
• func ProcessorMetricViews(configType string, legacyViews []*view.View) []*view.View
• func ProcessorMetricsDataAccepted(processorCtx context.Context, md consumerdata.MetricsData)
• func ProcessorMetricsDataDropped(processorCtx context.Context, md consumerdata.MetricsData)
• func ProcessorMetricsDataRefused(processorCtx context.Context, md consumerdata.MetricsData)
• func ProcessorTraceDataAccepted(processorCtx context.Context, td consumerdata.TraceData)
• func ProcessorTraceDataDropped(processorCtx context.Context, td consumerdata.TraceData)
• func ProcessorTraceDataRefused(processorCtx context.Context, td consumerdata.TraceData)
• func ReceiverContext(ctx context.Context, receiver string, transport string, legacyName string) context.Context
• func StartMetricsExportOp(operationCtx context.Context, exporter string) context.Context
• func StartMetricsReceiveOp(operationCtx context.Context, receiver string, transport string, ...) context.Context
• func StartTraceDataExportOp(operationCtx context.Context, exporter string) context.Context
• func StartTraceDataReceiveOp(operationCtx context.Context, receiver string, transport string, ...) context.Context
• type StartReceiveOption
  • func WithLongLivedCtx() StartReceiveOption
• type StartReceiveOptions

Constants

const (
	// Key used to identify exporters in metrics and traces.
	ExporterKey = "exporter"

	// Key used to track spans sent by exporters.
	SentSpansKey = "sent_spans"
	// Key used to track spans that failed to be sent by exporters.
	FailedToSendSpansKey = "send_failed_spans"

	// Key used to track metric points sent by exporters.
	SentMetricPointsKey = "sent_metric_points"
	// Key used to track metric points that failed to be sent by exporters.
	FailedToSendMetricPointsKey = "send_failed_metric_points"
)

const (
	// Key used to identify processors in metrics and traces.
	ProcessorKey = "processor"

	// Key used to identify spans dropped by the Collector.
	DroppedSpansKey = "dropped_spans"

	// Key used to identify metric points dropped by the Collector.
	DroppedMetricPointsKey = "dropped_metric_points"
)

const (
	// Key used to identify receivers in metrics and traces.
	ReceiverKey = "receiver"
	// Key used to identify the transport used to received the data.
	TransportKey = "transport"
	// Key used to identify the format of the data received.
	FormatKey = "format"

	// Key used to identify spans accepted by the Collector.
	AcceptedSpansKey = "accepted_spans"
	// Key used to identify spans refused (ie.: not ingested) by the Collector.
	RefusedSpansKey = "refused_spans"

	// Key used to identify metric points accepted by the Collector.
	AcceptedMetricPointsKey = "accepted_metric_points"
	// Key used to identify metric points refused (ie.: not ingested) by the
	// Collector.
	RefusedMetricPointsKey = "refused_metric_points"
)

Functions

func BuildProcessorCustomMetricName

func BuildProcessorCustomMetricName(configType, metric string) string

BuildProcessorCustomMetricName is used to be build a metric name following the standards used in the Collector. The configType should be the same value used to identify the type on the config.

func Configure

func Configure(
	generateLegacy, generateNew bool,
) (views []*view.View)

Configure is used to control the settings that will be used by the obsreport package.

func CountMetricPoints

func CountMetricPoints(md consumerdata.MetricsData) (numTimeSeries int, numPoints int)

CountMetricPoints is a helper to count the "amount" of metrics data.

func EndMetricsExportOp

func EndMetricsExportOp(
	exporterCtx context.Context,
	numExportedPoints int,
	numExportedTimeSeries int,
	numDroppedTimeSeries int,
	err error,
)

EndMetricsExportOp completes the export operation that was started with StartMetricsExportOp.

func EndMetricsReceiveOp

func EndMetricsReceiveOp(
	receiverCtx context.Context,
	format string,
	numReceivedPoints int,
	numReceivedTimeSeries int,
	err error,
)

EndMetricsReceiveOp completes the receive operation that was started with StartMetricsReceiveOp.

func EndTraceDataExportOp

func EndTraceDataExportOp(
	exporterCtx context.Context,
	numExportedSpans int,
	numDroppedSpans int,
	err error,
)

EndTraceDataExportOp completes the export operation that was started with StartTraceDataExportOp.

func EndTraceDataReceiveOp

func EndTraceDataReceiveOp(
	receiverCtx context.Context,
	format string,
	numReceivedSpans int,
	err error,
)

EndTraceDataReceiveOp completes the receive operation that was started with StartTraceDataReceiveOp.

func ExporterContext

func ExporterContext(
	ctx context.Context,
	exporter string,
) context.Context

ExporterContext adds the keys used when recording observability metrics to the given context returning the newly created context. This context should be used in related calls to the obsreport functions so metrics are properly recorded.

func ProcessorContext

func ProcessorContext(
	ctx context.Context,
	processor string,
) context.Context

ProcessorContext adds the keys used when recording observability metrics to the given context returning the newly created context. This context should be used in related calls to the obsreport functions so metrics are properly recorded.

func ProcessorMetricViews

func ProcessorMetricViews(configType string, legacyViews []*view.View) []*view.View

ProcessorMetricViews builds the metric views for custom metrics of processors.

func ProcessorMetricsDataAccepted

func ProcessorMetricsDataAccepted(
	processorCtx context.Context,
	md consumerdata.MetricsData,
)

ProcessorMetricsDataAccepted reports that the metrics were accepted.

func ProcessorMetricsDataDropped

func ProcessorMetricsDataDropped(
	processorCtx context.Context,
	md consumerdata.MetricsData,
)

ProcessorMetricsDataDropped reports that the metrics were dropped.

func ProcessorMetricsDataRefused

func ProcessorMetricsDataRefused(
	processorCtx context.Context,
	md consumerdata.MetricsData,
)

ProcessorMetricsDataRefused reports that the metrics were refused.

func ProcessorTraceDataAccepted

func ProcessorTraceDataAccepted(
	processorCtx context.Context,
	td consumerdata.TraceData,
)

ProcessorTraceDataAccepted reports that the trace data was accepted.

func ProcessorTraceDataDropped

func ProcessorTraceDataDropped(
	processorCtx context.Context,
	td consumerdata.TraceData,
)

ProcessorTraceDataDropped reports that the trace data was dropped.

func ProcessorTraceDataRefused

func ProcessorTraceDataRefused(
	processorCtx context.Context,
	td consumerdata.TraceData,
)

ProcessorTraceDataRefused reports that the trace data was refused.

func ReceiverContext

func ReceiverContext(
	ctx context.Context,
	receiver string,
	transport string,
	legacyName string,
) context.Context

ReceiverContext adds the keys used when recording observability metrics to the given context returning the newly created context. This context should be used in related calls to the obsreport functions so metrics are properly recorded.

func StartMetricsExportOp

func StartMetricsExportOp(
	operationCtx context.Context,
	exporter string,
) context.Context

StartMetricsExportOp is called at the start of an Export operation. The returned context should be used in other calls to the obsreport functions dealing with the same export operation.

func StartMetricsReceiveOp

func StartMetricsReceiveOp(
	operationCtx context.Context,
	receiver string,
	transport string,
	opt ...StartReceiveOption,
) context.Context

StartMetricsReceiveOp is called when a \request is received from a client. The returned context should be used in other calls to the obsreport functions dealing with the same receive operation.

func StartTraceDataExportOp

func StartTraceDataExportOp(
	operationCtx context.Context,
	exporter string,
) context.Context

StartTraceDataExportOp is called at the start of an Export operation. The returned context should be used in other calls to the obsreport functions dealing with the same export operation.

func StartTraceDataReceiveOp

func StartTraceDataReceiveOp(
	operationCtx context.Context,
	receiver string,
	transport string,
	opt ...StartReceiveOption,
) context.Context

StartTraceDataReceiveOp is called when a \request is received from a client. The returned context should be used in other calls to the obsreport functions dealing with the same receive operation.

Types

type StartReceiveOption

type StartReceiveOption func(*StartReceiveOptions)

StartReceiveOption function applues changes to StartReceiveOptions.

func WithLongLivedCtx

func WithLongLivedCtx() StartReceiveOption

WithLongLivedCtx indicates that the context passed in the call outlives the receive operation at hand. Typically the long lived context is associated to a connection, eg.: a gRPC stream or a TCP connection, for which many batches of data are received in individual operations without a corresponding new context per operation.

Example:

func (r *receiver) ClientConnect(ctx context.Context, rcvChan <-chan consumerdata.TraceData) {
    longLivedCtx := obsreport.ReceiverContext(ctx, r.config.Name(), r.transport, "")
    for {
        // Since the context outlives the individual receive operations call obsreport using
        // WithLongLivedCtx().
        ctx := obsreport.StartTraceDataReceiveOp(
            longLivedCtx,
            r.config.Name(),
            r.transport,
            obsreport.WithLongLivedCtx())

        td, ok := <-rcvChan
        var err error
        if ok {
            err = r.nextConsumer.ConsumeTraceData(ctx, td)
        }
        obsreport.EndTraceDataReceiveOp(
            ctx,
            r.format,
            len(td.Spans),
            err)
        if !ok {
            break
        }
    }
}

type StartReceiveOptions

type StartReceiveOptions struct {
	// LongLivedCtx when true indicates that the context passed in the call
	// outlives the individual receive operation. See WithLongLivedCtx() for
	// more information.
	LongLivedCtx bool
}

StartReceiveOptions has the options related to starting a receive operation.