spanmetricsprocessor

README

Span Metrics Processor

Status
Stability	development
Supported pipeline types	traces
Distributions	contrib

Note: Currently experimental and subject to breaking changes (e.g. change from processor to exporter/translator component). See: https://github.com/asserts/opentelemetry-collector-contrib/issues/403.

Aggregates Request, Error and Duration (R.E.D) metrics from span data.

Request counts are computed as the number of spans seen per unique set of dimensions, including Errors. For example, the following metric shows 142 calls:

calls_total{http_method="GET",http_status_code="200",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 142

Multiple metrics can be aggregated if, for instance, a user wishes to view call counts just on service_name and operation.

Error counts are computed from the Request counts which have an "Error" Status Code metric dimension. For example, the following metric indicates 220 errors:

calls_total{http_method="GET",http_status_code="503",operation="/checkout",service_name="frontend",span_kind="SPAN_KIND_CLIENT",status_code="STATUS_CODE_ERROR"} 220

Duration is computed from the difference between the span start and end times and inserted into the relevant latency histogram time bucket for each unique set dimensions. For example, the following latency buckets indicate the vast majority of spans (9K) have a 100ms latency:

latency_bucket{http_method="GET",http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="2"} 327
latency_bucket{http_method="GET",http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="6"} 751
latency_bucket{http_method="GET",http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="10"} 1195
latency_bucket{http_method="GET",http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="100"} 10180
latency_bucket{http_method="GET",http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="250"} 10180
...

Each metric will have at least the following dimensions because they are common across all spans:

• Service name
• Operation
• Span kind
• Status code

This processor lets traces to continue through the pipeline unmodified.

The following settings are required:

• metrics_exporter: the name of the exporter that this processor will write metrics to. This exporter must be present in a pipeline.

The following settings can be optionally configured:

• latency_histogram_buckets: the list of durations defining the latency histogram buckets.

  • Default: [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]

• dimensions: the list of dimensions to add together with the default dimensions defined above.

  Each additional dimension is defined with a name which is looked up in the span's collection of attributes or resource attributes (AKA process tags) such as ip, host.name or region.

  If the named attribute is missing in the span, the optional provided default is used.

  If no default is provided, this dimension will be omitted from the metric.

• dimensions_cache_size: the max items number of metric_key_to_dimensions_cache. If not provided, will use default value size 1000.

• aggregation_temporality: Defines the aggregation temporality of the generated metrics. One of either AGGREGATION_TEMPORALITY_CUMULATIVE or AGGREGATION_TEMPORALITY_DELTA.

  • Default: AGGREGATION_TEMPORALITY_CUMULATIVE

• namespace: Defines the namespace of the generated metrics. If namespace provided, generated metric name will be added namespace. prefix.

Examples

The following is a simple example usage of the spanmetrics processor.

For configuration examples on other use cases, please refer to More Examples.

The full list of settings exposed for this processor are documented here.

receivers:
  jaeger:
    protocols:
      thrift_http:
        endpoint: "0.0.0.0:14278"

  # Dummy receiver that's never used, because a pipeline is required to have one.
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: "localhost:12345"

  otlp:
    protocols:
      grpc:
        endpoint: "localhost:55677"

processors:
  batch:
  spanmetrics:
    metrics_exporter: otlp/spanmetrics
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
    dimensions:
      - name: http.method
        default: GET
      - name: http.status_code
    dimensions_cache_size: 1000
    aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"     

exporters:
  jaeger:
    endpoint: localhost:14250

  otlp/spanmetrics:
    endpoint: "localhost:55677"
    tls:
      insecure: true

  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    traces:
      receivers: [jaeger]
      processors: [spanmetrics, batch]
      exporters: [jaeger]

    # The exporter name must match the metrics_exporter name.
    # The receiver is just a dummy and never used; added to pass validation requiring at least one receiver in a pipeline.
    metrics/spanmetrics:
      receivers: [otlp/spanmetrics]
      exporters: [otlp/spanmetrics]

    metrics:
      receivers: [otlp]
      exporters: [prometheus]

More Examples

For more example configuration covering various other use cases, please visit the testdata directory.

Documentation

Index

• func NewFactory() processor.Factory
• type Config
  • func (c Config) GetAggregationTemporality() pmetric.AggregationTemporality
• type Dimension

Functions

func NewFactory

func NewFactory() processor.Factory

NewFactory creates a factory for the spanmetrics processor.

Types

type Config

type Config struct {
	// MetricsExporter is the name of the metrics exporter to use to ship metrics.
	MetricsExporter string `mapstructure:"metrics_exporter"`

	// LatencyHistogramBuckets is the list of durations representing latency histogram buckets.
	// See defaultLatencyHistogramBucketsMs in processor.go for the default value.
	LatencyHistogramBuckets []time.Duration `mapstructure:"latency_histogram_buckets"`

	// Dimensions defines the list of additional dimensions on top of the provided:
	// - service.name
	// - operation
	// - span.kind
	// - status.code
	// The dimensions will be fetched from the span's attributes. Examples of some conventionally used attributes:
	// https://github.com/open-telemetry/opentelemetry-collector/blob/main/model/semconv/opentelemetry.go.
	Dimensions []Dimension `mapstructure:"dimensions"`

	// DimensionsCacheSize defines the size of cache for storing Dimensions, which helps to avoid cache memory growing
	// indefinitely over the lifetime of the collector.
	// Optional. See defaultDimensionsCacheSize in processor.go for the default value.
	DimensionsCacheSize int `mapstructure:"dimensions_cache_size"`

	AggregationTemporality string `mapstructure:"aggregation_temporality"`

	// MetricsEmitInterval is the time period between when metrics are flushed or emitted to the configured MetricsExporter.
	MetricsFlushInterval time.Duration `mapstructure:"metrics_flush_interval"`

	// Namespace is the namespace to use for the metrics.
	Namespace string `mapstructure:"namespace"`
	// contains filtered or unexported fields
}

Config defines the configuration options for spanmetricsprocessor.

func (Config) GetAggregationTemporality

func (c Config) GetAggregationTemporality() pmetric.AggregationTemporality

GetAggregationTemporality converts the string value given in the config into a AggregationTemporality. Returns cumulative, unless delta is correctly specified.

type Dimension

type Dimension struct {
	Name    string  `mapstructure:"name"`
	Default *string `mapstructure:"default"`
}

Dimension defines the dimension name and optional default value if the Dimension is missing from a span attribute.