filterprocessor

README

Filter Processor

Status
Stability	alpha: traces, metrics, logs
Distributions	core, contrib
Warnings	Orphaned Telemetry, Other
Issues
Code Owners	@TylerHelmuth, @boostchicken

The filterprocessor allows dropping spans, span events, metrics, datapoints, and logs from the collector.

Configuration

The filterprocessor utilizes the OpenTelemetry Transformation Language to create conditions that determine when telemetry should be dropped. If any condition is met, the telemetry is dropped (each condition is ORed together). Each configuration option corresponds with a different type of telemetry and OTTL Context. See the table below for details on each context and the fields it exposes.

Config	OTTL Context
traces.span	Span
traces.spanevent	SpanEvent
metrics.metric	Metric
metrics.datapoint	DataPoint
logs.log_record	Log

The OTTL allows the use of and, or, and () in conditions. See OTTL Boolean Expressions for more details.

For conditions that apply to the same signal, such as spans and span events, if the "higher" level telemetry matches a condition and is dropped, the "lower" level condition will not be checked. This means that if a span is dropped but a span event condition was defined, the span event condition will not be checked for that span. The same relationship applies to metrics and datapoints.

If all span events for a span are dropped, the span will be left intact. If all datapoints for a metric are dropped, the metric will also be dropped.

The filter processor also allows configuring an optional field, error_mode, which will determine how the processor reacts to errors that occur while processing an OTTL condition.

error_mode	description
ignore	The processor ignores errors returned by conditions, logs them, and continues on to the next condition. This is the recommended mode.
silent	The processor ignores errors returned by conditions, does not log them, and continues on to the next condition.
propagate	The processor returns the error up the pipeline. This will result in the payload being dropped from the collector.

If not specified, propagate will be used.

Examples

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'attributes["container.name"] == "app_container_1"'
        - 'resource.attributes["host.name"] == "localhost"'
        - 'name == "app_3"'
      spanevent:
        - 'attributes["grpc"] == true'
        - 'IsMatch(name, ".*grpc.*")'
    metrics:
      metric:
          - 'name == "my.metric" and resource.attributes["my_label"] == "abc123"'
          - 'type == METRIC_DATA_TYPE_HISTOGRAM'
      datapoint:
          - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
          - 'resource.attributes["service.name"] == "my_service_name"'
    logs:
      log_record:
        - 'IsMatch(body, ".*password.*")'
        - 'severity_number < SEVERITY_NUMBER_WARN'

Dropping data based on a resource attribute

processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - IsMatch(resource.attributes["k8s.pod.name"], "my-pod-name.*")

Dropping metrics with invalid type

processors:
  filter:
    error_mode: ignore
    metrics:
      metric:
        - type == METRIC_DATA_TYPE_NONE

Dropping specific metric and value

processors:
  filter:
    error_mode: ignore
    metrics:
      datapoint:
        - metric.name == "k8s.pod.phase" && value_int == 4

Dropping non-HTTP spans

processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - attributes["http.request.method"] == nil

Dropping HTTP spans

processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - attributes["http.request.method"] != nil

OTTL Functions

The filter processor has access to all OTTL Converter functions

In addition, the processor defines a few of its own functions:

Metrics only functions

• HasAttrKeyOnDatapoint
• HasAttrOnDatapoint

HasAttrKeyOnDatapoint

HasAttrKeyOnDatapoint(key)

Returns true if the given key appears in the attribute map of any datapoint on a metric. key must be a string. You must use the metrics.metric context.

Examples:

• HasAttrKeyOnDatapoint("http.method")

# Drops metrics containing the 'bad.metric' attribute key
filter/keep_good_metrics:
  error_mode: ignore
  metrics:
    metric:
      - 'HasAttrKeyOnDatapoint("bad.metric")'

HasAttrOnDatapoint

HasAttrOnDatapoint(key, value)

Returns true if the given key and value appears in the attribute map of any datapoint on a metric. key and value must both be strings. If the value of the attribute on the datapoint is not a string, value will be compared to "". You must use the metrics.metric context.

Examples:

• HasAttrOnDatapoint("http.method", "GET")

# Drops metrics containing the 'bad.metric' attribute key and 'true' value
filter/keep_good_metrics:
  error_mode: ignore
  metrics:
    metric:
      - 'HasAttrOnDatapoint("bad.metric", "true")'

Warnings

In general, understand your data before using the filter processor.

• When using the filterprocessor make sure you understand the look of your incoming data and test the configuration thoroughly. In general, use as specific a configuration as possible to lower the risk of the wrong data being dropped.
• Orphaned Telemetry: The processor allows dropping spans. Dropping a span may lead to orphaned spans if the dropped span is a parent. Dropping a span may lead to orphaned logs if the log references the dropped span.

Documentation

Overview

Package filterprocessor implements a processor for filtering (dropping) metrics and/or spans by various properties.

Index

• func NewFactory() processor.Factory
• type Config
  • func (cfg *Config) Validate() error
• type LogFilters
• type LogMatchProperties
• type LogMatchType
• type LogSeverityNumberMatchProperties
• type MetricFilters
• type TraceFilters

Functions

func NewFactory

func NewFactory() processor.Factory

NewFactory returns a new factory for the Filter processor.

Types

type Config

type Config struct {
	// ErrorMode determines how the processor reacts to errors that occur while processing an OTTL condition.
	// Valid values are `ignore` and `propagate`.
	// `ignore` means the processor ignores errors returned by conditions and continues on to the next condition. This is the recommended mode.
	// `propagate` means the processor returns the error up the pipeline.  This will result in the payload being dropped from the collector.
	// The default value is `propagate`.
	ErrorMode ottl.ErrorMode `mapstructure:"error_mode"`

	Metrics MetricFilters `mapstructure:"metrics"`

	Logs LogFilters `mapstructure:"logs"`

	Spans filterconfig.MatchConfig `mapstructure:"spans"`

	Traces TraceFilters `mapstructure:"traces"`
}

Config defines configuration for Resource processor.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate checks if the processor configuration is valid

type LogFilters

type LogFilters struct {
	// Include match properties describe logs that should be included in the Collector Service pipeline,
	// all other logs should be dropped from further processing.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Include *LogMatchProperties `mapstructure:"include"`
	// Exclude match properties describe logs that should be excluded from the Collector Service pipeline,
	// all other logs should be included.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Exclude *LogMatchProperties `mapstructure:"exclude"`

	// LogConditions is a list of OTTL conditions for an ottllog context.
	// If any condition resolves to true, the log event will be dropped.
	// Supports `and`, `or`, and `()`
	LogConditions []string `mapstructure:"log_record"`
}

LogFilters filters by Log properties.

type LogMatchProperties

type LogMatchProperties struct {
	// LogMatchType specifies the type of matching desired
	LogMatchType LogMatchType `mapstructure:"match_type"`

	// ResourceAttributes defines a list of possible resource attributes to match logs against.
	// A match occurs if any resource attribute matches all expressions in this given list.
	ResourceAttributes []filterconfig.Attribute `mapstructure:"resource_attributes"`

	// RecordAttributes defines a list of possible record attributes to match logs against.
	// A match occurs if any record attribute matches at least one expression in this given list.
	RecordAttributes []filterconfig.Attribute `mapstructure:"record_attributes"`

	// SeverityTexts is a list of strings that the LogRecord's severity text field must match
	// against.
	SeverityTexts []string `mapstructure:"severity_texts"`

	// SeverityNumberProperties defines how to match against a log record's SeverityNumber, if defined.
	SeverityNumberProperties *LogSeverityNumberMatchProperties `mapstructure:"severity_number"`

	// LogBodies is a list of strings that the LogRecord's body field must match
	// against.
	LogBodies []string `mapstructure:"bodies"`
}

LogMatchProperties specifies the set of properties in a log to match against and the type of string pattern matching to use.

type LogMatchType

type LogMatchType string

LogMatchType specifies the strategy for matching against `plog.Log`s.

type LogSeverityNumberMatchProperties

type LogSeverityNumberMatchProperties struct {
	// Min is the minimum severity needed for the log record to match.
	// This corresponds to the short names specified here:
	// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#displaying-severity
	// this field is case-insensitive ("INFO" == "info")
	Min logSeverity `mapstructure:"min"`

	// MatchUndefined lets logs records with "unknown" severity match.
	// If MinSeverity is not set, this field is ignored, as fields are not matched based on severity.
	MatchUndefined bool `mapstructure:"match_undefined"`
}

type MetricFilters

type MetricFilters struct {
	// Include match properties describe metrics that should be included in the Collector Service pipeline,
	// all other metrics should be dropped from further processing.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Include *filterconfig.MetricMatchProperties `mapstructure:"include"`

	// Exclude match properties describe metrics that should be excluded from the Collector Service pipeline,
	// all other metrics should be included.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Exclude *filterconfig.MetricMatchProperties `mapstructure:"exclude"`

	// RegexpConfig specifies options for the regexp match type
	RegexpConfig *regexp.Config `mapstructure:"regexp"`

	// MetricConditions is a list of OTTL conditions for an ottlmetric context.
	// If any condition resolves to true, the metric will be dropped.
	// Supports `and`, `or`, and `()`
	MetricConditions []string `mapstructure:"metric"`

	// DataPointConditions is a list of OTTL conditions for an ottldatapoint context.
	// If any condition resolves to true, the datapoint will be dropped.
	// Supports `and`, `or`, and `()`
	DataPointConditions []string `mapstructure:"datapoint"`
}

MetricFilters filters by Metric properties.

type TraceFilters

type TraceFilters struct {
	// SpanConditions is a list of OTTL conditions for an ottlspan context.
	// If any condition resolves to true, the span will be dropped.
	// Supports `and`, `or`, and `()`
	SpanConditions []string `mapstructure:"span"`

	// SpanEventConditions is a list of OTTL conditions for an ottlspanevent context.
	// If any condition resolves to true, the span event will be dropped.
	// Supports `and`, `or`, and `()`
	SpanEventConditions []string `mapstructure:"spanevent"`
}

TraceFilters filters by OTTL conditions