filterprocessor

package module
v0.135.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 8, 2025 License: Apache-2.0 Imports: 36 Imported by: 27

README

Filter Processor

Status
Stability alpha: traces, metrics, logs
Distributions core, contrib, k8s
Warnings Orphaned Telemetry, Other
Issues Open issues Closed issues
Code coverage codecov
Code Owners @TylerHelmuth, @boostchicken, @evan-bradley, @edmocosta

The filterprocessor allows dropping spans, span events, metrics, datapoints, and logs from the collector.

Configuration

The filterprocessor utilizes the OpenTelemetry Transformation Language to create conditions that determine when telemetry should be dropped. If any condition is met, the telemetry is dropped (each condition is ORed together). Each configuration option corresponds with a different type of telemetry and OTTL Context. See the table below for details on each context and the fields it exposes.

Config OTTL Context
traces.span Span
traces.spanevent SpanEvent
metrics.metric Metric
metrics.datapoint DataPoint
logs.log_record Log

The OTTL allows the use of and, or, and () in conditions. See OTTL Boolean Expressions for more details.

For conditions that apply to the same signal, such as spans and span events, if the "higher" level telemetry matches a condition and is dropped, the "lower" level condition will not be checked. This means that if a span is dropped but a span event condition was defined, the span event condition will not be checked for that span. The same relationship applies to metrics and datapoints.

If all span events for a span are dropped, the span will be left intact. If all datapoints for a metric are dropped, the metric will also be dropped.

The filter processor also allows configuring an optional field, error_mode, which will determine how the processor reacts to errors that occur while processing an OTTL condition.

error_mode description
ignore The processor ignores errors returned by conditions, logs them, and continues on to the next condition. This is the recommended mode.
silent The processor ignores errors returned by conditions, does not log them, and continues on to the next condition.
propagate The processor returns the error up the pipeline. This will result in the payload being dropped from the collector.

If not specified, propagate will be used.

Examples
processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'attributes["container.name"] == "app_container_1"'
        - 'resource.attributes["host.name"] == "localhost"'
        - 'name == "app_3"'
      spanevent:
        - 'attributes["grpc"] == true'
        - 'IsMatch(name, ".*grpc.*")'
    metrics:
      metric:
          - 'name == "my.metric" and resource.attributes["my_label"] == "abc123"'
          - 'type == METRIC_DATA_TYPE_HISTOGRAM'
      datapoint:
          - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
          - 'resource.attributes["service.name"] == "my_service_name"'
    logs:
      log_record:
        - 'IsMatch(body, ".*password.*")'
        - 'severity_number < SEVERITY_NUMBER_WARN'
Dropping data based on a resource attribute
processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - IsMatch(resource.attributes["k8s.pod.name"], "my-pod-name.*")
Dropping metrics with invalid type
processors:
  filter:
    error_mode: ignore
    metrics:
      metric:
        - type == METRIC_DATA_TYPE_NONE
Dropping specific metric and value
processors:
  filter:
    error_mode: ignore
    metrics:
      datapoint:
        - metric.name == "k8s.pod.phase" and value_int == 4
Dropping non-HTTP spans
processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - attributes["http.request.method"] == nil
Dropping HTTP spans
processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - attributes["http.request.method"] != nil
Dropping non-error spans with a duration of less than 1 second
processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - (end_time - start_time) < Duration("1s") and status.code != STATUS_CODE_ERROR
OTTL Functions

The filter processor has access to all OTTL Converter functions

In addition, the processor defines a few of its own functions:

Metrics only functions

HasAttrKeyOnDatapoint

HasAttrKeyOnDatapoint(key)

Returns true if the given key appears in the attribute map of any datapoint on a metric. key must be a string. You must use the metrics.metric context.

Examples:

  • HasAttrKeyOnDatapoint("http.method")
# Drops metrics containing the 'bad.metric' attribute key
filter/keep_good_metrics:
  error_mode: ignore
  metrics:
    metric:
      - 'HasAttrKeyOnDatapoint("bad.metric")'
HasAttrOnDatapoint

HasAttrOnDatapoint(key, value)

Returns true if the given key and value appears in the attribute map of any datapoint on a metric. key and value must both be strings. If the value of the attribute on the datapoint is not a string, value will be compared to "". You must use the metrics.metric context.

Examples:

  • HasAttrOnDatapoint("http.method", "GET")
# Drops metrics containing the 'bad.metric' attribute key and 'true' value
filter/keep_good_metrics:
  error_mode: ignore
  metrics:
    metric:
      - 'HasAttrOnDatapoint("bad.metric", "true")'

Troubleshooting

When using OTTL you can enable debug logging in the collector to print out useful information, such as if the condition matched and the TransformContext used in the condition, to help you troubleshoot why a condition is not behaving as you expect. This feature is very verbose, but provides you an accurate view into how OTTL views the underlying data.

receivers:
  filelog:
    start_at: beginning
    include: [ /Users/tylerhelmuth/projects/opentelemetry-collector-contrib/local/test.log ]


processors:
  filter:
    error_mode: ignore
    logs:
      log_record:
        - body == "test"

exporters:
  debug:

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    logs:
      receivers:
        - filelog
      processors:
        - filter
      exporters:
        - debug

2024-05-29T16:47:04.362-0600    debug   ottl@v0.101.0/parser.go:338     condition evaluation result     {"kind": "processor", "name": "filter", "pipeline": "logs", "condition": "body == \"test\"", "match": true, "TransformContext": {"resource": {"attributes": {}, "dropped_attribute_count": 0}, "scope": {"attributes": {}, "dropped_attribute_count": 0, "name": "", "version": ""}, "log_record": {"attributes": {"log.file.name": "test.log"}, "body": "test", "dropped_attribute_count": 0, "flags": 0, "observed_time_unix_nano": 1717022824262063000, "severity_number": 0, "severity_text": "", "span_id": "", "time_unix_nano": 0, "trace_id": ""}, "cache": {}}}

Warnings

In general, understand your data before using the filter processor.

  • When using the filterprocessor make sure you understand the look of your incoming data and test the configuration thoroughly. In general, use as specific a configuration as possible to lower the risk of the wrong data being dropped.
  • Orphaned Telemetry: The processor allows dropping spans. Dropping a span may lead to orphaned spans if the dropped span is a parent. Dropping a span may lead to orphaned logs if the log references the dropped span.

Documentation

Overview

Package filterprocessor implements a processor for filtering (dropping) metrics and/or spans by various properties.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DefaultDataPointFunctions added in v0.133.0 

func DefaultDataPointFunctions() []ottl.Factory[ottldatapoint.TransformContext]

func DefaultLogFunctions added in v0.133.0 

func DefaultLogFunctions() []ottl.Factory[ottllog.TransformContext]

func DefaultMetricFunctions added in v0.133.0 

func DefaultMetricFunctions() []ottl.Factory[ottlmetric.TransformContext]

func DefaultSpanEventFunctions added in v0.133.0 

func DefaultSpanEventFunctions() []ottl.Factory[ottlspanevent.TransformContext]

func DefaultSpanFunctions added in v0.133.0 

func DefaultSpanFunctions() []ottl.Factory[ottlspan.TransformContext]

func NewFactory 

func NewFactory() processor.Factory

NewFactory returns a new factory for the Filter processor.

func NewFactoryWithOptions added in v0.133.0 

func NewFactoryWithOptions(options ...FactoryOption) processor.Factory

NewFactoryWithOptions can receive FactoryOption like With*Functions to register non-default OTTL functions in the resulting processor.

Types

type Config 

type Config struct {
	// ErrorMode determines how the processor reacts to errors that occur while processing an OTTL condition.
	// Valid values are `ignore` and `propagate`.
	// `ignore` means the processor ignores errors returned by conditions and continues on to the next condition. This is the recommended mode.
	// `propagate` means the processor returns the error up the pipeline.  This will result in the payload being dropped from the collector.
	// The default value is `propagate`.
	ErrorMode ottl.ErrorMode `mapstructure:"error_mode"`

	Metrics MetricFilters `mapstructure:"metrics"`

	Logs LogFilters `mapstructure:"logs"`

	Spans filterconfig.MatchConfig `mapstructure:"spans"`

	Traces TraceFilters `mapstructure:"traces"`
	// contains filtered or unexported fields
}

Config defines configuration for Resource processor.

func (*Config) Validate 

func (cfg *Config) Validate() error

Validate checks if the processor configuration is valid

type FactoryOption added in v0.133.0 

type FactoryOption func(factory *filterProcessorFactory)

FactoryOption applies changes to filterProcessorFactory.

func WithDataPointFunctions added in v0.133.0 

func WithDataPointFunctions(dataPointFunctions []ottl.Factory[ottldatapoint.TransformContext]) FactoryOption

WithDataPointFunctions will override the default OTTL datapoint context functions with the provided dataPointFunctions in resulting processor. Subsequent uses of WithDataPointFunctions will merge the provided dataPointFunctions with the previously registered functions.

func WithLogFunctions added in v0.133.0 

func WithLogFunctions(logFunctions []ottl.Factory[ottllog.TransformContext]) FactoryOption

WithLogFunctions will override the default OTTL log context functions with the provided logFunctions in the resulting processor. Subsequent uses of WithLogFunctions will merge the provided logFunctions with the previously registered functions.

func WithMetricFunctions added in v0.133.0 

func WithMetricFunctions(metricFunctions []ottl.Factory[ottlmetric.TransformContext]) FactoryOption

WithMetricFunctions will override the default OTTL metric context functions with the provided metricFunctions in the resulting processor. Subsequent uses of WithMetricFunctions will merge the provided metricFunctions with the previously registered functions.

func WithSpanEventFunctions added in v0.133.0 

func WithSpanEventFunctions(spanEventFunctions []ottl.Factory[ottlspanevent.TransformContext]) FactoryOption

WithSpanEventFunctions will override the default OTTL spanevent context functions with the provided spanEventFunctions in the resulting processor. Subsequent uses of WithSpanEventFunctions will merge the provided spanEventFunctions with the previously registered functions.

func WithSpanFunctions added in v0.133.0 

func WithSpanFunctions(spanFunctions []ottl.Factory[ottlspan.TransformContext]) FactoryOption

WithSpanFunctions will override the default OTTL span context functions with the provided spanFunctions in the resulting processor. Subsequent uses of WithSpanFunctions will merge the provided spanFunctions with the previously registered functions.

type LogFilters 

type LogFilters struct {
	// Include match properties describe logs that should be included in the Collector Service pipeline,
	// all other logs should be dropped from further processing.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Include *LogMatchProperties `mapstructure:"include"`
	// Exclude match properties describe logs that should be excluded from the Collector Service pipeline,
	// all other logs should be included.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Exclude *LogMatchProperties `mapstructure:"exclude"`

	// LogConditions is a list of OTTL conditions for an ottllog context.
	// If any condition resolves to true, the log event will be dropped.
	// Supports `and`, `or`, and `()`
	LogConditions []string `mapstructure:"log_record"`
}

LogFilters filters by Log properties.

type LogMatchProperties added in v0.36.0 

type LogMatchProperties struct {
	// LogMatchType specifies the type of matching desired
	LogMatchType LogMatchType `mapstructure:"match_type"`

	// ResourceAttributes defines a list of possible resource attributes to match logs against.
	// A match occurs if any resource attribute matches all expressions in this given list.
	ResourceAttributes []filterconfig.Attribute `mapstructure:"resource_attributes"`

	// RecordAttributes defines a list of possible record attributes to match logs against.
	// A match occurs if any record attribute matches at least one expression in this given list.
	RecordAttributes []filterconfig.Attribute `mapstructure:"record_attributes"`

	// SeverityTexts is a list of strings that the LogRecord's severity text field must match
	// against.
	SeverityTexts []string `mapstructure:"severity_texts"`

	// SeverityNumberProperties defines how to match against a log record's SeverityNumber, if defined.
	SeverityNumberProperties *LogSeverityNumberMatchProperties `mapstructure:"severity_number"`

	// LogBodies is a list of strings that the LogRecord's body field must match
	// against.
	LogBodies []string `mapstructure:"bodies"`
}

LogMatchProperties specifies the set of properties in a log to match against and the type of string pattern matching to use.

type LogMatchType added in v0.36.0 

type LogMatchType string

LogMatchType specifies the strategy for matching against `plog.Log`s.

type LogSeverityNumberMatchProperties added in v0.58.0 

type LogSeverityNumberMatchProperties struct {
	// Min is the minimum severity needed for the log record to match.
	// This corresponds to the short names specified here:
	// https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#displaying-severity
	// this field is case-insensitive ("INFO" == "info")
	Min logSeverity `mapstructure:"min"`

	// MatchUndefined lets logs records with "unknown" severity match.
	// If MinSeverity is not set, this field is ignored, as fields are not matched based on severity.
	MatchUndefined bool `mapstructure:"match_undefined"`
}

type MetricFilters 

type MetricFilters struct {
	// Include match properties describe metrics that should be included in the Collector Service pipeline,
	// all other metrics should be dropped from further processing.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Include *filterconfig.MetricMatchProperties `mapstructure:"include"`

	// Exclude match properties describe metrics that should be excluded from the Collector Service pipeline,
	// all other metrics should be included.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Exclude *filterconfig.MetricMatchProperties `mapstructure:"exclude"`

	// RegexpConfig specifies options for the regexp match type
	RegexpConfig *regexp.Config `mapstructure:"regexp"`

	// MetricConditions is a list of OTTL conditions for an ottlmetric context.
	// If any condition resolves to true, the metric will be dropped.
	// Supports `and`, `or`, and `()`
	MetricConditions []string `mapstructure:"metric"`

	// DataPointConditions is a list of OTTL conditions for an ottldatapoint context.
	// If any condition resolves to true, the datapoint will be dropped.
	// Supports `and`, `or`, and `()`
	DataPointConditions []string `mapstructure:"datapoint"`
}

MetricFilters filters by Metric properties.

type TraceFilters added in v0.65.0 

type TraceFilters struct {
	// SpanConditions is a list of OTTL conditions for an ottlspan context.
	// If any condition resolves to true, the span will be dropped.
	// Supports `and`, `or`, and `()`
	SpanConditions []string `mapstructure:"span"`

	// SpanEventConditions is a list of OTTL conditions for an ottlspanevent context.
	// If any condition resolves to true, the span event will be dropped.
	// Supports `and`, `or`, and `()`
	SpanEventConditions []string `mapstructure:"spanevent"`
}

TraceFilters filters by OTTL conditions

Source Files

View all Source files

Directories

Path Synopsis
internal
metadata
metadatatest

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.