prometheusremotewriteexporter

README

Prometheus Remote Write Exporter

Status
Stability	beta: metrics
Distributions	core, contrib
Issues
Code Owners	@Aneurysm9, @rapphil, @dashpole, @ArthurSens

Prometheus Remote Write Exporter sends OpenTelemetry metrics to Prometheus remote write compatible backends such as Cortex, Mimir, and Thanos. By default, this exporter requires TLS and offers queued retry capabilities.

⚠ Non-cumulative monotonic, histogram, and summary OTLP metrics are dropped by this exporter.

A design doc is available to document in detail how this exporter works.

Getting Started

The following settings are required:

• endpoint (no default): The remote write URL to send remote write samples.

By default, TLS is enabled and must be configured under tls::

• insecure (default = false): whether to enable client transport security for the exporter's connection.

As a result, the following parameters are also required under tls::

• cert_file (no default): path to the TLS cert to use for TLS required connections. Should only be used if insecure is set to false.
• key_file (no default): path to the TLS key to use for TLS required connections. Should only be used if insecure is set to false.

The following settings can be optionally configured:

• external_labels: map of labels names and values to be attached to each metric data point
• headers: additional headers attached to each HTTP request.
  • Note the following headers cannot be changed: Content-Encoding, Content-Type, X-Prometheus-Remote-Write-Version, and User-Agent.
• namespace: prefix attached to each exported metric name.
• add_metric_suffixes: If set to false, type and unit suffixes will not be added to metrics. Default: true.
• send_metadata: If set to true, prometheus metadata will be generated and sent. Default: false.
• remote_write_queue: fine tuning for queueing and sending of the outgoing remote writes.
  • enabled: enable the sending queue (default: true)
  • queue_size: number of OTLP metrics that can be queued. Ignored if enabled is false (default: 10000)
  • num_consumers: minimum number of workers to use to fan out the outgoing requests. (default: 5)
• resource_to_telemetry_conversion
  • enabled (default = false): If enabled is true, all the resource attributes will be converted to metric labels by default.
• target_info: customize target_info metric
  • enabled (default = true): If enabled is true, a target_info metric will be generated for each resource metric (see https://github.com/open-telemetry/opentelemetry-specification/pull/2381).
• export_created_metric: WARNING Deprecated and planned for removal in v0.116.0. See related issue for more information.
  • enabled (default = false): If enabled is true, a _created metric is exported for Summary, Histogram, and Monotonic Sum metric points if StartTimeUnixNano is set.
• max_batch_size_bytes (default = 3000000 -> ~2.861 mb): Maximum size of a batch of samples to be sent to the remote write endpoint. If the batch size is larger than this value, it will be split into multiple batches.

Example:

exporters:
  prometheusremotewrite:
    endpoint: "https://my-cortex:7900/api/v1/push"
    wal: # Enabling the Write-Ahead-Log for the exporter.
      directory: ./prom_rw # The directory to store the WAL in
      buffer_size: 100 # Optional count of elements to be read from the WAL before truncating; default of 300
      truncate_frequency: 45s # Optional frequency for how often the WAL should be truncated. It is a time.ParseDuration; default of 1m
    resource_to_telemetry_conversion:
      enabled: true # Convert resource attributes to metric labels

Example:

exporters:
  prometheusremotewrite:
    endpoint: "https://my-cortex:7900/api/v1/push"
    external_labels:
      label_name1: label_value1
      label_name2: label_value2

Advanced Configuration

Several helper files are leveraged to provide additional capabilities automatically:

• HTTP settings
• TLS and mTLS settings
• Retry and timeout settings, note that the exporter doesn't support sending_queue but provides remote_write_queue.

Feature gates

This exporter has feature gate: exporter.prometheusremotewritexporter.RetryOn429. When this feature gate is enable the prometheus remote write exporter will retry on 429 http status code with the provided retry configuration. It currently doesn't support respecting the http header Retry-After if provided since the retry library used doesn't support this feature.

To enable it run collector with enabled feature gate exporter.prometheusremotewritexporter.RetryOn429. This can be done by executing it with one additional parameter - --feature-gates=telemetry.useOtelForInternalMetrics.

Metric names and labels normalization

OpenTelemetry metric names and attributes are normalized to be compliant with Prometheus naming rules. Details on this normalization process are described in the Prometheus translator module.

Setting resource attributes as metric labels

By default, resource attributes are added to a special metric called target_info. To select and group by metrics by resource attributes, you need to do join on target_info. For example, to select metrics with k8s_namespace_name attribute equal to my-namespace:

app_ads_ad_requests_total * on (job, instance) group_left target_info{k8s_namespace_name="my-namespace"}

Or to group by a particular attribute (for ex. k8s_namespace_name):

sum by (k8s_namespace_name) (app_ads_ad_requests_total * on (job, instance) group_left(k8s_namespace_name) target_info)

This is not a common pattern, and we recommend copying the most common resource attributes into metric labels. You can do this through the transform processor:

processor:
  transform:
    metric_statements:
      - context: datapoint
        statements:
        - set(attributes["namespace"], resource.attributes["k8s.namespace.name"])
        - set(attributes["container"], resource.attributes["k8s.container.name"])
        - set(attributes["pod"], resource.attributes["k8s.pod.name"])

After this, grouping or selecting becomes as simple as:

app_ads_ad_requests_total{namespace="my-namespace"}

sum by (namespace) (app_ads_ad_requests_total)

Documentation

Overview

Package prometheusremotewriteexporter sends metrics data to Prometheus Remote Write (PRW) endpoints.

Index

• func NewFactory() exporter.Factory
• type Config
  • func (cfg *Config) Validate() error
• type CreatedMetric
• type RemoteWriteQueue
• type TargetInfo
• type WALConfig

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a new Prometheus Remote Write exporter.

Types

type Config

type Config struct {
	TimeoutSettings           exporterhelper.TimeoutConfig `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
	configretry.BackOffConfig `mapstructure:"retry_on_failure"`

	// prefix attached to each exported metric name
	// See: https://prometheus.io/docs/practices/naming/#metric-names
	Namespace string `mapstructure:"namespace"`

	// QueueConfig allows users to fine tune the queues
	// that handle outgoing requests.
	RemoteWriteQueue RemoteWriteQueue `mapstructure:"remote_write_queue"`

	// ExternalLabels defines a map of label keys and values that are allowed to start with reserved prefix "__"
	ExternalLabels map[string]string `mapstructure:"external_labels"`

	ClientConfig confighttp.ClientConfig `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.

	// maximum size in bytes of time series batch sent to remote storage
	MaxBatchSizeBytes int `mapstructure:"max_batch_size_bytes"`

	// ResourceToTelemetrySettings is the option for converting resource attributes to telemetry attributes.
	// "Enabled" - A boolean field to enable/disable this option. Default is `false`.
	// If enabled, all the resource attributes will be converted to metric labels by default.
	ResourceToTelemetrySettings resourcetotelemetry.Settings `mapstructure:"resource_to_telemetry_conversion"`
	WAL                         *WALConfig                   `mapstructure:"wal"`

	// TargetInfo allows customizing the target_info metric
	TargetInfo *TargetInfo `mapstructure:"target_info,omitempty"`

	// CreatedMetric allows customizing creation of _created metrics
	// Deprecated[0.114.0]: The feature doesn't provide the expected behavior. Use Prometheus remote-write v2 to enable sending Created Timestamps.
	// This feature is planned to be removed in v0.116.0
	CreatedMetric *CreatedMetric `mapstructure:"export_created_metric,omitempty"`

	// AddMetricSuffixes controls whether unit and type suffixes are added to metrics on export
	AddMetricSuffixes bool `mapstructure:"add_metric_suffixes"`

	// SendMetadata controls whether prometheus metadata will be generated and sent
	SendMetadata bool `mapstructure:"send_metadata"`
}

Config defines configuration for Remote Write exporter.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate checks if the exporter configuration is valid

type CreatedMetric

type CreatedMetric struct {
	// Enabled if true the _created metrics could be exported
	Enabled bool `mapstructure:"enabled"`
}

type RemoteWriteQueue

type RemoteWriteQueue struct {
	// Enabled if false the queue is not enabled, the export requests
	// are executed synchronously.
	Enabled bool `mapstructure:"enabled"`

	// QueueSize is the maximum number of OTLP metric batches allowed
	// in the queue at a given time. Ignored if Enabled is false.
	QueueSize int `mapstructure:"queue_size"`

	// NumWorkers configures the number of workers used by
	// the collector to fan out remote write requests.
	NumConsumers int `mapstructure:"num_consumers"`
}

RemoteWriteQueue allows to configure the remote write queue.

type TargetInfo

type TargetInfo struct {
	// Enabled if false the target_info metric is not generated by the exporter
	Enabled bool `mapstructure:"enabled"`
}

type WALConfig

type WALConfig struct {
	Directory         string        `mapstructure:"directory"`
	BufferSize        int           `mapstructure:"buffer_size"`
	TruncateFrequency time.Duration `mapstructure:"truncate_frequency"`
}