exporterhelper

package
Version: v0.37.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2021 License: Apache-2.0 Imports: 27 Imported by: 67

README

Exporter Helper

This is a helper exporter that other exporters can depend on. Today, it primarily offers queued retries and resource attributes to metric labels conversion.

⚠️ This exporter should not be added to a service pipeline.

Configuration

The following configuration options can be modified:

  • retry_on_failure
    • enabled (default = true)
    • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
    • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
    • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false
  • sending_queue
    • enabled (default = true)
    • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
    • queue_size (default = 5000): Maximum number of batches kept in memory before dropping; ignored if enabled is false User should calculate this as num_seconds * requests_per_second where:
      • num_seconds is the number of seconds to buffer in case of a backend outage
      • requests_per_second is the average number of requests per seconds.
  • resource_to_telemetry_conversion
    • enabled (default = false): If enabled is true, all the resource attributes will be converted to metric labels by default.
  • timeout (default = 5s): Time to wait per individual attempt to send data to a backend.

The full list of settings exposed for this helper exporter are documented here.

Persistent Queue

Status: under development

⚠️ The capability is under development and currently can be enabled only in OpenTelemetry Collector Contrib with enable_unstable build tag set.

With this build tag set, additional configuration option can be enabled:

  • sending_queue
    • persistent_storage_enabled (default = false): When set, enables persistence via a file storage extension (note, enable_unstable build tag needs to be enabled first, see below for more details)

The maximum number of batches stored to disk can be controlled using sending_queue.queue_size parameter (which, similarly as for in-memory buffering, defaults to 5000 batches).

When persistent_storage_enabled is set to true, the queue is being buffered to disk using file storage extension. If collector instance is killed while having some items in the persistent queue, on restart the items are being picked and the exporting is continued.

                                                              ┌─Consumer #1─┐
                                                              │    ┌───┐    │
                              ──────Deleted──────        ┌───►│    │ 1 │    ├───► Success
        Waiting in channel    x           x     x        │    │    └───┘    │
        for consumer ───┐     x           x     x        │    │             │
                        │     x           x     x        │    └─────────────┘
                        ▼     x           x     x        │
┌─────────────────────────────────────────x─────x───┐    │    ┌─Consumer #2─┐
│                             x           x     x   │    │    │    ┌───┐    │
│     ┌───┐     ┌───┐ ┌───┐ ┌─x─┐ ┌───┐ ┌─x─┐ ┌─x─┐ │    │    │    │ 2 │    ├───► Permanent -> X
│ n+1 │ n │ ... │ 6 │ │ 5 │ │ 4 │ │ 3 │ │ 2 │ │ 1 │ ├────┼───►│    └───┘    │      failure
│     └───┘     └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ │    │    │             │
│                                                   │    │    └─────────────┘
└───────────────────────────────────────────────────┘    │
   ▲              ▲     ▲           ▲                    │    ┌─Consumer #3─┐
   │              │     │           │                    │    │    ┌───┐    │
   │              │     │           │                    │    │    │ 3 │    ├───► (in progress)
 write          read    └─────┬─────┘                    ├───►│    └───┘    │
 index          index         │                          │    │             │
   ▲                          │                          │    └─────────────┘
   │                          │                          │
   │                      currently                      │    ┌─Consumer #4─┐
   │                      dispatched                     │    │    ┌───┐    │     Temporary
   │                                                     └───►│    │ 4 │    ├───►  failure
   │                                                          │    └───┘    │         │
   │                                                          │             │         │
   │                                                          └─────────────┘         │
   │                                                                 ▲                │
   │                                                                 └── Retry ───────┤
   │                                                                                  │
   │                                                                                  │
   └────────────────────────────────────── Requeuing  ◄────── Retry limit exceeded ───┘

Example:

receivers:
  otlp:
    protocols:
      grpc:
exporters:
  otlp:
    endpoint: <ENDPOINT>
    sending_queue:
      persistent_storage_enabled: true
extensions:
  file_storage:
    directory: /var/lib/storage/otc
    timeout: 10s
service:
  extensions: [file_storage]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      exporters: [otlp]

Documentation

Overview

Package exporterhelper provides helper functions for exporters.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewFactory added in v0.7.0

func NewFactory(
	cfgType config.Type,
	createDefaultConfig CreateDefaultConfig,
	options ...FactoryOption) component.ExporterFactory

NewFactory returns a component.ExporterFactory.

func NewLogsExporter

NewLogsExporter creates an LogsExporter that records observability metrics and wraps every request with a Span.

func NewMetricsExporter

NewMetricsExporter creates an MetricsExporter that records observability metrics and wraps every request with a Span.

func NewThrottleRetry added in v0.7.0

func NewThrottleRetry(err error, delay time.Duration) error

NewThrottleRetry creates a new throttle retry error.

func NewTracesExporter added in v0.25.0

NewTracesExporter creates a TracesExporter that records observability metrics and wraps every request with a Span.

Types

type CreateDefaultConfig added in v0.7.0

type CreateDefaultConfig func() config.Exporter

CreateDefaultConfig is the equivalent of component.ExporterFactory.CreateDefaultConfig()

type CreateLogsExporter added in v0.7.0

CreateLogsExporter is the equivalent of component.ExporterFactory.CreateLogsExporter()

type CreateMetricsExporter added in v0.7.0

CreateMetricsExporter is the equivalent of component.ExporterFactory.CreateMetricsExporter()

type CreateTracesExporter added in v0.24.0

CreateTracesExporter is the equivalent of component.ExporterFactory.CreateTracesExporter()

type FactoryOption added in v0.7.0

type FactoryOption func(o *factory)

FactoryOption apply changes to ExporterOptions.

func WithLogs added in v0.7.0

func WithLogs(createLogsExporter CreateLogsExporter) FactoryOption

WithLogs overrides the default "error not supported" implementation for CreateLogsReceiver.

func WithMetrics added in v0.7.0

func WithMetrics(createMetricsExporter CreateMetricsExporter) FactoryOption

WithMetrics overrides the default "error not supported" implementation for CreateMetricsReceiver.

func WithTraces added in v0.7.0

func WithTraces(createTracesExporter CreateTracesExporter) FactoryOption

WithTraces overrides the default "error not supported" implementation for CreateTracesReceiver.

type Option added in v0.16.0

type Option func(*baseSettings)

Option apply changes to baseSettings.

func WithCapabilities added in v0.27.0

func WithCapabilities(capabilities consumer.Capabilities) Option

WithCapabilities overrides the default Capabilities() function for a Consumer. The default is non-mutable data. TODO: Verify if we can change the default to be mutable as we do for processors.

func WithQueue added in v0.7.0

func WithQueue(queueSettings QueueSettings) Option

WithQueue overrides the default QueueSettings for an exporter. The default QueueSettings is to disable queueing.

func WithRetry added in v0.7.0

func WithRetry(retrySettings RetrySettings) Option

WithRetry overrides the default RetrySettings for an exporter. The default RetrySettings is to disable retries.

func WithShutdown

func WithShutdown(shutdown componenthelper.ShutdownFunc) Option

WithShutdown overrides the default Shutdown function for an exporter. The default shutdown function does nothing and always returns nil.

func WithStart added in v0.6.0

func WithStart(start componenthelper.StartFunc) Option

WithStart overrides the default Start function for an exporter. The default start function does nothing and always returns nil.

func WithTimeout added in v0.7.0

func WithTimeout(timeoutSettings TimeoutSettings) Option

WithTimeout overrides the default TimeoutSettings for an exporter. The default TimeoutSettings is 5 seconds.

type QueueSettings added in v0.7.0

type QueueSettings struct {
	// Enabled indicates whether to not enqueue batches before sending to the consumerSender.
	Enabled bool `mapstructure:"enabled"`
	// NumConsumers is the number of consumers from the queue.
	NumConsumers int `mapstructure:"num_consumers"`
	// QueueSize is the maximum number of batches allowed in queue at a given time.
	QueueSize int `mapstructure:"queue_size"`
}

QueueSettings defines configuration for queueing batches before sending to the consumerSender.

func DefaultQueueSettings added in v0.16.0

func DefaultQueueSettings() QueueSettings

DefaultQueueSettings returns the default settings for QueueSettings.

type RetrySettings added in v0.7.0

type RetrySettings struct {
	// Enabled indicates whether to not retry sending batches in case of export failure.
	Enabled bool `mapstructure:"enabled"`
	// InitialInterval the time to wait after the first failure before retrying.
	InitialInterval time.Duration `mapstructure:"initial_interval"`
	// MaxInterval is the upper bound on backoff interval. Once this value is reached the delay between
	// consecutive retries will always be `MaxInterval`.
	MaxInterval time.Duration `mapstructure:"max_interval"`
	// MaxElapsedTime is the maximum amount of time (including retries) spent trying to send a request/batch.
	// Once this value is reached, the data is discarded.
	MaxElapsedTime time.Duration `mapstructure:"max_elapsed_time"`
}

RetrySettings defines configuration for retrying batches in case of export failure. The current supported strategy is exponential backoff.

func DefaultRetrySettings added in v0.16.0

func DefaultRetrySettings() RetrySettings

DefaultRetrySettings returns the default settings for RetrySettings.

type TimeoutSettings added in v0.7.0

type TimeoutSettings struct {
	// Timeout is the timeout for every attempt to send data to the backend.
	Timeout time.Duration `mapstructure:"timeout"`
}

TimeoutSettings for timeout. The timeout applies to individual attempts to send data to the backend.

func DefaultTimeoutSettings added in v0.16.0

func DefaultTimeoutSettings() TimeoutSettings

DefaultTimeoutSettings returns the default settings for TimeoutSettings.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL