exporterhelper

package
v0.123.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 31, 2025 License: Apache-2.0 Imports: 18 Imported by: 289

README

Exporter Helper

This is a helper exporter that other exporters can depend on. Today, it primarily offers queued retry capabilities.

⚠ This exporter should not be added to a service pipeline.

Configuration

The following configuration options can be modified:

  • retry_on_failure
    • enabled (default = true)
    • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
    • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
    • max_elapsed_time (default = 300s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false. If set to 0, the retries are never stopped.
  • sending_queue
    • enabled (default = true)
    • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
    • wait_for_result (default = false): determines if incoming requests are blocked until the request is processed or not.
    • block_on_overflow (default = false): If true, blocks the request until the queue has space otherwise rejects the data immediately; ignored if enabled is false
    • sizer (default = requests): How the queue and batching is measured. Available options:
      • requests: number of incoming batches of metrics, logs, traces (the most performant option);
      • items: number of the smallest parts of each signal (spans, metric data points, log records);
      • bytes: the size of serialized data in bytes (the least performant option).
    • queue_size (default = 1000): Maximum size the queue can accept. Measured in units defined by sizer
    • batch disabled by default if not defined
      • flush_timeout: time after which a batch will be sent regardless of its size.
      • min_size: the minimum size of a batch.
      • min_size: the maximum size of a batch, enables batch splitting.
  • timeout (default = 5s): Time to wait per individual attempt to send data to a backend

The initial_interval, max_interval, max_elapsed_time, and timeout options accept duration strings, valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

Persistent Queue

To use the persistent queue, the following setting needs to be set:

  • sending_queue
    • storage (default = none): When set, enables persistence and uses the component specified as a storage extension for the persistent queue. There is no in-memory queue when set.

The maximum number of batches stored to disk can be controlled using sending_queue.queue_size parameter (which, similarly as for in-memory buffering, defaults to 1000 batches).

When persistent queue is enabled, the batches are being buffered using the provided storage extension - filestorage is a popular and safe choice. If the collector instance is killed while having some items in the persistent queue, on restart the items will be picked and the exporting is continued.

                                                              ┌─Consumer #1─┐
                                                              │    ┌───┐    │
                              ──────Deleted──────        ┌───►│    │ 1 │    ├───► Success
        Waiting in channel    x           x     x        │    │    └───┘    │
        for consumer ───┐     x           x     x        │    │             │
                        │     x           x     x        │    └─────────────┘
                        ▼     x           x     x        │
┌─────────────────────────────────────────x─────x───┐    │    ┌─Consumer #2─┐
│                             x           x     x   │    │    │    ┌───┐    │
│     ┌───┐     ┌───┐ ┌───┐ ┌─x─┐ ┌───┐ ┌─x─┐ ┌─x─┐ │    │    │    │ 2 │    ├───► Permanent -> X
│ n+1 │ n │ ... │ 6 │ │ 5 │ │ 4 │ │ 3 │ │ 2 │ │ 1 │ ├────┼───►│    └───┘    │      failure
│     └───┘     └───┘ └───┘ └───┘ └───┘ └───┘ └───┘ │    │    │             │
│                                                   │    │    └─────────────┘
└───────────────────────────────────────────────────┘    │
   ▲              ▲     ▲           ▲                    │    ┌─Consumer #3─┐
   │              │     │           │                    │    │    ┌───┐    │
   │              │     │           │                    │    │    │ 3 │    ├───► (in progress)
 write          read    └─────┬─────┘                    ├───►│    └───┘    │
 index          index         │                          │    │             │
                              │                          │    └─────────────┘
                              │                          │
                          currently                      │    ┌─Consumer #4─┐
                          dispatched                     │    │    ┌───┐    │     Temporary
                                                         └───►│    │ 4 │    ├───►  failure
                                                              │    └───┘    │         │
                                                              │             │         │
                                                              └─────────────┘         │
                                                                     ▲                │
                                                                     └── Retry ───────┤
                                                                                      │
                                                                                      │
                                                   X  ◄────── Retry limit exceeded ───┘

Example:

receivers:
  otlp:
    protocols:
      grpc:
exporters:
  otlp:
    endpoint: <ENDPOINT>
    sending_queue:
      storage: file_storage/otc
extensions:
  file_storage/otc:
    directory: /var/lib/storage/otc
    timeout: 10s
service:
  extensions: [file_storage]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      exporters: [otlp]

Documentation

Overview

Package exporterhelper provides helper functions for exporters.

Index

Constants

This section is empty.

Variables

View Source 
var (
	RequestSizerTypeBytes    = request.SizerTypeBytes
	RequestSizerTypeItems    = request.SizerTypeItems
	RequestSizerTypeRequests = request.SizerTypeRequests
)
View Source 
var ErrQueueIsFull = queuebatch.ErrQueueIsFull
View Source 
var NewDefaultBatcherConfig = internal.NewDefaultBatcherConfig

Deprecated: [v0.123.0] use WithQueueBatch.

View Source 
var NewDefaultQueueConfig = internal.NewDefaultQueueConfig

NewDefaultQueueConfig returns the default config for QueueBatchConfig. By default, the queue stores 1000 requests of telemetry and is non-blocking when full.

Functions

func NewLogs added in v0.112.0 

func NewLogs(
	ctx context.Context,
	set exporter.Settings,
	cfg component.Config,
	pusher consumer.ConsumeLogsFunc,
	options ...Option,
) (exporter.Logs, error)

NewLogs creates an exporter.Logs that records observability logs and wraps every request with a Span.

func NewLogsRequest added in v0.112.0 

func NewLogsRequest(
	_ context.Context,
	set exporter.Settings,
	converter RequestConverterFunc[plog.Logs],
	pusher RequestConsumeFunc,
	options ...Option,
) (exporter.Logs, error)

NewLogsRequest creates new logs exporter based on custom LogsConverter and Sender. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

func NewMetrics added in v0.112.0 

func NewMetrics(
	ctx context.Context,
	set exporter.Settings,
	cfg component.Config,
	pusher consumer.ConsumeMetricsFunc,
	options ...Option,
) (exporter.Metrics, error)

NewMetrics creates an exporter.Metrics that records observability metrics and wraps every request with a Span.

func NewMetricsRequest added in v0.112.0 

func NewMetricsRequest(
	_ context.Context,
	set exporter.Settings,
	converter RequestConverterFunc[pmetric.Metrics],
	pusher RequestConsumeFunc,
	options ...Option,
) (exporter.Metrics, error)

NewMetricsRequest creates a new metrics exporter based on a custom MetricsConverter and Sender. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

func NewThrottleRetry 

func NewThrottleRetry(err error, delay time.Duration) error

NewThrottleRetry creates a new throttle retry error.

func NewTraces added in v0.112.0 

func NewTraces(
	ctx context.Context,
	set exporter.Settings,
	cfg component.Config,
	pusher consumer.ConsumeTracesFunc,
	options ...Option,
) (exporter.Traces, error)

NewTraces creates an exporter.Traces that records observability metrics and wraps every request with a Span.

func NewTracesRequest added in v0.112.0 

func NewTracesRequest(
	_ context.Context,
	set exporter.Settings,
	converter RequestConverterFunc[ptrace.Traces],
	pusher RequestConsumeFunc,
	options ...Option,
) (exporter.Traces, error)

NewTracesRequest creates a new traces exporter based on a custom TracesConverter and Sender. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

Types

type BatchConfig added in v0.123.0 

type BatchConfig = queuebatch.BatchConfig

BatchConfig defines a configuration for batching requests based on a timeout and a minimum number of items.

type BatcherConfig deprecated added in v0.123.0 

type BatcherConfig = internal.BatcherConfig

Deprecated: [v0.123.0] use WithQueueBatch.

type Option 

type Option = internal.Option

Option apply changes to BaseExporter.

func WithBatcher deprecated added in v0.97.0 

func WithBatcher(cfg BatcherConfig) Option

Deprecated: [v0.123.0] use WithQueueBatch.

func WithCapabilities 

func WithCapabilities(capabilities consumer.Capabilities) Option

WithCapabilities overrides the default Capabilities() function for a Consumer. The default is non-mutable data. TODO: Verify if we can change the default to be mutable as we do for processors.

func WithQueue 

func WithQueue(config QueueBatchConfig) Option

WithQueue overrides the default QueueBatchConfig for an exporter. The default QueueBatchConfig is to disable queueing. This option cannot be used with the new exporter helpers New[Traces|Metrics|Logs]RequestExporter.

func WithQueueBatch added in v0.123.0 

func WithQueueBatch(cfg QueueBatchConfig, set QueueBatchSettings) Option

WithQueueBatch enables queueing and batching for an exporter. This option should be used with the new exporter helpers New[Traces|Metrics|Logs]RequestExporter. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

func WithRequestQueue deprecated added in v0.94.0 

func WithRequestQueue(cfg QueueBatchConfig, encoding QueueBatchEncoding[Request]) Option

Deprecated: [v0.123.0] use WithQueueBatch.

func WithRetry 

func WithRetry(config configretry.BackOffConfig) Option

WithRetry overrides the default configretry.BackOffConfig for an exporter. The default configretry.BackOffConfig is to disable retries.

func WithShutdown 

func WithShutdown(shutdown component.ShutdownFunc) Option

WithShutdown overrides the default Shutdown function for an exporter. The default shutdown function does nothing and always returns nil.

func WithStart 

func WithStart(start component.StartFunc) Option

WithStart overrides the default Start function for an exporter. The default start function does nothing and always returns nil.

func WithTimeout 

func WithTimeout(timeoutConfig TimeoutConfig) Option

WithTimeout overrides the default TimeoutConfig for an exporter. The default TimeoutConfig is 5 seconds.

type QueueBatchConfig added in v0.123.0 

type QueueBatchConfig = queuebatch.Config

QueueBatchConfig defines configuration for queueing and batching for the exporter.

type QueueBatchEncoding added in v0.123.0 

type QueueBatchEncoding[T any] interface {
	// Marshal is a function that can marshal a request into bytes.
	Marshal(T) ([]byte, error)

	// Unmarshal is a function that can unmarshal bytes into a request.
	Unmarshal([]byte) (T, error)
}

QueueBatchEncoding defines the encoding to be used if persistent queue is configured. Duplicate definition with queuebatch.Encoding since aliasing generics is not supported by default.

type QueueBatchSettings added in v0.123.0 

type QueueBatchSettings = internal.QueueBatchSettings[Request]

QueueBatchSettings are settings for the QueueBatch component. They include things line Encoding to be used with persistent queue, or the available Sizers, etc.

func NewLogsQueueBatchSettings added in v0.123.0 

func NewLogsQueueBatchSettings() QueueBatchSettings

NewLogsQueueBatchSettings returns a new QueueBatchSettings to configure to WithQueueBatch when using plog.Logs. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

func NewMetricsQueueBatchSettings added in v0.123.0 

func NewMetricsQueueBatchSettings() QueueBatchSettings

NewMetricsQueueBatchSettings returns a new QueueBatchSettings to configure to WithQueueBatch when using pmetric.Metrics. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

func NewTracesQueueBatchSettings added in v0.123.0 

func NewTracesQueueBatchSettings() QueueBatchSettings

NewTracesQueueBatchSettings returns a new QueueBatchSettings to configure to WithQueueBatch when using ptrace.Traces. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type QueueConfig deprecated added in v0.110.0 

type QueueConfig = QueueBatchConfig

Deprecated: [v0.123.0] use QueueBatchConfig.

type Request added in v0.84.0 

type Request = request.Request

Request represents a single request that can be sent to an external endpoint. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type RequestConsumeFunc added in v0.122.0 

type RequestConsumeFunc = sender.SendFunc[Request]

RequestConsumeFunc processes the request. After the function returns, the request is no longer accessible, and accessing it is considered undefined behavior.

type RequestConverterFunc added in v0.122.0 

type RequestConverterFunc[T any] func(context.Context, T) (Request, error)

RequestConverterFunc converts pdata telemetry into a user-defined Request. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type RequestErrorHandler added in v0.89.0 

type RequestErrorHandler = request.ErrorHandler

RequestErrorHandler is an optional interface that can be implemented by Request to provide a way handle partial temporary failures. For example, if some items failed to process and can be retried, this interface allows to return a new Request that contains the items left to be sent. Otherwise, the original Request should be returned. If not implemented, the original Request will be returned assuming the error is applied to the whole Request. Experimental: This API is at the early stage of development and may change without backward compatibility until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.

type RequestSizer added in v0.123.0 

type RequestSizer = request.Sizer[Request]

RequestSizer is an interface that returns the size of the given request.

func NewRequestsSizer added in v0.123.0 

func NewRequestsSizer() RequestSizer

NewRequestsSizer returns a RequestSizer that counts the requests by the number of requests, always returning 1.

type RequestSizerType added in v0.123.0 

type RequestSizerType = request.SizerType

type SizeConfig deprecated added in v0.123.0 

type SizeConfig = internal.SizeConfig

Deprecated: [v0.123.0] use WithQueueBatch.

type TimeoutConfig added in v0.110.0 

type TimeoutConfig = internal.TimeoutConfig

func NewDefaultTimeoutConfig added in v0.110.0 

func NewDefaultTimeoutConfig() TimeoutConfig

NewDefaultTimeoutConfig returns the default config for TimeoutConfig.

Source Files

View all Source files

Directories

Path Synopsis
experr
hosttest
metadata
metadatatest
oteltest
queuebatch
Package queuebatch provides helper functions for exporter's queueing and batching.
 Package queuebatch provides helper functions for exporter's queueing and batching.
request
requesttest
sender
sendertest
sizer
storagetest

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.