opensearchexporter

README

OpenSearch Exporter

Status
Stability	alpha: traces, logs
Distributions	contrib
Issues
Code coverage
Code Owners	@ps48 | Seeking more code owners!
Emeritus	@Aneurysm9, @MitchellGale, @MaxKsyunz, @YANG-DB

OpenSearch exporter supports sending OpenTelemetry signals as documents to OpenSearch.

The documents are sent using observability catalog schema.

Configuration options

Indexing Options

The Observability indices would follow the recommended pattern for immutable data stream ingestion using the data_stream concepts. Index pattern will follow the next naming template ss4o_{type}-{dataset}-{namespace}

• dataset (default=default) a user-provided label to classify source of telemetry. It is used to construct the name of the destination index or data stream.
• namespace (default=namespace) a user-provided label to group telemetry. It is used to construct the name of the destination index or data stream.

Dynamic Indexing

The OpenSearch exporter supports dynamic index names for both logs and traces using placeholders. You can use any attribute or context key as a placeholder to construct index names dynamically per record.

Caution: In practice, OpenSearch clusters can become unstable or even break down when index counts reach a very high level. Introducing attributes with high cardinality resulting in many separate indices can significantly impact the stability of your target cluster.

Configuration Options:

• logs_index - Custom index name pattern for logs
• logs_index_fallback - Fallback value when placeholder is missing (default: unknown)
• logs_index_time_format - Time suffix format for logs
• traces_index - Custom index name pattern for traces
• traces_index_fallback - Fallback value when placeholder is missing (default: unknown)
• traces_index_time_format - Time suffix format for traces

Placeholder Syntax:

• Placeholder: %{key}
  • Example: otel-logs-%{service.name} or otel-traces-%{service.name}-%{env}
  • Multiple placeholders are supported per index name
  • The value is looked up from item attributes (log/span), scope attributes, and resource attributes (in that precedence order)
  • If the key is missing, the fallback value is used
  • Generated index names must adhere to OpenSearch index naming restrictions

Time Suffix Format:

Both logs and traces support time-formatted suffixes using *_time_format options. The time suffix only supports UTC.

• Valid tokens (case-sensitive):
  • yyyy (4-digit year), yy (2-digit year)
  • MM (2-digit month), dd (2-digit day)
  • HH (2-digit hour, 24h), mm (2-digit minute), ss (2-digit second)
• Allowed separators: -, ., _, +
• Examples: yyyy.MM.dd → 2024.06.07, yyyy-MM → 2024-06, yyMMdd → 240607

Default Behavior:

If custom index names are not set, the exporter uses default patterns:

• Logs: ss4o_logs-{dataset}-{namespace}
• Traces: ss4o_traces-{dataset}-{namespace}

Example Configuration

exporters:
  opensearch:
    http:
      endpoint: http://opensearch.example.com:9200
    # Logs configuration
    logs_index: "otel-logs-%{service.name}-%{env}"
    logs_index_fallback: "default"
    logs_index_time_format: "yyyy.MM.dd"
    # Traces configuration
    traces_index: "otel-traces-%{service.name}-%{deployment.environment}"
    traces_index_fallback: "unknown"
    traces_index_time_format: "yyyy.MM.dd"
    sending_queue:
      batch:

This configuration will create:

• Log indexes like: otel-logs-myservice-prod-2024.06.07
• Trace indexes like: otel-traces-myservice-production-2024.06.07

If any placeholder key is missing, the fallback value is used e.g.:

• otel-logs-myservice-default-2024.06.07).
• otel-traces-unknown-production-2024.06.07

OpenSearch document mapping

The mapping mode can be controlled via the scope attribute opensearch.mapping.mode.

The OpenSearch exporter supports several document schemas and preprocessing behaviors, which may be configured through the following settings:

• mapping:
  • mode (default=ss4o): Configures the field mappings. Supported modes are:
    • ss4o: Exports logs in the Simple Schema for Observability standard.
    • ecs: Maps fields defined in the OpenTelemetry Semantic Conventions to the Elastic Common Schema
    • flatten_attributes: Uses the ECS mapping but flattens all resource and log attributes in the record to the top-level.
    • bodymap: uses the "body" of a log record as the exact content of the OpenSearch document, without any transformation. This mapping mode is intended for use cases where the client wishes to have complete control over the OpenSearch document structure.
    • otel-v1: exports logs and traces using the Data Prepper OTel v1 schema, compatible with OpenSearch Observability dashboards.
  • timestamp_field: (optional) Field to store the timestamp in. If not set, uses the default @timestamp.
  • unix_timestamp: (optional) Whether to store the timestamp in epoch milliseconds.
  • dedup: (optional) removes fields from the document, that have duplicate keys. The filtering only keeps the last value for a key.
  • dedot: (optional) convert dotted keys into nested JSON objects.

SS4O mapping mode

The default Simple Schema for Observability mapping mode.

In ss4o mapping mode, the OpenSearch exporter stores documents using the SS4O schema, which is designed for observability data in OpenSearch. Documents use standardized field names and structure to facilitate integration with OpenSearch dashboards and tools.

Signal	Supported
Logs	✅
Traces	✅

ECS mapping mode

[!WARNING] The ECS mapping mode is currently undergoing changes, and its behaviour is unstable.

In ecs mapping mode, the OpenSearch exporter maps fields from OpenTelemetry Semantic Conventions to Elastic Common Schema (ECS) where possible. This mode may be used for compatibility with dashboards and tools that expect ECS.

Signal	ecs
Logs	✅
Traces	🚫

Flatten attributes mapping mode

[!WARNING] The Flatten attributes mapping mode is currently undergoing changes, and its behaviour is unstable.

In flatten_attributes mapping mode, the OpenSearch exporter uses the ECS mapping but flattens all resource and log attributes in the record to the top-level of the document.

Signal	flatten_attributes
Logs	✅
Traces	🚫

Bodymap mapping mode

In bodymap mapping mode, the OpenSearch exporter supports only logs and uses the "body" of a log record as the exact content of the OpenSearch document, without any transformation. This mapping mode is intended for use cases where the client wishes to have complete control over the OpenSearch document structure.

The bodymap mapping mode only supports log records where the body is of type Map. If the log body is not a Map, encoding will fail with an error. This ensures that only structured map data can be used as the document content in bodymap mode.

Signal	bodymap
Logs	✅
Traces	🚫

OTel v1 mapping mode

In otel-v1 mapping mode, the OpenSearch exporter produces documents compatible with Data Prepper's OTel v1 index schemas. This enables interoperability with OpenSearch Observability dashboards that consume Data Prepper indices.

Default index names:

• Traces: otel-v1-apm-span
• Logs: otel-v1-logs

These defaults can be overridden using traces_index and logs_index configuration options.

Signal	otel-v1
Logs	✅
Traces	✅

Schema references:

• otel-v1-apm-span index template
• logs-otel-v1 index template

[!NOTE] The exporter emits nanosecond-precision timestamps in the document body, but to materialize them as date_nanos (and apply the rest of the recommended field mappings) you must install the matching index templates before indexing begins. Without a template OpenSearch's dynamic mapping will infer date (millisecond precision) for timestamp fields. Install the templates out-of-band for now; see also the schema references above.

Example Configuration

exporters:
  opensearch:
    http:
      endpoint: http://opensearch.example.com:9200
    mapping:
      mode: "otel-v1"
    sending_queue:
      batch:

HTTP Connection Options

OpenSearch export supports standard HTTP client settings.

• http.endpoint (required) <url>:<port> of OpenSearch node to send data to.

TLS settings

Supports standard TLS settings as part of HTTP settings. See TLS Configuration/Client Settings.

Retry Options

• retry_on_failure: See retry_on_failure

Sending Queue Options

• sending_queue: See sending_queue

Timeout Options

• timeout : See timeout

Bulk Indexer Options

• bulk_action (optional): the action for ingesting data. Only create and index are allowed here.
• pipeline (optional): the ID of an ingest pipeline to apply when indexing documents. When set, all documents sent via the bulk API will be processed by the specified pipeline before being indexed. The ingest pipeline must exist in the cluster and there must be at least one node with the ingest node role assigned.

Example

extensions:
  basicauth/client:
    client_auth:
      username: username
      password: password

exporters:
  opensearch/trace:
    http:
      endpoint: https://opensearch.example.com:9200
      auth:
        authenticator: basicauth/client
    sending_queue:
      batch:
# ······
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [opensearch/trace]

Documentation

Overview

Package opensearchexporter contains an opentelemetry-collector exporter for OpenSearch.

Index

• func NewFactory() exporter.Factory
• type Config
  • func (cfg *Config) Validate() error
• type MappingMode
  • func (m MappingMode) String() string
• type MappingsSettings

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for OpenSearch exporter.

Types

type Config

type Config struct {
	confighttp.ClientConfig   `mapstructure:"http"`
	configretry.BackOffConfig `mapstructure:"retry_on_failure"`
	TimeoutSettings           exporterhelper.TimeoutConfig `mapstructure:",squash"`
	MappingsSettings          `mapstructure:"mapping"`
	QueueConfig               configoptional.Optional[exporterhelper.QueueBatchConfig] `mapstructure:"sending_queue"`

	// The Observability indices would follow the recommended for immutable data stream ingestion pattern using
	// the data_stream concepts. See https://opensearch.org/docs/latest/dashboards/im-dashboards/datastream/
	// Index pattern will follow the next naming template ss4o_{type}-{dataset}-{namespace}
	Dataset   string `mapstructure:"dataset"`
	Namespace string `mapstructure:"namespace"`

	// LogsIndex configures the index, index alias, or data stream name logs should be indexed in.
	// https://opensearch.org/docs/latest/im-plugin/index/
	// https://opensearch.org/docs/latest/dashboards/im-dashboards/datastream/
	LogsIndex           string `mapstructure:"logs_index"`
	LogsIndexFallback   string `mapstructure:"logs_index_fallback"`
	LogsIndexTimeFormat string `mapstructure:"logs_index_time_format"`

	// TracesIndex configures the index, index alias, or data stream name traces should be indexed in.
	// https://opensearch.org/docs/latest/im-plugin/index/
	// https://opensearch.org/docs/latest/dashboards/im-dashboards/datastream/
	TracesIndex           string `mapstructure:"traces_index"`
	TracesIndexFallback   string `mapstructure:"traces_index_fallback"`
	TracesIndexTimeFormat string `mapstructure:"traces_index_time_format"`

	// BulkAction configures the action for ingesting data. Only `create` and `index` are allowed here.
	// If not specified, the default value `create` will be used.
	BulkAction string `mapstructure:"bulk_action"`

	// Pipeline is the optional ID of an ingest pipeline to apply when indexing documents.
	// https://opensearch.org/docs/latest/ingest-pipelines/
	Pipeline string `mapstructure:"pipeline"`
}

Config defines configuration for OpenSearch exporter.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate validates the opensearch server configuration.

type MappingMode

type MappingMode int

const (
	MappingSS4O MappingMode = iota
	MappingECS
	MappingFlattenAttributes
	MappingBodyMap
	MappingOTelV1
)

Enum values for MappingMode.

func (MappingMode) String

func (m MappingMode) String() string

type MappingsSettings

type MappingsSettings struct {
	// Mode configures the field mappings.
	// Supported modes are the following:
	//
	//   ss4o: exports logs in the Simple Schema for Observability standard.
	//   This mode is enabled by default.
	//   See: https://opensearch.org/docs/latest/observing-your-data/ss4o/
	//
	//   ecs: maps fields defined in the OpenTelemetry Semantic Conventions
	//   to the Elastic Common Schema.
	//   See: https://www.elastic.co/guide/en/ecs/current/index.html
	//
	//   flatten_attributes: uses the ECS mapping but flattens all resource and
	//   log attributes in the record to the top-level.
	//
	//   bodymap: supports only logs and uses the "body" of a log record as the exact content
	//   of the OpenSearch document, without any transformation.
	//   This mapping mode is intended for use cases where the client wishes to have complete control over the
	//   OpenSearch document structure.
	//
	//   otel-v1: exports logs and traces using the OTel v1 schema published by the
	//   OpenSearch Data Prepper project (https://github.com/opensearch-project/data-prepper).
	//   Documents are compatible with OpenSearch Observability dashboards that consume
	//   Data Prepper indices. See the upstream index templates for the canonical field mappings:
	//     https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/index-template/otel-v1-apm-span-index-standard-template.json
	//     https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/src/main/resources/index-template/logs-otel-v1-index-standard-template.json
	Mode string `mapstructure:"mode"`

	// Additional field mappings.
	Fields map[string]string `mapstructure:"fields"`

	// File to read additional fields mappings from.
	File string `mapstructure:"file"`

	// Field to store timestamp in.  If not set uses the default @timestamp
	TimestampField string `mapstructure:"timestamp_field"`

	// Whether to store timestamp in Epoch milliseconds
	UnixTimestamp bool `mapstructure:"unix_timestamp"`

	// Try to find and remove duplicate fields
	Dedup bool `mapstructure:"dedup"`

	Dedot bool `mapstructure:"dedot"`
}