splunkhecexporter

README

Splunk HTTP Event Collector (HEC) Exporter

Status
Stability	beta: traces, metrics, logs
Distributions	contrib
Issues
Code Owners	@atoulme, @dmitryax

Configuration

The following configuration options are required:

• token (no default): HEC requires a token to authenticate incoming traffic. To procure a token, please refer to the Splunk documentation.
• endpoint (no default): Splunk HEC URL.

The following configuration options can also be configured:

• source (no default): Optional Splunk source: https://docs.splunk.com/Splexicon:Source
• sourcetype (no default): Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
• index (no default): Splunk index, optional name of the Splunk index targeted
• use_multi_metric_format (default: false): Combines metrics with the same metadata to reduce ingest using the multiple-metric JSON format. Applicable in the metrics pipeline only.
• disable_compression (default: false): Whether to disable gzip compression over HTTP.
• timeout (default: 10s): HTTP timeout when sending data.
• http2_read_idle_timeout (default = 10s): Send a ping frame for a health check if the connection has been idle for the configured value. 0s means http/2 health check will be disabled.
• http2_ping_timeout (default = 10s): Triggered by http2_read_idle_timeout; When there's no response to the ping within the configured value, the connection will be closed. If this value is set to 0, it will default to 15s.
• insecure_skip_verify (default: false): Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS.
• ca_file (no default) Path to the CA cert to verify the server being connected to.
• cert_file (no default) Path to the TLS cert to use for client connections when TLS client auth is required.
• key_file (no default) Path to the TLS key to use for TLS required connections.
• max_content_length_logs (default: 2097152): Maximum log payload size in bytes. Log batches of bigger size will be broken down into several requests. Default value is 2097152 bytes (2 MiB). Maximum allowed value is 838860800 (~ 800 MB). When set to 0, it will treat as infinite length and it will create only 1 request per batch.
• max_content_length_metrics (default: 2097152): Maximum metric payload size in bytes. Metric batches of bigger size will be broken down into several requests. Default value is 2097152 bytes (2 MiB). Maximum allowed value is 838860800 (~ 800 MB). When set to 0, it will treat as infinite length and it will create only one request per batch.
• max_content_length_traces (default: 2097152): Maximum trace payload size in bytes. Trace batches of bigger size will be broken down into several requests. Default value is 2097152 bytes (2 MiB). Maximum allowed value is 838860800 (~ 800 MB). When set to 0, it will treat as infinite length and it will create only one request per batch.
• max_event_size (default: 5242880): Maximum raw uncompressed individual event size in bytes. Maximum allowed value is 838860800 (~ 800 MB).
• splunk_app_name (default: "OpenTelemetry Collector Contrib") App name is used to track telemetry information for Splunk App's using HEC by App name.
• splunk_app_version (default: Current OpenTelemetry Collector Contrib Build Version): App version is used to track telemetry information for Splunk App's using HEC by App version.
• log_data_enabled (default: true): Specifies whether the log data is exported. Set it to false if you want the log data to be dropped instead. Applicable in the logs pipeline only.
• profiling_data_enabled (default: true): Specifies whether the profiling data is exported. Set it to false if you want the profiling data to be dropped instead. Applicable in the logs pipeline only.
• health_path (default = '/services/collector/health'): The path reporting health checks.
• health_check_enabled (default = false): Whether to perform Splunk HEC Health Check during the exporter's startup.
• export_raw (default = false): send only the log's body, targeting a Splunk HEC raw endpoint.
• otel_attrs_to_hec_metadata/source (default = 'com.splunk.source'): Specifies the mapping of a specific unified model attribute value to the standard source field of a HEC event.
• otel_attrs_to_hec_metadata/sourcetype (default = 'com.splunk.sourcetype'): Specifies the mapping of a specific unified model attribute value to the standard sourcetype field of a HEC event.
• otel_attrs_to_hec_metadata/index (default = 'com.splunk.index'): Specifies the mapping of a specific unified model attribute value to the standard index field of a HEC event.
• otel_attrs_to_hec_metadata/host (default = 'host.name'): Specifies the mapping of a specific unified model attribute value to the standard host field and the host.name field of a HEC event.
• hec_metadata_to_otel_attrs/source (default = 'com.splunk.source'): Specifies the mapping of a specific unified model attribute value to the standard source field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/source.
• hec_metadata_to_otel_attrs/sourcetype (default = 'com.splunk.sourcetype'): Specifies the mapping of a specific unified model attribute value to the standard sourcetype field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/sourcetype.
• hec_metadata_to_otel_attrs/index (default = 'com.splunk.index'): Specifies the mapping of a specific unified model attribute value to the standard index field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/index.
• hec_metadata_to_otel_attrs/host (default = 'host.name'): Specifies the mapping of a specific unified model attribute value to the standard host field and the host.name field of a HEC event. Deprecated (v0.116.0): prefer otel_attrs_to_hec_metadata/host.
• otel_to_hec_fields/severity_text (default = otel.log.severity.text): Specifies the name of the field to map the severity text field of log events.
• otel_to_hec_fields/severity_number (default = otel.log.severity.number): Specifies the name of the field to map the severity number field of log events.
• otel_to_hec_fields/name (default = "otel.log.name): Specifies the name of the field to map the name field of log events.
• heartbeat/interval (no default): Specifies the interval of sending hec heartbeat to the destination. If not specified, heartbeat is not enabled.
• heartbeat/startup (default: false): Check heartbeat at start up time. This action enforces a synchronous heartbeat action during the collector start up sequence. The collector will fail to start if the heartbeat returns an error.
• telemetry/enabled (default: false): Specifies whether to enable telemetry inside splunk hec exporter.
• telemetry/override_metrics_names (default: empty map): Specifies the metrics name to overrides in splunk hec exporter.
• telemetry/extra_attributes (default: empty map): Specifies the extra metrics attributes in splunk hec exporter.
• batcher(Experimental, disabled by default): Specifies batching configuration on the exporter.

In addition, this exporter offers queued retry which is enabled by default. For more information, see the queued retry options in the exporter documentation.
If you are getting throttled due to high volume of events the collector might experience memory issues, in those cases it is recommended to change the queued retry configuration to drop events more frequently, for example you can reduce the maximum amount of time spent trying to send a batch from 120s (default) to 60s:

exporters:
  splunk_hec:
    retry_on_failure:
      max_elapsed_time: 60

If that does not resolve the memory issues you can try to reduce it further and adjust the other queued retry parameters accordingly.
As a last resort after you have tried to solve the memory issues by adjusting the queued retry configuration you can disable it altogether:

exporters:
  splunk_hec:
    retry_on_failure:
      enabled: false

Example:

exporters:
  splunk_hec:
    # Splunk HTTP Event Collector token.
    token: "00000000-0000-0000-0000-0000000000000"
    # URL to a Splunk instance to send data to.
    endpoint: "https://splunk:8088/services/collector"
    # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
    source: "otel"
    # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
    sourcetype: "otel"
    # Splunk index, optional name of the Splunk index targeted.
    index: "metrics"
    # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
    max_idle_conns: 200
    # Whether to disable gzip compression over HTTP. Defaults to false.
    disable_compression: false
    # HTTP timeout when sending data. Defaults to 10s.
    timeout: 10s
    tls:
      # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
      insecure_skip_verify: false
      # Path to the CA cert to verify the server being connected to.
      ca_file: /certs/ExampleCA.crt
      # Path to the TLS cert to use for client connections when TLS client auth is required.
      cert_file: /certs/HECclient.crt
      # Path to the TLS key to use for TLS required connections.
      key_file: /certs/HECclient.key
    # Application name is used to track telemetry information for Splunk App's using HEC by App name.
    splunk_app_name: "OpenTelemetry-Collector Splunk Exporter"
    # Application version is used to track telemetry information for Splunk App's using HEC by App version.
    splunk_app_version: "v0.0.1"
    heartbeat:
      interval: 30s
    telemetry:
      enabled: true
      override_metrics_names:
        otelcol_exporter_splunkhec_heartbeats_sent: app_heartbeats_success_total
        otelcol_exporter_splunkhec_heartbeats_failed: app_heartbeats_failed_total
      extra_attributes:
        dataset_name: SplunkCloudBeaverStack
        custom_key: custom_value

The full list of settings exposed for this exporter are documented in config.go with detailed sample configurations in testdata/config.yaml.

This exporter also offers proxy support.

Advanced Configuration

Several helper files are leveraged to provide additional capabilities automatically:

• HTTP settings
• TLS and mTLS settings
• Queuing, retry and timeout settings

Documentation

Overview

Package splunkhecexporter implements an exporter that sends data to Splunk.

Index

• func NewFactory() exporter.Factory
• type Config
  • func (cfg *Config) Validate() error
• type HecHeartbeat
• type HecTelemetry
• type OtelToHecFields

Functions

func NewFactory

func NewFactory() exporter.Factory

NewFactory creates a factory for Splunk HEC exporter.

Types

type Config

type Config struct {
	confighttp.ClientConfig   `mapstructure:",squash"`
	QueueSettings             exporterhelper.QueueConfig `mapstructure:"sending_queue"`
	configretry.BackOffConfig `mapstructure:"retry_on_failure"`

	// Experimental: This configuration is at the early stage of development and may change without backward compatibility
	// until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.
	BatcherConfig exporterbatcher.Config `mapstructure:"batcher"`

	// LogDataEnabled can be used to disable sending logs by the exporter.
	LogDataEnabled bool `mapstructure:"log_data_enabled"`

	// ProfilingDataEnabled can be used to disable sending profiling data by the exporter.
	ProfilingDataEnabled bool `mapstructure:"profiling_data_enabled"`

	// HEC Token is the authentication token provided by Splunk: https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector.
	Token configopaque.String `mapstructure:"token"`

	// Optional Splunk source: https://docs.splunk.com/Splexicon:Source.
	// Sources identify the incoming data.
	Source string `mapstructure:"source"`

	// Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype.
	SourceType string `mapstructure:"sourcetype"`

	// Splunk index, optional name of the Splunk index.
	Index string `mapstructure:"index"`

	// Disable GZip compression. Defaults to false.
	DisableCompression bool `mapstructure:"disable_compression"`

	// Maximum log payload size in bytes. Default value is 2097152 bytes (2MiB).
	// Maximum allowed value is 838860800 (~ 800 MB).
	MaxContentLengthLogs uint `mapstructure:"max_content_length_logs"`

	// Maximum metric payload size in bytes. Default value is 2097152 bytes (2MiB).
	// Maximum allowed value is 838860800 (~ 800 MB).
	MaxContentLengthMetrics uint `mapstructure:"max_content_length_metrics"`

	// Maximum trace payload size in bytes. Default value is 2097152 bytes (2MiB).
	// Maximum allowed value is 838860800 (~ 800 MB).
	MaxContentLengthTraces uint `mapstructure:"max_content_length_traces"`

	// Maximum payload size, raw uncompressed. Default value is 5242880 bytes (5MiB).
	// Maximum allowed value is 838860800 (~ 800 MB).
	MaxEventSize uint `mapstructure:"max_event_size"`

	// App name is used to track telemetry information for Splunk App's using HEC by App name. Defaults to "OpenTelemetry Collector Contrib".
	SplunkAppName string `mapstructure:"splunk_app_name"`

	// App version is used to track telemetry information for Splunk App's using HEC by App version. Defaults to the current OpenTelemetry Collector Contrib build version.
	SplunkAppVersion string `mapstructure:"splunk_app_version"`

	// OtelAttrsToHec creates a mapping from attributes to HEC specific metadata: source, sourcetype, index and host.
	OtelAttrsToHec splunk.HecToOtelAttrs `mapstructure:"otel_attrs_to_hec_metadata"`

	// HecToOtelAttrs creates a mapping from attributes to HEC specific metadata: source, sourcetype, index and host.
	// Deprecated: [v0.113.0] Use OtelAttrsToHec instead.
	HecToOtelAttrs splunk.HecToOtelAttrs `mapstructure:"hec_metadata_to_otel_attrs"`
	// HecFields creates a mapping from attributes to HEC fields.
	HecFields OtelToHecFields `mapstructure:"otel_to_hec_fields"`

	// HealthPath for health API, default is '/services/collector/health'
	HealthPath string `mapstructure:"health_path"`

	// HecHealthCheckEnabled can be used to verify Splunk HEC health on exporter's startup
	HecHealthCheckEnabled bool `mapstructure:"health_check_enabled"`

	// ExportRaw to send only the log's body, targeting a Splunk HEC raw endpoint.
	ExportRaw bool `mapstructure:"export_raw"`

	// UseMultiMetricFormat combines metric events to save space during ingestion.
	UseMultiMetricFormat bool `mapstructure:"use_multi_metric_format"`

	// Heartbeat is the configuration to enable heartbeat
	Heartbeat HecHeartbeat `mapstructure:"heartbeat"`

	// Telemetry is the configuration for splunk hec exporter telemetry
	Telemetry HecTelemetry `mapstructure:"telemetry"`
}

Config defines configuration for Splunk exporter.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate checks if the exporter configuration is valid.

type HecHeartbeat

type HecHeartbeat struct {
	// Interval represents the time interval for the heartbeat interval. If nothing or 0 is set,
	// heartbeat is not enabled.
	// A heartbeat is an event sent to _internal index with metadata for the current collector/host.
	Interval time.Duration `mapstructure:"interval"`

	// Startup is used to send heartbeat events on exporter's startup.
	Startup bool `mapstructure:"startup"`
}

HecHeartbeat defines the heartbeat information for the exporter

type HecTelemetry

type HecTelemetry struct {
	// Enabled is the bool to enable telemetry inside splunk hec exporter
	Enabled bool `mapstructure:"enabled"`

	// OverrideMetricsNames is the map to override metrics for internal metrics in splunk hec exporter
	OverrideMetricsNames map[string]string `mapstructure:"override_metrics_names"`

	// ExtraAttributes is the extra attributes for metrics inside splunk hex exporter
	ExtraAttributes map[string]string `mapstructure:"extra_attributes"`
}

HecTelemetry defines the telemetry configuration for the exporter

type OtelToHecFields

type OtelToHecFields struct {
	// SeverityText informs the exporter to map the severity text field to a specific HEC field.
	SeverityText string `mapstructure:"severity_text"`
	// SeverityNumber informs the exporter to map the severity number field to a specific HEC field.
	SeverityNumber string `mapstructure:"severity_number"`
}

OtelToHecFields defines the mapping of attributes to HEC fields