splunk-otel-collector

README

---

Getting Started   •   Getting Involved

Architecture   •   Components   •   Monitoring   •   Security   •   Sizing   •   Troubleshooting

---

Splunk OpenTelemetry Collector

Splunk OpenTelemetry Collector is a distribution of the OpenTelemetry Collector. It provides a unified way to receive, process, and export metric, trace, and log data for Splunk Observability Cloud:

• Splunk APM via the otlp_http exporter. More information available here.
• Splunk Infrastructure Monitoring via the signalfx exporter. More information available here.

Splunk OpenTelemetry Collector natively supports sending logs and metrics to Splunk Cloud and Splunk Enterprise. See Collect logs on Linux. Splunk Forwarders remain a supported alternative.

Current Status

• The Splunk Distribution of the OpenTelemetry Collector is production tested; it is in use by a number of customers in their production environments
• Customers that use our distribution can receive direct help from official Splunk support within SLA's
• Customers can use or migrate to the Splunk Distribution of the OpenTelemetry Collector without worrying about future breaking changes to its core configuration experience for metrics, traces, and log collection. There may be breaking changes to the Collector's own metrics.

Getting Started

The following resources are available:

• Architecture: How the Collector can be deployed
• Components: What the Collector supports with links to documentation
• Monitoring: How to ensure the Collector is healthy
• Security: How to ensure the Collector is secure
• Sizing: How to ensure the Collector is properly sized
• Troubleshooting: How to resolve common issues

All you need to get started is:

• Splunk Access Token
• Splunk Realm
• Agent or Gateway mode
• Confirm exposed ports to make sure your environment doesn't have conflicts and that firewalls are configured properly. Ports can be changed in the collector's configuration.

This distribution is supported on and packaged for a variety of platforms including:

• Kubernetes
  • Helm (recommended)
  • Operator (in alpha)
  • YAML
• HashiCorp Nomad
• Linux
  • Installer script (recommended for single-host demo/test environments)
  • Collect logs on Linux
  • Configuration management (recommended for multi-host production environments)
    • Ansible
    • Chef
    • Puppet
    • Salt
  • Platform as a Service
    • Heroku
  • Manual including DEB/RPM packages, Docker, and binary
• Windows
  • Installer script (recommended for single-host demo/test environments)
  • Run with a dedicated service account
  • Configuration management (recommended for multi-host production environments)
    • Ansible
    • Chef
    • Puppet
  • Manual including MSI with GUI and Powershell, Chocolatey, and Docker

You can consult additional use cases in the examples directory.

Advanced Configuration

A variety of default configuration files are provided:

• OpenTelemetry Collector see full_config_linux.yaml for a commented configuration with links to full documentation. agent_config.yaml is the recommended starting configuration for most environments. To collect logs, see Collect logs on Linux or Collect logs on Windows.

In addition, the following components can be configured:

• Configuration sources
  • Environment variables
  • Etcd2
  • Include
  • Splunk Secret Storage
  • Vault
  • Zookeeper
• SignalFx Smart Agent
  • Extension offering Collectd and Python extensions
  • Receiver offering the complete set of Smart Agent monitors
  • Information about migrating from the SignalFx Smart Agent can be found here

You can use the environment variable SPLUNK_LISTEN_INTERFACE and associated installer option to configure the network interface on which the collector's receivers and telemetry endpoints will listen. The default value of SPLUNK_LISTEN_INTERFACE is set to 127.0.0.1 for the default agent configuration and 0.0.0.0 otherwise.

Upgrade guidelines

The following changes need to be done to configuration files for Splunk OTel Collector for specific version upgrades. We provide automated scripts included in the bundle that cover backward compatibility on the fly, but configuration files will not be overridden, so you need to update them manually before the backward compatibility is dropped. For every configuration update use the default agent config as a reference.

Upgrading to 0.160.0

The Linux DEB/RPM splunk-otel-auto-instrumentation package now installs the official OpenTelemetry injector (libotelinject.so) instead of the previous libsplunk.so shim. This is a breaking change for existing auto-instrumentation installations:

• Configuration moves from /etc/splunk/zeroconfig/ (java.conf, node.conf, dotnet.conf) to /etc/opentelemetry/injector/ (default_env.conf and injector.conf). Existing values are not migrated automatically. Before upgrading, migrate any customized OTEL_* and SPLUNK_* values from java.conf, node.conf, and dotnet.conf into default_env.conf; other variables, including JAVA_TOOL_OPTIONS, NODE_OPTIONS, CORECLR_*, and DOTNET_*, are ignored going forward. Agent activation paths are now configured in injector.conf instead. default_env.conf is shared by all runtimes, and runtime-specific values must be set through the relevant application or service environment; if a variable is set in both places, the value in default_env.conf takes precedence.
• During the package upgrade, the legacy /etc/splunk/zeroconfig/ files are preserved under /usr/lib/splunk-instrumentation/legacy-zeroconfig/ before that directory is removed, and the libsplunk.so entry in /etc/ld.so.preload is removed. To remove the preserved legacy configuration during an explicit package removal, set REMOVE_LEGACY_CONFIG=true when invoking the package manager; cleanup is skipped if the backup directory is absent. Restart instrumented applications or services, or reboot, after the upgrade.
• .NET agent files are now installed under splunk-otel-dotnet/glibc, with OTEL_DOTNET_AUTO_HOME set to that directory, and .NET auto-instrumentation now supports arm64 in addition to amd64.
• When using the install.sh script with --with-instrumentation or --with-systemd-instrumentation options the script requires an auto-instrumentation package version greater than 0.159.0 and during installation adds the libotelinject.so to the /etc/ld.so.preload file.
• When using Ansible, Chef, Puppet, or Salt, do not edit the generated default_env.conf directly. Move the settings into the corresponding deployment-tool parameters and upgrade to an injector-compatible version of that deployment module before promoting the new auto-instrumentation package.
• Support for the new package requires use of the following minimum versions:
  • Ansible playbook v1.3.0
  • Chef recipe v0.22.0
  • Puppet module v0.23.0
  • Salt module is published with the merge of the libotelinject.so migration

See the instrumentation README for full activation and configuration details.

From 0.158.0 to 0.159.0

Linux DEB and RPM packages and Windows installations through MSI or Chocolatey now use otelcollauncher as the service entrypoint instead of otelcol. By default, the launcher starts otelcol directly and passes through the existing service arguments, so upgrades preserve the previous Collector service behavior. The launcher allows the service to start either otelcol directly or the OpAMP Supervisor, enabling additional Fleet Management capabilities. See OpenTelemetry Fleet Management for details.

For a new Linux installation, pass --with-supervisor to the installer script. To enable the OpAMP Supervisor after upgrading, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=true in /etc/otel/collector/splunk-otel-collector.conf and restart the service.

For a new Windows installation, pass -with_supervisor $true to the installer script. To enable the OpAMP Supervisor after upgrading, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=true in the service's Environment value under HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector and restart the service.

To stop running the Collector under OpAMP Supervisor, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=false in:

• Linux: /etc/otel/collector/splunk-otel-collector.conf
• Windows: service's Environment value under HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector

Restart the service after changing the setting. The launcher will return to starting otelcol directly, without the supervisor. When switched back to direct mode, remote configuration delivered through the supervisor is no longer applied.

On Linux DEB and RPM installation or upgrade, the package also now recursively sets the ownership of /var/lib/otelcol to the service user and group. This ensures the Collector and OpAMP Supervisor can write files in existing and new subdirectories of the shared state directory.

From 0.157.0 to 0.158.0

OpenTelemetry has deprecated deployment.environment in favor of deployment.environment.name. The Collector's optional deployment environment configuration and the --deployment-environment (Linux) and -deployment_env (Windows) installer options now use deployment.environment.name.

If deployment.environment is still required, use that name instead in the applicable configuration. For instrumented applications, set OTEL_RESOURCE_ATTRIBUTES=deployment.environment=<value>. The Ansible, Chef, Puppet, and Salt integrations can pass the deprecated key through their generic resource-attribute option.

The installer deployment environment option will use the new attribute name. To keep the deprecated attribute, replace only that attribute in OTEL_RESOURCE_ATTRIBUTES, preserve all other attributes, and restart the instrumented applications or IIS.

From 0.150.0 to 0.151.0

The default Windows MSI artifact download URL has been updated: Splunk Observability Cloud domain transition guide

• https://dl.signalfx.com → https://dl.observability.splunkcloud.com

To keep using the legacy dl.signalfx.com download URL, pass -collector_msi_url explicitly to install.ps1 (replace <version> with the collector version):

Windows (install.ps1):

-collector_msi_url https://dl.signalfx.com/splunk-otel-collector/msi/release/splunk-otel-collector-<version>-amd64.msi

From 0.149.0 to 0.150.0

Default endpoint URLs have changed from *.signalfx.com to *.observability.splunkcloud.com. The legacy endpoints remaim functional but migrating to the new domain is highly recommended. See the domain transition guide for background.

Variable	Old default	New default
SPLUNK_API_URL	https://api.<realm>.signalfx.com	https://api.<realm>.observability.splunkcloud.com
SPLUNK_INGEST_URL	https://ingest.<realm>.signalfx.com	https://ingest.<realm>.observability.splunkcloud.com
SPLUNK_TRACE_URL	https://ingest.<realm>.signalfx.com/v2/trace	https://ingest.<realm>.observability.splunkcloud.com/v2/trace
SPLUNK_HEC_URL	https://ingest.<realm>.signalfx.com/v1/log	https://ingest.<realm>.observability.splunkcloud.com/v1/log

Note: If you have firewall rules or proxy allowlists scoped to *.signalfx.com, add
*.observability.splunkcloud.com (or the realm-specific hostnames
api.<realm>.observability.splunkcloud.com and ingest.<realm>.observability.splunkcloud.com)
before switching.

Package manager upgrade (apt, yum, choco upgrade)

Your existing environment configuration is preserved. The collector continues to use the endpoints you previously configured.

When you are ready to migrate, replace with your realm (e.g. us0):

Linux: Edit /etc/otel/collector/splunk-otel-collector.conf:

SPLUNK_API_URL=https://api.<realm>.observability.splunkcloud.com
SPLUNK_INGEST_URL=https://ingest.<realm>.observability.splunkcloud.com
SPLUNK_HEC_URL=https://ingest.<realm>.observability.splunkcloud.com/v1/log

Then restart the service: sudo systemctl restart splunk-otel-collector

Windows: Update the Environment value in the registry at HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector with the same URLs, then restart the service.

SPLUNK_TRACE_URL is derived automatically from SPLUNK_INGEST_URL at runtime unless you have explicitly set it. If you have, update it as well.

MSI upgrade (Windows)

Upgrading via msiexec applies the new *.observability.splunkcloud.com defaults. To keep the legacy endpoints, pass them as MSI properties:

msiexec /i <path\to\msi> SPLUNK_API_URL=https://api.<realm>.signalfx.com SPLUNK_INGEST_URL=https://ingest.<realm>.signalfx.com SPLUNK_HEC_URL=https://ingest.<realm>.signalfx.com/v1/log

New installation

New installations default to *.observability.splunkcloud.com. If your environment requires the legacy endpoints, pass them explicitly:

install.sh (add these flags):

--api-url https://api.<realm>.signalfx.com \
--ingest-url https://ingest.<realm>.signalfx.com

install.ps1 (add these flags):

-api_url https://api.<realm>.signalfx.com `
-ingest_url https://ingest.<realm>.signalfx.com

Chocolatey (add these params):

choco install splunk-otel-collector --params "'/SPLUNK_API_URL:https://api.<realm>.signalfx.com /SPLUNK_INGEST_URL:https://ingest.<realm>.signalfx.com /SPLUNK_HEC_URL:https://ingest.<realm>.signalfx.com/v1/log'"

msiexec (add these properties):

SPLUNK_API_URL=https://api.<realm>.signalfx.com SPLUNK_INGEST_URL=https://ingest.<realm>.signalfx.com SPLUNK_HEC_URL=https://ingest.<realm>.signalfx.com/v1/log

From 0.117.0 to 0.118.0

• The deprecated syntax for config source expansion is no longer supported.

  Strings like $ENV or $include:/path/to/file.yaml will no longer be expanded. Instead, use the ${env:ENV} or ${include:/path/to/file.yaml} syntax. There are only two symbols allowed after $: { and $. The collector will log an error and fail to start if it encounters a bare config source.

  Please update your configuration files to use the correct syntax.

From 0.114.0 to 0.115.0

• The sapm exporter still works as before but has been deprecated. Use the otlphttp exporter instead

1. Replace the sapm exporter with otlphttp exporter using the following configuration

   otlphttp:
     traces_endpoint: "${SPLUNK_INGEST_URL}/v2/trace/otlp"
     headers:
       "X-SF-Token": "${SPLUNK_ACCESS_TOKEN}"
   

2. Update traces pipeline to use otlphttp exporter instead of sapm:

   service:
     pipelines:
       traces:
         exporters: [otlphttp]
   

From 0.96.1 to 0.97.0

• memory_ballast is no longer effective. The garbage collection is now controlled by the soft memory limit set to 90% of total memory (SPLUNK_MEMORY_TOTAL_MIB env var) by default.

  If you haven't customized the memory_ballast, just remove it from the configuration.

  If you have customized it via SPLUNK_BALLAST_SIZE_MIB (or extensions::memory_ballast::size_mib config), you should remove the memory_ballast extension and use the GOMEMLIMIT environment variable to set a custom soft memory limit:

  • To decrease frequency of garbage collections: set GOMEMLIMIT to a higher value than the default 90% of total memory.
  • To increase frequency of garbage collections: set GOMEMLIMIT to a lower value than the default 90% of total memory.

From 0.68.0 to 0.69.0

• gke and gce resource detectors in resourcedetection processor are replaced with gcp resource detector. If you have gke and gce detectors configured in the resourcedetection processor, please update your configuration accordingly. More details: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/10347

From 0.41.0 to 0.42.0

• The Splunk OpenTelemetry Collector used to evaluate user configuration twice and this required escaping of each $ symbol with $$ to prevent unwanted environment variable expansion. The issue was fixed in 0.42.0 version. Any occurrences of $$ in your configuration should be replaced with $.

From 0.35.0 to 0.36.0

• Configuration parameter "exporters -> otlp -> insecure" is moved to "exporters -> otlp -> tls -> insecure".

  More details: https://github.com/open-telemetry/opentelemetry-collector/pull/4063/.

  Configuration part for otlp exporter should look like this:

  exporters:
    otlp:
      endpoint: "${SPLUNK_GATEWAY_URL}:4317"
      tls:
        insecure: true
  

From 0.34.0 to 0.35.0

• ballast_size_mib parameter moved from memory_limiter processor to memory_ballast extension as size_mib.

  More details: https://github.com/signalfx/splunk-otel-collector/pull/567.

  Remove ballast_size_mib parameter from memory_limiter and make sure that it's added to memory_ballast extension as size_mib parameter instead:

  extensions:
    memory_ballast:
      size_mib: ${SPLUNK_BALLAST_SIZE_MIB}
  

Using Upstream OpenTelemetry Collector

It is possible to use the upstream OpenTelemetry Collector instead of this distribution. The following features are not available upstream at this time:

• Packaging
  • Installer scripts for Linux and Windows
  • Configuration management via Ansible or Puppet
• Configuration sources
• Several SignalFx Smart Agent capabilities

⚠ Splunk only provides best-effort support for upstream OpenTelemetry

In order to use the upstream OpenTelemetry Collector:

• Use the contrib distribution as commercial exporters must reside in contrib
• Properly configure the Collector for your particular metrics, traces, and logs use cases, as only a minimal default configuration is provided by the contrib release.

An example configuration for upstream, that ensures infrastructure correlation is properly configured, is available here.

License

Apache Software License version 2.0.

ℹ️  SignalFx was acquired by Splunk in October 2019. See Splunk SignalFx for more information.