splunkenterprisereceiver

README

Splunk Enterprise Receiver

The Splunk Enterprise Receiver is a pull based tool which enables the ingestion of performance metrics describing the operational status of a user's Splunk Enterprise deployment to an appropriate observability tool. It is designed to leverage several different data sources to gather these metrics including the introspection api endpoint and serializing results from ad-hoc searches. Because of this, care must be taken by users when enabling metrics as running searches can effect your Splunk Enterprise Deployment and introspection may fail to report for Splunk Cloud deployments. The primary purpose of this receiver is to empower those tasked with the maintenance and care of a Splunk Enterprise deployment to leverage opentelemetry and their observability toolset in their jobs.

Status
Stability	alpha: metrics
Distributions	contrib
Issues
Code coverage
Code Owners	@shalper2, @MovieStoreGuy, @greatestusername

Configuration

By default the Splunk Enterprise receiver is not configured to gather any metrics other than splunk.health

The following settings are required, omitting them will either cause your receiver to fail to compile or result in 4/5xx return codes during scraping.

NOTE: These must be set for each Splunk instance type (indexer, search head, or cluster master) from which you wish to pull metrics. At present, only one of each type is accepted, per configured receiver instance. This means, for example, that if you have three different "indexer" type instances that you would like to pull metrics from you will need to configure three different splunkenterprise receivers for each indexer node you wish to monitor.

• basicauth (from basicauthextension): A configured stanza for the basicauthextension.
• auth (no default): String name referencing your auth extension.
• endpoint (no default): your Splunk Enterprise host's endpoint.

The following settings are optional:

• collection_interval (default: 10m): The time between scrape attempts.
• timeout (default: 60s): The time the scrape function will wait for a response before returning empty.
• build_version_info (default: false): Elect to run an additional scrape which will retrieve build and version info for the configured endpoints and attach this as attributes to the selected metrics. A value of false will report an empty string as the attribute value but will speed up the receiver slightly.

Basic Example:

extensions:
    basicauth/indexer:
        client_auth:
            username: admin
            password: securityFirst
    basicauth/cluster_master:
        client_auth:
            username: admin
            password: securityFirst

receivers:
    splunkenterprise:
        indexer:
            auth: 
              authenticator: basicauth/indexer
            endpoint: "https://localhost:8089"
            timeout: 45s
        cluster_master:
            auth: 
              authenticator: basicauth/cluster_master
            endpoint: "https://localhost:8089"
            timeout: 45s

exporters:
  debug:
    verbosity: basic

service:
  extensions: [basicauth/indexer, basicauth/cluster_master]
  pipelines:
    metrics:
      receivers: [splunkenterprise]
      exporters: [debug]

Individual Cluster Config Example (with specific metrics enabled/disabled):

extensions:
  basicauth/indexer:
    client_auth:
      username: admin_user
      password: XXXXXXXXX
  basicauth/cluster_master:
    client_auth:
      username: admin_user
      password: XXXXXXXXX
  basicauth/search_head:
    client_auth:
      username: admin_user
      password: XXXXXXXXX
  basicauth/monitoring_console:
    client_auth:
      username: admin_user
      password: XXXXXXXXX

receivers:
  splunkenterprise/indexer:
    indexer:
      auth:
        authenticator: basicauth/indexer
      endpoint: "https://ip.ip.ip.ip:8089"
      tls:
        insecure_skip_verify: true
    metrics:
      splunk.indexer.throughput:
        enabled: true
      splunk.data.indexes.extended.total.size:
        enabled: true
      splunk.data.indexes.extended.event.count:
        enabled: true
      splunk.data.indexes.extended.raw.size:
        enabled: true
      splunk.data.indexes.extended.bucket.event.count:
        enabled: true
      splunk.data.indexes.extended.bucket.count:
        enabled: true
      splunk.data.indexes.extended.bucket.hot.count:
        enabled: true
      splunk.data.indexes.extended.bucket.warm.count:
        enabled: true
      splunk.server.introspection.queues.current:
        enabled: true
      splunk.server.introspection.queues.current.bytes:
        enabled: true
      splunk.health:
        enabled: false
      

  splunkenterprise/cluster_master:
    cluster_master:
      auth:
        authenticator: basicauth/cluster_master
      endpoint: "https://ip.ip.ip.ip:8089"
      tls:
        insecure_skip_verify: true
    metrics:
      splunk.parse.queue.ratio:  
        enabled: true
      splunk.indexer.avg.rate:
        enabled: true
      splunk.pipeline.set.count:
        enabled: true
      splunk.aggregation.queue.ratio:
        enabled: true
      splunk.typing.queue.ratio:
        enabled: true
      splunk.indexer.queue.ratio:
        enabled: true
      splunk.indexes.bucket.count:
        enabled: true
      splunk.indexes.size:
        enabled: true
      splunk.indexer.raw.write.time:  
        enabled: true
      splunk.indexer.cpu.time:
        enabled: true
      splunk.indexes.avg.size:
        enabled: true
      splunk.indexes.avg.usage:
        enabled: true
      splunk.indexes.median.data.age: 
        enabled: true
      splunk.scheduler.avg.execution.latency:
        enabled: true
      splunk.scheduler.completion.ratio:
        enabled: true
      splunk.scheduler.avg.run.time:
        enabled: true
      splunk.buckets.searchable.status:
        enabled: true
      splunk.health:
        enabled: true


  splunkenterprise/monitoring_console:
    cluster_master:
      auth:
        authenticator: basicauth/monitoring_console
      endpoint: "https://ip.ip.ip.ip:8089"
      tls:
        insecure_skip_verify: true
    metrics:
      splunk.parse.queue.ratio:  
        enabled: true
      splunk.indexer.avg.rate:
        enabled: true
      splunk.pipeline.set.count:
        enabled: true
      splunk.aggregation.queue.ratio:
        enabled: true
      splunk.typing.queue.ratio:
        enabled: true
      splunk.indexer.queue.ratio:
        enabled: true
      splunk.indexes.bucket.count:
        enabled: true
      splunk.indexes.size:
        enabled: true
      splunk.indexer.raw.write.time:  
        enabled: true
      splunk.indexer.cpu.time:
        enabled: true
      splunk.indexes.avg.size:
        enabled: true
      splunk.indexes.avg.usage:
        enabled: true
      splunk.indexes.median.data.age: 
        enabled: true
      splunk.scheduler.avg.execution.latency:
        enabled: true
      splunk.scheduler.completion.ratio:
        enabled: true
      splunk.scheduler.avg.run.time:
        enabled: true
      splunk.buckets.searchable.status:
        enabled: true
      splunk.health:
        enabled: true

  splunkenterprise/search_head:
    search_head:
      auth:
        authenticator: basicauth/search_head
      endpoint: "https://ip.ip.ip.ip:8089"
      tls:
        insecure_skip_verify: true
    metrics:
      splunk.server.searchartifacts.adhoc:
        enabled: true
      splunk.server.searchartifacts.scheduled:
        enabled: true
      splunk.server.searchartifacts.completed:
        enabled: true
      splunk.server.searchartifacts.incomplete:
        enabled: true
      splunk.server.searchartifacts.invalid:
        enabled: true
      splunk.server.searchartifacts.savedsearches:
        enabled: true
      splunk.server.searchartifacts.job.cache.size:
        enabled: true
      splunk.server.searchartifacts.job.cache.count:
        enabled: true
      splunk.server.searchartifacts.adhoc.size:
        enabled: true
      splunk.server.searchartifacts.scheduled.size:
        enabled: true
      splunk.server.searchartifacts.completed.size:
        enabled: true
      splunk.server.searchartifacts.incomplete.size:
        enabled: true
      splunk.search.duration:
        enabled: true
      splunk.search.initiation:
        enabled: true
      splunk.search.status:
        enabled: true
      splunk.search.success:
        enabled: true
      splunk.kvstore.status:
        enabled: true
      splunk.kvstore.replication.status:
        enabled: true
      splunk.kvstore.backup.status:
        enabled: true
      splunk.health:
        enabled: false

service:
  extensions: [basicauth/indexer, basicauth/cluster_master, basicauth/monitoring_console, basicauth/search_head]
  pipelines:
    metrics:
      receivers: [splunkenterprise/indexer, splunkenterprise/cluster_master, splunkenterprise/monitoring_console, splunkenterprise/search_head]
      exporters: [otlp_grpc]

Custom SPL Searches

The receiver supports running arbitrary SPL searches against configured Splunk endpoints. This allows you to create custom metrics from any data available in your Splunk deployment.

WARNING: Custom searches execute ad-hoc searches on your Splunk deployment. Be mindful of the performance impact, especially with complex searches or short collection intervals.

Search Configuration

Each custom search requires:

• spl (required): The SPL search string. Can be provided as a single line or multi-line YAML block.
• target (required): The endpoint type to run the search against. Must be one of: indexer, search_head, or cluster_master.
• earliest (optional): The earliest time for the search. Defaults to -{collection_interval} (e.g., -10m if collection_interval is 10m). Ignored if SPL already contains earliest=.
• latest (optional): The latest time for the search. Defaults to now. Ignored if SPL already contains latest=.
• metrics (required): A list of metric definitions that map search result columns to OTel metrics.

Note on time ranges:

• For regular searches (like index=_internal | stats count), the receiver automatically adds earliest and latest time modifiers.
• For generating commands that start with | (other than | tstats), no time range is added since these commands handle time differently.
  • Receiver will correctly add a time range to searches that begin with | tstats

Each metric definition supports:

• metric_name (required): The name of the metric to emit (e.g., splunk.custom.event_count).
• value_column (required): The column name from the SPL results that contains the metric value.
• attribute_columns (optional): List of column names to include as metric attributes.
• value_type (optional): Either int (default) or double.
• unit (optional): The unit of measurement (e.g., {count}, By, %).
• description (optional): A description of the metric.
• static_attributes (optional): Key-value pairs to add as static attributes to all data points.

Different Collection Intervals

All custom searches run at the same collection_interval as the built-in metrics. If you need searches to run at different intervals, configure multiple receiver instances:

receivers:
  # Fast searches - 1 minute interval
  splunkenterprise/fast:
    collection_interval: 1m
    cluster_master:
      auth:
        authenticator: basicauth/cm
      endpoint: "https://cm:8089"
    searches:
      - spl: "| tstats count where index=* by index"
        target: cluster_master
        metrics:
          - metric_name: splunk.custom.index_event_count
            value_column: count
            attribute_columns: [index]

  # Slow searches - 10 minute interval
  splunkenterprise/slow:
    collection_interval: 10m
    cluster_master:
      auth:
        authenticator: basicauth/cm
      endpoint: "https://cm:8089"
    searches:
      - spl: |
          search index=_internal sourcetype=splunkd
          | stats count by host, component
        target: cluster_master
        metrics:
          - metric_name: splunk.custom.component_events
            value_column: count
            attribute_columns: [host, component]

Custom Search Example

receivers:
  splunkenterprise:
    collection_interval: 10m
    cluster_master:
      auth:
        authenticator: basicauth/cm
      endpoint: "https://cm.example.com:8089"

    searches:
      # Regular search - time range auto-added from collection_interval
      - spl: "index=_internal | stats count by host"
        target: cluster_master
        metrics:
          - metric_name: splunk.custom.internal_events
            value_column: count
            attribute_columns: [host]
            value_type: int
            unit: "{events}"
            description: "Count of _internal events by host"

      # Custom time range - look back 1 hour
      - spl: "index=_audit | stats count by action"
        target: cluster_master
        earliest: "-1h"
        latest: "now"
        metrics:
          - metric_name: splunk.custom.audit_actions
            value_column: count
            attribute_columns: [action]
            value_type: int

      # tstats for fast indexed field searches - receiver injects earliest/latest automatically
      - spl: "| tstats count where index=* by index"
        target: cluster_master
        metrics:
          - metric_name: splunk.custom.index_event_count
            value_column: count
            attribute_columns: [index]
            value_type: int

      # REST API search - non-tstats generating commands (| rest, | makeresults, etc.) do NOT
      # receive automatic time injection; include time modifiers in the SPL if needed
      - spl: "| rest /services/server/info | fields serverName, numberOfCores"
      - spl: "| rest /services/server/info | fields serverName, numberOfCores"
        target: cluster_master
        metrics:
          - metric_name: splunk.custom.server_cores
            value_column: numberOfCores
            attribute_columns: [serverName]
            value_type: int
            static_attributes:
              search_source: "rest_api"

For a full list of settings exposed by this receiver please look in config.go with a detailed configuration in testdata/config.yaml.

Documentation

Index

• Constants
• func NewFactory() receiver.Factory
• type Config
  • func (cfg *Config) Validate() (errors error)
• type MetricConfig
  • func (m MetricConfig) Validate() error
• type SearchConfig
  • func (s SearchConfig) TargetType() string
  • func (s SearchConfig) Validate() error

Constants

const (
	TargetIndexer       = "indexer"
	TargetSearchHead    = "search_head"
	TargetClusterMaster = "cluster_master"

	MetricValueTypeInt    = "int"
	MetricValueTypeDouble = "double"
)

const (
	// StateDone indicates a dispatch state of Done
	StateDone = "DONE"
	// StateFailed indicates a dispatch state of Failed
	StateFailed = "FAILED"
	// StateFinalizing indicates a dispatch state of Finalizing
	StateFinalizing = "FINALIZING"
	// StateParsing indicates a dispatch state of Parsing
	StateParsing = "PARSING"
	// StatePaused indicates a dispatch state of Paused
	StatePaused = "PAUSED"
	// StateQueued indicates a dispatch state of Queued
	StateQueued = "QUEUED"
	// StateRunning indicates a dispatch state of Running
	StateRunning = "RUNNING"
	// ControlError indicates a dispatch state of Error
	ControlError = "CONTROL_ERROR"
)

Functions

func NewFactory

func NewFactory() receiver.Factory

Types

type Config

type Config struct {
	scraperhelper.ControllerConfig `mapstructure:",squash"`
	metadata.MetricsBuilderConfig  `mapstructure:",squash"`
	IdxEndpoint                    confighttp.ClientConfig `mapstructure:"indexer"`
	SHEndpoint                     confighttp.ClientConfig `mapstructure:"search_head"`
	CMEndpoint                     confighttp.ClientConfig `mapstructure:"cluster_master"`
	VersionInfo                    bool                    `mapstructure:"build_version_info"`
	Searches                       []SearchConfig          `mapstructure:"searches"`
}

func (*Config) Validate

func (cfg *Config) Validate() (errors error)

type MetricConfig

type MetricConfig struct {
	MetricName       string            `mapstructure:"metric_name"`
	ValueColumn      string            `mapstructure:"value_column"`
	AttributeColumns []string          `mapstructure:"attribute_columns"`
	ValueType        string            `mapstructure:"value_type"`
	Unit             string            `mapstructure:"unit"`
	Description      string            `mapstructure:"description"`
	StaticAttributes map[string]string `mapstructure:"static_attributes"`
}

func (MetricConfig) Validate

func (m MetricConfig) Validate() error

type SearchConfig

type SearchConfig struct {
	SPL      string         `mapstructure:"spl"`
	Target   string         `mapstructure:"target"`
	Earliest string         `mapstructure:"earliest"`
	Latest   string         `mapstructure:"latest"`
	Metrics  []MetricConfig `mapstructure:"metrics"`
}

func (SearchConfig) TargetType

func (s SearchConfig) TargetType() string

func (SearchConfig) Validate

func (s SearchConfig) Validate() error