discovery

README

Discovery confmap.Provider

This feature currently has an alpha stability level.
Backwards incompatible changes to components and custom discovery configuration may occur.

The Discovery confmap.Provider provides the ability to define Collector service config through individual component yaml mappings in a config.d directory:

graph LR
  config.d[/config.d/] --> 1>service.yaml]
  subgraph 1a[service.yaml]
    1 --> 1a1[[pipelines:<br>metrics:<br>receivers:<br>- otlp<br>exporters:<br>- debug]]
  end
  config.d --> 2[/exporters/]
  subgraph 2a[exporters]
    2 --> 2a1>otlp.yaml]
    2a1 --> 2b1[[otlp:<br>endpoint: 1.2.3.4:2345]]
    2 --> 2a2>logging.yaml]
    2a2 --> 2b2[[debug:<br>verbosity: detailed]]
  end
  config.d --> 3[/extensions/]
  subgraph 3a[extensions]
    3 --> 3a1>zpages.yaml]
    3a1 --> 3b1[[zpages:<br>endpoint: 0.0.0.0:12345]]
    3 --> 3a2>health-check.yaml]
    3a2 --> 3b2[[health_check:<br>path: /health]]
  end
  config.d --> 4[/processors/]
  subgraph 4a[processors]
    4 --> 4a1>batch.yaml]
    4a1 --> 4b1[[batch:<br>]]
    4 --> 4a2>resource-detection.yaml]
    4a2 --> 4b2[[resourcedetection:<br>detectors:<br>- system]]
  end
  config.d --> 5[/receivers/]
  subgraph 5a[receivers]
    5 --> 5a1>otlp.yaml]
    5a1 --> 5b1[[otlp:<br>protocols:<br>grpc:]]
  end

This component is currently supported in the Collector settings via the following commandline options:

option	environment variable	default	description
--configd	none	disabled	Whether to enable config.d functionality for final Collector config content.
--config-dir	SPLUNK_CONFIG_DIR	/etc/otel/collector/config.d	The root config.d directory to walk for component directories and yaml mapping files.
--dry-run	none	disabled	Whether to report the final assembled config contents to stdout before immediately exiting. This can be used with or without config.d

To source only config.d content and not an additional or default configuration file, the --config option or SPLUNK_CONFIG environment variable must be set to /dev/null or an arbitrary empty file:

$ # run the Collector without a config file using components from a local ./config.d config directory,
$ # printing the config to stdout before exiting instead of starting the Collector service:
$ bin/otelcol --config /dev/null --configd --config-dir ./config.d --dry-run
2023/02/24 19:54:23 settings.go:331: Set config to [/dev/null]
2023/02/24 19:54:23 settings.go:384: Set ballast to 168 MiB
2023/02/24 19:54:23 settings.go:400: Set memory limit to 460 MiB
exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: 1.2.3.4:2345
extensions:
  health_check:
    path: /health
  zpages:
    endpoint: 0.0.0.0:1234
processors:
  batch: {}
  resourcedetection:
    detectors:
    - system
receivers:
  otlp:
    protocols:
      grpc: null
service:
  pipelines:
    metrics:
      exporters:
      - debug
      receivers:
      - otlp

Discovery Mode

This component also provides a --discovery [--dry-run] [--discovery-properties=<properties.yaml>] option compatible with config.d that attempts to instantiate any .discovery.yaml receivers using corresponding .discovery.yaml observers in a "preflight" Collector service. Discovery mode will:

1. Load and attempt to start any observers in config.d/extensions/<name>.discovery.yaml.
2. Load and attempt to start any receiver blocks in config.d/receivers/<name>.discovery.yaml in a Discovery Receiver instance to receive discovery events from all successfully started observers.
3. Wait 10s or the configured SPLUNK_DISCOVERY_DURATION environment variable time.Duration.
4. Embed any receiver instances' configs resulting in a discovery.status of successful inside a receiver_creator/discovery receiver's configuration to be passed to the final Collector service config in a new or existing service::pipelines::metrics::receivers sequence (or outputted w/ --dry-run). Any required observers will be added to service::extensions.
5. Log any receiver resulting in a discovery.status of partial with the configured guidance for setting any relevant discovery properties.
6. Stop all temporary components before continuing on to the actual Collector service (or exiting early with --dry-run).

Unlike config.d component files, which are direct configuration entries for the desired component, Discovery component configs have an enabled boolean and config parent mapping field to determine use and configure the functionality of the components:

# <some-observer-type-with-optional-name.discovery.yaml>
<observer_type>(/<observer_name>):
  enabled: <true | false> # true by default
  config:
    <embedded observer config>

# <some-receiver-type-with-optional-name.discovery.yaml>
<receiver_type>(/<receiver_name>):
  enabled: <true | false> # true by default
  rule:
    <observer_type>(/<observer_name>): <receiver creator rule for this observer>
  config:
    default:
      <default embedded receiver config>
    <observer_type>(/<observer_name>):
      <observer-specific config items, merged with `default`>
  status:
    metrics:
      <discovery receiver metric status entries>
    statements:
      <discovery receiver statement status entries>

By default, the discovery mode is provided with pre-made discovery config components in bundle.d.

The following components have bundled discovery configurations in the last Splunk OpenTelemetry Collector release:

I. Receivers

• mysql (Linux and Windows)
• oracledb (Linux and Windows)
• postgresql (Linux and Windows)
• redis (Linux and Windows)
• smartagent with collectd/mysql monitor type (Linux)
• smartagent with collectd/nginx monitor type (Linux)
• smartagent with postgresql monitor type (Linux and Windows)

II. Extensions

• docker_observer (Linux and Windows)
• host_observer (Linux and Windows)
• k8s_observer (Linux and Windows)

Discovery properties

Configuring discovery components is performed by merging discovery properties with the config.d receivers and extensions *.discovery.yaml files. Discovery properties are of the form:

splunk.discovery.receivers.<receiver-type(/name)>.config.<field>(<::subfield>)*: <value>
splunk.discovery.extensions.<observer-type(/name)>.config.<field>(<::subfield>)*: <value>
splunk.discovery.receivers.<receiver-type(/name)>.enabled: <true or false>
splunk.discovery.extensions.<observer-type(/name)>.enabled: <true or false>

# Examples
splunk.discovery.receivers.prometheus_simple.config.labels::my_label: my_label_value
splunk.discovery.receivers.prometheus_simple.enabled: true

splunk.discovery.extensions.docker_observer.config.endpoint: tcp://localhost:8080
splunk.discovery.extensions.k8s_observer.enabled: false

These properties can be in config.d/properties.discovery.yaml or specified at run time with --set command line options.

You can also specify a --discovery-properties=<filepath.yaml> argument to disregard config.d/properties.discovery.yaml properties and load properties not to be shared with another Collector service, while still benefiting from existing discovery component definitions.

The config.d/properties.discovery.yaml file supports specifying the property values directly as well within a mapped form. The property values can also be environment variables:

# --set form will take priority to mapped values
splunk.discovery.receivers.prometheus_simple.config.labels::my_label: my_label_value
splunk.discovery.receivers.prometheus_simple.enabled: true

splunk.discovery.receivers.smartagent/postgresql.config.params::username: '${env:PG_USERNAME}'
splunk.discovery.receivers.smartagent/postgresql.config.params::password: '${env:PG_PASSWORD}'

# mapped property form
splunk.discovery:
  extensions:
    docker_observer:
      enabled: false
      config:
        endpoint: tcp://localhost:54321
  receivers:
    prometheus_simple:
      enabled: false # will be overwritten by above --set form (discovery is attempted for the receiver)

Each discovery property also has an equivalent environment variable form using _x<hex pair>_ encoded delimiters for non-word characters [^a-zA-Z0-9_]:

SPLUNK_DISCOVERY_RECEIVERS_receiver_x2d_type_x2f_receiver_x2d_name_CONFIG_field_x3a__x3a_subfield=value
SPLUNK_DISCOVERY_EXTENSIONS_observer_x2d_type_x2f_observer_x2d_name_CONFIG_field_x3a__x3a_subfield=value
SPLUNK_DISCOVERY_RECEIVERS_receiver_x2d_type_x2f_receiver_x2d_name_ENABLED=<true or false>
SPLUNK_DISCOVERY_EXTENSIONS_observer_x2d_type_x2f_observer_x2d_name_ENABLED=<true or false>

# Examples
SPLUNK_DISCOVERY_RECEIVERS_prometheus_simple_CONFIG_labels_x3a__x3a_my_label="my_username"
SPLUNK_DISCOVERY_RECEIVERS_prometheus_simple_ENABLED=true

SPLUNK_DISCOVERY_EXTENSIONS_docker_observer_CONFIG_endpoint="tcp://localhost:8080"
SPLUNK_DISCOVERY_EXTENSIONS_k8s_observer_ENABLED=false

The priority order for discovery config values from lowest to highest is:

1. Pre-made bundle.d component config content (lowest).
2. config.d/<receivers or extensions>/*.discovery.yaml component config file content.
3. config.d/properties.discovery.yaml properties file mapped form content.
4. config.d/properties.discovery.yaml properties file --set form content.
5. SPLUNK_DISCOVERY_<xyz> property environment variables available to the collector process.
6. --set splunk.discovery.<xyz> property commandline options (highest).

Documentation

Index

• type Config
  • func NewConfig(logger *zap.Logger) *Config
  • func (c *Config) Load(configDPath string) error
  • func (c *Config) LoadFS(dirfs fs.FS) error
  • func (c *Config) LoadProperties(path string) error
• type Entry
  • func (e Entry) Self() Entry
  • func (e Entry) ToStringMap() map[string]any
• type ExporterEntry
  • func (ExporterEntry) ErrorF(path string, err error) error
• type ExtensionEntry
  • func (ExtensionEntry) ErrorF(path string, err error) error
• type ObserverEntry
  • func (ObserverEntry) ErrorF(path string, err error) error
• type ProcessorEntry
  • func (ProcessorEntry) ErrorF(path string, err error) error
• type PropertiesEntry
  • func (PropertiesEntry) ErrorF(path string, err error) error
• type Provider
  • func New() (Provider, error)
• type ReceiverEntry
  • func (ReceiverEntry) ErrorF(path string, err error) error
• type ReceiverToDiscoverEntry
  • func (ReceiverToDiscoverEntry) ErrorF(path string, err error) error
  • func (r ReceiverToDiscoverEntry) ToStringMap() map[string]any
• type ServiceEntry
  • func (ServiceEntry) ErrorF(path string, err error) error

Types

type Config

type Config struct {

	// Service is for pipelines and final settings.
	// It must be in the root config directory and named "service.yaml"
	Service ServiceEntry
	// Exporters is a map of exporters to use in final config.
	// They must be in `config.d/exporters` directory.
	Exporters map[component.ID]ExporterEntry
	// Extensions is a map of extensions to use in final config.
	// They must be in `config.d/extensions` directory.
	Extensions map[component.ID]ExtensionEntry
	// DiscoveryObservers is a map of observer extensions to use in discovery.
	// They must be in `config.d/extensions` directory and end with ".discovery.yaml".
	DiscoveryObservers map[component.ID]ObserverEntry
	// Processors is a map of extensions to use in final config.
	// They must be in `config.d/processors` directory.
	Processors map[component.ID]ProcessorEntry
	// Receivers is a map of receiver entries to use in final config
	// They must be in `config.d/receivers` directory.
	Receivers map[component.ID]ReceiverEntry
	// ReceiversToDiscover is a map of receiver entries to use in discovery mode's
	// underlying discovery receiver. They must be in `config.d/receivers` directory and
	// end with ".discovery.yaml".
	ReceiversToDiscover map[component.ID]ReceiverToDiscoverEntry
	// DiscoveryProperties is a mapping of discovery properties to their values for
	// configuring discovery mode components.
	// It must be in the root config directory and named "properties.discovery.yaml".
	DiscoveryProperties PropertiesEntry
	// contains filtered or unexported fields
}

Config is a model for stitching together the final Collector configuration with additional discovery component fields for use w/ discovery mode. It allows individual yaml files to be added to a config.d directory and be sourced in the final config such that small changes don't apply to a central configuration file, and possibly eliminates the need for one overall (still in design and dependent on aliasing and array insertion operators).

func NewConfig

func NewConfig(logger *zap.Logger) *Config

func (*Config) Load

func (c *Config) Load(configDPath string) error

Load will walk the file tree from the configDPath root, loading the component files as they are discovered, determined by their parent directory and filename.

func (*Config) LoadFS

func (c *Config) LoadFS(dirfs fs.FS) error

LoadFS will walk the provided filesystem, loading the component files as they are discovered, determined by their parent directory and filename.

func (*Config) LoadProperties

func (c *Config) LoadProperties(path string) error

type Entry

type Entry map[string]any

func (Entry) Self

func (e Entry) Self() Entry

func (Entry) ToStringMap

func (e Entry) ToStringMap() map[string]any

type ExporterEntry

type ExporterEntry struct {
	Entry `yaml:",inline"`
}

func (ExporterEntry) ErrorF

func (ExporterEntry) ErrorF(path string, err error) error

type ExtensionEntry

type ExtensionEntry struct {
	Entry `yaml:",inline"`
}

func (ExtensionEntry) ErrorF

func (ExtensionEntry) ErrorF(path string, err error) error

type ObserverEntry

type ObserverEntry struct {
	Enabled *bool
	Config  Entry
	Entry   `yaml:",inline"`
}

func (ObserverEntry) ErrorF

func (ObserverEntry) ErrorF(path string, err error) error

type ProcessorEntry

type ProcessorEntry struct {
	Entry `yaml:",inline"`
}

func (ProcessorEntry) ErrorF

func (ProcessorEntry) ErrorF(path string, err error) error

type PropertiesEntry

type PropertiesEntry struct {
	Entry `yaml:",inline"`
}

func (PropertiesEntry) ErrorF

func (PropertiesEntry) ErrorF(path string, err error) error

type Provider

type Provider interface {
	ConfigDScheme() string
	ConfigDProvider() confmap.Provider
	DiscoveryModeScheme() string
	DiscoveryModeProvider() confmap.Provider
	PropertyScheme() string
	PropertyProvider() confmap.Provider
	PropertiesFileScheme() string
	PropertiesFileProvider() confmap.Provider
}

func New

func New() (Provider, error)

type ReceiverEntry

type ReceiverEntry struct {
	Entry `yaml:",inline"`
}

func (ReceiverEntry) ErrorF

func (ReceiverEntry) ErrorF(path string, err error) error

type ReceiverToDiscoverEntry

type ReceiverToDiscoverEntry struct {
	// Receiver creator rules by observer extension ID
	Rule map[component.ID]string
	// Platform/observer specific config by observer extension ID.
	// These are merged w/ "default" component.ID in a "config" map
	Config map[component.ID]map[string]any
	// Whether to attempt to discover this receiver
	Enabled *bool
	// The remaining items used to merge applicable rule and config
	Entry `yaml:",inline"`
}

func (ReceiverToDiscoverEntry) ErrorF

func (ReceiverToDiscoverEntry) ErrorF(path string, err error) error

func (ReceiverToDiscoverEntry) ToStringMap

func (r ReceiverToDiscoverEntry) ToStringMap() map[string]any

type ServiceEntry

type ServiceEntry struct {
	Entry `yaml:",inline"`
}

func (ServiceEntry) ErrorF

func (ServiceEntry) ErrorF(path string, err error) error