discovery

README

Discovery confmap.Provider (Experimental)

This component should not be considered stable. At this time its functionality is provided for testing and validation purposes only.

The Discovery confmap.Provider provides the ability to define Collector service config through individual component yaml mappings in a config.d directory:

graph LR
  config.d[/config.d/] --> 1>service.yaml]
  subgraph 1a[service.yaml]
    1 --> 1a1[[pipelines:<br>metrics:<br>receivers:<br>- otlp<br>exporters:<br>- logging]]
  end
  config.d --> 2[/exporters/]
  subgraph 2a[exporters]
    2 --> 2a1>otlp.yaml]
    2a1 --> 2b1[[otlp:<br>endpoint: 1.2.3.4:2345]]
    2 --> 2a2>logging.yaml]
    2a2 --> 2b2[[logging:<br>verbosity: detailed]]
  end
  config.d --> 3[/extensions/]
  subgraph 3a[extensions]
    3 --> 3a1>zpages.yaml]
    3a1 --> 3b1[[zpages:<br>endpoint: 0.0.0.0:12345]]
    3 --> 3a2>health-check.yaml]
    3a2 --> 3b2[[health_check:<br>path: /health]]
  end
  config.d --> 4[/processors/]
  subgraph 4a[processors]
    4 --> 4a1>batch.yaml]
    4a1 --> 4b1[[batch:<br>]]
    4 --> 4a2>resource-detection.yaml]
    4a2 --> 4b2[[resourcedetection:<br>detectors:<br>- system]]
  end
  config.d --> 5[/receivers/]
  subgraph 5a[receivers]
    5 --> 5a1>otlp.yaml]
    5a1 --> 5b1[[otlp:<br>protocols:<br>grpc:]]
  end

This component is currently supported in the Collector settings via the following commandline options:

option	environment variable	default	description
--configd	none	disabled	Whether to enable config.d functionality for final Collector config content.
--config-dir	SPLUNK_CONFIG_DIR	/etc/otel/collector/config.d	The root config.d directory to walk for component directories and yaml mapping files.
--dry-run	none	disabled	Whether to report the final assembled config contents to stdout before immediately exiting. This can be used with or without config.d

To source only config.d content and not an additional or default configuration file, the --config option or SPLUNK_CONFIG environment variable must be set to /dev/null or an arbitrary empty file:

$ # run the Collector without a config file using components from a local ./config.d config directory,
$ # printing the config to stdout before exiting instead of starting the Collector service:
$ bin/otelcol --config /dev/null --configd --config-dir ./config.d --dry-run
2023/02/24 19:54:23 settings.go:331: Set config to [/dev/null]
2023/02/24 19:54:23 settings.go:384: Set ballast to 168 MiB
2023/02/24 19:54:23 settings.go:400: Set memory limit to 460 MiB
exporters:
  logging:
    verbosity: detailed
  otlp:
    endpoint: 1.2.3.4:2345
extensions:
  health_check:
    path: /health
  zpages:
    endpoint: 0.0.0.0:1234
processors:
  batch: {}
  resourcedetection:
    detectors:
    - system
receivers:
  otlp:
    protocols:
      grpc: null
service:
  pipelines:
    metrics:
      exporters:
      - logging
      receivers:
      - otlp

Discovery Mode

This component also provides a --discovery [--dry-run] option compatible with config.d that attempts to instantiate any .discovery.yaml receivers using corresponding .discovery.yaml observers in a "preflight" Collector service. Discovery mode will:

1. Load and attempt to start any observers in config.d/extensions/<name>.discovery.yaml.
2. Load and attempt to start any receiver blocks in config.d/receivers/<name>.discovery.yaml in a Discovery Receiver instance to receive discovery events from all successfully started observers.
3. Wait 10s or the configured SPLUNK_DISCOVERY_DURATION environment variable time.Duration.
4. Embed any receiver instances' configs resulting in a discovery.status of successful inside a receiver_creator/discovery receiver's configuration to be passed to the final Collector service config (or outputted w/ --dry-run).
5. Log any receiver resulting in a discovery.status of partial with the configured guidance for setting any relevant discovery properties.
6. Stop all temporary components before continuing on to the actual Collector service (or exiting early with --dry-run).

By default, the Discovery mode is provided with pre-made discovery config components in bundle.d.

Discovery properties

Configuring discovery components is performed by merging discovery properties with the config.d receivers and extensions *.discovery.yaml files. Discovery properties are of the form:

splunk.discovery.receivers.<receiver-type(/name)>.config.<field>(<::subfield>)*: <value>
splunk.discovery.extensions.<observer-type(/name)>.config.<field>(<::subfield>)*: <value>
splunk.discovery.receivers.<receiver-type(/name)>.enabled: <true or false>
splunk.discovery.extensions.<observer-type(/name)>.enabled: <true or false>

# Examples
splunk.discovery.receivers.prometheus_simple.config.labels::my_label: my_label_value
splunk.discovery.receivers.prometheus_simple.enabled: true

splunk.discovery.extensions.docker_observer.config.endpoint: tcp://localhost:8080
splunk.discovery.extensions.k8s_observer.enabled: false

These properties can be in config.d/properties.discovery.yaml or specified at run time with --set command line options.

Each discovery property also has an equivalent environment variable form using _x<hex pair>_ encoded delimiters for non-word characters [^a-zA-Z0-9_]:

SPLUNK_DISCOVERY_RECEIVERS_receiver_x2d_type_x2f_receiver_x2d_name_CONFIG_field_x3a__x3a_subfield=value
SPLUNK_DISCOVERY_EXTENSIONS_observer_x2d_type_x2f_observer_x2d_name_CONFIG_field_x3a__x3a_subfield=value
SPLUNK_DISCOVERY_RECEIVERS_receiver_x2d_type_x2f_receiver_x2d_name_ENABLED=<true or false>
SPLUNK_DISCOVERY_EXTENSIONS_observer_x2d_type_x2f_observer_x2d_name_ENABLED=<true or false>

# Examples
SPLUNK_DISCOVERY_RECEIVERS_prometheus_simple_CONFIG_labels_x3a__x3a_my_label="my_username"
SPLUNK_DISCOVERY_RECEIVERS_prometheus_simple_ENABLED=true

SPLUNK_DISCOVERY_EXTENSIONS_docker_observer_CONFIG_endpoint="tcp://localhost:8080"
SPLUNK_DISCOVERY_EXTENSIONS_k8s_observer_ENABLED=false

The priority order for discovery config content from lowest to highest is:

1. config.d/<receivers or extensions>/*.discovery.yaml file content (lowest).
2. config.d/properties.discovery.yaml file content.
3. SPLUNK_DISCOVERY_<xyz> environment variables available to the collector process.
4. --set splunk.discovery.<xyz> commandline options (highest).

Documentation

Index

• type Config
  • func NewConfig(logger *zap.Logger) *Config
  • func (c *Config) Load(configDPath string) error
  • func (c *Config) LoadFS(dirfs fs.FS) error
• type Entry
  • func (e Entry) Self() Entry
  • func (e Entry) ToStringMap() map[string]any
• type ExporterEntry
  • func (ExporterEntry) ErrorF(path string, err error) error
• type ExtensionEntry
  • func (ExtensionEntry) ErrorF(path string, err error) error
• type ObserverEntry
  • func (ObserverEntry) ErrorF(path string, err error) error
• type ProcessorEntry
  • func (ProcessorEntry) ErrorF(path string, err error) error
• type PropertiesEntry
  • func (PropertiesEntry) ErrorF(path string, err error) error
• type Provider
  • func New() (Provider, error)
• type ReceiverEntry
  • func (ReceiverEntry) ErrorF(path string, err error) error
• type ReceiverToDiscoverEntry
  • func (ReceiverToDiscoverEntry) ErrorF(path string, err error) error
  • func (r ReceiverToDiscoverEntry) ToStringMap() map[string]any
• type ServiceEntry
  • func (ServiceEntry) ErrorF(path string, err error) error

Types

type Config

type Config struct {

	// Service is for pipelines and final settings.
	// It must be in the root config directory and named "service.yaml"
	Service ServiceEntry
	// Exporters is a map of exporters to use in final config.
	// They must be in `config.d/exporters` directory.
	Exporters map[component.ID]ExporterEntry
	// Extensions is a map of extensions to use in final config.
	// They must be in `config.d/extensions` directory.
	Extensions map[component.ID]ExtensionEntry
	// DiscoveryObservers is a map of observer extensions to use in discovery.
	// They must be in `config.d/extensions` directory and end with ".discovery.yaml".
	DiscoveryObservers map[component.ID]ExtensionEntry
	// Processors is a map of extensions to use in final config.
	// They must be in `config.d/processors` directory.
	Processors map[component.ID]ProcessorEntry
	// Receivers is a map of receiver entries to use in final config
	// They must be in `config.d/receivers` directory.
	Receivers map[component.ID]ReceiverEntry
	// ReceiversToDiscover is a map of receiver entries to use in discovery mode's
	// underlying discovery receiver. They must be in `config.d/receivers` directory and
	// end with ".discovery.yaml".
	ReceiversToDiscover map[component.ID]ReceiverToDiscoverEntry
	// DiscoveryProperties is a mapping of discovery properties to their values for
	// configuring discovery mode components.
	// It must be in the root config directory and named "properties.discovery.yaml".
	DiscoveryProperties PropertiesEntry
	// contains filtered or unexported fields
}

Config is a model for stitching together the final Collector configuration with additional discovery component fields for use w/ discovery mode. It allows individual yaml files to be added to a config.d directory and be sourced in the final config such that small changes don't apply to a central configuration file, and possibly eliminates the need for one overall (still in design and dependent on aliasing and array insertion operators).

func NewConfig

func NewConfig(logger *zap.Logger) *Config

func (*Config) Load

func (c *Config) Load(configDPath string) error

Load will walk the file tree from the configDPath root, loading the component files as they are discovered, determined by their parent directory and filename.

func (*Config) LoadFS

func (c *Config) LoadFS(dirfs fs.FS) error

LoadFS will walk the provided filesystem, loading the component files as they are discovered, determined by their parent directory and filename.

type Entry

type Entry map[string]any

func (Entry) Self

func (e Entry) Self() Entry

func (Entry) ToStringMap

func (e Entry) ToStringMap() map[string]any

type ExporterEntry

type ExporterEntry struct {
	Entry `yaml:",inline"`
}

func (ExporterEntry) ErrorF

func (ExporterEntry) ErrorF(path string, err error) error

type ExtensionEntry

type ExtensionEntry struct {
	Entry `yaml:",inline"`
}

func (ExtensionEntry) ErrorF

func (ExtensionEntry) ErrorF(path string, err error) error

type ObserverEntry

type ObserverEntry struct {
	Entry `yaml:",inline"`
}

func (ObserverEntry) ErrorF

func (ObserverEntry) ErrorF(path string, err error) error

type ProcessorEntry

type ProcessorEntry struct {
	Entry `yaml:",inline"`
}

func (ProcessorEntry) ErrorF

func (ProcessorEntry) ErrorF(path string, err error) error

type PropertiesEntry

type PropertiesEntry struct {
	Entry `yaml:",inline"`
}

func (PropertiesEntry) ErrorF

func (PropertiesEntry) ErrorF(path string, err error) error

type Provider

type Provider interface {
	ConfigDScheme() string
	ConfigDProvider() confmap.Provider
	DiscoveryModeScheme() string
	DiscoveryModeProvider() confmap.Provider
	PropertyScheme() string
	PropertyProvider() confmap.Provider
}

func New

func New() (Provider, error)

type ReceiverEntry

type ReceiverEntry struct {
	Entry `yaml:",inline"`
}

func (ReceiverEntry) ErrorF

func (ReceiverEntry) ErrorF(path string, err error) error

type ReceiverToDiscoverEntry

type ReceiverToDiscoverEntry struct {
	// Receiver creator rules by observer extension ID
	Rule map[component.ID]string
	// Platform/observer specific config by observer extension ID.
	// These are merged w/ "default" component.ID in a "config" map
	Config map[component.ID]map[string]any
	// The remaining items used to merge applicable rule and config
	Entry `yaml:",inline"`
}

func (ReceiverToDiscoverEntry) ErrorF

func (ReceiverToDiscoverEntry) ErrorF(path string, err error) error

func (ReceiverToDiscoverEntry) ToStringMap

func (r ReceiverToDiscoverEntry) ToStringMap() map[string]any

type ServiceEntry

type ServiceEntry struct {
	Entry `yaml:",inline"`
}

func (ServiceEntry) ErrorF

func (ServiceEntry) ErrorF(path string, err error) error