confmap

package module
v1.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2024 License: Apache-2.0 Imports: 17 Imported by: 199

README

High Level Design

This document is work in progress.

Conf

The Conf represents the raw configuration for a service (e.g. OpenTelemetry Collector).

Provider

The Provider provides configuration, and allows to watch/monitor for changes. Any Provider has a <scheme> associated with it, and will provide configs for configURI that follow the ":<opaque_data>" format. This format is compatible with the URI definition (see RFC 3986). The <scheme> MUST be always included in the configURI. The scheme for any Provider MUST be at least 2 characters long to avoid conflicting with a driver-letter identifier as specified in file URI syntax.

Converter

The Converter allows implementing conversion logic for the provided configuration. One of the most common use-case is to migrate/transform the configuration after a backwards incompatible change.

Resolver

The Resolver handles the use of multiple Providers and Converters simplifying configuration parsing, monitoring for updates, and the overall life-cycle of the used config providers. The Resolver provides two main functionalities: Configuration Resolving and Watching for Updates.

Configuration Resolving

The Resolver receives as input a set of Providers, a list of Converters, and a list of configuration identifier configURI that will be used to generate the resulting, or effective, configuration in the form of a Conf, that can be used by code that is oblivious to the usage of Providers and Converters.

Providers are used to provide an entire configuration when the configURI is given directly to the Resolver, or an individual value (partial configuration) when the configURI is embedded into the Conf as a values using the syntax ${configURI}.

Limitation:

  • When embedding a ${configURI} the uri cannot contain dollar sign ("$") character unless it embeds another uri.
  • The number of URIs is limited to 100.
              Resolver                   Provider
   Resolve       │                          │
────────────────►│                          │
                 │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │          Conf            │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Merge                 │
              │  │◄──┘                      │
              └─ │                          │
              ┌─ │        Retrieve          │
              │  ├─────────────────────────►│
              │  │    Partial Conf Value    │
              │  │◄─────────────────────────┤
  foreach     │  │                          │
  embedded    │  │                          │
  configURI   │  ├───┐                      │
              │  │   │Replace               │
              │  │◄──┘                      │
              └─ │                          │
                 │            Converter     │
              ┌─ │     Convert    │         │
              │  ├───────────────►│         │
    foreach   │  │                │         │
   Converter  │  │◄───────────────┤         │
              └─ │                          │
                 │                          │
◄────────────────┤                          │

The Resolve method proceeds in the following steps:

  1. Start with an empty "result" of Conf type.
  2. For each config URI retrieves individual configurations, and merges it into the "result".
  3. For each embedded config URI retrieves individual value, and replaces it into the "result".
  4. For each "Converter", call "Convert" for the "result".
  5. Return the "result", aka effective, configuration.
Watching for Updates

After the configuration was processed, the Resolver can be used as a single point to watch for updates in the configuration retrieved via the Provider used to retrieve the “initial” configuration and to generate the “effective” one.

         Resolver              Provider
            │                     │
   Watch    │                     │
───────────►│                     │
            │                     │
            .                     .
            .                     .
            .                     .
            │      onChange       │
            │◄────────────────────┤
◄───────────┤                     │

The Resolver does that by passing an onChange func to each Provider.Retrieve call and capturing all watch events.

Troubleshooting

Null Maps

Due to how our underlying merge library, koanf, behaves, configuration resolution will treat configuration such as

processors:

as null, which is a valid value. As a result if you have configuration A:

receivers:
    nop:

processors:
    nop:

exporters:
    nop:

extensions:
    nop:

service:
    extensions: [nop]
    pipelines:
        traces:
            receivers: [nop]
            processors: [nop]
            exporters: [nop]

and configuration B:

processors:

and do ./otelcorecol --config A.yaml --config B.yaml

The result will be an error:

Error: invalid configuration: service::pipelines::traces: references processor "nop" which is not configured
2024/06/10 14:37:14 collector server run finished with error: invalid configuration: service::pipelines::traces: references processor "nop" which is not configured

This happens because configuration B sets processors to null, removing the nop processor defined in configuration A, so the nop processor referenced in configuration A's pipeline no longer exists.

This situation can be remedied 2 ways:

  1. Use {} when you want to represent an empty map, such as processors: {} instead of processors:.
  2. Omit configuration like processors: from your configuration.

Documentation

Overview

Example (EmbeddedManualUnmarshaling)

We can unmarshal an embedded struct with a custom `Unmarshal` method.

package main

import (
	"fmt"
	"slices"

	"go.opentelemetry.io/collector/confmap"
)

type NetworkScrape struct {
	Enabled  bool     `mapstructure:"enabled"`
	Networks []string `mapstructure:"networks"`
	Wifi     bool     `mapstructure:"wifi"`
}

func (n *NetworkScrape) Unmarshal(c *confmap.Conf) error {
	if err := c.Unmarshal(n, confmap.WithIgnoreUnused()); err != nil {
		return err
	}
	if slices.Contains(n.Networks, "wlan0") {
		n.Wifi = true
	}
	return nil
}

type RouterScrape struct {
	NetworkScrape `mapstructure:",squash"`
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"networks": []string{"eth0", "eth1", "wlan0"},
		"enabled":  true,
	})
	scrapeInfo := &RouterScrape{}
	if err := conf.Unmarshal(scrapeInfo); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nNetworks: %q\nWifi: %v\nEnabled: %v\n", scrapeInfo.Networks, scrapeInfo.Wifi, scrapeInfo.Enabled)
}
Output:

Configuration contains the following:
Networks: ["eth0" "eth1" "wlan0"]
Wifi: true
Enabled: true
Example (EmbeddedUnmarshaling)

We can unmarshal embedded structs with mapstructure field annotations.

package main

import (
	"fmt"
	"time"

	"go.opentelemetry.io/collector/confmap"
)

type DiskScrape struct {
	Disk   string        `mapstructure:"disk"`
	Scrape time.Duration `mapstructure:"scrape"`
}

type CPUScrape struct {
	Enabled bool `mapstructure:"enabled"`
}

type ComputerScrape struct {
	DiskScrape `mapstructure:",squash"`
	CPUScrape  `mapstructure:",squash"`
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"disk":    "c",
		"scrape":  "5s",
		"enabled": true,
	})
	scrapeInfo := &ComputerScrape{}
	if err := conf.Unmarshal(scrapeInfo); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nDisk: %q\nScrape: %s\nEnabled: %v\n", scrapeInfo.Disk, scrapeInfo.Scrape, scrapeInfo.Enabled)
}
Output:

Configuration contains the following:
Disk: "c"
Scrape: 5s
Enabled: true
Example (ManualUnmarshaling)
package main

import (
	"fmt"
	"time"

	"go.opentelemetry.io/collector/confmap"
)

type ManualScrapeInfo struct {
	Disk   string
	Scrape time.Duration
}

func (m *ManualScrapeInfo) Unmarshal(c *confmap.Conf) error {
	m.Disk = c.Get("disk").(string)
	if c.Get("vinyl") == "33" {
		m.Scrape = 10 * time.Second
	} else {
		m.Scrape = 2 * time.Second
	}
	return nil
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"disk":  "Beatles",
		"vinyl": "33",
	})
	scrapeInfo := &ManualScrapeInfo{}
	if err := conf.Unmarshal(scrapeInfo, confmap.WithIgnoreUnused()); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nDisk: %q\nScrape: %s\n", scrapeInfo.Disk, scrapeInfo.Scrape)
}
Output:

Configuration contains the following:
Disk: "Beatles"
Scrape: 10s
Example (SimpleUnmarshaling)

We can annotate a struct with mapstructure field annotations.

package main

import (
	"fmt"
	"time"

	"go.opentelemetry.io/collector/confmap"
)

type DiskScrape struct {
	Disk   string        `mapstructure:"disk"`
	Scrape time.Duration `mapstructure:"scrape"`
}

func main() {
	conf := confmap.NewFromStringMap(map[string]any{
		"disk":   "c",
		"scrape": "5s",
	})
	scrapeInfo := &DiskScrape{}
	if err := conf.Unmarshal(scrapeInfo); err != nil {
		panic(err)
	}
	fmt.Printf("Configuration contains the following:\nDisk: %q\nScrape: %s\n", scrapeInfo.Disk, scrapeInfo.Scrape)
}
Output:

Configuration contains the following:
Disk: "c"
Scrape: 5s

Index

Examples

Constants

View Source
const (
	// KeyDelimiter is used as the default key delimiter in the default koanf instance.
	KeyDelimiter = "::"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type ChangeEvent

type ChangeEvent struct {
	// Error is nil if the config is changed and needs to be re-fetched.
	// Any non-nil error indicates that there was a problem with watching the config changes.
	Error error
}

ChangeEvent describes the particular change event that happened with the config.

type CloseFunc

type CloseFunc func(context.Context) error

CloseFunc a function equivalent to Retrieved.Close.

type Conf

type Conf struct {
	// contains filtered or unexported fields
}

Conf represents the raw configuration map for the OpenTelemetry Collector. The confmap.Conf can be unmarshalled into the Collector's config using the "service" package.

func New

func New() *Conf

New creates a new empty confmap.Conf instance.

func NewFromStringMap

func NewFromStringMap(data map[string]any) *Conf

NewFromStringMap creates a confmap.Conf from a map[string]any.

func (*Conf) AllKeys

func (l *Conf) AllKeys() []string

AllKeys returns all keys holding a value, regardless of where they are set. Nested keys are returned with a KeyDelimiter separator.

func (*Conf) Get

func (l *Conf) Get(key string) any

Get can retrieve any value given the key to use.

func (*Conf) IsSet

func (l *Conf) IsSet(key string) bool

IsSet checks to see if the key has been set in any of the data locations.

func (*Conf) Marshal

func (l *Conf) Marshal(rawVal any, _ ...MarshalOption) error

Marshal encodes the config and merges it into the Conf.

func (*Conf) Merge

func (l *Conf) Merge(in *Conf) error

Merge merges the input given configuration into the existing config. Note that the given map may be modified.

func (*Conf) Sub

func (l *Conf) Sub(key string) (*Conf, error)

Sub returns new Conf instance representing a sub-config of this instance. It returns an error is the sub-config is not a map[string]any (use Get()), and an empty Map if none exists.

func (*Conf) ToStringMap

func (l *Conf) ToStringMap() map[string]any

ToStringMap creates a map[string]any from a Parser.

func (*Conf) Unmarshal

func (l *Conf) Unmarshal(result any, opts ...UnmarshalOption) error

Unmarshal unmarshalls the config into a struct using the given options. Tags on the fields of the structure must be properly set.

type Converter

type Converter interface {
	// Convert applies the conversion logic to the given "conf".
	Convert(ctx context.Context, conf *Conf) error
}

Converter is a converter interface for the confmap.Conf that allows distributions (in the future components as well) to build backwards compatible config converters.

type ConverterFactory added in v0.99.0

type ConverterFactory = moduleFactory[Converter, ConverterSettings]

ConverterFactory defines a factory that can be used to instantiate new instances of a Converter.

func NewConverterFactory added in v0.99.0

func NewConverterFactory(f CreateConverterFunc) ConverterFactory

NewConverterFactory can be used to create a ConverterFactory.

type ConverterSettings added in v0.94.0

type ConverterSettings struct {
	// Logger is a zap.Logger that will be passed to Converters.
	// Converters should be able to rely on the Logger being non-nil;
	// when instantiating a Converter with a ConverterFactory,
	// nil Logger references should be replaced with a no-op Logger.
	Logger *zap.Logger
}

ConverterSettings are the settings to initialize a Converter.

type CreateConverterFunc added in v0.99.0

type CreateConverterFunc = createConfmapFunc[Converter, ConverterSettings]

CreateConverterFunc is a function that creates a Converter instance.

type CreateProviderFunc added in v0.99.0

type CreateProviderFunc = createConfmapFunc[Provider, ProviderSettings]

CreateProviderFunc is a function that creates a Provider instance.

type MarshalOption

type MarshalOption interface {
	// contains filtered or unexported methods
}

type Marshaler

type Marshaler interface {
	// Marshal the config into a Conf in a custom way.
	// The Conf will be empty and can be merged into.
	Marshal(component *Conf) error
}

Marshaler defines an optional interface for custom configuration marshaling. A configuration struct can implement this interface to override the default marshaling.

type Provider

type Provider interface {
	// Retrieve goes to the configuration source and retrieves the selected data which
	// contains the value to be injected in the configuration and the corresponding watcher that
	// will be used to monitor for updates of the retrieved value.
	//
	// `uri` must follow the "<scheme>:<opaque_data>" format. This format is compatible
	// with the URI definition (see https://datatracker.ietf.org/doc/html/rfc3986). The "<scheme>"
	// must be always included in the `uri`. The "<scheme>" supported by any provider:
	//   - MUST consist of a sequence of characters beginning with a letter and followed by any
	//     combination of letters, digits, plus ("+"), period ("."), or hyphen ("-").
	//     See https://datatracker.ietf.org/doc/html/rfc3986#section-3.1.
	//   - MUST be at least 2 characters long to avoid conflicting with a driver-letter identifier as specified
	//     in https://tools.ietf.org/id/draft-kerwin-file-scheme-07.html#syntax.
	//   - For testing, all implementation MUST check that confmaptest.ValidateProviderScheme returns no error.
	//
	// `watcher` callback is called when the config changes. watcher may be called from
	// a different go routine. After watcher is called Retrieved.Get should be called
	// to get the new config. See description of Retrieved for more details.
	// watcher may be nil, which indicates that the caller is not interested in
	// knowing about the changes.
	//
	// If ctx is cancelled should return immediately with an error.
	// Should never be called concurrently with itself or with Shutdown.
	Retrieve(ctx context.Context, uri string, watcher WatcherFunc) (*Retrieved, error)

	// Scheme returns the location scheme used by Retrieve.
	Scheme() string

	// Shutdown signals that the configuration for which this Provider was used to
	// retrieve values is no longer in use and the Provider should close and release
	// any resources that it may have created.
	//
	// This method must be called when the Collector service ends, either in case of
	// success or error. Retrieve cannot be called after Shutdown.
	//
	// Should never be called concurrently with itself or with Retrieve.
	// If ctx is cancelled should return immediately with an error.
	Shutdown(ctx context.Context) error
}

Provider is an interface that helps to retrieve a config map and watch for any changes to the config map. Implementations may load the config from a file, a database or any other source.

The typical usage is the following:

r, err := provider.Retrieve("file:/path/to/config")
// Use r.Map; wait for watcher to be called.
r.Close()
r, err = provider.Retrieve("file:/path/to/config")
// Use r.Map; wait for watcher to be called.
r.Close()
// repeat retrieve/wait/close cycle until it is time to shut down the Collector process.
// ...
provider.Shutdown()

type ProviderFactory added in v0.99.0

type ProviderFactory = moduleFactory[Provider, ProviderSettings]

ProviderFactory defines a factory that can be used to instantiate new instances of a Provider.

func NewProviderFactory added in v0.99.0

func NewProviderFactory(f CreateProviderFunc) ProviderFactory

NewProviderFactory can be used to create a ProviderFactory.

type ProviderSettings added in v0.94.0

type ProviderSettings struct {
	// Logger is a zap.Logger that will be passed to Providers.
	// Providers should be able to rely on the Logger being non-nil;
	// when instantiating a Provider with a ProviderFactory,
	// nil Logger references should be replaced with a no-op Logger.
	Logger *zap.Logger
}

ProviderSettings are the settings to initialize a Provider.

type Resolver

type Resolver struct {
	// contains filtered or unexported fields
}

Resolver resolves a configuration as a Conf.

func NewResolver

func NewResolver(set ResolverSettings) (*Resolver, error)

NewResolver returns a new Resolver that resolves configuration from multiple URIs.

To resolve a configuration the following steps will happen:

  1. Retrieves individual configurations from all given "URIs", and merge them in the retrieve order.
  2. Once the Conf is merged, apply the converters in the given order.

After the configuration was resolved the `Resolver` can be used as a single point to watch for updates in the configuration data retrieved via the config providers used to process the "initial" configuration and to generate the "effective" one. The typical usage is the following:

Resolver.Resolve(ctx)
Resolver.Watch() // wait for an event.
Resolver.Resolve(ctx)
Resolver.Watch() // wait for an event.
// repeat Resolve/Watch cycle until it is time to shut down the Collector process.
Resolver.Shutdown(ctx)

`uri` must follow the "<scheme>:<opaque_data>" format. This format is compatible with the URI definition (see https://datatracker.ietf.org/doc/html/rfc3986). An empty "<scheme>" defaults to "file" schema.

func (*Resolver) Resolve

func (mr *Resolver) Resolve(ctx context.Context) (*Conf, error)

Resolve returns the configuration as a Conf, or error otherwise. Should never be called concurrently with itself, Watch or Shutdown.

func (*Resolver) Shutdown

func (mr *Resolver) Shutdown(ctx context.Context) error

Shutdown signals that the provider is no longer in use and the that should close and release any resources that it may have created. It terminates the Watch channel.

Should never be called concurrently with itself or Get.

func (*Resolver) Watch

func (mr *Resolver) Watch() <-chan error

Watch blocks until any configuration change was detected or an unrecoverable error happened during monitoring the configuration changes.

Error is nil if the configuration is changed and needs to be re-fetched. Any non-nil error indicates that there was a problem with watching the configuration changes.

Should never be called concurrently with itself or Get.

type ResolverSettings

type ResolverSettings struct {
	// URIs locations from where the Conf is retrieved, and merged in the given order.
	// It is required to have at least one location.
	URIs []string

	// ProviderFactories is a slice of Provider factories.
	// It is required to have at least one factory.
	ProviderFactories []ProviderFactory

	// DefaultScheme is the scheme that is used if ${} syntax is used but no schema is provided.
	// If no DefaultScheme is set, ${} with no schema will not be expanded.
	// It is strongly recommended to set "env" as the default scheme to align with the
	// OpenTelemetry Configuration Specification
	DefaultScheme string

	// ProviderSettings contains settings that will be passed to Provider
	// factories when instantiating Providers.
	ProviderSettings ProviderSettings

	// ConverterFactories is a slice of Converter creation functions.
	ConverterFactories []ConverterFactory

	// ConverterSettings contains settings that will be passed to Converter
	// factories when instantiating Converters.
	ConverterSettings ConverterSettings
}

ResolverSettings are the settings to configure the behavior of the Resolver.

type Retrieved

type Retrieved struct {
	// contains filtered or unexported fields
}

Retrieved holds the result of a call to the Retrieve method of a Provider object.

func NewRetrieved

func NewRetrieved(rawConf any, opts ...RetrievedOption) (*Retrieved, error)

NewRetrieved returns a new Retrieved instance that contains the data from the raw deserialized config. The rawConf can be one of the following types:

  • Primitives: int, int32, int64, float32, float64, bool, string;
  • []any;
  • map[string]any;

func NewRetrievedFromYAML added in v0.103.0

func NewRetrievedFromYAML(yamlBytes []byte, opts ...RetrievedOption) (*Retrieved, error)

NewRetrievedFromYAML returns a new Retrieved instance that contains the deserialized data from the yaml bytes. * yamlBytes the yaml bytes that will be deserialized. * opts specifies options associated with this Retrieved value, such as CloseFunc.

func (*Retrieved) AsConf

func (r *Retrieved) AsConf() (*Conf, error)

AsConf returns the retrieved configuration parsed as a Conf.

func (*Retrieved) AsRaw

func (r *Retrieved) AsRaw() (any, error)

AsRaw returns the retrieved configuration parsed as an any which can be one of the following types:

  • Primitives: int, int32, int64, float32, float64, bool, string;
  • []any - every member follows the same rules as the given any;
  • map[string]any - every value follows the same rules as the given any;

func (*Retrieved) AsString added in v0.103.0

func (r *Retrieved) AsString() (string, error)

AsString returns the retrieved configuration as a string. If the retrieved configuration is not convertible to a string unambiguously, an error is returned. If the retrieved configuration is a string, the string is returned. This method is used to resolve ${} references in inline position.

func (*Retrieved) Close

func (r *Retrieved) Close(ctx context.Context) error

Close and release any watchers that Provider.Retrieve may have created.

Should block until all resources are closed, and guarantee that `onChange` is not going to be called after it returns except when `ctx` is cancelled.

Should never be called concurrently with itself.

type RetrievedOption

type RetrievedOption interface {
	// contains filtered or unexported methods
}

RetrievedOption options to customize Retrieved values.

func WithRetrievedClose

func WithRetrievedClose(closeFunc CloseFunc) RetrievedOption

WithRetrievedClose overrides the default Retrieved.Close function. The default Retrieved.Close function does nothing and always returns nil.

type UnmarshalOption

type UnmarshalOption interface {
	// contains filtered or unexported methods
}

func WithIgnoreUnused added in v0.92.0

func WithIgnoreUnused() UnmarshalOption

WithIgnoreUnused sets an option to ignore errors if existing keys in the original Conf were unused in the decoding process (extra keys).

type Unmarshaler

type Unmarshaler interface {
	// Unmarshal a Conf into the struct in a custom way.
	// The Conf for this specific component may be nil or empty if no config available.
	// This method should only be called by decoding hooks when calling Conf.Unmarshal.
	Unmarshal(component *Conf) error
}

Unmarshaler interface may be implemented by types to customize their behavior when being unmarshaled from a Conf.

type WatcherFunc

type WatcherFunc func(*ChangeEvent)

Directories

Path Synopsis
Package confmaptest helps loading confmap.Conf to test packages implementing using the configuration.
Package confmaptest helps loading confmap.Conf to test packages implementing using the configuration.
converter
internal
provider
envprovider Module
fileprovider Module
httpprovider Module
httpsprovider Module
yamlprovider Module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL