config

package module
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 1, 2020 License: MIT Imports: 16 Imported by: 4

README

Nacelle Config GoDoc CircleCI Coverage Status

Configuration loading and validation for nacelle.

Often, initializers and processes will need external configuration during their startup process. These values can be pulled from a configuration loader backed by a particular source (e.g. environment or disk) and assigned to tagged fields of a configuration struct.

You can see an additional example of loading configuration in the example repository: definition and loading.

Configuration Struct Definition

Configuration structs are defined by the application or library developer with the fields needed by the package in which they are defined. Each field is tagged with a source hint (e.g. an environment variable name, a key in a YAML file) and, optionally, default values and basic validation. Tagged fields must be exported in order for this package to assign to them.

The following example defines configuration for a hypothetical worker process. For the application to start successfully, the address of an API must be supplied. All other configuration values are optional.

type Config struct {
    APIAddr        string   `env:"api_addr" required:"true"`
    CassandraHosts []string `env:"cassandra_hosts"`
    NumWorkers     int      `env:"num_workers" default:"10"`
    BufferSize     int      `env:"buffer_size" default:"1024"`
}
Configuration Loading

At initialization time of an application component, the particular subset of configuration variables should be populated, validated, and stored on the service that will later require them.

func (p *Process) Init(config nacelle.Config) error {
    appConfig := &Config{}
    if err := config.Load(appConfig); err != nil {
        return err
    }

    // Use populated appConfig
    return nil
}

The Load method fails if a value from the source cannot be converted into the correct type, a value from the source cannot be unmarshalled, or is required and not supplied. After each successful load of a configuration struct, the loaded configuration values will are logged. This, however, may be a concern for application secrets. In order to hide sensitive configuration values, add the mask:"true" struct tag to the field. This will omit that value from the log message. Additionally, configuration loader object can be initialized with a blacklist of values that should be masked (values printed as ***** rather than their real value) instead of omitted. These values can be configured in the bootstrapper.

JSON Unmarshalling

The values that are loaded into non-string field of a configuration struct are interpreted as JSON. Supplying the environment CASSANDRA_HOSTS='["host1", "host2", "host3"]' to the configuration struct above will populate the CassandraHosts field with the values host1, host2, and host3. The defaults for such fields can also be supplied as a JSON-encoded string, but must be escaped to preserve the format of the struct tag.

type Config struct {
    CassandraHosts []string `env:"cassandra_hosts" default:"[\"host1\", \"host2\", \"host3\"]"`
}
Conversion and Validation

After successful loading of a configuration struct, the method named PostLoad will be called if it is defined. This allows a place for additional validation (such as mutually exclusive settings, regex value matching, etc) and deserialization of more complex types (such enums from strings, durations from integers, etc). The following example parses and stores a text/template from a user-supplied string.

import "text/template"

type Config struct {
    RawTemplate    string `env:"template" default:"Hello, {{.Name}}!"`
    ParsedTemplate *template.Template
}

func (c *Config) PostLoad() (err error) {
    c.ParsedTemplate, err = template.New("ConfigTemplate").Parse(c.RawTemplate)
    return
}

An error returned by PostLoad will be returned via the Load method.

Anonymous Structs

Loading configuration values also works with structs containing composite fields. The following example shows the definition of multiple configuration structs with a set of shared fields.

type StreamConfig struct {
    StreamName string `env:"stream_name" required:"true"`
}

type StreamProducerConfig struct {
    StreamConfig
    PublishAttempts int `env:"publish_attempts" default:"3"`
    PublishDelay    int `env:"publish_delay" default:"1"`
}

type StreamConsumerConfig struct {
    StreamConfig
    FetchLimit int `env:"fetch_limit" default:"100"`
}
Sourcers

A sourcer reads values from a particular source based on a configuration struct's tags. Sourcers declare the struct tags that determine their behavior when loading configuration structs. The examples above only work with the environment sourcer. All sources support the default and required tags (which are mutually exclusive). Tagged fields must be exported. The following six sourcers are supplied. Additional behavior can be added by conforming to the Sourcer interface.

Environment Sourcer
An environment sourcer reads the env tag and looks up the corresponding value in the process's environment. An expected prefix may be supplied in order to namespace application configuration from the rest of the system. A sourcer instantiated with NewEnvSourcer("APP") will load the env tag fetch_limit from the environment variable APP_FETCH_LIMIT and falling back to the environment variable FETCH_LIMIT.
Test Environment Sourcer
A test environment sourcer reads the env tag but looks up the corresponding value from a literal map. This sourcer can be used in unit tests where the full construction of a nacelle process is too burdensome.
Flag Sourcer
A flag sourcer reads the flag tag and looks up the corresponding value attached to the process's command line arguments.
File Sourcer
A file sourcer reads the file tag and returns the value at the given path. A filename and a file parser musts be supplied on instantiation. Both ParseYAML and ParseTOML are supplied file parsers -- note that as JSON is a subset of YAML, ParseYAML will also correctly parse JSON files. If a nil file parser is supplied, one is chosen by the filename extension. A file sourcer will load the file tag api.timeout from the given file by parsing it into a map of values and recursively walking the (keys separated by dots). This can return a primitive type or a structured map, as long as the target field has a compatible type. The constructor NewOptionalFileSourcer will return a no-op sourcer if the filename does not exist.
Multi sourcer
A multi-sourcer is a sourcer wrapping one or more other sourcers. For each configuration struct field, each sourcer is queried in reverse order of registration and the first value to exist is returned. This is useful to allow a chain of configuration files in which some files or directories take precedence over others, or to allow environment variables to take precedence over files.
Directory Sourcer
A directory sourcer creates a multi-sourcer by reading each file in a directory in alphabetical order. The constructor NewOptionalDirectorySourcer will return a no-op sourcer if the directory does not exist.
Glob Sourcer
A glob sourcer creates a multi-sourcer by reading each file that matches a given glob pattern. Each matching file creates a distinct file sourcer and does so in alphabetical order.
Tag Modifiers

A tag modifier dynamically alters the tags of a configuration struct. The following five tag modifiers are supplied. Additional behavior can be added by conforming to the TagModifier interface.

Default Tag Setter
A default tag setter sets the default tag for a particular field. This is useful when the default values supplied by a library are inappropriate for a particular application. This would otherwise require a source change in the library.
Display Tag Setter
A display tag setter sets the display tag to the value of the env tag. This tag modifier can be used to provide sane defaults to the tag without doubling the length of the struct tag definition.
Flag Tag Setter
A flag tag setter sets the flag tag to the value of the env tag. This tag modifier can be used to provide sane defaults to the tag without doubling the length of the struct tag definition.
File Tag Setter
A file tag setter sets the file tag to the value of the env tag. This tag modifier can be used to provide sane defaults to the tag without doubling the length of the struct tag definition.
Env Tag Prefixer
A environment tag prefixer inserts a prefix on each env tags. This is useful when two distinct instances of the same configuration are required, and each one should be configured independently from the other (for example, using the same abstraction to consume from two different event busses with the same consumer code).
Flag Tag Prefixer
A flag tag prefixer inserts a prefix on each flag tag. This effectively looks in a distinct top-level namespace in the parsed configuration. This is similar to the env tag prefixer.
File Tag Prefixer
A file tag prefixer inserts a prefix on each file tag. This effectively looks in a distinct top-level namespace in the parsed configuration. This is similar to the env tag prefixer.

Tag modifiers are supplied at the time that a configuration struct is loaded. In the following example, each env tag is prefixed with ACME_, and the CassandraHosts field is given a default. Notice that you supply the field name to the tag modifier (not a tag value) when targeting a particular field value.

if err := config.Load(
    appConfig,
    NewEnvTagPrefixer("ACME"),
    NewDefaultTagSetter("CassandraHosts", "[127.0.0.1:9042]"),
); err != nil {
    // handle error
}

Documentation

Index

Constants

View Source 
const (
	DefaultTag  = "default"
	RequiredTag = "required"
	EnvTag      = "env"
	FlagTag     = "flag"
	FileTag     = "file"
	DisplayTag  = "display"
	MaskTag     = "mask"
)

Variables

This section is empty.

Functions

func ApplyTagModifiers 

func ApplyTagModifiers(obj interface{}, modifiers ...TagModifier) (modified interface{}, err error)

ApplyTagModifiers returns a new struct with a dynamic type whose fields are equivalent to the given object but whose field tags are run through each tag modifier in sequence.

func ParseTOML 

func ParseTOML(content []byte) (map[string]interface{}, error)

ParseTOML parses the given content as JSON.

func ParseYAML 

func ParseYAML(content []byte) (map[string]interface{}, error)

ParseYAML parses the given content as YAML.

Types

type Config 

type Config interface {
	// Init prepares state required by the registered sourcer. This
	// method should be called before calling any other method.
	Init() error

	// Load populates a configuration object. The given tag modifiers
	// are applied to the configuration object pre-load. If the target
	// value conforms to the PostLoadConfig interface, the PostLoad
	// function may be called multiple times.
	Load(interface{}, ...TagModifier) error

	// MustInject calls Injects and panics on error.
	MustLoad(interface{}, ...TagModifier)

	// Assets returns a list of names of assets that compose the
	// underlying sourcer. This can be a list of matched files that are
	// read, or a token that denotes a fixed source.
	Assets() []string

	// Dump returns the full content of the underlying sourcer. This
	// is used by the logging package to show the content of the
	// environment and config files when a value is missing or otherwise
	// illegal.
	Dump() map[string]string
}

Config is a structure that can populate the exported fields of a struct based on the value of the field `env` tags.

func NewConfig 

func NewConfig(sourcer Sourcer) Config

NewConfig creates a config loader with the given sourcer.

func NewLoggingConfig 

func NewLoggingConfig(config Config, logger Logger, maskedKeys []string) Config

NewLoggingConfig wraps a config object with logging. After each successful load, the populated configuration object is serialized as fields and output at the info level.

type DirectorySourcerConfigFunc added in v1.1.0 

type DirectorySourcerConfigFunc func(*directorySourcerOptions)

DirectorySourcerConfigFunc is a function used to configure instances of directory sourcers.

func WithDirectorySourcerFS added in v1.1.0 

func WithDirectorySourcerFS(fs FileSystem) DirectorySourcerConfigFunc

WithDirectorySourcerFS sets the FileSystem instance.

type FileParser 

type FileParser func(content []byte) (map[string]interface{}, error)

type FileSourcerConfigFunc added in v1.1.0 

type FileSourcerConfigFunc func(*fileSourcerOptions)

FileSourcerConfigFunc is a function used to configure instances of file sourcers.

func WithFileSourcerFS added in v1.1.0 

func WithFileSourcerFS(fs FileSystem) FileSourcerConfigFunc

WithFileSourcerFS sets the FileSystem instance.

type FileSystem added in v1.1.0 

type FileSystem interface {
	// Exists determines if the given path exists.
	Exists(path string) (bool, error)

	// ListFiles returns the names of the files that are a direct
	// child of the directory at the given path.
	ListFiles(path string) ([]string, error)

	// Glob returns the paths that the given pattern matches.
	Glob(pattern string) ([]string, error)

	// ReadFile returns the content of the file at the given path.
	ReadFile(path string) ([]byte, error)
}

FileSystem is an interface wrapping filesystem operations for sourcers that read information from disk. This is necessary in order to allow remote and in-memory filesystems that may be present in some application deployments. Third-party libraries such as spf13/afero that provide FS-like functionality can be shimmed into this interface.

type FlagSourcerConfigFunc added in v1.2.0 

type FlagSourcerConfigFunc func(*flagSourcerOptions)

FlagSourcerConfigFunc is a function used to configure instances of flag sourcers.

func WithFlagSourcerArgs added in v1.2.0 

func WithFlagSourcerArgs(args []string) FlagSourcerConfigFunc

WithFlagSourcerArgs sets raw command line arguments.

type GlobSourcerConfigFunc added in v1.1.0 

type GlobSourcerConfigFunc func(*globSourcerOptions)

GlobSourcerConfigFunc is a function used to configure instances of glob sourcers.

func WithGlobSourcerFS added in v1.1.0 

func WithGlobSourcerFS(fs FileSystem) GlobSourcerConfigFunc

WithGlobSourcerFS sets the FileSystem instance.

type Logger 

type Logger interface {
	// Printf logs a message. Arguments should be handled in the manner of fmt.Printf.
	Printf(format string, args ...interface{})
}

Logger is an interface to the logger where config values are printed.

type PostLoadConfig 

type PostLoadConfig interface {
	PostLoad() error
}

PostLoadConfig is a marker interface for configuration objects which should do some post-processing after being loaded. This can perform additional casting (e.g. ints to time.Duration) and more sophisticated validation (e.g. enum or exclusive values).

type ReflectField 

type ReflectField struct {
	Field     reflect.Value
	FieldType reflect.StructField
}

type Sourcer 

type Sourcer interface {
	// Init is a hook for certain classes of sourcers to read and normalize
	// the source data. This gives a canonical place where external errors
	// can occur that are not directly related to validation.
	Init() error

	// Tags returns a list of tags which are required to get a value from
	// the source. Order matters.
	Tags() []string

	// Get will retrieve a value from the source with the given tag values.
	// The tag values passed to this method will be in the same order as
	// returned from the Tags method. The flag return value directs config
	// population whether or not this value should be treated as missing or
	// skippable.
	Get(values []string) (string, SourcerFlag, error)

	// Assets returns a list of names of assets that compose the sourcer.
	// This can be a list of matched files that are read, or a token that
	// denotes a fixed source.
	Assets() []string

	// Dump returns the full content of the sourcer. This is used by the
	// logging package to show the content of the environment and config
	// files when a value is missing or otherwise illegal.
	Dump() map[string]string
}

Sourcer pulls requested names from a variable source. This can be the environment, a file, a remote server, etc. This can be done on-demand per variable, or a cache of variables can be built on startup and then pulled from a cached mapping as requested.

func NewDirectorySourcer 

func NewDirectorySourcer(dirname string, parser FileParser, configs ...DirectorySourcerConfigFunc) Sourcer

NewDirectorySourcer creates a sourcer that reads files from a directory. For details on parsing format, refer to NewFileParser. Each file in a directory is read in alphabetical order. Nested directories are ignored when reading directory content, and each found regular file is assumed to be parseable by the given FileParser.

func NewEnvSourcer 

func NewEnvSourcer(prefix string) Sourcer

NewEnvSourcer creates a Sourcer that pulls values from the environment. The environment variable {PREFIX}_{NAME} is read before and, if empty, the environment variable {NAME} is read as a fallback. The prefix is normalized by replacing all non-alpha characters with an underscore, removing leading and trailing underscores, and collapsing consecutive underscores with a single character.

func NewFileSourcer 

func NewFileSourcer(filename string, parser FileParser, configs ...FileSourcerConfigFunc) Sourcer

NewFileSourcer creates a sourcer that reads content from a file. The format of the file is read by the given FileParser. The content of the file must be an encoding of a map from string keys to JSON-serializable values. If a nil parser is supplied, one will be selected based on the extension of the file. JSON, YAML, and TOML files are supported.

func NewFlagSourcer added in v1.2.0 

func NewFlagSourcer(configs ...FlagSourcerConfigFunc) Sourcer

NewFlagSourcer creates a Sourcer that pulls values from the application flags.

func NewGlobSourcer 

func NewGlobSourcer(pattern string, parser FileParser, configs ...GlobSourcerConfigFunc) Sourcer

NewGlobSourcer creates a sourcer that reads all files that match the given glob pattern. Each matching file is read in alphabetical order of path. Each matching pathis assumed to be parseable by the given FileParser.

func NewMultiSourcer 

func NewMultiSourcer(sourcers ...Sourcer) Sourcer

NewMultiSourcer creates a sourcer that reads form each sourcer. The last value found is returned - sourcers should be provided from low priority to high priority.

func NewOptionalDirectorySourcer 

func NewOptionalDirectorySourcer(dirname string, parser FileParser, configs ...DirectorySourcerConfigFunc) Sourcer

NewOptionalDirectorySourcer creates a directory sourcer that does not error on init if the directory does not exist. In this case, the underlying sourcer returns no values.

func NewOptionalFileSourcer 

func NewOptionalFileSourcer(filename string, parser FileParser, configs ...FileSourcerConfigFunc) Sourcer

NewOptionalFileSourcer creates a file sourcer that does not error on init if the file does not exist. The underlying sourcer returns no values.

func NewTOMLFileSourcer 

func NewTOMLFileSourcer(filename string) Sourcer

NewTOMLFileSourcer creates a file sourcer that parses conent as TOML.

func NewTestEnvSourcer 

func NewTestEnvSourcer(values map[string]string) Sourcer

NewTestEnvSourcer creates a Sourcer that returns values from a given map.

func NewYAMLFileSourcer 

func NewYAMLFileSourcer(filename string) Sourcer

NewYAMLFileSourcer creates a file sourcer that parses conent as YAML.

type SourcerFlag 

type SourcerFlag int
const (
	FlagUnknown SourcerFlag = iota
	FlagFound
	FlagMissing
	FlagSkip
)

type TagModifier 

type TagModifier interface {
	// AlterFieldTag modifies the tags reference by setting or deleting
	// values from the given tag wrapper. The given tag wrapper is the
	// parsed version of the tag for the given field. Returns an error
	// if there is an internal consistency problem.
	AlterFieldTag(field reflect.StructField, tags *structtag.Tags) error
}

TagModifier is an interface that rewrites a set of tags for a struct field. This interface is used by the ApplyTagModifiers function.

func NewDefaultTagSetter 

func NewDefaultTagSetter(field string, defaultValue string) TagModifier

NewDefaultTagSetter creates a new TagModifier which sets the value of the default tag for a particular field. This is used to change the default values provided by third party libraries (for which a source change would be otherwise required).

func NewDisplayTagSetter 

func NewDisplayTagSetter() TagModifier

NewDisplayTagSetter creates a new TagModifier which sets the value of the display tag to be the same as the env tag.

func NewEnvTagPrefixer 

func NewEnvTagPrefixer(prefix string) TagModifier

NewEnvTagPrefixer creates a new TagModifier which adds a prefix to the values of `env` tags. This can be used to register one config multiple times and have their initialization be read from different environment variables.

func NewFileTagPrefixer 

func NewFileTagPrefixer(prefix string) TagModifier

NewFileTagPrefixer creates a new TagModifier which adds a prefix to the values of `file` tags. This can be used to register one config multiple times and have their initialization be read from different keys in a config file.

func NewFileTagSetter 

func NewFileTagSetter() TagModifier

NewFileTagSetter creates a new TagModifier which sets the value of the file tag to be the same as the env tag.

func NewFlagTagPrefixer added in v1.2.0 

func NewFlagTagPrefixer(prefix string) TagModifier

NewFlagTagPrefixer creates a new TagModifier which adds a prefix to the values of `flag` tags.

func NewFlagTagSetter added in v1.2.0 

func NewFlagTagSetter() TagModifier

NewFlagTagSetter creates a new TagModifier which sets the value of the flag tag to be the same as the env tag.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.