config

Documentation

Overview

Package config is used to configure software systems. A configuration managed by package config is called a profile. The binary that loads a profile declares a set of named, global objects through the APIs in this package. A profile configures these objects (objects may depend on each other, forming a DAG) and lets the user retrieve configured objects through its API.

The semantics of profiles provide the kind of flexibility that is often required in operational contexts. Profiles define a principled overriding so that a base configuration can be extended by the user, either by composing multiple configuration or by editing the configuration through a command-line integration. Profiles may also derive multiple instances from the same base instance in order to provide small variations on instance configuration. Profiles define a concrete syntax so that they may be stored (e.g., centrally) or transmitted over a network connection (e.g., to bootstrap a remote binary with a particular configuration). Profiles are also self-documenting in the manner of Go's flag package. Profiles are resolved lazily, and thus maintain configuration for unknown instances, so long as these are never retrieved. This permits a single profile to be reused across many binaries without concern for compatibility.

Profile syntax

A profile contains a set of clauses, or directives. Each clause either declares a new instance or configures an existing instance. Clauses are interpreted in order, top-to-bottom, and later configurations override earlier configurations. These semantics accommodate for "overlays", where for example a user profile is loaded after a base profile to provide customization. Within GRAIL, a base profile is declared in the standard package github.com/grailbio/base/grail, which also loads a user profile from $HOME/grail/profile.

A parameter is set by the directive param. For example, the following sets the parallelism parameter on the instance bigslice to 1024:

param bigslice parallelism = 1024

The values supported by profiles are: integers, strings, booleans, floats, and indirections (naming other instances). The following shows an example of each:

param bigslice load-factor = 0.8
param bigmachine/ec2system username = "marius"
param bigmachine/ec2system on-demand = false
param s3 retries = 8

As a shortcut, parameters for the same instance may be grouped together. For example, the two parameters on the instance bigmachine/ec2system may be grouped together as follows:

param bigmachine/ec2system (
	username = "marius"
	on-demand = false
)

Instances may refer to each other by name. The following configures the aws/ticket instance to use a particular ticket path and region; it then configures bigmachine/ec2system to use this AWS session.

param aws/ticket (
	path = "eng/dev/aws"
	region = "us-west-2"
)

param bigmachine/ec2system aws = aws/ticket

Profiles may also define new instances with different configurations. This is done via the instance directive. For example, if we wanted to declare a new bigmachine/ec2system that used on-demand instances instead of spot instances, we could define a profile as follows:

instance bigmachine/ec2ondemand bigmachine/ec2system

param bigmachine/ec2ondemand on-demand = false

Since it is common to declare an instance and configure it, the profile syntax provides an affordance for combining the two, also through grouping. The above is equivalent to:

instance bigmachine/ec2ondemand bigmachine/ec2system (
	on-demand = false
	username = "marius-ondemand"
	// (any other configuration to be changed from the base)
)

New instances may depend on any instance. For example, the above may be further customized as follows.

instance bigmachine/ec2ondemand-anonymous bigmachine/ec2ondemand (
	username = "anonymous"
)

Customization through flags

Profile parameters may be adjusted via command-line flags. Profile provides utility methods to register flags and interpret them. See the appropriate methods for more details. Any parameter may be set through the provided command-line flags by specifying the path to the parameter. As an example, the following invocations customize aspects of the above profile.

# Override the ticket path and the default ec2system username.
# -set flags are interpreted in order, and the following is equivalent
# to the clauses
# 	param aws/ticket path = "eng/prod/aws"
#	param bigmachine/ec2system username = "anonymous"
$ program -set aws/ticket.path=eng/prod/aws -set bigmachine/ec2system.username=anonymous

# User the aws/env instance instead of aws/ticket, as above.
# The type of a flag is interpreted based on underlying type, so
# the following is equivalent to the clause
# 	param bigmachine/ec2system aws = aws/env
$ program -set bigmachine/ec2system.aws=aws/env

Default profile

Package config also defines a default profile and a set of package-level methods that operate on this profile. Most users should make use only of the default profile. This package also exports an http handler on the path /debug/profile on the default (global) ServeMux, which returns the global profile in parseable form.

Index

• Variables
• func Default(name, instance string)
• func Get(path string) (value string, ok bool)
• func Instance(name string, ptr interface{}) error
• func Merge(p *Profile)
• func Must(name string, ptr interface{})
• func Parse(r io.Reader) error
• func ProcessFlags() error
• func Register[T any](name string, configure func(*Constructor[T]))
• func RegisterFlags(prefix string, defaultProfilePath string)
• func Set(path, value string) error
• type Constructor
  • func (c *Constructor[_]) Bool(name string, value bool, help string) *bool
  • func (c *Constructor[_]) BoolVar(ptr *bool, name string, value bool, help string)
  • func (c *Constructor[_]) Float(name string, value float64, help string) *float64
  • func (c *Constructor[_]) FloatVar(ptr *float64, name string, value float64, help string)
  • func (c *Constructor[_]) InstanceVar(ptr interface{}, name string, value string, help string)
  • func (c *Constructor[_]) Int(name string, value int, help string) *int
  • func (c *Constructor[_]) IntVar(ptr *int, name string, value int, help string)
  • func (c *Constructor[_]) String(name string, value string, help string) *string
  • func (c *Constructor[_]) StringVar(ptr *string, name string, value string, help string)
• type Nil
• type Profile
  • func Application() *Profile
  • func New() *Profile
  • func (p *Profile) Get(path string) (value string, ok bool)
  • func (p *Profile) Instance(name string, ptr interface{}) error
  • func (p *Profile) InstanceNames() map[string]struct{}
  • func (p *Profile) Merge(q *Profile)
  • func (p *Profile) NeedProcessFlags() bool
  • func (p *Profile) Parse(r io.Reader) error
  • func (p *Profile) PrintTo(w io.Writer) error
  • func (p *Profile) ProcessFlags() error
  • func (p *Profile) RegisterFlags(fs *flag.FlagSet, prefix string, defaultProfilePath string)
  • func (p *Profile) Set(path string, value string) error

Variables

var NewDefault = New

NewDefault is used to initialize the default profile. It can be set by a program before the application profile has been created in order to support asynchronous profile retrieval.

Functions

func Default

func Default(name, instance string)

Default declares a new derived instance. It is a convenience function used to provide a default implementation among multiple choices, and is equivalent to the the profile directive

instance name instance

Default panics if name is already the name of an instance, or if the specified parent instance does not exist.

func Get

func Get(path string) (value string, ok bool)

Get retrieves the value of the parameter named by the provided path on the default profile.

func Instance

func Instance(name string, ptr interface{}) error

Instance retrieves the instance with the provided name into the provided pointer from the default profile. See Profile.Instance for more details.

func Merge

func Merge(p *Profile)

Merge merges profile p into the default profile. See Profile.Merge for more details.

func Must

func Must(name string, ptr interface{})

Must is a version of get which calls log.Fatal on error.

func Parse

func Parse(r io.Reader) error

Parse parses the profile in reader r into the default profile. See Profile.Parse for more details.

func ProcessFlags

func ProcessFlags() error

ProcessFlags processes the flags as registered by RegisterFlags.

func Register

func Register[T any](name string, configure func(*Constructor[T]))

Register registers a constructor and later invokes the provided function whenever a new profile instance is created. Register panics if multiple constructors are registered with the same name. Constructors should typically be registered in package init functions, and the configure function must define at least Constructor.New. For example, the following configures a constructor with a single parameter, n, which simply returns its value.

config.Register("config/test", func(constr *config.Constructor[int]) {
	n := constr.Int("n", 32, "the number configured")
	constr.New = func() (int, error) {
		return *n, nil
	}
	constr.Doc = "a customizable integer"
})

func RegisterFlags

func RegisterFlags(prefix string, defaultProfilePath string)

RegisterFlags registers the default profile on flag.CommandLine with the provided prefix. See Profile.RegisterFlags for details.

func Set

func Set(path, value string) error

Set sets the value of the parameter named by the provided path on the default profile. See Profile.Set for more details.

Types

type Constructor

type Constructor[T any] struct {
	New func() (T, error)
	Doc string
	// contains filtered or unexported fields
}

Constructor defines a constructor, as configured by Register. Typically a constructor registers a set of parameters through the flags-like methods provided by Constructor. The value returned by New is configured by these parameters.

func (*Constructor[_]) Bool

func (c *Constructor[_]) Bool(name string, value bool, help string) *bool

Bool registers a boolean parameter with a default value. The returned pointer points to its value.

func (*Constructor[_]) BoolVar

func (c *Constructor[_]) BoolVar(ptr *bool, name string, value bool, help string)

BoolVar registers a boolean parameter with a default value. The parameter's value written to the location pointed to by ptr.

func (*Constructor[_]) Float

func (c *Constructor[_]) Float(name string, value float64, help string) *float64

Float registers floating point parameter with a default value. The returned pointer points to its value.

func (*Constructor[_]) FloatVar

func (c *Constructor[_]) FloatVar(ptr *float64, name string, value float64, help string)

FloatVar register a floating point parameter with a default value. The parameter's value is written to the provided pointer.

func (*Constructor[_]) InstanceVar

func (c *Constructor[_]) InstanceVar(ptr interface{}, name string, value string, help string)

InstanceVar registers a parameter that is satisfied by another instance; the method panics if ptr is not a pointer. The default value is always an indirection; if it is left empty it is taken as the nil value: it remains uninitialized by default.

func (*Constructor[_]) Int

func (c *Constructor[_]) Int(name string, value int, help string) *int

Int registers an integer parameter with a default value. The returned pointer points to its value.

func (*Constructor[_]) IntVar

func (c *Constructor[_]) IntVar(ptr *int, name string, value int, help string)

IntVar registers an integer parameter with a default value. The parameter's value written to the location pointed to by ptr.

func (*Constructor[_]) String

func (c *Constructor[_]) String(name string, value string, help string) *string

String registers a string parameter with a default value. The returned pointer points to its value.

func (*Constructor[_]) StringVar

func (c *Constructor[_]) StringVar(ptr *string, name string, value string, help string)

StringVar registers a string parameter with a default value. The parameter's value written to the location pointed to by ptr.

type Nil

type Nil interface {
	// contains filtered or unexported methods
}

Nil is an interface type with no implementations. Constructor[Nil] indicates an instance is created just for its side effects.

type Profile

type Profile struct {
	// contains filtered or unexported fields
}

Profile stores a set of parameters and configures instances based on these. It is the central data structure of this package as detailed in the package docs. Each Profile instance maintains its own set of instances. Most users should use the package-level methods that operate on the default profile.

func Application

func Application() *Profile

Application returns the default application profile. The default instance is initialized during the first call to Application (and thus of the package-level methods that operate on the default profile). Because of this, Application (and the other package-level methods operating on the default profile) should not be called during package initialization as doing so means that some global objects may not yet have been registered.

func New

func New() *Profile

New creates and returns a new profile, installing all currently registered global objects. Global objects registered after a call to New are not reflected in the returned profile.

func (*Profile) Get

func (p *Profile) Get(path string) (value string, ok bool)

Get returns the value of the configured parameter at the provided dot-separated path.

func (*Profile) Instance

func (p *Profile) Instance(name string, ptr interface{}) error

Instance retrieves the named instance from this profile into the pointer ptr. All of its parameters are fully resolved and the underlying global object is instantiated according to the desired parameterization. Instance panics if ptr is not a pointer type. If the type of the instance cannot be assigned to the value pointed to by ptr, an error is returned. Since such errors may occur transitively (e.g., the type of an instance required by another instance may be wrong), the source location of the type mismatch is included in the error to help with debugging. Instances are cached and are only initialized the first time they are requested.

If ptr is nil, the instance is created without populating the pointer.

func (*Profile) InstanceNames

func (p *Profile) InstanceNames() map[string]struct{}

InstanceNames returns the set of names of instances provided by p.

func (*Profile) Merge

func (p *Profile) Merge(q *Profile)

Merge merges the instance parameters in profile q into p, so that parameters defined in q override those in p.

func (*Profile) NeedProcessFlags

func (p *Profile) NeedProcessFlags() bool

NeedProcessFlags returns true when a call to p.ProcessFlags should not be delayed -- i.e., the flag values have user-visible side effects.

func (*Profile) Parse

func (p *Profile) Parse(r io.Reader) error

Parse parses a profile from the provided reader into p. On success, the instances defined by the profile in src are merged into profile p. If the reader implements

Name() string

then the result of calling Name is used as a filename to provide positional information in errors.

func (*Profile) PrintTo

func (p *Profile) PrintTo(w io.Writer) error

func (*Profile) ProcessFlags

func (p *Profile) ProcessFlags() error

ProcessFlags processes the flags as registered by RegisterFlags, and is documented by that method.

func (*Profile) RegisterFlags

func (p *Profile) RegisterFlags(fs *flag.FlagSet, prefix string, defaultProfilePath string)

RegisterFlags registers a set of flags on the provided FlagSet. These flags configure the profile when ProcessFlags is called (after flag parsing). The flags are:

-profile path
	Parses and loads the profile at the given path. This flag may be
	repeated, loading each profile in turn. If no -profile flags are
	specified, then the provided default path is loaded instead. If
	the default path does not exist, it is skipped; other profile loading
	errors cause ProcessFlags to return an error.

-set key=value
	Sets the value of the named parameter. See Profile.Set for
	details. This flag may be repeated.

-profileinline text
	Parses the argument. This is equivalent to writing the text to a file
	and using -profile.

-profiledump
	Writes the profile (after processing the above flags) to standard
	error and exits.

The flag names are prefixed with the provided prefix.

func (*Profile) Set

func (p *Profile) Set(path string, value string) error

Set sets the value of the parameter at the provided path to the provided value, which is intepreted according to the type of the parameter at that path. Set returns an error if the parameter does not exist or if the value cannot be parsed into the expected type. The path is a set of identifiers separated by dots ("."). Paths may traverse multiple indirections.