Documentation

Overview

Package config is used to configure software systems. A configuration managed by package config is called a profile. The binary that loads a profile declares a set of named, global objects through the APIs in this package. A profile configures these objects (objects may depend on each other, forming a DAG) and lets the user retrieve configured objects through its API.

The semantics of profiles provide the kind of flexibility that is often required in operational contexts. Profiles define a principled overriding so that a base configuration can be extended by the user, either by composing multiple configuration or by editing the configuration through a command-line integration. Profiles may also derive multiple instances from the same base instance in order to provide small variations on instance configuration. Profiles define a concrete syntax so that they may be stored (e.g., centrally) or transmitted over a network connection (e.g., to bootstrap a remote binary with a particular configuration). Profiles are also self-documenting in the manner of Go's flag package. Profiles are resolved lazily, and thus maintain configuration for unknown instances, so long as these are never retrieved. This permits a single profile to be reused across many binaries without concern for compatibility.

Profile syntax

A profile contains a set of clauses, or directives. Each clause either declares a new instance or configures an existing instance. Clauses are interpreted in order, top-to-bottom, and later configurations override earlier configurations. These semantics accommodate for "overlays", where for example a user profile is loaded after a base profile to provide customization. Within GRAIL, a base profile is declared in the standard package github.com/grailbio/base/grail, which also loads a user profile from $HOME/grail/profile.

A parameter is set by the directive param. For example, the following sets the parallelism parameter on the instance bigslice to 1024:

param bigslice parallelism = 1024

The values supported by profiles are: integers, strings, booleans, floats, and indirections (naming other instances). The following shows an example of each:

param bigslice load-factor = 0.8
param bigmachine/ec2system username = "marius"
param bigmachine/ec2system on-demand = false
param s3 retries = 8

As a shortcut, parameters for the same instance may be grouped together. For example, the two parameters on the instance bigmachine/ec2system may be grouped together as follows:

param bigmachine/ec2system (
	username = "marius"
	on-demand = false
)

Instances may refer to each other by name. The following configures the aws/ticket instance to use a particular ticket path and region; it then configures bigmachine/ec2system to use this AWS session.

param aws/ticket (
	path = "eng/dev/aws"
	region = "us-west-2"
)

param bigmachine/ec2system aws = aws/ticket

Profiles may also define new instances with different configurations. This is done via the instance directive. For example, if we wanted to declare a new bigmachine/ec2system that used on-demand instances instead of spot instances, we could define a profile as follows:

instance bigmachine/ec2ondemand bigmachine/ec2system

param bigmachine/ec2ondemand on-demand = false

Since it is common to declare an instance and configure it, the profile syntax provides an affordance for combining the two, also through grouping. The above is equivalent to:

instance bigmachine/ec2ondemand bigmachine/ec2system (
	on-demand = false
	username = "marius-ondemand"
	// (any other configuration to be changed from the base)
)

New instances may depend on any instance. For example, the above may be further customized as follows.

instance bigmachine/ec2ondemand-anonymous bigmachine/ec2ondemand (
	username = "anonymous"
)

Customization through flags

Profile parameters may be adjusted via command-line flags. Profile provides utility methods to register flags and interpret them. See the appropriate methods for more details. Any parameter may be set through the provided command-line flags by specifying the path to the parameter. As an example, the following invocations customize aspects of the above profile.

# Override the ticket path and the default ec2system username.
# -set flags are interpreted in order, and the following is equivalent
# to the clauses
# 	param aws/ticket path = "eng/prod/aws"
#	param bigmachine/ec2system username = "anonymous"
$ program -set aws/ticket.path=eng/prod/aws -set bigmachine/ec2system.username=anonymous

# User the aws/env instance instead of aws/ticket, as above.
# The type of a flag is interpreted based on underlying type, so
# the following is equivalent to the clause
# 	param bigmachine/ec2system aws = aws/env
$ program -set bigmachine/ec2system.aws=aws/env

Default profile

Package config also defines a default profile and a set of package-level methods that operate on this profile. Most users should make use only of the default profile. This package also exports an http handler on the path /debug/profile on the default (global) ServeMux, which returns the global profile in parseable form.

Index

Constants

This section is empty.

Variables

View Source
var NewDefault = New

    NewDefault is used to initialize the default profile. It can be set by a program before the application profile has been created in order to support asynchronous profile retrieval.

    Functions

    func Default

    func Default(name, instance string)

      Default declares a new derived instance. It is a convenience function used to provide a default implementation among multiple choices, and is equivalent to the the profile directive

      instance name instance
      

      Default panics if name is already the name of an instance, or if the specified parent instance does not exist.

      func Get

      func Get(path string) (value string, ok bool)

        Get retrieves the value of the parameter named by the provided path on the default profile.

        func Instance

        func Instance(name string, ptr interface{}) error

          Instance retrieves the instance with the provided name into the provided pointer from the default profile. See Profile.Instance for more details.

          func Merge

          func Merge(p *Profile)

            Merge merges profile p into the default profile. See Profile.Merge for more details.

            func Must

            func Must(name string, ptr interface{})

              Must is a version of get which calls log.Fatal on error.

              func Parse

              func Parse(r io.Reader) error

                Parse parses the profile in reader r into the default profile. See Profile.Parse for more details.

                func ProcessFlags

                func ProcessFlags() error

                  ProcessFlags processes the flags as registered by RegisterFlags.

                  func Register

                  func Register(name string, configure func(*Constructor))

                    Register registers a constructor and later invokes the provided function whenever a new profile instance is created. Register panics if multiple constructors are registered with the same name. Constructors should typically be registered in package init functions, and the configure function must define at least Constructor.New. For example, the following configures a constructor with a single parameter, n, which simply returns its value.

                    config.Register("config/test", func(constr *config.Constructor) {
                    	n := constr.Int("n", 32, "the number configured")
                    	constr.New = func() (interface{}, error) {
                    		return *n, nil
                    	}
                    	constr.Doc = "a customizable integer"
                    })
                    

                    func RegisterFlags

                    func RegisterFlags(prefix string, defaultProfilePath string)

                      RegisterFlags registers the default profile on flag.CommandLine with the provided prefix. See Profile.RegisterFlags for details.

                      func Set

                      func Set(path, value string) error

                        Set sets the value of the parameter named by the provided path on the default profile. See Profile.Set for more details.

                        Types

                        type Constructor

                        type Constructor struct {
                        	// New instantiates the value provided by this instance, and
                        	// configured by the flags registered.
                        	//
                        	// TODO(marius): consider making this an interface{} so that we can
                        	// inspect the return type (typechecking can be done in Register)
                        	// and allow us to perform better type checking.
                        	New func() (interface{}, error)
                        
                        	// Doc is a string describing the instance.
                        	Doc string
                        	// contains filtered or unexported fields
                        }

                          Constructor defines a constructor, as configured by Register. Typically a constructor registers a set of parameters through the flags-like methods provided by Constructor. The value returned by New is configured by these parameters.

                          func (*Constructor) Bool

                          func (c *Constructor) Bool(name string, value bool, help string) *bool

                            Bool registers a boolean parameter with a default value. The returned pointer points to its value.

                            func (*Constructor) BoolVar

                            func (c *Constructor) BoolVar(ptr *bool, name string, value bool, help string)

                              BoolVar registers a boolean parameter with a default value. The parameter's value written to the location pointed to by ptr.

                              func (*Constructor) Float

                              func (c *Constructor) Float(name string, value float64, help string) *float64

                                Float registers floating point parameter with a default value. The returned pointer points to its value.

                                func (*Constructor) FloatVar

                                func (c *Constructor) FloatVar(ptr *float64, name string, value float64, help string)

                                  FloatVar register a floating point parameter with a default value. The parameter's value is written to the provided pointer.

                                  func (*Constructor) InstanceVar

                                  func (c *Constructor) InstanceVar(ptr interface{}, name string, value string, help string)

                                    InstanceVar registers a parameter that is satisfied by another instance; the method panics if ptr is not a pointer. The default value is always an indirection; if it is left empty it is taken as the nil value: it remains uninitialized by default.

                                    func (*Constructor) Int

                                    func (c *Constructor) Int(name string, value int, help string) *int

                                      Int registers an integer parameter with a default value. The returned pointer points to its value.

                                      func (*Constructor) IntVar

                                      func (c *Constructor) IntVar(ptr *int, name string, value int, help string)

                                        IntVar registers an integer parameter with a default value. The parameter's value written to the location pointed to by ptr.

                                        func (*Constructor) String

                                        func (c *Constructor) String(name string, value string, help string) *string

                                          String registers a string parameter with a default value. The returned pointer points to its value.

                                          func (*Constructor) StringVar

                                          func (c *Constructor) StringVar(ptr *string, name string, value string, help string)

                                            StringVar registers a string parameter with a default value. The parameter's value written to the location pointed to by ptr.

                                            type Profile

                                            type Profile struct {
                                            	// contains filtered or unexported fields
                                            }

                                              Profile stores a set of parameters and configures instances based on these. It is the central data structure of this package as detailed in the package docs. Each Profile instance maintains its own set of instances. Most users should use the package-level methods that operate on the default profile.

                                              func Application

                                              func Application() *Profile

                                                Application returns the default application profile. The default instance is initialized during the first call to Application (and thus of the package-level methods that operate on the default profile). Because of this, Application (and the other package-level methods operating on the default profile) should not be called during package initialization as doing so means that some global objects may not yet have been registered.

                                                func New

                                                func New() *Profile

                                                  New creates and returns a new profile, installing all currently registered global objects. Global objects registered after a call to New are not reflected in the returned profile.

                                                  func (*Profile) Get

                                                  func (p *Profile) Get(path string) (value string, ok bool)

                                                    Get returns the value of the configured parameter at the provided dot-separated path.

                                                    func (*Profile) Instance

                                                    func (p *Profile) Instance(name string, ptr interface{}) error

                                                      Instance retrieves the named instance from this profile into the pointer ptr. All of its parameters are fully resolved and the underlying global object is instantiated according to the desired parameterization. Instance panics if ptr is not a pointer type. If the type of the instance cannot be assigned to the value pointed to by ptr, an error is returned. Since such errors may occur transitively (e.g., the type of an instance required by another instance may be wrong), the source location of the type mismatch is included in the error to help with debugging. Instances are cached and are only initialized the first time they are requested.

                                                      If ptr is nil, the instance is created without populating the pointer.

                                                      func (*Profile) Merge

                                                      func (p *Profile) Merge(q *Profile)

                                                        Merge merges the instance parameters in profile q into p, so that parameters defined in q override those in p.

                                                        func (*Profile) NeedProcessFlags

                                                        func (p *Profile) NeedProcessFlags() bool

                                                          NeedProcessFlags returns true when a call to p.ProcessFlags should not be delayed -- i.e., the flag values have user-visible side effects.

                                                          func (*Profile) Parse

                                                          func (p *Profile) Parse(r io.Reader) error

                                                            Parse parses a profile from the provided reader into p. On success, the instances defined by the profile in src are merged into profile p. If the reader implements

                                                            Name() string
                                                            

                                                            then the result of calling Name is used as a filename to provide positional information in errors.

                                                            func (*Profile) PrintTo

                                                            func (p *Profile) PrintTo(w io.Writer) error

                                                            func (*Profile) ProcessFlags

                                                            func (p *Profile) ProcessFlags() error

                                                              ProcessFlags processes the flags as registered by RegisterFlags, and is documented by that method.

                                                              func (*Profile) RegisterFlags

                                                              func (p *Profile) RegisterFlags(fs *flag.FlagSet, prefix string, defaultProfilePath string)

                                                                RegisterFlags registers a set of flags on the provided FlagSet. These flags configure the profile when ProcessFlags is called (after flag parsing). The flags are:

                                                                -profile path
                                                                	Parses and loads the profile at the given path. This flag may be
                                                                	repeated, loading each profile in turn. If no -profile flags are
                                                                	specified, then the provided default path is loaded instead. If
                                                                	the default path does not exist, it is skipped; other profile loading
                                                                	errors cause ProcessFlags to return an error.
                                                                
                                                                -set key=value
                                                                	Sets the value of the named parameter. See Profile.Set for
                                                                	details. This flag may be repeated.
                                                                
                                                                -profiledump
                                                                	Writes the profile (after processing the above flags) to standard
                                                                	error and exits.
                                                                

                                                                The flag names are prefixed with the provided prefix.

                                                                func (*Profile) Set

                                                                func (p *Profile) Set(path string, value string) error

                                                                  Set sets the value of the parameter at the provided path to the provided value, which is intepreted according to the type of the parameter at that path. Set returns an error if the parameter does not exist or if the value cannot be parsed into the expected type. The path is a set of identifiers separated by dots ("."). Paths may traverse multiple indirections.

                                                                  Directories

                                                                  Path Synopsis
                                                                  Package http defines profile providers for local HTTP servers.
                                                                  Package http defines profile providers for local HTTP servers.