templig

README

templig

templig (pronounced [ˈtɛmplɪç]) is a non-intrusive configuration library that utilizes the text-templating engine of Go and the functions best known from Helm charts, originating from Masterminds/sprig.

Its primary goal is to enable dynamic configuration files, that have access to the system environment to fill information using functions like env and read. To facilitate different environments, overlays can be defined that amend a base configuration with environment-specific attributes and changes. Configurations that implement the Validator interface also have automated checking enabled upon loading.

This is not the first configuration library and surely will not be the last. There exist alternatives, the most elaborate of them may be viper. The difference to basically all of these is that they burden the developer to provide all the means to gather the configuration information. So if the developer does not foresee a means to read a value from the environment, a user cannot use this. templig turns that around and gives the developer a simple interface to do what he wants—read a config—and gives the user the means to compile his configuration in whatever way he sees fit. Experience shows that the target system environments can be extremely diverse, and limiting the possibilities of end users directly limits the spectrum of application.

Installation

To install templig, you can use the following command:

$ go get github.com/AlphaOne1/templig

Getting Started

Simple Case

Having a configuration file like the following:

id:   23
name: Interesting Name

The code to read that file would look like this:

package main

import (
	"fmt"

	"github.com/AlphaOne1/templig"
)

// Config is the configuration structure
type Config struct {
	ID   int    `yaml:"id"`
	Name string `yaml:"name"`
}

// main will read and display the configuration
func main() {
	c, confErr := templig.FromFile[Config]("my_config.yaml")

	fmt.Printf("read errors: %v", confErr)

	if confErr == nil {
		fmt.Printf("ID:   %v\n", c.Get().ID)
		fmt.Printf("Name: %v\n", c.Get().Name)
	}
}

The Get method gives a pointer to the internally held Config structure that the user supplied. The pointer is always non-nil, so additional nil-checks are not necessary. Running that program would give:

read errors: <nil>
ID:   23
Name: Interesting Name

Reading with Overlays

Having a base configuration file my_config.yaml like the following:

id:   23
name: Interesting DevName

and a file that contains specific configuration for e.g. the production environment my_prod_overlay.yaml:

name: Important ProdName

The code to read those files would look like this:

package main

import (
	"fmt"

	"github.com/AlphaOne1/templig"
)

// Config is the configuration structure
type Config struct {
	ID   int    `yaml:"id"`
	Name string `yaml:"name"`
}

// main will read and display the configuration
func main() {
	c, confErr := templig.FromFile[Config](
		"my_config.yaml",
		"my_prod_overlay.yaml",
	)

	fmt.Printf("read errors: %v", confErr)

	if confErr == nil {
		fmt.Printf("ID:   %v\n", c.Get().ID)
		fmt.Printf("Name: %v\n", c.Get().Name)
	}
}

That way, the different configuration files are read in order, with the first one as the base. Every additional file gives changes to all the ones read before. In this example, changing the name. Running this program would give:

read errors: <nil>
ID:   23
Name: Important ProdName

As expected, the value of Name was replaced by the one provided in overlay configuration.

Template Functionality

Overview

templig supports templating the configuration files. In addition to the basic templating functions provided by the Go text/template library, templig includes the functions from sprig, which are perhaps best known for their use in Helm charts. On top of that, the following functions are provided for convenience:

Function	Description	Example
arg	reads the value of the command line argument with the given name	Link
hasArg	true if an argument with the given name is present, false otherwise	Link
required	checks that its second argument is not zero length or nil	Link
read	reads the content of a file	Link

The expansion of the templated parts is done before overlaying takes place. Any errors of templating will thus be displayed in their respective source locations.

Reading Environment

Having a templated configuration file like this one:

id:   23
name: Interesting Name
pass: {{ env "PASSWORD" | required "password required" | quote }}

or this one:

id:   23
name: Interesting Name
pass: {{ read "pass.txt" | required "password required" | quote }}

One can see the templating code between the double curly braces {{ and }}. The following program is essentially the same as in the Simple Case. It just adds the pass field to the configuration:

package main

import (
	"fmt"
	"strings"

	"github.com/AlphaOne1/templig"
)

// Config is the configuration structure
type Config struct {
	ID   int    `yaml:"id"`
	Name string `yaml:"name"`
	Pass string `yaml:"pass"`
}

// main will read and display the configuration
func main() {
	c, confErr := templig.FromFile[Config]("my_config.yaml")

	fmt.Printf("read errors: %v", confErr)

	if confErr == nil {
		fmt.Printf("ID:   %v\n", c.Get().ID)
		fmt.Printf("Name: %v\n", c.Get().Name)
		fmt.Printf("Pass: %v\n", strings.Repeat("*", len(c.Get().Pass)))
	}
}

Validation

The templating facilities allow also for a wide range of tests, but depend on the configuration file read. As it is most likely user supplied, possible consistency checks are not reliable in the form of template code. For this purpose, templig also allows for the configuration structure to implement the Validator interface. Implementing types provide a function Validate that allows templig to check after the configuration was read, if its structure should be considered valid and report errors accordingly.

package main

import (
    "errors"
    "fmt"

	"github.com/AlphaOne1/templig"
)

// Config is the configuration structure
type Config struct {
    ID   int    `yaml:"id"`
    Name string `yaml:"name"`
}

// Validate fulfills the Validator interface provided by templig.
// This method is called, if it is defined. It influences the outcome of the configuration reading.
func (c *Config) Validate() error {
    var result []error

	if len(c.Name) == 0 {
		result = append(result, errors.New("name is required"))
	}

	if c.ID < 0 {
		result = append(result, errors.New("id greater than zero is required"))
	}

	return errors.Join(result...)
}

// main will read and display the configuration
func main() {
	c, confErr := templig.FromFile[Config]("my_config_good.yaml")

	if confErr == nil {
		fmt.Printf("ID:   %v\n", c.Get().ID)
		fmt.Printf("Name: %v\n", c.Get().Name)
	}
}

Validation functionality can be as simple as in this example. But as the complexity of the configuration grows, automated tools to generate the configuration structure and basic consistency checks could be employed. These use e.g. JSON Schema or its embedded form in OpenAPI 2 or 3.

A non-exhaustive list of these:

• https://github.com/xeipuuv/gojsonschema (JSON Schema)
• https://github.com/atombender/go-jsonschema (JSON Schema)
• https://github.com/ogen-go/ogen (OpenAPI 3.x)
• https://github.com/go-swagger/go-swagger (OpenAPI 2.0 / Swagger 2.0)

An example combining generation and templating can be found here.

Output & Secret Hiding

On program start, it is advisable to output the basic parameters controlling the following execution. However, many configurations contain secrets, credentials for databases, access tokens etc. These should normally not be printed in plain text to any location.

templig offers several possibilities to write the final configuration to a Writer:

1. To writes the configuration completely, that is including secrets, to the given Writer.

   c, _ := FromFile[Config]("my_config.yaml")
   c.To(os.Stdout)
   

   This program will produce the following, structurally identical output to the input configuration:

   id:   23
   name: Interesting Name
   passes:
     - secretPass0
     - alternativePass1
   

2. ToSecretsHidden writes the configuration, hiding secrets recognized using the SecretRE regular expression. The example of 1. will then become:

   c, _ := FromFile[Config]("my_config.yaml")
   c.ToSecretsHidden(os.Stdout)
   

   With the new output to be:

   id:   23
   name: Interesting Name
   pass: '*'
   

3. ToSecretsHiddenStructured writes the configuration, hiding secrets, but letting their structure recognizable. The example of 1. will then become:

   c, _ := FromFile[Config]("my_config.yaml")
   c.ToSecretsHiddenStructured(os.Stdout)
   

   With the new output to be:

   id:   23
   name: Interesting Name
   pass:
     - '***********'
     - '****************'
   

Single passwords are always replaced by a string of * of equal length. An example usage can be found here.

Documentation

Index

• Constants
• Variables
• func HideSecrets(n *yaml.Node, hideStructure bool)
• func MergeYAMLNodes(a, b *yaml.Node) (*yaml.Node, error)
• type Config
  • func From[T any](r ...io.Reader) (*Config[T], error)
  • func FromFile[T any](paths ...string) (*Config[T], error)
  • func FromFiles[T any](paths []string) (*Config[T], error)
  • func (c *Config[T]) Get() *T
  • func (c *Config[T]) To(w io.Writer) error
  • func (c *Config[T]) ToFile(path string) error
  • func (c *Config[T]) ToSecretsHidden(w io.Writer) error
  • func (c *Config[T]) ToSecretsHiddenStructured(w io.Writer) error
  • func (c *Config[T]) Validate() error
• type Validator

Constants

const SecretDefaultRE = "(key)|(secret)|(pass)|(password)|(cert)|(certificate)"

SecretDefaultRE is the default regular expression used to identify secret values automatically. All input to the regular expression is preprocessed using the strings.ToLower function, to not unnecessarily complicate the regexp.

Variables

var SecretRE = regexp.MustCompile(SecretDefaultRE)

SecretRE is the regular expression used to identify secret values automatically. In case there are different properties to identify secrets, extend it.

var TemplateFunctions = template.FuncMap{
	"arg":      argumentValue,
	"hasArg":   argumentPresent,
	"required": required,
	"read":     readFile,
}

TemplateFunctions is a template.FuncMap that allows to globally remove the additional templig template functions or even add own functions on top of what is already provided. For user provided functions, please use the prefix `uP`, that is guaranteed to never be used as a templig provided function.

Functions

func HideSecrets

func HideSecrets(n *yaml.Node, hideStructure bool)

HideSecrets hides secrets in the given YAML node structure. Secrets are identified using the SecretRE. Depending on the parameter `hideStructure` the structure of the secret is hidden too (`true`) or visible (`false`).

func MergeYAMLNodes

func MergeYAMLNodes(a, b *yaml.Node) (*yaml.Node, error)

MergeYAMLNodes merges the content of node `b` into node `a`. If `a` contains already an element with the same name and of the same kind as `b`, they are merged recursively.

Types

type Config

type Config[T any] struct {
	// contains filtered or unexported fields
}

Config is the generic structure holding the configuration information for the specified type.

func From

func From[T any](r ...io.Reader) (*Config[T], error)

From reads a configuration from the given set of io.Reader.

func FromFile

func FromFile[T any](paths ...string) (*Config[T], error)

FromFile loads a series of configuration files. The first file is considered the base, all others are loaded on top of that one using the MergeYAMLNodes functionality.

func FromFiles

func FromFiles[T any](paths []string) (*Config[T], error)

FromFiles loads a series of configuration files. The first file is considered the base, all others are loaded on top of that one using the MergeYAMLNodes functionality.

Deprecated: As of version 0.6.0 this function is deprecated and will be removed in the next major release.

func (*Config[T]) Get

func (c *Config[T]) Get() *T

Get gives a pointer to the deserialized configuration.

func (*Config[T]) To

func (c *Config[T]) To(w io.Writer) error

To writes a configuration to the given io.Writer.

func (*Config[T]) ToFile

func (c *Config[T]) ToFile(path string) error

ToFile saves a configuration to a file with the given name, replacing it in case.

func (*Config[T]) ToSecretsHidden

func (c *Config[T]) ToSecretsHidden(w io.Writer) error

ToSecretsHidden writes the configuration to the given io.Writer and hides secret values using the SecretRE. Strings are replaced with the number of * corresponding to their length. Substructures containing secrets, are replaced with a single '*'. The following example

id: id0
secrets:
  - secret0
  - secret1

thus will be replaced by

id: id0
secrets: *

func (*Config[T]) ToSecretsHiddenStructured

func (c *Config[T]) ToSecretsHiddenStructured(w io.Writer) error

ToSecretsHiddenStructured writes the configuration to the given io.Writer and hides secret values using the SecretRE. Strings are replaced with the number of * corresponding to their length. Substructures containing secrets, are replaced with a corresponding structure of '*'. The following example

id: id0
secrets:
  - secret0
  - secret1

thus will be replaced by

id: id0
secrets:
  - *******
  - *******

func (*Config[T]) Validate

func (c *Config[T]) Validate() error

Validate checks if the configuration is valid, if the content fulfills the Validator interface.

type Validator

type Validator interface {
	// Validate is used to Validate a configuration.
	Validate() error
}

Validator is the interface to facility validity checks on configuration types.