skipper

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 14, 2024 License: MIT Imports: 20 Imported by: 4

README

Go Reference


Logo

Skipper

Inventory based templated configuration library inspired by the kapitan project


Although Skipper is already used in production, the code on main should be considered unstable. Skipper is currently undergoing a heavy rework as the current implementation only served as POC.

The current development branch is feat/data-abstraction-layer.
The corresponding PR is Data Abstraction Layer

What is skipper?

Skipper is a library which helps you to manage complex configuration and enables you to use your large data-set inside templates. Having one - central - set of agnostic configuration files will make managing your clusters, infrastrucutre stages, etc. much easier. You can rely on the inventory of data, modify it to target-specific needs, use the data in templates, and be sure that whatever you're generating is always in sync with your inventoyu. Whether you generate only a single file, or manage multi-stage multi-region infrastructure deployments doesn't matter. Skipper is a library which enables you to easily build your own - company or project specific - configuration management.

Skipper is heavily inspired by the Kapitan project. The difference is that skipper is a library which does not make any assumptions of your needs (aka. not opinionated). This allows you for example to have way more control over how you want to process your inventory.

Skipper is not meant to be a one-size-fits-all solution. The goal of Skipper is to enable you to create the own - custom built - template and inventory engine, without having to do the heavy lifing.

Core Concepts

Skipper has a few concepts, but not all of them are necessary to understand how Skipper works. More in-depth informatation about Skippers concepts can be found in our docs.

Inventory

The inventory is the heart of every Skipper-enabled project. It is your data storage, the single source of truth. It is a user-defined collection of YAML files (classes and targets).

Classes

Classes are YAML files in which you can define information about every part of your project. These classes become your building blocks and therefore the heart of your project.

Targets

A target represents an instance of your project. Targets are defined with YAML files as well. They use skipper-keywords to includ classes, relevant for that instance. Inside a target config you are also able to overwrite any kind of information (change the location in which your resources are deployed for example).

Templates

Templates (Skipper is using go templates) have access to your target and classes. You can build generic templates and aggregate your data into it, without having to re-write files for different stages. Having a documentation, specific to an instance (stage) of your project, can be quite useful and is easy to implement with Skipper.

Idea collection

  • Allow static file copying instead of rendering it as template (e.g. copy a zip file from templates to compiled)
  • Add timing stats (benchmark, 'compiled in xxx') to compare with kapitan
  • Class inheritance. Currently only targets can use classes but it would be nice if classes could also use different classes
    • This would introduce a higher level of inheritance which users can set-up for their inventory.

Documentation

This documentation is

Classes

A class is a yaml file which defines arbitrary information about your project.

There is only two rules for classes:

  • The filename of the class must be the root key of the yaml struct
  • The filename cannot be target.yaml, resulting in a root key of target.
    • Although this will not return an error, you simply will not be able to use the class as the actual target will overwrite it completely.

This means that if your class is called pizza.yaml, the class must look like this:

pizza:
  # any value

Targets

A target usually is a speparate environment in your infrastructure or a single namespace in your Kubernetes cluster. Targets use classes to pull in the required innventory data in order to produce the correct tree which is required in order to render the templates.

On any given run, Skipper only allows to set one target. This is to ensure that the generated map of data is consistent.

The way a target makes uses of the inventory is by using the use clause which tells Skipper which classes to include in the assembly of the target inventory.

Naming

The name of the target is given by its filename. So if your target is called development.yaml, the target name will be development.

The structure of a target file is pretty simple, there are only two rules:

  • The top-level key of the target must be target
  • There must be a key target.use which has to be an array and tells Skipper which classes this particular target requires.

Below you'll find the most basic example of a target. The target does not define values itself, it just uses values from a class project.common. The class must be located in the classPath passed into Inventory.Load(), where path separators are replaced by a dot.

So if your classPath is ./inventory/classes, referencing foo.bar will make Skipper attempt to load ./inventory/classes/foo/bar.yaml.

target:
  use:
    project.common

Variables

Variables in Skipper always have the same format: ${variable_name}

Skipper has three different types of variables.

  1. Dynamic Variables
  2. Predefined Variables
  3. User-defined Variables
Dynamic Variables

Dynamic variables are variables which use a selector path to point to existing values which are defined in your inventory.

Consider the following class images.yaml

images:
  base_image: some/image

  production:
    image: ${images:base_image}:v1.0.0
  staging:
    image: ${images:base_image}:v2.0.0-rc1
  development:
    image: ${images:base_image}:latest

Once the class is processed, the class looks like this:

images:
  base_image: some/image

  production:
    image: some/image:v1.0.0
  staging:
    image: some/image:v2.0.0-rc1
  development:
    image: some/image:latest

The name of the variable uses common dot-notation, except that we're using ':' instead of dots. We chose to use colons because they are easier to read inside the curly braces.

Predefined Variables

Predefined variables could also be considered constants - at least from a user perspective. The predefined variables can easily be defined as map[string]interface{}, where the keys are the variable names.

You have to pass your predefined variables to the Inventory.Data() call, then they are evaluated. If you do not pass these variables, the function will return an error as it will attempt to treat them as dynamic variables.

Consider the following example (your main.go)

// ...

predefinedVariables := map[string]interface{}{
    "target_name": "develop",
    "output_path": "foo/bar/baz",
    "company_name": "AcmeCorp"
}

data, err := inventory.Data(target, predefinedVariables)
if err != nil {
    panic(err)
}

// ...

You will now be able to use the variables ${target_name}, ${output_path} and ${company_name} throughout your yaml files and have Skipper replace them dynamically once you call Data() on the inventory.

User-defined Variables

TODO

Acknowledgments

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrEmptyFunctionName error = fmt.Errorf("empty function name")
)
View Source 
var (
	ErrFilePathEmpty = fmt.Errorf("file path is empty")
)

Functions

func CopyFile 

func CopyFile(fs afero.Fs, sourcePath, targetPath string) error

CopyFile will copy the given sourcePath to the targetPath inside the passed afero.Fs.

func CopyFileFsToFs 

func CopyFileFsToFs(sourceFs afero.Fs, targetFs afero.Fs, sourcePath, targetPath string) error

CopyFileFsToFs will copy a file from the given sourceFs and sourcePath to the targetFs and targetPath

func CopyFilesByConfig 

func CopyFilesByConfig(fs afero.Fs, configs []CopyConfig, sourceBasePath, targetBasePath string) error

CopyFilesByConfig uses a list of CopyConfigs and calls the CopyFile func on them.

func DefaultTemplateContext 

func DefaultTemplateContext(data Data, targetName string) any

DefaultTemplateContext returns the default template context with an 'Inventory' field where the Data is located. Additionally it adds the 'TargetName' field for convenience.

func ReplaceSecret 

func ReplaceSecret(data Data, secret *Secret) error

ReplaceSecret will replace the given secret inside Data with the actual secret value.

func ReplaceVariables 

func ReplaceVariables(data Data, classFiles []*Class, predefinedVariables map[string]interface{}) (err error)

ReplaceVariables searches and replaces variables defined in data. The classFiles are used for local referencing variables (class internal references). predefinedVariables can be used to provide global user-defined variables.

func WriteFile 

func WriteFile(fs afero.Fs, targetPath string, data []byte, mode fs.FileMode) error

WriteFile ensures that `targetPath` exists in the `fs` and then writes `data` into it.

func YamlFileLoader 

func YamlFileLoader(fileSystem afero.Fs, basePath string, loader YamlFileLoaderFunc) error

YamlFileLoader is used to load Skipper specific yaml files from the inventory. It searches the basePath inside the given fileSystem for yaml files and loads them. Empty files are skipped. A path relative to the given pasePath is constructed. The loaded yaml file and the relative path are then passed to the given YamlFileLoaderFunc which is responsible for creating specific types from the YamlFile.

Types

type Call 

type Call struct {
	// Identifier points to wherever the call is used in the [Data] map
	Identifier   []interface{}
	FunctionName string
	Param        string
	// contains filtered or unexported fields
}

func FindCalls 

func FindCalls(data Data) ([]*Call, error)

func NewCall 

func NewCall(functionName string, param string, path []interface{}) (*Call, error)

func NewRawCall 

func NewRawCall(callString string) (*Call, bool, error)

func (*Call) Execute 

func (c *Call) Execute() string

func (Call) FullName 

func (c Call) FullName() string

func (Call) Path 

func (c Call) Path() string

func (*Call) RawString 

func (c *Call) RawString() string

type CallFunc 

type CallFunc func(param string) string

type Class 

type Class struct {
	// File is the underlying file in the filesystem.
	File *YamlFile
	// Name is the relative path of the file inside the inventory which uniquely identifies this class.
	// Because the name is path based, no two classes with the same name can exist.
	// For the name, the path-separator is replaced with '.' and the file extension is stripped.
	// Example: 'something/foo/bar.yaml' will have the name 'something.foo.bar'
	//
	// The Name is also what is used to reference classes throughout Skpper.
	Name string
	// Configuration holds Skipper-relevant configuration inside the class
	Configuration *SkipperConfig
}

Class represents a single file containing a YAML struct which makes up the inventory.

func NewClass 

func NewClass(file *YamlFile, relativeClassPath string) (*Class, error)

NewClass will create a new class, given a raw YamlFile and the relative filePath from inside the inventory. If your class file is at `foo/bar/inventory/classes/myClass.yaml`, the relativeClassPath will be `myClass.yaml`

func (*Class) Data 

func (c *Class) Data() *Data

Data returns the underlying class file-data map as Data

func (*Class) NameAsIdentifier 

func (c *Class) NameAsIdentifier() (id []interface{})

NameAsIdentifier returns the class name as an identifier used by skipper. The name is a dot-separated list of values (e.g. 'foo.bar.baz'). The returned identifier is a []interface which the values and can be used to address the class in Data.

func (*Class) RootKey 

func (c *Class) RootKey() string

RootKey returns the root key name of the class.

type ComponentConfig 

type ComponentConfig struct {
	OutputPath string         `yaml:"output_path"`
	InputPaths []string       `yaml:"input_paths"`
	Renames    []RenameConfig `yaml:"rename,omitempty"`
}

type CopyConfig 

type CopyConfig struct {
	// SourcePath is the source file to copy, relative to the template-root
	SourcePath string `yaml:"source"`
	// TargetPath is the target to copy the source file to, relative to the compile-root
	TargetPath string `yaml:"target"`
}

type Data 

type Data map[string]interface{}

Data is an arbitrary map of values which makes up the inventory.

func NewData 

func NewData(input interface{}) (Data, error)

NewData attempts to convert any given interface{} into Data. This is done by first using `yaml.Marshal` and then `yaml.Unmarshal`. If the given interface is compatible with Data, these steps will succeed.

func (Data) Bytes 

func (d Data) Bytes() []byte

Bytes returns a `[]byte` representation of Data

func (Data) FindValues 

func (d Data) FindValues(valueFunc FindValueFunc, target *[]interface{}) (err error)

FindValues can be used to find specific 'leaf' nodes, aka values. The Data is iterated recursively and once a plain value is found, the given FindValueFunc is called. It's the responsibility of the FindValueFunc to determine if the value is what is searched for. The FindValueFunc can return any data, which is aggregated and written into the passed `*[]interface{}`. The callee is then responsible of handling the returned value and ensuring the correct types were returned.

func (Data) Get 

func (d Data) Get(key string) Data

Get returns the value at `Data[key]` as Data. Note that his function does not support paths like `HasKey("foo.bar.baz")`. For that you can use [GetPath]

func (Data) GetPath 

func (d Data) GetPath(path ...interface{}) (tree interface{}, err error)

GetPath allows path based indexing into Data. A path is a slice of interfaces which are used as keys in order. Supports array indexing (arrays start at 0) Examples of valid paths:

  • ["foo", "bar"]
  • ["foo", "bar", 0]

func (Data) HasKey 

func (d Data) HasKey(key string) bool

HasKey returns true if Data[key] exists. Note that his function does not support paths like `HasKey("foo.bar.baz")`. For that you can use [GetPath]

func (Data) MergeReplace 

func (d Data) MergeReplace(data Data) Data

MergeReplace merges the existing Data with the given. If a key already exists, the passed data has precedence and it's value will be used.

func (*Data) SetPath 

func (d *Data) SetPath(value interface{}, path ...interface{}) (err error)

SetPath uses the same path slices as [GetPath], only that it can set the value at the given path. Supports array indexing (arrays start at 0)

func (Data) String 

func (d Data) String() string

String returns the string result of `yaml.Marshal`. Can be useful for debugging or just dumping the inventory.

type File 

type File struct {
	Path  string
	Mode  fs.FileMode
	Bytes []byte
}

File is just an arbitrary description of a path and the data of the File to which Path points to. Note that the used filesystem is not relevant, only at the time of loading a File.

func NewFile 

func NewFile(path string) (*File, error)

func (*File) Exists 

func (f *File) Exists(fs afero.Fs) bool

Exists returns true if the File exists in the given filesystem, false otherwise.

func (*File) Load 

func (f *File) Load(fs afero.Fs) (err error)

Load will attempt to read the File from the given filesystem implementation. The loaded data is stored in `File.Bytes`

type FindValueFunc 

type FindValueFunc func(value string, path []interface{}) (interface{}, error)

FindValueFunc is a callback used to find values inside a Data map. `value` is the actual found value; `path` are the path segments which point to that value The function returns the extracted value and an error (if any).

type Inventory 

type Inventory struct {
	// contains filtered or unexported fields
}

Inventory is the collection of classes and targets. The inventory wraps everything together and is capable of producing a single, coherent Data which can then be used inside the templates.

func NewInventory 

func NewInventory(fs afero.Fs, classPath, targetPath, secretPath string) (*Inventory, error)

NewInventory creates a new Inventory with the given afero.Fs. At least one extension must be provided, otherwise an error is returned.

func (*Inventory) AddExternalClass 

func (inv *Inventory) AddExternalClass(data map[string]any, classFilePath string) error

AddExternalClass can be used to dynamically create class files. The given data will be written into `classFilePath`, overwriting any existing file.

The class path is first normalized to match the existing `Inventory.classPath`.

After that, the root-key of the data is adjusted to match the fileName which is extracted from `classFilePath`. This has to be done in order to comply Skipper rules where the class-filename must also be the root-key of any given class.

A new file inside the Skipper class path is created which makes it available for loading. In order to prevent confusion, a file header is added to indicate that the class was generated.

func (*Inventory) Data 

func (inv *Inventory) Data(targetName string, predefinedVariables map[string]interface{}, skipSecretHandling, revealSecrets bool) (data Data, err error)

Data loads the required inventory data map given the target. This is where variables and secrets are handled and eventually replaced. The resulting Data is what can be passed to the templates.

func (*Inventory) GetAllClasses 

func (inv *Inventory) GetAllClasses() []*Class

GetAllClasses returns all discovered Classes

func (*Inventory) GetAllTargets 

func (inv *Inventory) GetAllTargets() []*Target

GetAllTargets returns all discovered Targets

func (*Inventory) GetClass 

func (inv *Inventory) GetClass(name string) *Class

GetClass attempts to return a Class, given a name. If the class does not exist, nil is returned.

func (*Inventory) GetSkipperConfig 

func (inv *Inventory) GetSkipperConfig(targetName string) (config *SkipperConfig, err error)

GetSkipperConfig merges SkipperConfig of the target and it's used classes into one effective configuration.

func (*Inventory) GetTarget 

func (inv *Inventory) GetTarget(name string) *Target

GetTarget attempts to return a target struct given a target name. If the target could not be found, nil is returned.

func (*Inventory) GetUsedClasses 

func (inv *Inventory) GetUsedClasses(targetName string) ([]*Class, error)

GetUsedClasses returns the loaded classes which are used by the given target.

type RenameConfig 

type RenameConfig struct {
	InputPath string `yaml:"input_path"`
	Filename  string `yaml:"filename"`
}

type Secret 

type Secret struct {
	*SecretFile
	Driver          secret.Driver
	DriverName      string
	AlternativeCall *Call
	Identifier      []interface{}
}

func FindOrCreateSecrets 

func FindOrCreateSecrets(data Data, secretFiles SecretFileList, secretPath string, fs afero.Fs) ([]*Secret, error)

FindSecrets will leverage the `FindValues` function of Data to recursively search for secrets. All returned values are converted to *Secret and then returned as []*Secret.

func NewSecret 

func NewSecret(secretFile *SecretFile, driver string, alternative *Call, path []interface{}) (*Secret, error)

func (Secret) FullName 

func (s Secret) FullName() string

FullName returns the full secret name as it would be expected to ocurr in a class/target.

func (*Secret) Load 

func (s *Secret) Load(fs afero.Fs) error

Load is used to load the actual secret files and ensure that they are correctly formatted. Load does NOT load the actual value, it just ensures that it could be loaded using the secret.Value() call.

func (Secret) Path 

func (s Secret) Path() string

func (*Secret) Value 

func (s *Secret) Value() (string, error)

Value returns the actual secret value.

type SecretFile 

type SecretFile struct {
	*YamlFile
	Data         SecretFileData
	RelativePath string
}

func NewSecretFile 

func NewSecretFile(file *YamlFile, relativeSecretPath string) (*SecretFile, error)

func (*SecretFile) LoadSecretFileData 

func (sf *SecretFile) LoadSecretFileData(fs afero.Fs) error

type SecretFileData 

type SecretFileData struct {
	Data string `yaml:"data"`
	Type string `yaml:"type"`
	Key  string `yaml:"key"`
}

SecretFileData describes the generic structure of secret files.

func NewSecretData 

func NewSecretData(data string, driver string, key string) (*SecretFileData, error)

NewSecretData constructs a Data map as it is required for secrets.

type SecretFileList 

type SecretFileList []*SecretFile

func (SecretFileList) GetSecretFile 

func (sfl SecretFileList) GetSecretFile(path string) *SecretFile

type SkipperConfig 

type SkipperConfig struct {
	Classes     []string          `yaml:"use,omitempty"`
	Components  []ComponentConfig `mapstructure:"components,omitempty"`
	Copies      []CopyConfig      `yaml:"copy,omitempty"`
	IgnoreRegex []string          `yaml:"ignore_regex,omitempty"`
	Renames     []RenameConfig    `yaml:"rename,omitempty"`
}

func LoadSkipperConfig 

func LoadSkipperConfig(file *YamlFile, rootKey string) (*SkipperConfig, error)

LoadSkipperConfig attempts to load a SkipperConfig from the given YamlFile with the passed rootKey

func MergeSkipperConfig 

func MergeSkipperConfig(merge ...*SkipperConfig) (mergedConfig *SkipperConfig)

MergeSkipperConfig merges a list of configs into one

func (*SkipperConfig) IsSet 

func (config *SkipperConfig) IsSet() bool

IsSet returns true if the config is not nil. The function is useful because LoadSkipperConfig can return nil.

type Target 

type Target struct {
	File *YamlFile
	// Name is the relative path of the file inside the inventory
	// where '/' is replaced with '.' and without file extension.
	Name string
	// UsedWildcardClasses holds all resolved wildcard class imports as specified in the `targets.skipper.use` key.
	UsedWildcardClasses []string
	// Configuration is the skipper-internal configuration which needs to be present on every target.
	Configuration TargetConfig
	// SkipperConfig is the generic Skipper configuration which can be use throughout targets and classes
	SkipperConfig *SkipperConfig
}

Target defines which classes to use for the compilation.

func NewTarget 

func NewTarget(file *YamlFile, inventoryPath string) (*Target, error)

func (*Target) Data 

func (t *Target) Data() Data

func (*Target) ReloadConfiguration 

func (t *Target) ReloadConfiguration()

type TargetConfig 

type TargetConfig struct {
	Secrets TargetSecretConfig `mapstructure:"secrets,omitempty"`
}

type TargetSecretConfig 

type TargetSecretConfig struct {
	Drivers map[string]interface{} `mapstructure:"drivers"`
}

type Templater 

type Templater struct {
	Files []*File

	IgnoreRegex []*regexp.Regexp
	// contains filtered or unexported fields
}

func NewTemplater 

func NewTemplater(fileSystem afero.Fs, templateRootPath, outputRootPath string, userFuncMap map[string]any, ignoreRegex []string) (*Templater, error)

func (*Templater) DiscoverPartials added in v0.2.0 

func (t *Templater) DiscoverPartials()

DiscoverPartials will iterate over all registered files and check each of them whether an additional template is defined (e.g. using the 'define' directive). If so, the file is added to the list of partial templates which is made available during template execution. This ensures that every template can access partial templates. Subsequent calls to this method will reset the 'partialTemplates' field.

func (*Templater) Execute 

func (t *Templater) Execute(template *File, data any, allowNoValue bool, renameConfig []RenameConfig) error

Execute is responsible of parsing and executing the given template, using the passed data context. If execution is successful, the template is written to it's desired target location. If allowNoValue is true, the template is rendered even if it contains variables which are not defined.

func (*Templater) ExecuteAll 

func (t *Templater) ExecuteAll(data any, allowNoValue bool, renameConfig []RenameConfig) error

ExecuteAll is just a convenience function to execute all templates in `Templater.Files`

func (*Templater) ExecuteComponents 

func (t *Templater) ExecuteComponents(data any, components []ComponentConfig, allowNoValue bool) error

ExecuteComponents will only execute the templates as they are defined in the given components.

type Variable 

type Variable struct {
	// Name of the variable is whatever string is between ${}.
	// + For dynamic variables, this can be a ':' separated string which points somewhere into the Data map.
	// 	 The reason we use ':' is to improve readability between curly braces.
	// + For predefined variables, this can be any string and must not be a path into the Data map.
	Name string
	// Identifier is the list of keys which point to the variable itself within the data set in which it is used.
	Identifier []interface{}
}

Variable is a keyword which self-references the Data map it is defined in. A Variable has the form ${key:key}.

func FindVariables 

func FindVariables(data Data) ([]Variable, error)

FindVariables leverages the [FindValues] function of the given Data to extract all variables by using the [variableFindValueFunc] as callback.

func (Variable) FullName 

func (v Variable) FullName() string

func (Variable) NameAsIdentifier 

func (v Variable) NameAsIdentifier() (id []interface{})

func (Variable) Path 

func (v Variable) Path() string

type YamlFile 

type YamlFile struct {
	File
	Data Data
}

YamlFile is what is used for all inventory-relevant files (classes, secrets and targets).

func CreateNewYamlFile 

func CreateNewYamlFile(fs afero.Fs, path string, data []byte) (*YamlFile, error)

CreateNewFile can be used to manually create a File inside the given filesystem. This is useful for dynamically creating classes or targets.

The given path is attempted to be created and a file written.

func DiscoverYamlFiles 

func DiscoverYamlFiles(fileSystem afero.Fs, rootPath string) ([]*YamlFile, error)

DiscoverYamlFiles iterates over a given rootPath recursively, filters out all files with the appropriate file fileExtensions and finally creates a YamlFile slice which is then returned.

func NewYamlFile 

func NewYamlFile(path string) (*YamlFile, error)

NewYamlFile returns a newly initialized `YamlFile`.

func (*YamlFile) Load 

func (f *YamlFile) Load(fs afero.Fs) error

Load will first load the underlying raw file-data and then attempt to `yaml.Unmarshal` it into `Data` The resulting Data is stored in `YamlFile.Data`.

func (*YamlFile) UnmarshalPath 

func (f *YamlFile) UnmarshalPath(target interface{}, path ...interface{}) error

UnmarshalPath can be used to unmarshall only a sub-map of the Data inside YamlFile. The function errors if the file has not been loaded.

type YamlFileLoaderFunc 

type YamlFileLoaderFunc func(file *YamlFile, relativePath string) error

YamlFileLoaderFunc is a function used to create specific types from a YamlFile and a relative path to that file.

Source Files

View all Source files

Directories

Path Synopsis
examples
external_classes Module
keyvault Module
secrets Module
terraform_azure Module
driver

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.