mimic

README

Mimic - Define your Configuration, Infrastructure and Deployments as Go Code

mimic: A Go module for defining, templating and generating configuration in Go:

• Define your Configuration (e.g Envoy, Prometheus, Alertmanager, Nginx, Prometheus Alerts, Rules, Grafana Dashaboards etc.)
• Define your Infrastructure (e.g Terraform, Ansible, Puppet, Chef, Kubernetes etc)
• Define any other file that you can imagine.

...using simple, typed and testable Go code!

mimic: Mimic is a super-human with the ability to copy the powers and abilities of others.

Getting Started

1. Create a new .go file for your config e.g config/example.go
2. Import mimic using Go 1.17+ e.g go get github.com/bwplotka/mimic@latest
3. Instantiate mimic and defer generation in your main function using the mimic module:

  package main

  import ( 
  	"github.com/bwplotka/mimic"
  )

  func main() {
      generator := mimic.New() 
  	
  	// Defer Generate to ensure we generate the output.
      defer generator.Generate()
        
  	//...

1. Add typed configuration in your main to each file using encoding you want using With method: generator.With("config").Add("some.yaml", encoding.GhodssYAML(set))
2. Run go run config/example.go generate
3. You will now see the generated Kubernetes YAML file here: cat gen/config/some.yaml

See full example here:

   package main
   
   import (
   	"github.com/bwplotka/mimic"
   	"github.com/bwplotka/mimic/encoding"
   	"github.com/go-openapi/swag"
   	appsv1 "k8s.io/api/apps/v1"
   	corev1 "k8s.io/api/core/v1"
   	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
   )
   
   func main() {
   	generator := mimic.New()
   
   	// Defer Generate to ensure we generate the output.
   	defer generator.Generate()
   
   	// Hook in your config below.
   	// As an example Kubernetes configuration!
   	const name = "some-statefulset"
   
   	// Create some containers ... (imagine for now).
   	var container1, container2, container3 corev1.Container
   	var volume1 corev1.Volume
   	
   	// Configure a statefulset using native Kubernetes structs.
   	set := appsv1.StatefulSet{
   		TypeMeta: metav1.TypeMeta{
   			Kind:       "StatefulSet",
   			APIVersion: "apps/v1",
   		},
   		ObjectMeta: metav1.ObjectMeta{
   			Name: name,
   			Labels: map[string]string{
   				"app": name,
   			},
   		},
   		Spec: appsv1.StatefulSetSpec{
   			Replicas:    swag.Int32(2),
   			ServiceName: name,
   			Template: corev1.PodTemplateSpec{
   				ObjectMeta: metav1.ObjectMeta{
   					Labels: map[string]string{
   						"app": name,
   					},
   				},
   				Spec: corev1.PodSpec{
   					InitContainers: []corev1.Container{container1},
   					Containers:     []corev1.Container{container2, container3},
   					Volumes:        []corev1.Volume{volume1},
   				},
   			},
   			Selector: &metav1.LabelSelector{
   				MatchLabels: map[string]string{
   					"app": name,
   				},
   			},
   		},
   	}
   	// Now Add some-statefulset.yaml to the config folder.
   	generator.With("config").Add(name+".yaml", encoding.GhodssYAML(set))
   }

Now you are ready to start defining your own resources!

Other examples can be found in here.

What is mimic?

mimic is a package that allows you to define your configuration using Go and generate this into configuration files your application and tooling understands.

Why was mimic created?

mimic has been built from our past experience using this concept to configure our applications and infrastructure.

It offers not only to show the concept and an implementation example but also to share what we have learned, best practice and patterns that we believe are valuable for everyone.

But Why Go?

Why you should define your templates/infra/configs in Go?

• Configuration as code ... like actual code, not json, yaml or tf.

• Go is a strongly typed language. This means that compiler and IDE of your choice will massively help you find what config fields are allowed, what values enum expects and what is the type of each field.

• Unit/Integration test your configuration, infrastructure and deployment.
  For example:

  • Test your PromQL queries in Prometheus alerts works as expected? Just write unit test for those using e.g this
  • Enforce conventions such as service naming conventions via tests.

• Import configuration structs directly from the project you are configurating for example bring in Kubernestes, Prometheus or any other structs that are exported. Allowing you to ensure your config matches the project.

  No more blind searches and surprises. It cannot be safer or simpler than this.

• Versioning and dependency management. Utilize go modules to ensure you are using the correct version of the configuration for the project version you are running.

• Documentation of your config, Go recommends goDoc formatting, so you can leverage native comments for each struct's fields to document behaviour or details related to the config field. Giving you visibility in your config of exactly what your defining. See this great Kubernetes struct as an example.

• Quick feedback loop. Catch most mistakes and incompatibilities in Go compile time, before you even deploy it further. As you probably know one of Go goal is to have very fas compilation time, which feels like you are running a script.

• Keep the set of the languages used in your organization to a minimum - just one: Go, one of the cleanest, simplest and developer friendly languages around.

What mimic is NOT

• It does NOT implement any deployment/distribution logic.
• It is NOT intended to trigger any changes. Instead use the right tool for the job e.g. kubectl apply, ansible, puppet, chef, terraform
• It is NOT (yet) a registry for reusable templates, however we encourage the community to create public repositories for those!

What does mimic include?

• providers go structs representing configuration for applications, infrastructure and container management.
  • Included are a set of go providers that do not have native structs OR are not easily importable (yet).
• encoding a way to transform your go config struts into specific file types.
  • Included are json, yaml and jsonpb.
• abstractions a way to abstract concepts to a higher level if really needed (see best practice).
• Examples:
  • Infra definitions for Prometheus remote read benchmarks on Kubernetes
  • You want to add your own example here? Write to us on Slack or file GH issue!

Want to help us and Contribute?

Please do!

First start defining your configuration, infra and deployment as Go code!

Share with the community:

• Share the Go templates you create.
• Share the Go configuration structs for non-Go projects.
• Share the Go unit/integration/acceptance tests against certain providers's definitions.
• Share best practices and your experience!

Please use GitHub issues and our slack #mimic for feedback and questions.

As always pull requests are VERY welcome as well!

Have a project written in Go?

If you maintain your own project using Go it would be great to help the effort of making config as go a reality by exposing your configuration structs for users to import.

How: * Maintain and export your config structs like Kubernetes does (it is an API and well documented types) * Define configuration file via protobuf e.g like envoy here

Problems:

• What if project has only json schema? or no schema at all, just project written in different language:

  Answer: Generate it yourself from YAML (e.g using this online tool). Answer2: At some point if this concept will be big enough anyone can maintain useful Go module with typed, documented and testable config for some providers like we have in providers package

• Importing native Go structs is the dream, however:

  • Not all project's config are prepared to be imported (config tied to implementation, huge deps, secret masked, no marshaling etc): See: https://github.com/prometheus/alertmanager/pull/1804
  • This is where providers come in and we can define a set of structs to meet the config specified for your needs.

Documentation

• Best Practice

Other solutions

• Cue
• mixins
• jsonnet
• Pullumi (Paid)

Documentation

Overview

`mimic`: A Go module for defining and generating config in Go:

* Define your Configuration (e.g Envoy, Prometheus, Alertmanager, Nginx, Prometheus Alerts, Rules, Grafana Dashaboards etc.) * Define your Infrastructure (e.g Terraform, Ansible, Puppet, Chef, etc) * Define your Container Management (e.g Docker compose, Kubernetes, etc) * Define any other file that you can imagine

... using simple, typed and testable Go code!

Index

• func PanicErr(err error)
• func PanicOnErr(err error)
• func Panicf(format string, a ...interface{})
• type FilePool
  • func (f *FilePool) Add(fileName string, r io.Reader)
• type Generator
  • func New(injs ...func(cmd *kingpin.CmdClause)) *Generator
  • func (g *Generator) Generate()
  • func (g *Generator) With(parts ...string) *Generator

Functions

func PanicErr

func PanicErr(err error)

PanicErr allows to panic because of certain error.

func PanicOnErr

func PanicOnErr(err error)

PanicOnErr allows to panic on error.

func Panicf

func Panicf(format string, a ...interface{})

Panicf allows panic error propagation using sprintf-like formatting.

Types

type FilePool

type FilePool struct {
	Logger log.Logger
	// contains filtered or unexported fields
}

FilePool is a struct for storing and managing files to be generated as part of generation.

func (*FilePool) Add

func (f *FilePool) Add(fileName string, r io.Reader)

Add adds a file to the file pool at the current path. The file is identified by filename. Content of the file is passed via an io.Reader.

If the file with the given name has already been added at this path the code will `panic`. NOTE: See mimic/encoding for different marshallers to use as io.Reader.

type Generator

type Generator struct {
	FilePool
	// contains filtered or unexported fields
}

Generator manages a pool of generated files.

func New

func New(injs ...func(cmd *kingpin.CmdClause)) *Generator

New returns a new Generator that parses os.Args as command line arguments. It allows passing closure BEFORE parsing the flags to allow defining additional flags.

NOTE: Read README.md before using. This is intentionally NOT following Go library patterns like: * It uses panics as the main error handling way. * It creates CLI command inside constructor. * It does not allow custom loggers etc

func (*Generator) Generate

func (g *Generator) Generate()

Generate generates the configuration files that have been defined and added to a generator.

func (*Generator) With

func (g *Generator) With(parts ...string) *Generator

With behaves like linux `cd` command. It allows to "walk" & organize output files in a desired way for ease of use. Example:

```

gen := gen.With("mycompany.com", "production", "eu1", "kubernetes", "thanos")

``` Giving the path `mycompany.com/production/eu1/kubernetes/thanos`.

With return a Generator pointing at the specified path which can be specified even further: Example: ```

gen := mimic.New()
// gen/
...
gen = gen.With('foo')
// gen/foo
...
{
  gen := gen.With('bar')
  // gen/foo/bar
}
// gen/foo

```