kamino

README

kamino

Manage development databases lifecycle described in 'recipes'

Kamino can be used to automatically:

• create database instances
• create the database schema (via sql-migrate migration)
• import initial dataset (from other database or files)
• generate configuration file for tools using these databases
• call shell script with information on these databases

Kamino is intend to manage databases for development, testing or CI/CD environments using the 'as-code' devops motto in simple and idempotent way

A common 'recipe' would implies the following steps:

1. Run a sql script as admin to create the database
2. Run a sql script as admin to create roles and/or users
3. Run a sql script as admin to create a schema
4. Run sql-migrate to populate the database schema
5. Synchronize tables from other database or files
6. Run a sql script as admin to give rights on tables/schemas
7. Generate a configuration file from a golang template with information of the database
8. Call a shell script to post a completion message to slack

Concepts

Kamino combine several type of 'objects' to manage the database lifecycle

Datasource

Each environment can be composed of several 'datasources'. This datasource is connection informations needed for a recipe (server, users/password, engine type, database name and/or schema if relevant). Each datasource is described in files in JSON, TOML or YAML format

Step

A step is an action to be done on the selected datasources. The action could be:

• sql : run a templated sql script
• migrate : apply sql-migrate migration, the actions will be different if Kamino is in apply or migrate mode.
• sync : Synchronize data from an other database or files
• shell : run a shell script
• template: Create a file from a template

More than one step can share the same 'name'. When user runs Kamino in apply mode, he can select a liste of step name or a step type to be run, only the steps corresponding to this criterias will be executed.

Each steps are described in files in JSON,TOML or YAML format. The needed attributes depend on the type of action.

Note: All relative path provided in attribute of a step are relative to the recipe folder that contains the recipe.

Idempotency

In apply and sync mode, before running a step of type migration, sql or sync, Kamino will determine if the action is necessary by running a provided SQL query, if this query returns a count different of 0, the step will be skipped. Once the action done, the query is re-run and the results must be different or Kamino will stop with an error. This condition can be removed with the --force option.

In migrate mode, the condition will never be enforced.

For step of type shell, it is the script responsability to be idempotent.

The idempotency is not garanteed for step of type template

Tags

Each datasource have tags (list of tag) that can be shared among them. These tags are used by steps to select on which datasource the action will occur.

Steps can have a list of tag selectors, if a datasource correspond to one selector of the list, it will be choosed by the step, the negation will remove datasource from the list of choosen ones.

Tags containing one ":" character will be considered has "named tags", their usage for selection is not different, but for template they can be referenced by Tags array or by the NamedTags map where the tag is splitted in two (name of tag before the ":", value of tag after).

A tag selector can be:

• A single tag (myTag): All datasources with this tag will be selected
• Intersection of tag (myTag1.myTag2): Only datasource with all this tag will be selected
• A negation of a tag selector (!ta1.ta2): All datasources complying to this tag selector will be excluded of the final list of datasource impacted by the step.

Tag selector can contains "globbing characters" (like in shell)

Recipe

A recipe is a collection of steps (a folders that contains the steps), the orders of step will be determine by the priority of the step (lower will be the first one), if more than one step have the same priority, they will be run in parallel.

Usage

Create a folder named after the name of your recipe, in this folder populate at least two folders (datasources and steps) with corresponding files and run kamino apply.

Refer to the documentation for a comprehensive description of the tool options

Example

The testdata folder of this repository contains an exemple of a recipe (pokemon) with all possible actions/options and a docker-compose.yml file. To see kamino in action run

docker-compose -f testdata/pokemon/docker-compose.yml up -d
kamino --connection-timeout 10s --connection-retry 5   -c testdata  apply pokemon

That will spin-up

• three mysql containers (root/adminpw)
• one postgres (postgres/adminpw)
• phpmyadmin (http://localhost:12348)
• phppgadmin (http://localhost:12349).

That will also start kamino using the testdata folder as base folder for recipe and applying the pokemon recipe.

The --connection--* parameters force to kamino to retry the first connection to datasources 5 times at 10 seconds interval, which is needed on this case because docker-compose will return when all containers are up, but database engines accept network connections only several seconds after the start of the containers.

This example is meant to show all the possibilities of kamino, your recipe does not have to be so complicated.

Contribution

I've made this project as a real use case to learn Golang. I've tried to adopt the Go mindset but I'm sure that other gophers could do better.

If this project can be useful for you, feel free to open issues, propose documentation updates or even pull request.

Contributions are of course always welcome!

1. Fork marema31/kamino (https://github.com/marema31/kamino/fork)
2. Create a feature branch
3. Commit your changes
4. Run test using go test ./...
5. Create a Pull Request

See CONTRIBUTING.md for details.

License

Copyright (c) 2019-present Marc Carmier

Licensed under BSD-2 Clause License