maestro

package module
v0.0.0-...-9443c12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2015 License: GPL-2.0 Imports: 15 Imported by: 0

README

Maestro

maestro

A smart deploy system for lazy software developers

Maestro is a development system able to build and deploy applications (docker images) on a distrubuted CoreOS cluster. Applications are described using a simple JSON configuration and converted into runnable Fleet units and all the burden about namespaces, DNS, container linking, etc is automatically handled by Maestro

Maestro is based on
Maestro is a set of tools written in
  • Golang
  • BASH
  • Make
  • HTML
Maestro components / tools / docker containers
GoDoc
Keynote
Prerequisites

Install Vagrant and Fleetctl and Golang for your architecture.

Installation
$ git clone https://github.com/crisidev/maestro
$ cd maestro
$ make
go clean .
go clean ./cmd/maestro
rm -rf maestro bindata.go
ln -s config/maestro-metrics.json maestro.json || echo "continuing..."
ln -s vagrant/maestro-vagrant-registry/user-data.sample vagrant/maestro-vagrant-registry/user-data || echo "continuing..."
ln -s vagrant/maestro-vagrant-cluster/user-data.sample vagrant/maestro-vagrant-cluster/user-data || echo "continuing..."
go-bindata -pkg maestro templates
go build .
go build ./cmd/maestro
go install ./cmd/maestro
$ maestro
usage: maestro [<flags>] <command> [<args> ...]

friendly generates and deploy systemd unit files on a CoreOS maestro cluster https://github.com/crisidev/maestro

Flags:
  --help           Show help (also see --help-long and --help-man).
  -d, --debug      enable debug mode
  -c, --config="maestro.json"
                   configuration file
  -V, --volumesdir="/share/maestro"
                   directory on the coreos host for shared volumes
  -m, --maestrodir=MAESTRODIR
                   directory on the local host for configs and temporary files (default to $USER/.maestro)
  --domain="maestro.io"
                   domain used to deal with etcd, skydns, spartito and violino
  -e, --etcd=ETCD  etcd / fleet endpoints to connect
  -F, --fleetopts=FLEETOPTS
                   fleetctl options
  -A, --fleetaddr="172.17.8.101"
                   fleetctl tunnel address and port

Commands:
  help [<command>...]
    Show help.

  corestatus
    report coreos cluster status

  exec <args>...
    exec an arbitrary command through fleet, returning output as stdout and exit code

  etcd [<flags>] [<name>]
    get maestro related list of keys from etcd

  run [<name>]
    run current app on coreos (this will build unit files, submit and run them)

  stop [<name>]
    stop current app without cleaning unit files on coreos

  nuke [<flags>] [<name>]
    stop current app and clean unit files on coreos

  status [<name>]
    show the global app status (systemctl status unitfiles)

  journal [<flags>] [<name>]
    show the journal (journalctl -xu unit) of one app s component

  user
    get current user name

  config
    print json configuration for current app

  build
    locally build app units

  buildimages [<unit>]
    run a container build and registry push on the cluster

  buildstatus [<name>]
    check status of a container build and registry push on the cluster

  buildnuke [<name>]
    check status of a container build and registry push on the cluster
Usage
DNS Resolution In Details
Configuration
A Basic Example

Let's say we want to run docker container which will ping google.com. Maestro config:

{
  "username": "crisidev",
  "app": "pinger",
  "stages": [
    {
      "name": "prod",
      "components": [
        {
          "name": "pinger",
          "src": "hub.maestro.io:5000/crisidev/busybox",
          "gitsrc": "https://github.com/crisidev/maestro-busybox",
          "cmd": "ping google.com",
        }
      ]
    }
  ]
}
A Complex Example

Let's say we want to run a complete monitoring system for Maestro, using Prometheus as timeseries database and Grafana as visualiser. DNS metrics will be gathered from SkyDNS, container metrics from Cadvisor and node metrics from Prometheus Node Exporter. Grafana and Prometheus the components will share a volume (MacOSX only).

{
  "username": "crisidev",
  "app": "metrics",
  "stages": [
    {
      "name": "prod",
      "components": [
        {
          "name": "prometheus",
          "dns": "prometheus",
          "frontend": true,
          "src": "hub.maestro.io:5000/crisidev/prometheus",
          "gitsrc": "https://github.com/crisidev/maestro-prometheus",
          "ports": [9090],
          "volumes": [
            "/sharedvol"
          ]
        },
        {
          "name": "grafana",
          "dns": "grafana",
          "frontend": true,
          "src": "hub.maestro.io:5000/crisidev/grafana",
          "gitsrc": "https://github.com/crisidev/maestro-grafana",
          "ports": [3001],
           "volumes": [
            "/sharedvol"
           ]
        },
        {
          "name": "cadvisor",
          "frontend": true,
          "global": true,
          "dns": "cadvisor",
          "src": "google/cadvisor:latest",
          "ports": [8080],
          "docker_args": "-v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro"
        },
        {
          "name": "exporter",
          "global": true,
          "src": "prom/node-exporter",
          "docker_args": "--net=host"
        }
      ]
    }
  ]
}

Our application will be reachable at http://prometheus.maestro.io and http://grafana.maestro.io

Configuration Structure
// MaestroComponent structure
type MaestroComponent struct {
	App           string   `json:"app"`
	BuildUnitPath string   `json:"build_unitpath"`
	Cmd           string   `json:"cmd"`
	ContainerName string   `json:"container_name"`
	DNS           string   `json:"dns"`
	DockerArgs    string   `json:"docker_args"`
	Env           []string `json:"env"`
	Frontend      bool     `json:"frontend"`
	GitSrc        string   `json:"gitsrc"`
	Global        bool     `json:"global"`
	InternalDNS   string   `json:"internal_dns"`
	KeepOnExit    bool     `json:"keep_on_exit"`
	Name          string   `json:"name"`
	Ports         []int    `json:"ports"`
	Scale         int      `json:"scale"`
	Single        bool     `json:"single"`
	Src           string   `json:"src"`
	Stage         string   `json:"stage"`
	UnitName      string   `json:"unitname"`
	UnitPath      string   `json:"unitpath"`
	Username      string   `json:"username"`
	Volumes       []string `json:"volumes"`
	VolumesDir    string   `json:"volumes_dir"`
}

// MaestroConfig structure
type MaestroConfig struct {
	App    string `json:"app"`
	Stages []struct {
		Components []MaestroComponent `json:"components"`
		Name       string             `json:"name"`
	} `json:"stages"`
	Username string `json:"username"`
}

Documentation

Overview

Maestro is a development system able to build and deploy applications (docker images) on a distrubuted CoreOS cluster. Applications are described using a simple JSON configuration and converted into runnable Fleet units and all the burden about namespaces, DNS, container linking, etc is automatically handled by Maestro

Prerequisites

Install Vagrant and Fleetctl and Golang for your architecture.

Installation and Usage

Please read README.md for full installation instructions and usage.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EtcdCheckExec 

func EtcdCheckExec()

Checks if etcdctl is available on the system.

func EtcdExec 

func EtcdExec(args []string, output chan string, exit chan int, key string)

Wrapper around etcdctl, able to run every command. It uses two channels to communicate output and return code of every command issued.

func EtcdPrepareArgs 

func EtcdPrepareArgs(key string) (etcdArgs []string)

Prepares etcdctl arguments with info based from command line

func EtcdPullKeys 

func EtcdPullKeys(skydns, all bool, key string) (exitCode int)

Pulls maestro related keys

func FleetBuildUnit 

func FleetBuildUnit(_, unitPath string) (exitCode int)

Wrapper to run a container build on the coreos cluster.

func FleetCheckExec 

func FleetCheckExec()

Checks if fleetctl is available on the system.

func FleetCheckPath 

func FleetCheckPath(unitPath string)

Checks if a unit path is valid, either build unit and run unit.

func FleetExec 

func FleetExec(args []string, output chan string, exit chan int)

Wrapper around fleetctl, able to run every command. It uses two channels to communicate output and return code of every command issued.

func FleetExecCommand 

func FleetExecCommand(cmd, unitPath string) (exitCode int)

Function able to run a command on a unit path. Output is processed and printed and fleetctl exit code is returned.

func FleetIsUnitRunning 

func FleetIsUnitRunning(unitPath string) (ret bool)

Utility function to check if a unit is already running on the cluster.

func FleetPrepareArgs 

func FleetPrepareArgs(args []string) (fleetArgs []string)

Prepares fleetctl arguments with info based from command line

func FleetProcessOutput 

func FleetProcessOutput(output chan string, exit chan int) (exitCode int)

Process output and exit channel from a fleetctl command.

func FleetRunUnit 

func FleetRunUnit(_, unitPath string) (exitCode int)

Wrapper to run a unit on the coreos cluster.

func GetTmpl 

func GetTmpl(name string) string

Return a string containing a template.

func GetUnitFd 

func GetUnitFd(filepath string) *os.File

Return a file descriptor to be used to render a template.

func HandleExit 

func HandleExit(exitCode int)

func Init 

func Init(maestroDir, domainName, address, volumes, endpoints string, options []string, debug bool)

func MaestroBuildContainers 

func MaestroBuildContainers(unit string) (exitCode int)

Build local unit files to build new docker images. After the unit is build, it will destroy, submit, load and start using fleetctl. The image will be pushed to the local registry.

func MaestroBuildLocalUnits 

func MaestroBuildLocalUnits()

Build local unit files for all components in configuration.

func MaestroBuildNuke 

func MaestroBuildNuke(unit string) (exitCode int)

Destroys all units used for building docker images. It can stop also a single unit, using `unit` argument.

func MaestroBuildStatus 

func MaestroBuildStatus(unit string) (exitCode int)

Check and prints the status of all units used to build new docker images.

func MaestroCommandExec 

func MaestroCommandExec(cmd *exec.Cmd, output chan string) (exitCode int)

func MaestroCoreStatus 

func MaestroCoreStatus() (exitCode int)

Executes a global coreos status, running `list-machines`, `list-units`, `list-unit-files`.

func MaestroExecBuild 

func MaestroExecBuild(fn MaestroCommand, cmd, unit string) (exitCode int)

Exec an arbitrary function on a build unit

func MaestroExecRun 

func MaestroExecRun(fn MaestroCommand, cmd, unit string) (exitCode int)

Exec an arbitrary function on a run unit

func MaestroJournal 

func MaestroJournal(unit string, follow, all bool) (exitCode int)

Prints the journal for all units in the current app It can also get the journal of a single unit, using `unit` argument.

func MaestroNuke 

func MaestroNuke(unit string) (exitCode int)

Destroys all units in the current app. It can stop also a single unit, using `unit` argument.

func MaestroNukeAll 

func MaestroNukeAll() (exitCode int)

func MaestroRun 

func MaestroRun(unit string) (exitCode int)

Function used to submit, load and start all the units inside the current app. It can start also a single unit, using `unit` argument. If the unit is already running, it will print a message and do nothing.

func MaestroStatus 

func MaestroStatus(unit string) (exitCode int)

Prints status for all units in the current app It can also get the status of a single unit, using `unit` argument.

func MaestroStop 

func MaestroStop(unit string) (exitCode int)

Stops all units in the current app. It can stop also a single unit, using `unit` argument.

func ProcessUnitTmpl 

func ProcessUnitTmpl(component MaestroComponent, unitName, unitPath, tmplName string)

Renders a template onto a file.

func SetupMaestroDir 

func SetupMaestroDir(dir string)

Setup directory ($CWD/.maestro) used to store user informations and temporary build file before submitting to coreos. Directory will be created if not existent.

Types

type MaestroCommand 

type MaestroCommand func(string, string) int

Function passed to commands

type MaestroComponent 

type MaestroComponent struct {
	After         string   `json:"after"`
	App           string   `json:"app"`
	BuildUnitPath string   `json:"build_unitpath"`
	Cmd           string   `json:"cmd"`
	ContainerName string   `json:"container_name"`
	DNS           string   `json:"dns"`
	DockerArgs    string   `json:"docker_args"`
	Env           []string `json:"env"`
	Frontend      bool     `json:"frontend"`
	GitSrc        string   `json:"gitsrc"`
	Global        bool     `json:"global"`
	InternalDNS   string   `json:"internal_dns"`
	KeepOnExit    bool     `json:"keep_on_exit"`
	Name          string   `json:"name"`
	Ports         []int    `json:"ports"`
	Scale         int      `json:"scale"`
	Single        bool     `json:"single"`
	Src           string   `json:"src"`
	Stage         string   `json:"stage"`
	UnitName      string   `json:"unitname"`
	UnitPath      string   `json:"unitpath"`
	Username      string   `json:"username"`
	Volumes       []string `json:"volumes"`
	VolumesDir    string   `json:"volumes_dir"`
}

MaestroComponent structure

type MaestroConfig 

type MaestroConfig struct {
	App    string `json:"app"`
	Stages []struct {
		Components []MaestroComponent `json:"components"`
		Name       string             `json:"name"`
	} `json:"stages"`
	Username string `json:"username"`
}

MaestroConfig structure

func BuildMaestroConfig 

func BuildMaestroConfig(cfg string) MaestroConfig

Public function used in the main to load the configuration.

func (*MaestroConfig) GetAfterUnit 

func (c *MaestroConfig) GetAfterUnit(name string) (after string)

func (*MaestroConfig) GetAppPath 

func (c *MaestroConfig) GetAppPath(stage string) string

Returns the local path for an app.

func (*MaestroConfig) GetContainerName 

func (c *MaestroConfig) GetContainerName(component *MaestroComponent) string

Returns the container name for a component (mainly for debugging purposes).

func (*MaestroConfig) GetNumberedUnitPath 

func (c *MaestroConfig) GetNumberedUnitPath(path, number string) string

Return a path for a numbered unit, used to start scalable components.

func (*MaestroConfig) GetUnitInternalDNS 

func (c *MaestroConfig) GetUnitInternalDNS(component *MaestroComponent, domain string) string

Returns an inernal DNS name for a unit (mainly for debugging purposes).

func (*MaestroConfig) GetUnitName 

func (c *MaestroConfig) GetUnitName(component *MaestroComponent, suffix string) string

Returns a name for a unit, starting from a `stage`, a `component` and a `suffix`.

func (*MaestroConfig) GetUnitPath 

func (c *MaestroConfig) GetUnitPath(component *MaestroComponent, suffix string) string

Returns the local path for a run unit. A run unit is a component of an application which will be run as docker container on the coreos cluster.

func (*MaestroConfig) GetUsername 

func (c *MaestroConfig) GetUsername()

func (*MaestroConfig) GetVolumePath 

func (c *MaestroConfig) GetVolumePath(stage, volume, volumesDir string) string

Returns a proper volume path, bound to the shared directory on the node.

func (*MaestroConfig) LoadMaestroConfig 

func (c *MaestroConfig) LoadMaestroConfig(path string) MaestroConfig

Parses JSON config file into MaestroConfig struct.

func (*MaestroConfig) Print 

func (c *MaestroConfig) Print()

Simple repr for MaestroConfig struct.

func (*MaestroConfig) SetMaestroComponentConfig 

func (c *MaestroConfig) SetMaestroComponentConfig()

Sets components config values.

func (*MaestroConfig) SetupMaestroAppDirs 

func (c *MaestroConfig) SetupMaestroAppDirs()

Creates directories for unit file building. Schema: $CWD/.maestro/$username/$stage/$app

func (*MaestroConfig) SetupUsername 

func (c *MaestroConfig) SetupUsername()

Manages username creation, loading and saving to file.

func (*MaestroConfig) UsernameWizard 

func (c *MaestroConfig) UsernameWizard()

Wizard to create a new username.

func (*MaestroConfig) WriteUsernameFile 

func (c *MaestroConfig) WriteUsernameFile()

Writes username JSON informations onto user file

type MaestroLog 

type MaestroLog struct {
	// contains filtered or unexported fields
}

Logger with function to print colors

func (MaestroLog) Debug 

func (l MaestroLog) Debug(msg string, prefix ...string)

Logs to standard error with colors

func (MaestroLog) Debug2 

func (l MaestroLog) Debug2(msg, suffix string, prefix ...string)

Logs to standard error with colors also in the message

func (MaestroLog) DebugError 

func (l MaestroLog) DebugError(err error)

Debugs an error

func (MaestroLog) Error 

func (l MaestroLog) Error(err error)

Logs an error

func (MaestroLog) Fatal 

func (l MaestroLog) Fatal(err error)

Logs an error and gracefully exit

func (MaestroLog) Out 

func (l MaestroLog) Out(msg string)

Logs to stardard output

func (MaestroLog) OutRaw 

func (l MaestroLog) OutRaw(msg string)

Logs to standard output without carriage return

func (MaestroLog) SetupBase 

func (l MaestroLog) SetupBase()

Setup "base" application name for debugging

func (MaestroLog) SetupDebug 

func (l MaestroLog) SetupDebug()

Setup debugging to stderr

func (MaestroLog) SetupPrefix 

func (l MaestroLog) SetupPrefix(prefix ...string) (base string)

Setup prefix for debugging with colors

type MaestroUser 

type MaestroUser struct {
	Name string `json:"name"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
maestro

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.