stretcher

README

Stretcher

A deployment tool with Consul / Serf event.

Example manifest

src: s3://example.com/app.tar.gz
checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93
dest: /home/stretcher/app
commands:
  pre:
    - echo 'staring deploy'
  post:
    - echo 'deploy done'
  success:
    - echo 'deploy success'
  failure:
    - echo 'deploy failed!!'
    - cat >> /path/to/failure.log
excludes:
  - "*.pid"
  - "*.socket"

Run

stretcher agent

$ stretcher -h
Usage of stretcher:
  -max-bandwidth string
        max bandwidth for download src archives (Bytes/sec)
  -random-delay float
        sleep [0,random-delay) sec on start
  -retry int
        retry count for download src archives
  -retry-wait int
        wait for retry download src archives (sec) (default 3)
  -rsync-verbose string
        rsync verbose option (default -v)
  -timeout int
        timeout for download src archives (sec)
  -v    show version
  -version
        show version

with Consul

A stretcher agent is designed as running under "consul watch" and will be kicked by Consul event.

$ consul watch -type event -name deploy /path/to/stretcher

• -name: your deployment identity name.

with Serf

A stretcher agent can be running as Serf event handler.

$ serf agent -event-handler="user:deploy=/path/to/stretcher >> /path/to/stretcher.log 2>&1"

Load AWS credentials

When you specify a S3 URL in manifest, requires a AWS credential setting one of below.

• ~/.aws/config and ~/.aws/credentials (overridden by AWS_CONFIG_FILE environment variable.)
  • AWS_DEFAULT_PROFILE is supported to select a profile from multiple credentials in file.
• AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION environment variable.
• EC2 IAM role.
  • requires AWS_DEFAULT_REGION environment variable.

Load GCP credentials

When you specify a GS(Google Cloud Storage) URL in manifest, requires a GCP credential setting one of below.

• ServiceAccount
  • requires GOOGLE_APPLICATION_CREDENTIALS=[PATH] environment variable.
  • Replace [PATH] with the file path of the JSON file that contains your service account key.
• DefaultAccount
  • If the environment variable isn't set, load the default service account that Compute Engine provide, for applications that run on those services.

Deployment process

Preparing

This process is not included in a stretcher agent.

1. Create a tar(or tar.gz) archive for deployment.
2. Upload the archive file to remote server (S3 or HTTP(S)).
3. Create a manifest file (YAML) and upload it to remote server.

Executing with Consul

Create a consul event to kick stretcher agents.

$ consul event -name [event_name] [manifest_url]

$ consul event -name deploy s3://example.com/deploy-20141117-112233.yml

• -name: consul event name (specified by consul watch -name)

Executing with Serf

Create a serf user event to kick stretcher agents.

$ serf event [event_name] [manifest_url]

$ serf event deploy s3://example.com/deploy-20141117-112233.yml

• event_name: user event name (specified by serf event handler).

Executing as command

Stretcher can read a manifest URL from stdin simply.

$ echo s3://example.com/deploy-20141117-112233.yml | stretcher

You can execute stretcher via ssh or any other methods.

Deployment process

A stretcher agent executes a following process.

1. Receive a manifest URL as Consul/Serf event's payload.
2. Get a manifest.
3. Get src URL and store it to a temporary file, and Check checksum.
4. Invoke pre commands.
5. Extract src archive to a temporary directory.
6. Sync files from extracted archive to dest directory.

• use rsync -a --delete or mv
• sync strategy is switched by sync_strategy

1. Invoke post commands.

• Invoke success commands when the deployment process succeeded.
• Invoke failure commands when the deployment process failed.

Manifest spec

src

Source archive URL.

• URL schema: 's3', 'http', 'file'
• Format: 'tar', 'tar.gz'

src: http://example.com/src/archive.tar.gz

checksum

Checksum of source archive.

• Type: 'md5', 'sha1', 'sha256', 'sha512'

checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93

dest

Destination directory.

dest: /home/stretcher/app

dest_mode

Destination directory mode. Default: 0755

dest_mode: 0711

Destination directory mode will be set as...

1. src archive includes . => same as . in the archive.
2. src archive does not include . => dest_mode

commands

• pre: Commands which will be invoked at before src archive extracted.
• post: Commands which will be invoked at after dest directory synced.
• success: Commands which will be invoked at deployment process is succeeded.
• failure: Commands which will be invoked at deployment process is failed.

commands:
  pre:
    - echo 'staring deploy'
  post:
    - echo 'deploy done'
  success:
    - echo 'deploy success'
  failure:
    - echo 'deploy failed!!'
    - cat >> /path/to/failure.log

stretcher agent logs will be passed to STDIN of success and failure commands.

excludes

Pass to rsync --exclude arguments.

excludes:
  - "*.pid"
  - "*.socket"

exclude_from

Pass to rsync --exclude-from arguments. The file must be included in src archive.

exclude_from: exclude.list

sync_strategy

A strategy for syncing src extracted directory to dest directory.

• rsync: Default
  • Use rsync(1) command with option -av --delete
• mv
  • Use os.Rename() of Golang.
  • Deployment will be failed if dest directory is already exists.

Requirements

• tar
• rsync

tar and rsync must be exist in PATH environment.

If you use stretcher under systemd, You can see unfinished stdout with journald. You should add RateLimitBurst=0 into /etc/systemd/journald.conf for getting stdout completely.

Commands execution only mode

If src is not defined in a manifest, Stretcher runs pre/post and success/failure commands simply.

LICENSE

The MIT License (MIT)

Copyright (c) 2014 FUJIWARA Shunichiro / (c) 2014 KAYAC Inc.

Documentation

Index

• Constants
• Variables
• func LoadAWSCredentials(profileName string) (aws.Auth, aws.Region, error)
• func RandomTime(delay float64) time.Duration
• func Run(conf Config) error
• func SetRsyncVerboseOpt(opt string) error
• type CommandLine
  • func (c CommandLine) Invoke() error
  • func (c CommandLine) InvokePipe(src io.Reader) error
  • func (c CommandLine) String() string
• type CommandLines
  • func (cs CommandLines) Invoke() error
  • func (cs CommandLines) InvokePipe(src *bytes.Buffer) error
• type Commands
• type Config
• type ConsulEvent
  • func ParseConsulEvents(in io.Reader) (*ConsulEvent, error)
• type ConsulEvents
• type Manifest
  • func ParseManifest(data []byte) (*Manifest, error)
  • func (m *Manifest) Deploy(conf Config) error
• type MvSyncStrategy
  • func (s *MvSyncStrategy) Sync(from, to string) error
• type RsyncStrategy
  • func (s *RsyncStrategy) Sync(from, to string) error
• type SyncStrategy
  • func NewSyncStrategy(m *Manifest) (SyncStrategy, error)

Constants

const (
	AWSDefaultRegionName  = "us-east-1"
	AWSDefaultProfileName = "default"
)

const Nanoseconds = 1000 * 1000 * 1000

Variables

var (
	AWSAuth   aws.Auth
	AWSRegion aws.Region
	LogBuffer bytes.Buffer
	Version   string
)

var (
	RsyncDefaultOpts = []string{"-a", "--delete"}
	RsyncVerboseOpt  = "-v"
)

var DefaultDestMode = os.FileMode(0755)

Functions

func LoadAWSCredentials

func LoadAWSCredentials(profileName string) (aws.Auth, aws.Region, error)

func RandomTime

func RandomTime(delay float64) time.Duration

func Run

func Run(conf Config) error

func SetRsyncVerboseOpt

func SetRsyncVerboseOpt(opt string) error

Types

type CommandLine

type CommandLine string

func (CommandLine) Invoke

func (c CommandLine) Invoke() error

func (CommandLine) InvokePipe

func (c CommandLine) InvokePipe(src io.Reader) error

func (CommandLine) String

func (c CommandLine) String() string

type CommandLines

type CommandLines []CommandLine

func (CommandLines) Invoke

func (cs CommandLines) Invoke() error

func (CommandLines) InvokePipe

func (cs CommandLines) InvokePipe(src *bytes.Buffer) error

type Commands

type Commands struct {
	Pre     CommandLines `yaml:"pre"`
	Post    CommandLines `yaml:"post"`
	Success CommandLines `yaml:"success"`
	Failure CommandLines `yaml:"failure"`
}

type Config

type Config struct {
	MaxBandWidth uint64
	Timeout      time.Duration
	InitSleep    time.Duration
	Retry        int
	RetryWait    time.Duration
}

type ConsulEvent

type ConsulEvent struct {
	ID      string `json:"ID"`
	Name    string `json:"Name"`
	Payload []byte `json:"Payload"`
	LTime   int    `json:"LTime"`
}

func ParseConsulEvents

func ParseConsulEvents(in io.Reader) (*ConsulEvent, error)

type ConsulEvents

type ConsulEvents []ConsulEvent

type Manifest

type Manifest struct {
	Src          string       `yaml:"src"`
	CheckSum     string       `yaml:"checksum"`
	Dest         string       `yaml:"dest"`
	DestMode     *os.FileMode `yaml:"dest_mode"`
	Commands     Commands     `yaml:"commands"`
	Excludes     []string     `yaml:"excludes"`
	ExcludeFrom  string       `yaml:"exclude_from"`
	SyncStrategy string       `yaml:"sync_strategy"`
}

func ParseManifest

func ParseManifest(data []byte) (*Manifest, error)

func (*Manifest) Deploy

func (m *Manifest) Deploy(conf Config) error

type MvSyncStrategy

type MvSyncStrategy struct {
}

func (*MvSyncStrategy) Sync

func (s *MvSyncStrategy) Sync(from, to string) error

type RsyncStrategy

type RsyncStrategy struct {
	*Manifest
}

func (*RsyncStrategy) Sync

func (s *RsyncStrategy) Sync(from, to string) error

type SyncStrategy

type SyncStrategy interface {
	Sync(from, to string) error
}

func NewSyncStrategy

func NewSyncStrategy(m *Manifest) (SyncStrategy, error)