lifecycled

README

Lifecycled - Gracefully handle EC2 scaling events

Lifecycled is designed to run on an AWS EC2 instance and listen for various state change mechanisms:

• AWS AutoScaling
• Spot Instance Termination Notices

When a termination notice is received, lifecycled runs a user-provided script (called a handler) and then proceeds with the shutdown. This script can be used to gracefully terminate any daemons you have running.

Installing with Systemd

Either install with go get -u github.com/buildkite/lifecycled or download a binary release for Linux or Windows. Install into /usr/bin/lifecycled.

# Install the binary
curl -Lf -o /usr/bin/lifecycled \
	https://github.com/buildkite/lifecycled/releases/download/${VERSION}/lifecycled-linux-amd64
chmod +x /usr/bin/lifecycled

# Install the systemd service
touch /etc/lifecycled
curl -Lf -o /etc/systemd/system/lifecycled.service \
	https://raw.githubusercontent.com/buildkite/lifecycled/${VERSION}/init/systemd/lifecycled.unit

Assuming your custom handler script is in /usr/local/bin/my_graceful_shutdown.sh and you've got an SNS topic for your EC2 Lifecycle Hooks, you would configure /etc/lifecycled with:

LIFECYCLED_HANDLER=/usr/local/bin/my_graceful_shutdown.sh
LIFECYCLED_SNS_TOPIC=arn:aws:sns:us-east-1:11111111:my-lifecycle-topic

Then start the daemon with:

systemctl daemon-reload
systemctl enable lifecycled
systemctl start lifecycled
systemctl status lifecycled

Handler script

Handler scripts are used for things like shutting down services that need some time to shutdown. Any example script that shuts down a service and waits for it to shutdown might look like:

#!/bin/bash
set -euo pipefail
function await_shutdown() {
  echo -n "Waiting for $1..."
  while systemctl is-active $1 > /dev/null; do
    sleep 1
  done
  echo "Done!"
}
systemctl stop myservice.service
await_shutdown myservice.service

The handler script is passed the event that was received and the instance id, e.g autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12 for lifecycle events, or ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z in the case of a spot termination.

Cleaning up Leftover SQS Queues

The lifecycled daemon should clean up the per-instance SQS queues that are created when it shuts down, but there has been a bug where this does not happen (See https://github.com/buildkite/lifecycled/issues/12). To mitigate this, you can run a cleanup tool:

go get -u github.com/buildkite/lifecycled/tools/lifecycled-queue-cleaner
lifecycled-queue-cleaner

Licence

See Licence.md (MIT)

Documentation

Index

• type AutoscalingClient
• type AutoscalingListener
  • func NewAutoscalingListener(instanceID string, queue *Queue, autoscaling AutoscalingClient) *AutoscalingListener
  • func (l *AutoscalingListener) Start(ctx context.Context, notices chan<- TerminationNotice, log *logrus.Entry) error
  • func (l *AutoscalingListener) Type() string
• type Config
• type Daemon
  • func New(config *Config, sess *session.Session, logger *logrus.Logger) *Daemon
  • func NewDaemon(config *Config, sqsClient SQSClient, snsClient SNSClient, ...) *Daemon
  • func (d *Daemon) AddListener(l Listener)
  • func (d *Daemon) Start(ctx context.Context) (notice TerminationNotice, err error)
• type Envelope
• type FileHandler
  • func NewFileHandler(file *os.File) *FileHandler
  • func (h *FileHandler) Execute(ctx context.Context, args ...string) error
• type Handler
• type Listener
• type Message
• type Queue
  • func NewQueue(queueName, topicArn string, sqsClient SQSClient, snsClient SNSClient) *Queue
  • func (q *Queue) Create() error
  • func (q *Queue) Delete() error
  • func (q *Queue) DeleteMessage(ctx context.Context, receiptHandle string) error
  • func (q *Queue) GetMessages(ctx context.Context) ([]*sqs.Message, error)
  • func (q *Queue) Subscribe() error
  • func (q *Queue) Unsubscribe() error
• type SNSClient
• type SQSClient
• type SpotListener
  • func NewSpotListener(instanceID string, metadata *ec2metadata.EC2Metadata, interval time.Duration) *SpotListener
  • func (l *SpotListener) Start(ctx context.Context, notices chan<- TerminationNotice, log *logrus.Entry) error
  • func (l *SpotListener) Type() string
• type TerminationNotice

Types

type AutoscalingClient

type AutoscalingClient autoscalingiface.AutoScalingAPI

AutoscalingClient for testing purposes

type AutoscalingListener

type AutoscalingListener struct {
	// contains filtered or unexported fields
}

AutoscalingListener ...

func NewAutoscalingListener

func NewAutoscalingListener(instanceID string, queue *Queue, autoscaling AutoscalingClient) *AutoscalingListener

NewAutoscalingListener ...

func (*AutoscalingListener) Start

func (l *AutoscalingListener) Start(ctx context.Context, notices chan<- TerminationNotice, log *logrus.Entry) error

Start the autoscaling lifecycle hook listener.

func (*AutoscalingListener) Type

func (l *AutoscalingListener) Type() string

Type returns a string describing the listener type.

type Config

type Config struct {
	InstanceID           string
	SNSTopic             string
	SpotListener         bool
	SpotListenerInterval time.Duration
}

Config for the Lifecycled Daemon.

type Daemon

type Daemon struct {
	// contains filtered or unexported fields
}

Daemon is what orchestrates the listening and execution of the handler on a termination notice.

func New

func New(config *Config, sess *session.Session, logger *logrus.Logger) *Daemon

New creates a new lifecycle Daemon.

func NewDaemon

func NewDaemon(
	config *Config,
	sqsClient SQSClient,
	snsClient SNSClient,
	asgClient AutoscalingClient,
	metadata *ec2metadata.EC2Metadata,
	logger *logrus.Logger,
) *Daemon

NewDaemon creates a new Daemon.

func (*Daemon) AddListener

func (d *Daemon) AddListener(l Listener)

AddListener to the Daemon.

func (*Daemon) Start

func (d *Daemon) Start(ctx context.Context) (notice TerminationNotice, err error)

Start the Daemon.

type Envelope

type Envelope struct {
	Type    string    `json:"Type"`
	Subject string    `json:"Subject"`
	Time    time.Time `json:"Time"`
	Message string    `json:"Message"`
}

Envelope ...

type FileHandler

type FileHandler struct {
	// contains filtered or unexported fields
}

FileHandler ...

func NewFileHandler

func NewFileHandler(file *os.File) *FileHandler

NewFileHandler ...

func (*FileHandler) Execute

func (h *FileHandler) Execute(ctx context.Context, args ...string) error

Execute the file handler.

type Handler

type Handler interface {
	Execute(ctx context.Context, args ...string) error
}

Handler ...

type Listener

type Listener interface {
	Type() string
	Start(context.Context, chan<- TerminationNotice, *logrus.Entry) error
}

Listener ...

type Message

type Message struct {
	Time        time.Time `json:"Time"`
	GroupName   string    `json:"AutoScalingGroupName"`
	InstanceID  string    `json:"EC2InstanceId"`
	ActionToken string    `json:"LifecycleActionToken"`
	Transition  string    `json:"LifecycleTransition"`
	HookName    string    `json:"LifecycleHookName"`
}

Message ...

type Queue

type Queue struct {
	// contains filtered or unexported fields
}

Queue manages the SQS queue and SNS subscription.

func NewQueue

func NewQueue(queueName, topicArn string, sqsClient SQSClient, snsClient SNSClient) *Queue

NewQueue returns a new... Queue.

func (*Queue) Create

func (q *Queue) Create() error

Create the SQS queue.

func (*Queue) Delete

func (q *Queue) Delete() error

Delete the SQS queue.

func (*Queue) DeleteMessage

func (q *Queue) DeleteMessage(ctx context.Context, receiptHandle string) error

DeleteMessage from the queue.

func (*Queue) GetMessages

func (q *Queue) GetMessages(ctx context.Context) ([]*sqs.Message, error)

GetMessages long polls for messages from the SQS queue.

func (*Queue) Subscribe

func (q *Queue) Subscribe() error

Subscribe the queue to an SNS topic

func (*Queue) Unsubscribe

func (q *Queue) Unsubscribe() error

Unsubscribe the queue from the SNS topic.

type SNSClient

type SNSClient snsiface.SNSAPI

SNSClient for testing purposes

type SQSClient

type SQSClient sqsiface.SQSAPI

SQSClient for testing purposes

type SpotListener

type SpotListener struct {
	// contains filtered or unexported fields
}

SpotListener ...

func NewSpotListener

func NewSpotListener(instanceID string, metadata *ec2metadata.EC2Metadata, interval time.Duration) *SpotListener

NewSpotListener ...

func (*SpotListener) Start

func (l *SpotListener) Start(ctx context.Context, notices chan<- TerminationNotice, log *logrus.Entry) error

Start the spot termination notice listener.

func (*SpotListener) Type

func (l *SpotListener) Type() string

Type returns a string describing the listener type.

type TerminationNotice

type TerminationNotice interface {
	Type() string
	Handle(context.Context, Handler, *logrus.Entry) error
}

TerminationNotice ...