drain

README

ecs-drain-lambda

Based on the original idea from AWS Blog post and GitHub. With the following differences:

• Autoscaling Hooks events are received via CloudWatch rules, which makes possible having one function for draining many ECS Clusters

• Serverless Framework based

• Written in Golang

• Supports the draining of Spot based ECS instances via Spot Instance Interruption Notice

Why?

When updating AMI for the ECS instances then ASG replaces them without "Draining" , which may cause a short downtime of deployed containers. This function automates the ECS Cluster Instances Drain process.

How does it work?

ecs-drain-lambda function:

• Receives CloudWatch event:

  • ANY AutoScaling Lifecycle Terminate event ( EC2 Auto Scaling Lifecycle Hooks for autoscaling:EC2_INSTANCE_TERMINATING event should be configured on your ASG ) from CloudWatch Events

  or

  • ANY Spot Instance Interruption Notice. Imporatant, AWS doesn't guarantees that instance will be drained in time, instance could be terminated before the notice arrival.

• Gets the ID of the instance that has to be terminated

• Looks for the ECS Cluster name in the UserData in the following format: ECS_CLUSTER=xxxxxxxxx

• If some ECS Tasks are running on the instance, starts the Drain process

• Waits for all the ECS Tasks to shutdown

• Completes Lifecycle Hook, which lets ASG proceed with instance termination

Requirements

• Serverless Framework

• Golang

• GNU Make

• Configured EC2 Auto Scaling Lifecycle Hooks for autoscaling:EC2_INSTANCE_TERMINATING event on your ASG

  Example CloudFormation resource:

    ASGTerminateHook:
       Type: "AWS::AutoScaling::LifecycleHook"
       Properties:
         AutoScalingGroupName: !Ref ECSAutoScalingGroup
         DefaultResult: "ABANDON"
         HeartbeatTimeout: "900"
         LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING"
  

How to use

• Clone the repo with git clone

• Enter the project directory cd ecs-drain-lambda

• Run make deploy

Note: by default us-east-1 region is selected, if you need to deploy it to the different region you can use sls deploy -v --region ${AWS_REGION}

Deploy with Terraform

If you want to deploy ecs-drain-lambda function with Terraform, there is ecs-drain-lambda Terraform Module.

Limitations

• Function waits for 15 minutes for Drain to complete and fails with the timeout after

• If function fails, then the default lifecycle hook action will be triggered (ABANDON or CONTINUE depending on your Hook configuration), either result will end up with eventual instance termination.

  Documentation

    If the instance is terminating, both ABANDON and CONTINUE allow the instance to terminate. However, ABANDON stops any remaining actions, such as other lifecycle hooks, while CONTINUE allows any other lifecycle hooks to complete.
  

Documentation

Index

• Constants
• Variables
• func Drain(ecsCluster, ec2Instance string) error
• func GetClusterNameFromInstanceUserData(ec2Instance string) (string, error)
• func HandleRequest(ctx context.Context, event *events.CloudWatchEvent) error
• type ASGLifecycleEventDetail
  • func (asgEvent *ASGLifecycleEventDetail) CompleteLifecycle() error
• type EC2SpotInterruptionEventDetail

Constants

const (
	EventASGTerminateDetailType       = "EC2 Instance-terminate Lifecycle Action"
	EventEC2SpotInteruptionDetailType = "EC2 Spot Instance Interruption Warning"
)

Variables

var (
	ErrMissingUserData             = errors.New("This instance seems not to have UserData")
	ErrMissingECSClusterInUserData = errors.New("This instance seems not to have EcsCluster definition in UserData")
	ErrInstanceTerminated          = errors.New("This instance is already terminated")
)

Functions

func Drain

func Drain(ecsCluster, ec2Instance string) error

func GetClusterNameFromInstanceUserData

func GetClusterNameFromInstanceUserData(ec2Instance string) (string, error)

func HandleRequest

func HandleRequest(ctx context.Context, event *events.CloudWatchEvent) error

Types

type ASGLifecycleEventDetail

type ASGLifecycleEventDetail struct {
	LifecycleActionToken string
	AutoScalingGroupName string
	LifecycleHookName    string
	EC2InstanceID        string
	LifecycleTransition  string
}

func (*ASGLifecycleEventDetail) CompleteLifecycle

func (asgEvent *ASGLifecycleEventDetail) CompleteLifecycle() error

type EC2SpotInterruptionEventDetail

type EC2SpotInterruptionEventDetail struct {
	InstanceID     string `json:"instance-id"`
	InstanceAction string `json:"instance-action"`
}