snitch

README

Snitch

Package github.com/shatil/snitch collects and (optionally) publishes ECS Cluster capacity metrics.

AWS SDK requires you to specify the AWS Region you wish to interact with, which you can do at runtime with environment variable AWS_REGION.

Development

git clone into your $GOPATH/src/, which may be ~/go/src/.

git clone \
    git@github.com:shatil/snitch.git \
    $GOPATH/src/github.com/shatil/snitch

For deployment-worth artifacts built locally, I recommend make docker-build. Summary of interesting Makefile targets:

Dependencies

make dep will fetch dependencies to vendor/.

Test

make test runs all tests and make cover-html will do that and generate HTML code coverage.

Run

You can build and run the binaries or pick a binary and:

AWS_REGION=ca-central-1 go run cmd/snitch/main.go

Build

make build builds all binaries in cmd/ and deposits them in this folder.

Install

make install installs the binaries to $GOPATH/bin/. You probably don't love this repo enough to do that--it's there mostly to simplify compiling this repository using Docker.

Documentation

Overview

Package snitch reads and optionally reports ECS Cluster resource consumption to assist with auto scaling.

Measurements are calculated from ECS Cluster data read by AWS SDK for Go, and if reporting is enabled, measurements are published to CloudWatch.

Example IAM permissions required to run (feel free to adjust "Resource" appropriately):

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "PermitReadingFromECS",
			"Effect": "Allow",
			"Action": [
				"ecs:DescribeContainerInstances",
				"ecs:ListClusters",
				"ecs:ListContainerInstances"
			],
			"Resource": [
				"*"
			]
		},
		{
			"Sid": "PermitWritingToCloudWatch",
			"Effect": "Allow",
			"Action": [
				"cloudwatch:GetMetricStatistics",
				"cloudwatch:PutMetricData"
			],
			"Resource": [
				"*"
			]
		}
	]
}

Authentication is done by AWS SDK for Go using your ~/.aws/credentials.

Index

• func ContainersPossible(cpu, memory int, resources []*ecs.Resource) (canSchedule int)
• func Run(sn *Snitcher)
• type ClusterResources
  • func NewClusterResources(cluster *string) *ClusterResources
  • func (cr *ClusterResources) ToMetricData() (metricData []*cloudwatch.MetricDatum)
• type Snitcher
  • func (sn *Snitcher) DescribeContainerInstances(cluster *string, instances []*string) []*ecs.ContainerInstance
  • func (sn *Snitcher) DescribeResourcesByInstanceType(cluster *string, instances []*string, cpu, memory int) []*cloudwatch.MetricDatum
  • func (sn *Snitcher) DiscoverClusters() <-chan *string
  • func (sn *Snitcher) DiscoverTasks(cluster *string) <-chan []*string
  • func (sn Snitcher) ListContainerInstances(cluster *string) []*string
  • func (sn *Snitcher) Measure() (metricData []*cloudwatch.MetricDatum)
  • func (sn *Snitcher) MeasureCluster(cluster *string) []*cloudwatch.MetricDatum
  • func (sn *Snitcher) MeasureResources(cluster *string, tasks []*string) (cpu, memory int)
  • func (sn *Snitcher) Publish(metricData []*cloudwatch.MetricDatum)
  • func (sn *Snitcher) WithAWS() *Snitcher
• Bugs

Functions

func ContainersPossible

func ContainersPossible(cpu, memory int, resources []*ecs.Resource) (canSchedule int)

ContainersPossible calculates how many containers are possible to launch.

This calculates how many containers can be scheduled per EC2 Instance, since array of ECS Resources is supplied per-Instance. cpu and memory provided indicate the number of CPU Units and Memory (RAM in MiB) a container will need to launch.

func Run

func Run(sn *Snitcher)

Run measures and maybe publishes findings.

During CLI or AWS Lambda usage, this is your entrypoint function. Lambda can use these handy environment variables in place of CLI arguments:

AWS_REGION for AWS Region (required unless ~/.aws/config sets it)

Types

type ClusterResources

type ClusterResources struct {
	Cluster    *string
	Resources  map[string]map[string]int
	CPU        map[string]int
	Memory     map[string]int
	Registered map[string]int
	Remaining  map[string]int
}

ClusterResources maps how many containers of lowest common multiple size can be launched by each EC2 Instance Type in an ECS Cluster.

"Lowest common multiple" means the largest container a cluster currently runs, whether it's the by largest CPU Unit count or Memory (RAM in MiB).

func NewClusterResources

func NewClusterResources(cluster *string) *ClusterResources

NewClusterResources creates a structure to map "RegisteredSchedulable" or "RemainingSchedulable" to count per *instanceType.

func (*ClusterResources) ToMetricData

func (cr *ClusterResources) ToMetricData() (metricData []*cloudwatch.MetricDatum)

ToMetricData formats metrics as AWS CloudWatch-compatible metric data.

type Snitcher

type Snitcher struct {
	// AWS clients from Go SDK, drawn from *iface to simplify testing.
	CloudWatch cloudwatchiface.CloudWatchAPI
	ECS        ecsiface.ECSAPI
	// Namespace in CloudWatch to publish metrics to.
	Namespace *string
	// Whether to publish metrics to CloudWatch.
	ShouldPublish *bool
}

Snitcher communicates with web services to collect or report data.

func (*Snitcher) DescribeContainerInstances

func (sn *Snitcher) DescribeContainerInstances(cluster *string, instances []*string) []*ecs.ContainerInstance

DescribeContainerInstances gathers descriptions of ECS Container Instances.

Requires IAM permission "ecs:DescribeContainerInstances".

func (*Snitcher) DescribeResourcesByInstanceType

func (sn *Snitcher) DescribeResourcesByInstanceType(cluster *string, instances []*string, cpu, memory int) []*cloudwatch.MetricDatum

DescribeResourcesByInstanceType collates an ECS Cluster's registered and remaining resources by EC2 Instance Type.

instances := sn.ListContainerInstances(cluster)
metricData := sn.DescribeResourcesByInstanceType(cluster, instances, cpu, memory)

EC2 Instance Type is gleaned from ECS Attribute "ecs.instance-type", which I think is supplied by ECS.

func (*Snitcher) DiscoverClusters

func (sn *Snitcher) DiscoverClusters() <-chan *string

DiscoverClusters reads ECS Clusters' ARNs like "arn:aws:ecs:ca-central-1:123456789012:cluster/my-cluster" and communicates derived Cluster nanme, like "my-cluster", to output channel.

Requires "ecs:ListClusters" IAM permission.

func (*Snitcher) DiscoverTasks

func (sn *Snitcher) DiscoverTasks(cluster *string) <-chan []*string

DiscoverTasks communicates pages of ECS Tasks' ARNs discovered in cluster.

While I'm no fan of arrays of string pointers, that's what AWS SDK outputs. Communication channel can safely be ranged over:

for tasks := range sn.DiscoverTasks(cluster) {
	log.Println(*cluster, "has", len(tasks), "tasks in cohort")
}

func (Snitcher) ListContainerInstances

func (sn Snitcher) ListContainerInstances(cluster *string) []*string

ListContainerInstances produces a cluster's container instance ARNs ("IDs").

Requires IAM permission "ecs:ListContainerInstances".

BUG(shatil): ListContainerInstances output isn't paginated, so we see first 100 containers' ARNs only.

func (*Snitcher) Measure

func (sn *Snitcher) Measure() (metricData []*cloudwatch.MetricDatum)

Measure how many containers an ECS Cluster can schedule.

func (*Snitcher) MeasureCluster

func (sn *Snitcher) MeasureCluster(cluster *string) []*cloudwatch.MetricDatum

MeasureCluster measures how many containers an ECS Cluster can schedule.

func (*Snitcher) MeasureResources

func (sn *Snitcher) MeasureResources(cluster *string, tasks []*string) (cpu, memory int)

MeasureResources finds "lowest common multiple" among reservable resources for specified tasks within a cluster.

Supply ECS cluster as aws.String() and ECS tasks are arrays communicated from DiscoverTasks.

func (*Snitcher) Publish

func (sn *Snitcher) Publish(metricData []*cloudwatch.MetricDatum)

Publish metrics to CloudWatch.

BUG(shatil): Publish must submit in batches of 20 MetricDatum because: https://github.com/aws/aws-sdk-go/issues/2019

func (*Snitcher) WithAWS

func (sn *Snitcher) WithAWS() *Snitcher

WithAWS adds AWS clients to Snitcher.

Notes

Bugs

• ListContainerInstances output isn't paginated, so we see first 100 containers' ARNs only.

• Publish must submit in batches of 20 MetricDatum because: https://github.com/aws/aws-sdk-go/issues/2019