aws

README

Cluster Autoscaler on AWS

The cluster autoscaler on AWS scales worker nodes within any specified autoscaling group. It will run as a Deployment in your cluster. This README will go over some of the necessary steps required to get the cluster autoscaler up and running.

Kubernetes Version

Cluster autoscaler must run on v1.3.0 or greater.

Permissions

The worker running the cluster autoscaler will need access to certain resources and actions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

Unfortunately AWS does not support ARNs for autoscaling groups yet so you must use "*" as the resource. More information here.

Deployment Specification

1 ASG Setup (min: 1, max: 10, ASG Name: k8s-worker-asg-1)

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      containers:
        - image: gcr.io/google_containers/cluster-autoscaler:v0.4.0
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --nodes=1:10:k8s-worker-asg-1
          env:
            - name: AWS_REGION
              value: us-east-1
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs/ca-certificates.crt
              readOnly: true
          imagePullPolicy: "Always"
      volumes:
        - name: ssl-certs
          hostPath:
            path: "/etc/ssl/certs/ca-certificates.crt"

Multiple ASG Setup

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      containers:
        - image: gcr.io/google_containers/cluster-autoscaler:v0.4.0
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --nodes=1:10:k8s-worker-asg-1
            - --nodes=1:3:k8s-worker-asg-2
          env:
            - name: AWS_REGION
              value: us-east-1
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs/ca-certificates.crt
              readOnly: true
          imagePullPolicy: "Always"
      volumes:
        - name: ssl-certs
          hostPath:
            path: "/etc/ssl/certs/ca-certificates.crt"

Common Notes and Gotchas:

• The /etc/ssl/certs/ca-certificates.crt should exist by default on your ec2 instance.
• Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. For more information, see https://github.com/kubernetes/contrib/pull/1552#discussion_r75533090.
• By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the --skip-nodes-with-system-pods=false flag.
• By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the --scale-down-delay flag. E.g. --scale-down-delay=5m to decrease the scale down delay to 5 minutes.
• If you're running multiple ASGs, the --expander flag supports three options: random, most-pods and least-waste. random will expand a random ASG on scale up. most-pods will scale up the ASG that will scheduable the most amount of pods. least-waste will expand the ASG that will waste the least amount of CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back to random.

Documentation

Index

• type Asg
  • func (asg *Asg) Belongs(node *apiv1.Node) (bool, error)
  • func (asg *Asg) Debug() string
  • func (asg *Asg) DecreaseTargetSize(delta int) error
  • func (asg *Asg) DeleteNodes(nodes []*apiv1.Node) error
  • func (asg *Asg) Id() string
  • func (asg *Asg) IncreaseSize(delta int) error
  • func (asg *Asg) MaxSize() int
  • func (asg *Asg) MinSize() int
  • func (asg *Asg) Nodes() ([]string, error)
  • func (asg *Asg) TargetSize() (int, error)
• type AwsCloudProvider
  • func BuildAwsCloudProvider(awsManager *AwsManager, specs []string) (*AwsCloudProvider, error)
  • func (aws *AwsCloudProvider) Name() string
  • func (aws *AwsCloudProvider) NodeGroupForNode(node *apiv1.Node) (cloudprovider.NodeGroup, error)
  • func (aws *AwsCloudProvider) NodeGroups() []cloudprovider.NodeGroup
• type AwsManager
  • func CreateAwsManager(configReader io.Reader) (*AwsManager, error)
  • func (m *AwsManager) DeleteInstances(instances []*AwsRef) error
  • func (m *AwsManager) GetAsgForInstance(instance *AwsRef) (*Asg, error)
  • func (m *AwsManager) GetAsgNodes(asg *Asg) ([]string, error)
  • func (m *AwsManager) GetAsgSize(asgConfig *Asg) (int64, error)
  • func (m *AwsManager) RegisterAsg(asg *Asg)
  • func (m *AwsManager) SetAsgSize(asg *Asg, size int64) error
• type AwsRef
  • func AwsRefFromProviderId(id string) (*AwsRef, error)

Types

type Asg

type Asg struct {
	AwsRef
	// contains filtered or unexported fields
}

Asg implements NodeGroup interfrace.

func (*Asg) Belongs

func (asg *Asg) Belongs(node *apiv1.Node) (bool, error)

Belongs returns true if the given node belongs to the NodeGroup.

func (*Asg) Debug

func (asg *Asg) Debug() string

Debug returns a debug string for the Asg.

func (*Asg) DecreaseTargetSize

func (asg *Asg) DecreaseTargetSize(delta int) error

DecreaseTargetSize decreases the target size of the node group. This function doesn't permit to delete any existing node and can be used only to reduce the request for new nodes that have not been yet fulfilled. Delta should be negative. It is assumed that cloud provider will not delete the existing nodes if the size when there is an option to just decrease the target.

func (*Asg) DeleteNodes

func (asg *Asg) DeleteNodes(nodes []*apiv1.Node) error

DeleteNodes deletes the nodes from the group.

func (*Asg) Id

func (asg *Asg) Id() string

Id returns asg id.

func (*Asg) IncreaseSize

func (asg *Asg) IncreaseSize(delta int) error

IncreaseSize increases Asg size

func (*Asg) MaxSize

func (asg *Asg) MaxSize() int

MaxSize returns maximum size of the node group.

func (*Asg) MinSize

func (asg *Asg) MinSize() int

MinSize returns minimum size of the node group.

func (*Asg) Nodes

func (asg *Asg) Nodes() ([]string, error)

Nodes returns a list of all nodes that belong to this node group.

func (*Asg) TargetSize

func (asg *Asg) TargetSize() (int, error)

TargetSize returns the current TARGET size of the node group. It is possible that the number is different from the number of nodes registered in Kuberentes.

type AwsCloudProvider

type AwsCloudProvider struct {
	// contains filtered or unexported fields
}

AwsCloudProvider implements CloudProvider interface.

func BuildAwsCloudProvider

func BuildAwsCloudProvider(awsManager *AwsManager, specs []string) (*AwsCloudProvider, error)

BuildAwsCloudProvider builds CloudProvider implementation for AWS.

func (*AwsCloudProvider) Name

func (aws *AwsCloudProvider) Name() string

Name returns name of the cloud provider.

func (*AwsCloudProvider) NodeGroupForNode

func (aws *AwsCloudProvider) NodeGroupForNode(node *apiv1.Node) (cloudprovider.NodeGroup, error)

NodeGroupForNode returns the node group for the given node.

func (*AwsCloudProvider) NodeGroups

func (aws *AwsCloudProvider) NodeGroups() []cloudprovider.NodeGroup

NodeGroups returns all node groups configured for this cloud provider.

type AwsManager

type AwsManager struct {
	// contains filtered or unexported fields
}

AwsManager is handles aws communication and data caching.

func CreateAwsManager

func CreateAwsManager(configReader io.Reader) (*AwsManager, error)

CreateAwsManager constructs awsManager object.

func (*AwsManager) DeleteInstances

func (m *AwsManager) DeleteInstances(instances []*AwsRef) error

DeleteInstances deletes the given instances. All instances must be controlled by the same ASG.

func (*AwsManager) GetAsgForInstance

func (m *AwsManager) GetAsgForInstance(instance *AwsRef) (*Asg, error)

GetAsgForInstance returns AsgConfig of the given Instance

func (*AwsManager) GetAsgNodes

func (m *AwsManager) GetAsgNodes(asg *Asg) ([]string, error)

GetAsgNodes returns Asg nodes.

func (*AwsManager) GetAsgSize

func (m *AwsManager) GetAsgSize(asgConfig *Asg) (int64, error)

GetAsgSize gets ASG size.

func (*AwsManager) RegisterAsg

func (m *AwsManager) RegisterAsg(asg *Asg)

RegisterAsg registers asg in Aws Manager.

func (*AwsManager) SetAsgSize

func (m *AwsManager) SetAsgSize(asg *Asg, size int64) error

SetAsgSize sets ASG size.

type AwsRef

type AwsRef struct {
	Name string
}

AwsRef contains a reference to some entity in AWS/GKE world.

func AwsRefFromProviderId

func AwsRefFromProviderId(id string) (*AwsRef, error)

AwsRefFromProviderId creates InstanceConfig object from provider id which must be in format: aws:///zone/name