README ¶
Cluster Autoscaler on AWS
The cluster autoscaler on AWS scales worker nodes within any specified autoscaling group. It will run as a Deployment
in your cluster. This README will go over some of the necessary steps required to get the cluster autoscaler up and running.
Kubernetes Version
Cluster autoscaler must run on v1.3.0 or greater.
Permissions
The worker running the cluster autoscaler will need access to certain resources and actions.
A minimum IAM policy would look like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}
If you'd like to auto-discover node groups by specifing the --node-group-auto-discover
flag, a DescribeTags
permission is also required:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}
AWS supports ARNs for autoscaling groups. More information here.
Deployment Specification
1 ASG Setup (min: 1, max: 10, ASG Name: k8s-worker-asg-1)
kubectl apply -f examples/cluster-autoscaler-one-asg.yaml
Multiple ASG Setup
kubectl apply -f examples/cluster-autoscaler-multi-asg.yaml
Master Node Setup
To run a CA pod in master node - CA deployment should tolerate the master taint
and nodeSelector
should be used to schedule the pods in master node.
kubectl apply -f examples/cluster-autoscaler-run-on-master.yaml
Auto-Discovery Setup
To run a cluster-autoscaler which auto-discovers ASGs with nodes use the --node-group-auto-discovery
flag and tag the ASGs with key k8s.io/cluster-autoscaler/enabled
with value yes
and key kubernetes.io/cluster/<YOUR CLUSTER NAME>
with value true
.
Note that:
kubernetes.io/cluster/<YOUR CLUSTER NAME>
is required whenk8s.io/cluster-autoscaler/enabled
is used across many clusters to prevent ASGs from different clusters recognized as the node groups- There are no
--nodes
flags passed to cluster-autoscaler because the node groups are automatically discovered by tags
kubectl apply -f examples/cluster-autoscaler-autodiscover.yaml
Scaling a node group to 0
From CA 0.6.1 - it is possible to scale a node group to 0 (and obviously from 0), assuming that all scale-down conditions are met.
If you are using nodeSelector
you need to tag the ASG with a node-template key "k8s.io/cluster-autoscaler/node-template/label/"
and "k8s.io/cluster-autoscaler/node-template/taint/"
if you are using taints.
For example for a node label of foo=bar
you would tag the ASG with:
{
"ResourceType": "auto-scaling-group",
"ResourceId": "foo.example.com",
"PropagateAtLaunch": true,
"Value": "bar",
"Key": "k8s.io/cluster-autoscaler/node-template/label/foo"
}
And for a taint of "dedicated": "foo:NoSchedule"
you would tag the ASG with:
{
"ResourceType": "auto-scaling-group",
"ResourceId": "foo.example.com",
"PropagateAtLaunch": true,
"Value": "foo:NoSchedule",
"Key": "k8s.io/cluster-autoscaler/node-template/taint/dedicated"
}
If you'd like to scale node groups from 0, a DescribeLaunchConfigurations
permission is also required:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeTags",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}
Common Notes and Gotchas:
- The
/etc/ssl/certs/ca-certificates.crt
should exist by default on your ec2 instance. - Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. For more information, see https://github.com/kubernetes/contrib/pull/1552#r75532949.
- By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the
--skip-nodes-with-system-pods=false
flag. - By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the
--scale-down-delay-after-add
,--scale-down-delay-after-delete
, and--scale-down-delay-after-failure
flag. E.g.--scale-down-delay-after-add=5m
to decrease the scale down delay to 5 minutes after a node has been added. - If you're running multiple ASGs, the
--expander
flag supports three options:random
,most-pods
andleast-waste
.random
will expand a random ASG on scale up.most-pods
will scale up the ASG that will scheduable the most amount of pods.least-waste
will expand the ASG that will waste the least amount of CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back torandom
.
Documentation ¶
Index ¶
- Constants
- Variables
- func BuildAwsCloudProvider(awsManager *AwsManager, resourceLimiter *cloudprovider.ResourceLimiter) (cloudprovider.CloudProvider, error)
- type Asg
- func (asg *Asg) Autoprovisioned() bool
- func (asg *Asg) Belongs(node *apiv1.Node) (bool, error)
- func (asg *Asg) Create() error
- func (asg *Asg) Debug() string
- func (asg *Asg) DecreaseTargetSize(delta int) error
- func (asg *Asg) Delete() error
- func (asg *Asg) DeleteNodes(nodes []*apiv1.Node) error
- func (asg *Asg) Exist() bool
- func (asg *Asg) Id() string
- func (asg *Asg) IncreaseSize(delta int) error
- func (asg *Asg) MaxSize() int
- func (asg *Asg) MinSize() int
- func (asg *Asg) Nodes() ([]string, error)
- func (asg *Asg) TargetSize() (int, error)
- func (asg *Asg) TemplateNodeInfo() (*schedulercache.NodeInfo, error)
- type AwsManager
- func (m *AwsManager) Cleanup()
- func (m *AwsManager) DeleteInstances(instances []*AwsRef) error
- func (m *AwsManager) GetAsgForInstance(instance *AwsRef) (*Asg, error)
- func (m *AwsManager) GetAsgNodes(asg *Asg) ([]string, error)
- func (m *AwsManager) GetAsgSize(asgConfig *Asg) (int64, error)
- func (m *AwsManager) Refresh() error
- func (m *AwsManager) RegisterAsg(asg *Asg) bool
- func (m *AwsManager) SetAsgSize(asg *Asg, size int64) error
- func (m *AwsManager) UnregisterAsg(asg *Asg) bool
- type AwsRef
Constants ¶
const (
// ProviderName is the cloud provider name for AWS
ProviderName = "aws"
)
Variables ¶
var InstanceTypes = map[string]*instanceType{}/* 122 elements not displayed */
InstanceTypes is a map of ec2 resources
Functions ¶
func BuildAwsCloudProvider ¶
func BuildAwsCloudProvider(awsManager *AwsManager, resourceLimiter *cloudprovider.ResourceLimiter) (cloudprovider.CloudProvider, error)
BuildAwsCloudProvider builds CloudProvider implementation for AWS.
Types ¶
type Asg ¶
type Asg struct { AwsRef // contains filtered or unexported fields }
Asg implements NodeGroup interface.
func (*Asg) Autoprovisioned ¶
Autoprovisioned returns true if the node group is autoprovisioned.
func (*Asg) DecreaseTargetSize ¶
DecreaseTargetSize decreases the target size of the node group. This function doesn't permit to delete any existing node and can be used only to reduce the request for new nodes that have not been yet fulfilled. Delta should be negative. It is assumed that cloud provider will not delete the existing nodes if the size when there is an option to just decrease the target.
func (*Asg) Delete ¶
Delete deletes the node group on the cloud provider side. This will be executed only for autoprovisioned node groups, once their size drops to 0.
func (*Asg) DeleteNodes ¶
DeleteNodes deletes the nodes from the group.
func (*Asg) Exist ¶
Exist checks if the node group really exists on the cloud provider side. Allows to tell the theoretical node group from the real one.
func (*Asg) IncreaseSize ¶
IncreaseSize increases Asg size
func (*Asg) TargetSize ¶
TargetSize returns the current TARGET size of the node group. It is possible that the number is different from the number of nodes registered in Kubernetes.
func (*Asg) TemplateNodeInfo ¶
func (asg *Asg) TemplateNodeInfo() (*schedulercache.NodeInfo, error)
TemplateNodeInfo returns a node template for this node group.
type AwsManager ¶
type AwsManager struct {
// contains filtered or unexported fields
}
AwsManager is handles aws communication and data caching.
func CreateAwsManager ¶
func CreateAwsManager(configReader io.Reader, discoveryOpts cloudprovider.NodeGroupDiscoveryOptions) (*AwsManager, error)
CreateAwsManager constructs awsManager object.
func (*AwsManager) DeleteInstances ¶
func (m *AwsManager) DeleteInstances(instances []*AwsRef) error
DeleteInstances deletes the given instances. All instances must be controlled by the same ASG.
func (*AwsManager) GetAsgForInstance ¶
func (m *AwsManager) GetAsgForInstance(instance *AwsRef) (*Asg, error)
GetAsgForInstance returns AsgConfig of the given Instance
func (*AwsManager) GetAsgNodes ¶
func (m *AwsManager) GetAsgNodes(asg *Asg) ([]string, error)
GetAsgNodes returns Asg nodes.
func (*AwsManager) GetAsgSize ¶
func (m *AwsManager) GetAsgSize(asgConfig *Asg) (int64, error)
GetAsgSize gets ASG size.
func (*AwsManager) Refresh ¶
func (m *AwsManager) Refresh() error
Refresh is called before every main loop and can be used to dynamically update cloud provider state. In particular the list of node groups returned by NodeGroups can change as a result of CloudProvider.Refresh().
func (*AwsManager) RegisterAsg ¶
func (m *AwsManager) RegisterAsg(asg *Asg) bool
RegisterAsg registers an ASG.
func (*AwsManager) SetAsgSize ¶
func (m *AwsManager) SetAsgSize(asg *Asg, size int64) error
SetAsgSize sets ASG size.
func (*AwsManager) UnregisterAsg ¶
func (m *AwsManager) UnregisterAsg(asg *Asg) bool
UnregisterAsg unregisters an ASG.
type AwsRef ¶
type AwsRef struct {
Name string
}
AwsRef contains a reference to some entity in AWS/GKE world.
func AwsRefFromProviderId ¶
AwsRefFromProviderId creates InstanceConfig object from provider id which must be in format: aws:///zone/name