Skip to main content

cluster-autoscaler


Prerequisites

ASG(Auto Scaling Group)

  • Auto Scaling 그룹에 아래 태그가 있어야합니다.
    • kubernetes.io/cluster/<cluster-name>: owned
    • k8s.io/cluster-autoscaler/<cluster-name>: owned
    • k8s.io/cluster-autoscaler/enabled: true
  • 사용자 데이터를 사용하여 Control plane node 연결, node label 설정을 자동화 해줍니다.
eks-asg-example.sh
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=xxx
API_SERVER_URL=xxx
K8S_CLUSTER_DNS_IP=xxx
/etc/eks/bootstrap.sh hhk-eks --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=hhk-eks,alpha.eksctl.io/nodegroup-name=worker-m5-large-ng,role=worker,eks.amazonaws.com/nodegroup-image=ami-0eb873de16468e55e,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=worker-m5-large-ng,eks.amazonaws.com/sourceLaunchTemplateId=lt-0676b3adb0ce8d3cc' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP

--//--

managedNodeGroup

  • EKS 클러스터의 노드 프로비저닝 및 수명 주기 관리 자동화
  • AWS 계정의 최신 Amazon EKS 최적화 AMI를 사용하여 실행
  • 노드 업데이트 및 종료시 노드를 적절하게 비워 애플리케이션을 계속 사용할 수 있도록 관리

nodeGroup(self-managed)

IAM

cluster-autoscaler-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeLaunchTemplateVersions"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
aws iam create-policy \
--policy-name AmazonEKSClusterAutoscalerPolicy \
--policy-document file://cluster-autoscaler-policy.json
eksctl create iamserviceaccount \
--cluster=<my-cluster> \
--namespace=kube-system \
--name=cluster-autoscaler \
--attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AmazonEKSClusterAutoscalerPolicy \
--override-existing-serviceaccounts \
--approve

설치

mkdir -p aws-cluster-autoscaler/base/patches
wget https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml \
-O aws-cluster-autoscaler/base/cluster-autoscaler-autodiscover.yaml
aws-cluster-autoscaler/base/cluster-autoscaler-autodiscover.yaml
# ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
app: cluster-autoscaler
spec:
replicas: 1
selector:
matchLabels:
app: cluster-autoscaler
template:
metadata:
labels:
app: cluster-autoscaler
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8085"
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
spec:
priorityClassName: system-cluster-critical
securityContext:
runAsNonRoot: true
runAsUser: 65534
fsGroup: 65534
serviceAccountName: cluster-autoscaler
containers:
- image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.0
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 600Mi
requests:
cpu: 100m
memory: 600Mi
command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<cluster-name>
- --balance-similar-node-groups
- --skip-nodes-with-system-pods=false
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs/ca-certificates.crt # /etc/ssl/certs/ca-bundle.crt for Amazon Linux Worker Nodes
readOnly: true
imagePullPolicy: "Always"
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs/ca-bundle.crt"
  • cluster-autoscaler.kubernetes.io/safe-to-evict: 'false': 제거 비용이 많이 드는 pod라는 표시로, 해당 pod이 node에 있는 경우 node는 scalie in 되지 않습니다.
aws-cluster-autoscaler/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster-autoscaler-autodiscover.yaml

IAM

mkdir -p aws-cluster-autoscaler/overlays/iam/patches
aws-cluster-autoscaler/overlays/iam/patches/role.yaml
- op: add
path: /metadata/annotations
value:
eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<AmazonEKSClusterAutoscalerRole>
danger

AmazonEKSClusterAutoscalerPolicy policy를 적용한 role의 ARN을 설정해야합니다.

aws-cluster-autoscaler/overlays/iam/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base

patches:
- path: patches/role.yaml
target:
kind: ServiceAccount
kubectl apply -k aws-cluster-autoscaler/overlays/iam

AWS Credential(Optional)

danger

On-premise 환경에서 사용하는 경우 아래와 같은 인증 방식을 사용할 수 있습니다. (보안 위협 요소가 될 수 있음)

mkdir -p aws-cluster-autoscaler/overlays/credentials/patches
aws-cluster-autoscaler/overlays/credentials/credentials.yaml
apiVersion: v1
kind: Secret
metadata:
name: aws-secret
namespace: kube-system
type: Opaque
data:
aws_access_key_id: <BASE64_OF_YOUR_AWS_ACCESS_KEY_ID>
aws_secret_access_key: <BASE64_OF_YOUR_AWS_SECRET_ACCESS_KEY>
aws-cluster-autoscaler/overlays/credentials/patches/env.yaml
- op: add
path: /spec/template/spec/containers/0/env
value:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-secret
key: aws_access_key_id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-secret
key: aws_secret_access_key
- name: AWS_REGION
value: <REGION>
aws-cluster-autoscaler/overlays/credentials/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- credentials.yaml
- ../../base

patches:
- path: patches/env.yaml
target:
kind: Deployment

Troubleshooting

kubectl logs -n kube-system -f deployment.apps/cluster-autoscaler

Reference