Skip to main content

cluster-autoscaler


사전 요구 사항​

ASG(Auto Scaling Group)​

  • Auto Scaling 그룹에 아래 태그가 있어야합니다.
    • kubernetes.io/cluster/<cluster-name>: owned
    • k8s.io/cluster-autoscaler/<cluster-name>: owned
    • k8s.io/cluster-autoscaler/enabled: true
  • 사용자 데이터를 사용하여 Control plane node 연결, node label 설정을 자동화 해줍니다.
eks-asg-example.sh
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=xxx
API_SERVER_URL=xxx
K8S_CLUSTER_DNS_IP=xxx
/etc/eks/bootstrap.sh hhk-eks --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=hhk-eks,alpha.eksctl.io/nodegroup-name=worker-m5-large-ng,role=worker,eks.amazonaws.com/nodegroup-image=ami-0eb873de16468e55e,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=worker-m5-large-ng,eks.amazonaws.com/sourceLaunchTemplateId=lt-0676b3adb0ce8d3cc' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP

--//--

managedNodeGroup​

  • EKS 클러스터의 노드 프로비저닝 및 수명 주기 관리 자동화
  • AWS 계정의 최신 Amazon EKS 최적화 AMI를 사용하여 실행
  • 노드 업데이트 및 종료시 노드를 적절하게 비워 애플리케이션을 계속 사용할 수 있도록 관리

nodeGroup(self-managed)​

IAM​

cluster-autoscaler-policy.json
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"autoscaling:DescribeAutoScalingGroups",
				"autoscaling:DescribeAutoScalingInstances",
				"autoscaling:DescribeLaunchConfigurations",
				"autoscaling:DescribeTags",
				"autoscaling:SetDesiredCapacity",
				"autoscaling:TerminateInstanceInAutoScalingGroup",
				"ec2:DescribeLaunchTemplateVersions"
			],
			"Resource": "*",
			"Effect": "Allow"
		}
	]
}
aws iam create-policy \
    --policy-name AmazonEKSClusterAutoscalerPolicy \
    --policy-document file://cluster-autoscaler-policy.json
eksctl create iamserviceaccount \
  --cluster=<my-cluster> \
  --namespace=kube-system \
  --name=cluster-autoscaler \
  --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AmazonEKSClusterAutoscalerPolicy \
  --override-existing-serviceaccounts \
  --approve

설치​

mkdir -p aws-cluster-autoscaler/base/patches
wget https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml \
    -O aws-cluster-autoscaler/base/cluster-autoscaler-autodiscover.yaml
aws-cluster-autoscaler/base/cluster-autoscaler-autodiscover.yaml
# ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8085"
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
    spec:
      priorityClassName: system-cluster-critical
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 65534
      serviceAccountName: cluster-autoscaler
      containers:
        - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.0
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 600Mi
            requests:
              memory: 600Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<cluster-name>
            - --balance-similar-node-groups
            - --skip-nodes-with-system-pods=false
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs/ca-certificates.crt # /etc/ssl/certs/ca-bundle.crt for Amazon Linux Worker Nodes
              readOnly: true
          imagePullPolicy: "Always"
      volumes:
        - name: ssl-certs
          hostPath:
            path: "/etc/ssl/certs/ca-bundle.crt"
  • cluster-autoscaler.kubernetes.io/safe-to-evict: 'false': 제거 비용이 많이 드는 pod라는 표시로, 해당 pod이 node에 있는 경우 node는 scalie in 되지 않습니다.
aws-cluster-autoscaler/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - cluster-autoscaler-autodiscover.yaml

IAM​

mkdir -p aws-cluster-autoscaler/overlays/iam/patches
aws-cluster-autoscaler/overlays/iam/patches/role.yaml
- op: add
  path: /metadata/annotations
  value:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<AmazonEKSClusterAutoscalerRole>
warning

AmazonEKSClusterAutoscalerPolicy policy를 적용한 role의 ARN을 설정해야합니다.

aws-cluster-autoscaler/overlays/iam/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base

patches:
  - path: patches/role.yaml
    target:
      kind: ServiceAccount
kubectl apply -k aws-cluster-autoscaler/overlays/iam

AWS Credential(Optional)​

warning

On-premise 환경에서 사용하는 경우 아래와 같은 인증 방식을 사용할 수 있습니다. (보안 위협 요소가 될 수 있음)

mkdir -p aws-cluster-autoscaler/overlays/credentials/patches
aws-cluster-autoscaler/overlays/credentials/credentials.yaml
apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: kube-system
type: Opaque
data:
  aws_access_key_id: <BASE64_OF_YOUR_AWS_ACCESS_KEY_ID>
  aws_secret_access_key: <BASE64_OF_YOUR_AWS_SECRET_ACCESS_KEY>
aws-cluster-autoscaler/overlays/credentials/patches/env.yaml
- op: add
  path: /spec/template/spec/containers/0/env
  value:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: aws-secret
          key: aws_access_key_id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: aws-secret
          key: aws_secret_access_key
    - name: AWS_REGION
      value: <REGION>
aws-cluster-autoscaler/overlays/credentials/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - credentials.yaml
  - ../../base

patches:
  - path: patches/env.yaml
    target:
      kind: Deployment

Troubleshooting​

kubectl logs -n kube-system -f deployment.apps/cluster-autoscaler

Reference​