본문으로 건너뛰기

Prometheus

Prometheus Operator Architecture

ServiceAccount

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: <name>
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs: ["get", "list", "watch"]
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
apiVersion: v1
kind: ServiceAccount
metadata:
name: <name>
namespace: <namespace>

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: <name>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: <name>
subjects:
- kind: ServiceAccount
name: <name>
namespace: <namespace>

Prometheus

spec에 맞는 prometheus StatefulSets을 생성합니다.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: <name>
namespace: <namespace>
spec:
serviceAccountName: <serviceAccountName>

podMetadata: {}

initContainers:
- name: "prometheus-permission"
image: "busybox"
command: ["/bin/chmod", "-R", "777", "/prometheus"]
volumeMounts:
- name: "prometheus-<name>-db"
mountPath: "/prometheus"

storage:
volumeClaimTemplate:
spec:
storageClassName: gp3
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: 30Gi
  • serviceAccountName: <serviceAccountName>
  • podMetadata
    • labels: {}
    • annotations: {}
  • resources: {}
  • initContainers: []
  • storage
    • volumeClaimTemplate: {}
  • affinity: {}
  • tolerations: []
  • retention: 24h

ServiceMonitor 수집하기

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
matchLabels:
loliot.net/<namespace>/<name>: enabled
  • serviceMonitorNamespaceSelector
    • LabelSelector
    • {}로 설정하면 모든 Namespace에서 ServiceMonitor를 찾습니다.
    • null로 설정하면 Prometheus가 선언된 Namespace에서만 ServiceMonitor를 찾습니다.
  • serviceMonitorSelector
    • LabelSelector
    • {}로 설정하면 모든 ServiceMonitor를 찾습니다.
    • null로 설정하면 ServiceMonitor를 찾지 않습니다.

PodMonitor 수집하기

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchLabels:
loliot.net/<namespace>/<name>: enabled
  • podMonitorNamespaceSelector
    • LabelSelector
    • {}로 설정하면 모든 Namespace에서 PodMonitor를 찾습니다.
    • null로 설정하면 Prometheus가 선언된 Namespace에서만 PodMonitor를 찾습니다.
  • podMonitorSelector
    • LabelSelector
    • {}로 설정하면 모든 PodMonitor를 찾습니다.
    • null로 설정하면 PodMonitor를 찾지 않습니다.

ScrapeConfig 수집하기

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
scrapeConfigNamespaceSelector: {}
scrapeConfigSelector:
matchLabels:
loliot.net/<namespace>/<name>: enabled
  • scrapeConfigNamespaceSelector
    • LabelSelector
    • {}로 설정하면 모든 Namespace에서 ScrapeConfig를 찾습니다.
    • null로 설정하면 Prometheus가 선언된 Namespace에서만 ScrapeConfig를 찾습니다.
  • scrapeConfigSelector
    • LabelSelector
    • {}로 설정하면 모든 ScrapeConfig를 찾습니다.
    • null로 설정하면 ScrapeConfig를 찾지 않습니다.
정보

Scrape 설정 추가를 아래와 같은 방법으로 직접 작성해 넣을 수 있습니다.

apiVersion: v1
kind: Secret
metadata:
name: additional-scrape-configs
namespace: monitoring
stringData:
prometheus-additional.yaml: |
- job_name: 'prometheus-other-cluster-1'
scheme: http
scrape_interval: 23s
scrape_timeout: 23s
honor_labels: true
metrics_path: '/federate'

params:
'match[]':
- '{job=~".+", job!="kublet"}'

static_configs:
- targets:
- 'prometheus-eks-2.monitoring:9090'

- job_name: 'prometheus-other-cluster-2'
scheme: http
scrape_interval: 21s
scrape_timeout: 21s
honor_labels: true
metrics_path: '/federate'

params:
'match[]':
- '{job="kublet"}'

static_configs:
- targets:
- 'prometheus-eks-2.monitoring:9090'
spec:
additionalScrapeConfigs:
name: additional-scrape-configs
key: prometheus-additional.yaml

Alertmanager 연결하기

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
alerting:
alertmanagers:
- namespace: <namespace>
name: <name>
port: <port>
  • alerting
    • alertmanagers: []
      • Alertmanager Service의 정보를 설정합니다.