본문으로 건너뛰기

Prometheus

Prometheus Operator Architecture

ServiceAccount​

Reference
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: <name>
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/metrics
      - services
      - endpoints
      - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - configmaps
    verbs: ["get"]
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - networking.k8s.io
    resources:
      - ingresses
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
apiVersion: v1
kind: ServiceAccount
metadata:
  name: <name>
  namespace: <namespace>

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: <name>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: <name>
subjects:
  - kind: ServiceAccount
    name: <name>
    namespace: <namespace>

Prometheus​

Reference

spec에 맞는 prometheus StatefulSets을 생성합니다.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: <name>
  namespace: <namespace>
spec:
  serviceAccountName: <serviceAccountName>

  podMetadata: {}

  initContainers:
    - name: "prometheus-permission"
      image: "busybox"
      command: ["/bin/chmod", "-R", "777", "/prometheus"]
      volumeMounts:
        - name: "prometheus-<name>-db"
          mountPath: "/prometheus"

  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: gp3
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 30Gi
  • serviceAccountName: <serviceAccountName>
  • podMetadata
    • labels: {}
    • annotations: {}
  • resources: {}
  • initContainers: []
  • storage
    • volumeClaimTemplate: {}
  • affinity: {}
  • tolerations: []
  • retention: 24h

ServiceMonitor 수집하기​

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      loliot.net/<namespace>/<name>: enabled
  • serviceMonitorNamespaceSelector
    • LabelSelector
    • {}로 설정하면 모든 Namespace에서 ServiceMonitor를 찾습니다.
    • null로 설정하면 Prometheus가 선언된 Namespace에서만 ServiceMonitor를 찾습니다.
  • serviceMonitorSelector
    • LabelSelector
    • {}로 설정하면 모든 ServiceMonitor를 찾습니다.
    • null로 설정하면 ServiceMonitor를 찾지 않습니다.

PodMonitor 수집하기​

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      loliot.net/<namespace>/<name>: enabled
  • podMonitorNamespaceSelector
    • LabelSelector
    • {}로 설정하면 모든 Namespace에서 PodMonitor를 찾습니다.
    • null로 설정하면 Prometheus가 선언된 Namespace에서만 PodMonitor를 찾습니다.
  • podMonitorSelector
    • LabelSelector
    • {}로 설정하면 모든 PodMonitor를 찾습니다.
    • null로 설정하면 PodMonitor를 찾지 않습니다.

ScrapeConfig 수집하기​

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
  scrapeConfigNamespaceSelector: {}
  scrapeConfigSelector:
    matchLabels:
      loliot.net/<namespace>/<name>: enabled
  • scrapeConfigNamespaceSelector
    • LabelSelector
    • {}로 설정하면 모든 Namespace에서 ScrapeConfig를 찾습니다.
    • null로 설정하면 Prometheus가 선언된 Namespace에서만 ScrapeConfig를 찾습니다.
  • scrapeConfigSelector
    • LabelSelector
    • {}로 설정하면 모든 ScrapeConfig를 찾습니다.
    • null로 설정하면 ScrapeConfig를 찾지 않습니다.
정보

Scrape 설정 추가를 아래와 같은 방법으로 직접 작성해 넣을 수 있습니다.

apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
  namespace: monitoring
stringData:
  prometheus-additional.yaml: |
    - job_name: 'prometheus-other-cluster-1'
      scheme: http
      scrape_interval: 23s
      scrape_timeout: 23s
      honor_labels: true
      metrics_path: '/federate'

      params:
        'match[]':
          - '{job=~".+", job!="kublet"}'

      static_configs:
        - targets:
          - 'prometheus-eks-2.monitoring:9090'

    - job_name: 'prometheus-other-cluster-2'
      scheme: http
      scrape_interval: 21s
      scrape_timeout: 21s
      honor_labels: true
      metrics_path: '/federate'

      params:
        'match[]':
          - '{job="kublet"}'

      static_configs:
        - targets:
          - 'prometheus-eks-2.monitoring:9090'
spec:
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

Alertmanager 연결하기​

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
  alerting:
    alertmanagers:
      - namespace: <namespace>
        name: <name>
        port: <port>
  • alerting
    • alertmanagers: []
      • Alertmanager Service의 정보를 설정합니다.