Ceph Shared File System(CephFS)

경고

Ceph Shared File System을 사용하면 클라이언트는 메타데이터 관리를 위해 MDS(Metadata Server)와 통신해야하므로 Ceph Rados Block Device(RBD)를 사용하는 것에 비해 성능이 떨어집니다.

생성

Operator 설정

rook-ceph-values.yaml

csi:
  enableCephfsDriver: true

  csiCephFSProvisionerResource: |
    - name : csi-provisioner
      resource: null
    - name : csi-resizer
      resource: null
    - name : csi-attacher
      resource: null
    - name : csi-snapshotter
      resource: null
    - name : csi-cephfsplugin
      resource: null
    - name : liveness-prometheus
      resource: null

  provisionerTolerations: []

  csiCephFSPluginResource: |
    - name : driver-registrar
      resource: null
    - name : csi-cephfsplugin
      resource: null
    - name : liveness-prometheus
      resource: null

  pluginTolerations: []

  kubeletDirPath: /var/lib/kubelet

CephFilesystem 생성

Reference

• Rook / Filesystem Storage Overview
• Rook / CephFilesystem CRD

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
spec:
  metadataPool: {}
  dataPool: []
  metadataServer:
    activeCount: 1
    activeStandby: true

• metadataPool
  • failureDomain: <crushType|osd>
    • 기본 값은 host입니다.
  • deviceClass: <deviceClass>
  • replicated
    • size: <size>
• dataPools: []
  • name: <name>
  • failureDomain: <crushType|osd>
    • 기본 값은 host입니다.
  • deviceClass: <deviceClass>
  • replicated
    • size: <size>
• metadataServer
  • activeCount: <count>
    • active 상태의 MDS의 수입니다.
    • active MDS + standby MDS 두 개가 쌍으로 생성되므로 총 MDS 수는 activeCount * 2 입니다.
  • activeStandby: <bool>
    • standby MDS를 standby_replay MDS로 설정할지 여부입니다.
    • standby_replay는 warm 캐시를 유지하므로 더 빠른 failover를 제공합니다.
  • resources
    • limits.memory설정은 mds_cache_memory_limit설정에 반영됩니다.
  • placement
    • nodeAffinity
    • podAffinity
    • podAntiAffinity
    • tolerations
    • topologySpreadConstraints

Reference

• Rook / FilesystemSubVolumeGroup CRD

apiVersion: ceph.rook.io/v1
kind: CephFilesystemSubVolumeGroup
spec:
  name: csi # 설정 안하면 .metadata.name이 사용됩니다.
  filesystemName: <cephFilesystemName>
  pinning:
    distributed: 1

StorageClass 생성

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
# operator에 CSI_DRIVER_NAME_PREFIX를 설정하지 않았다면 operator가 속한 namespace를 사용합니다.
provisioner: <operatorNamespace>.cephfs.csi.ceph.com
volumeBindingMode: Immediate
reclaimPolicy: Delete
allowVolumeExpansion: true
parameters:
  clusterID: <cephClusterNamespace>
  fsName: <cephFilesystemName>
  pool: <cephFilesystemName>-<cephFilesystem.spec.dataPools.name>
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: <cephClusterNamespace>
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: <cephClusterNamespace>
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: <cephClusterNamespace>

삭제

Reference

• Rook / Cleanup # Force Delete Resources

kubectl annotate cephfilesystemsubvolumegroups <name> \
    rook.io/force-deletion="true" \
    -n rook-ceph