Ceph Object Gateway, S3 호환 Object Storage

Operator 설정

rook-ceph-values.yaml

rbacAggregate:
  enableOBCs: true

CephObjectStore 생성

Reference

• Rook / Object Storage Overview

Local Object Store

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
spec:
  metadataPool:
  dataPool:
  gateway:
    port: 9000
    instances: 1
    priorityClassName: system-cluster-critical

• metadataPool
  • failureDomain: <crushType|osd>
    • 기본 값은 host입니다.
  • replicated
    • size: <size>
• dataPool
  • failureDomain: <crushType|osd>
    • 기본 값은 host입니다.
  • replicated
    • size: <size>
• gateway
  • port: <port>
  • instances: <replicas>
  • resources
  • placement
    • nodeAffinity
    • podAffinity
    • podAntiAffinity
    • tolerations
    • topologySpreadConstraints

External Object Store

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
spec:
  gateway:
    port: 9000
    externalRgwEndpoints:
      - ip: <objectStoreIP>
        # hostname: <objectStoreHostname>

• gateway
  • port: <port>
  • externalRgwEndpoints: []
    • ip 또는 hostname을 지정합니다.
    • ip: <objectStoreIP>
    • hostname: <objectStoreHostname>

노트

externalRgwEndpoints의 첫번째 ip 또는 hostname을 참조하는 리소스들이 많으므로, Provider Storage Cluster에 Object Store에 대한 LoadBalancer 또는 NodePort 타입의 Service를 만들어서 하나만 설정하는 것을 권장합니다.

StorageClass 생성

apiVersion: storage.k8s.io/v1
kind: StorageClass
provisioner: <cephClusterNamespace>.ceph.rook.io/bucket
volumeBindingMode: Immediate
reclaimPolicy: Delete
parameters:
  objectStoreName: <cephObjectStoreName>
  objectStoreNamespace: <cephObjectStoreNamespace>

경고

문서상에 provisioner로 <operatorNamespace>.ceph.rook.io/bucket을 사용하라고 되어 있지만, <cephClusterNamespace>.ceph.rook.io/bucket을 사용해야 작동합니다.

ObjectBucketClaim 생성

Reference

• Rook / Object Storage / Overview
• Rook / Object Storage / Bucket Claim

apiVersion: v1
kind: Secret
metadata:
  name: rgw-admin-ops-user
  namespace: <cephClusterNamespace>
type: kubernetes.io/rook
data:
  accessKey: <accessKey | base64>
  secretKey: <secretKey | base64>

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
spec:
  generateBucketName: <bucketNamePrefix>
  storageClassName: <storageClassName>

• bucketName: <bucketName>
• generateBucketName: <bucketNamePrefix>
  • 동일한 bucketName에 대한 충돌을 방지하려는 경우 권장됩니다.
• storageClassName: <storageClassName>
• additionalConfig
  • maxObjects: "1000"
  • maxSize: "2G"

정보

ObjectBucketClaim namespace와 name이 동일한 ConfigMap, Secret이 생성됩니다. ConfigMap에는 BUCKET_HOST, BUCKET_NAME, BUCKET_PORT, BUCKET_REGION, BUCKET_SUBREGION이 저장되어 있고, Secret에는 AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY가 저장되어 있습니다.

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: s5cmd
      image: docker.io/peakcom/s5cmd:v2.3.0
      imagePullPolicy: IfNotPresent
      command: [sleep, infinity]
      envFrom:
        - secretRef:
            name: <objectBucketClaimName>
        - configMapRef:
            name: <objectBucketClaimName>

/s5cmd --endpoint-url http://$BUCKET_HOST:$BUCKET_PORT ls