Learn Ceph - Ceph in Kubernetes & Cloud Native

#Introduction

Welcome to episode 16 of the Learn Ceph series! Up until episode 15 we treated Ceph as a standalone cluster. Now comes the question that's highly relevant in the modern era: how does Ceph coexist with Kubernetes and the cloud native ecosystem? This episode answers it thoroughly.

Kubernetes is the de facto orchestrator for container workloads, and storage is the most important part of its system. Ceph occupies a unique position because it provides both block and file storage via CSI, and can even run inside Kubernetes itself using operators like Rook.

By the end of this episode you'll understand how to use Ceph CSI for RBD and CephFS, get to know Rook as a storage operator, compare Ceph deployments inside and outside Kubernetes, and understand storage orchestration patterns in cloud native environments. Let's get started.

#Ceph CSI for RBD and CephFS

#Ceph CSI Architecture

The Container Storage Interface (CSI) is a standard that lets Kubernetes provision storage from any vendor. The Ceph CSI driver comes in two flavors: csi-rbd for block volumes and csi-cephfs for shared filesystems. Both drivers run as daemonsets and sidecars in the Kubernetes cluster.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: 3a5f0d12-6b2e-4c8a-9f1e-2d3c4b5a6f70
  pool: rbd-pool
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4

rbd.csi.ceph.com is the provisioner for RBD. The clusterID parameter points to the Ceph cluster's FSID, and pool determines where images are created. This StorageClass can be used directly by a PVC.

#PersistentVolumeClaims for RBD

Once the StorageClass is ready, create a PVC and the volume will be provisioned automatically:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-app
spec:
  storageClassName: ceph-rbd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

ReadWriteOnce means the volume can only be mounted by one node at a time — suitable for RBD. For workloads that need many readers at once, use a CephFS StorageClass with ReadWriteMany.

#CephFS CSI for Shared Storage

The CephFS driver uses the metadata and data pools of an existing Ceph filesystem:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-fs
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: 3a5f0d12-6b2e-4c8a-9f1e-2d3c4b5a6f70
  fsName: myfs
  csi.storage.k8s.io/fstype: ext4

cephfs.csi.ceph.com provisions a CephFS subvolume for each PVC. These volumes can be accessed by many pods at once, making them suitable for applications that share a filesystem.

#Rook and Cloud Native Integration

#What Is Rook

Rook is a Kubernetes operator that automates the deployment, configuration, and operation of Ceph inside a Kubernetes cluster. With Rook, the Ceph cluster configuration is expressed as CustomResourceDefinitions, and the operator handles the rest.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
  storage:
    useAllNodes: true
    useAllDevices: true

CephCluster is Rook's main resource. mon.count: 3 creates three MONs, and useAllDevices: true uses all empty disks for OSDs. The operator translates this spec into Ceph daemons running as pods.

#Rook vs. External Ceph

There are two legitimate patterns:

• Ceph inside Kubernetes (Rook): easy to set up and a good fit for deployments that are entirely on Kubernetes, but storage operations become tied to the cluster's lifecycle.
• External Ceph: a separate Ceph cluster where Kubernetes only uses CSI to connect. Better for large scale and for storage served as a standalone platform.

Choose based on your needs: if storage is the core product, keep it separate; if storage is a complement to your workloads, Rook is a healthy shortcut.

#Deploying Ceph in Kubernetes vs. Externally

#Operational Considerations

Ceph in Kubernetes leverages Kubernetes scheduling and self-healing, but adds a layer of complexity: upgrades must be synchronized, and Kubernetes node failures affect Ceph daemons. External Ceph gives you full control with more classic operations, isolated from the Kubernetes release cycle.

#Resource Considerations

Running Ceph inside Kubernetes means Ceph daemons share resources with other workloads. Because OSDs are sensitive to latency, crowded nodes can degrade performance. Many production deployments use dedicated nodes for storage even when Ceph runs in Kubernetes, to avoid noisy neighbors.

#Storage Orchestration Patterns

#The Operator Pattern

Operators like Rook are an example of the storage orchestration pattern: storage operational logic — provisioning, resize, backup, recovery — is written as code and run automatically by a controller. This reduces repetitive manual work and makes behavior consistent across deployments.

#Separating Control Plane and Data Plane

The best cloud native pattern is separating the control plane (operators, CSI controllers) from the data plane (OSDs). Storage daemons run on specially labeled nodes while control runs anywhere. This way, scaling and maintenance can be done per part without interfering with each other.

kubectl label node node-storage-1 role=storage

kubectl label node labels a dedicated storage node. Operators and schedulers can then ensure OSDs only run on labeled nodes, keeping storage performance quality under control.

#Conclusion

In this episode you've understood Ceph's position in the Kubernetes and cloud native ecosystem: using Ceph CSI for RBD and CephFS through StorageClasses and PVCs, getting to know Rook as a storage operator, comparing Ceph deployments inside and outside Kubernetes, and understanding storage orchestration patterns with operators and control-plane/data-plane separation.

The key takeaways:

• Ceph CSI provides block volumes via RBD and shared filesystems via CephFS.
• StorageClasses and PVCs provision volumes automatically without manual work.
• Rook automates Ceph operations inside Kubernetes through CRDs.
• Ceph in Kubernetes suits symbiosis; external Ceph suits scale and isolation.
• Storage nodes should be specially labeled so OSDs don't share resources with workloads.
• The operator pattern turns storage operations into repeatable, consistent code.

In the next episode, episode 17, we'll cover object lifecycle & S3 features — bucket lifecycle and versioning, object policies and multipart upload, GDPR and data retention features, and integrating object storage with analytics. Time to master the most advanced object storage features!