Learn OpenStack - Production Storage: Ceph Integration (RBD, RGW, CephFS)

#Introduction

In episode 8, you met Cinder with LVM and Ceph backends. In episode 14, production deployment was orchestrated by Kolla-Ansible. Episode 16 brings the two together on the layer that most determines cluster reliability: storage. In production, OpenStack's answer for storage isn't three separate clusters — it's one: Ceph.

Episode 16 covers why Ceph has become the dominant choice, how it provides block, object, and filesystem from a single cluster (unified storage), how to integrate it with Glance, Cinder, and Nova on RBD, and the sizing and deployment principles for a minimum three-node cluster.

#Why Ceph is the Best Backend for Production

#Unified Storage in a Single Cluster

Ceph provides three storage services from the same single cluster:

• RBD (RADOS Block Device): block storage for Cinder, Glance, and Nova.
• RGW (RADOS Gateway): object storage with the S3 API, a Swift replacement.
• CephFS: a shared filesystem for Manila.

Ceph Cluster (single RADOS pool)
   ├── RBD   → Cinder, Glance, Nova (block)
   ├── RGW   → Object storage S3 API (object)
   └── CephFS → Manila shared filesystem (file)

One infrastructure for three needs means one capacity to manage, one replication mechanism, and one team that understands it.

#Reliability and Self-Healing

Ceph is designed to be distributed and self-healing: every piece of data is replicated to several OSDs, data is spread evenly (CRUSH algorithm), and when one OSD fails, the cluster automatically reconstructs replicas elsewhere without downtime. That's what makes Ceph and OpenStack the standard production pairing.

#Integrating Ceph with OpenStack

#Glance Image Store on Ceph RBD

Putting Glance images on Ceph makes images directly available on all nodes — Nova doesn't need to download an image to local disk every time:

glance_backend_ceph: "yes"
glance_ceph_pool: "images"

glance_ceph_pool: "images" points to the Ceph pool that stores images. With this, cloning an image into an instance disk is done by Ceph instantly — no large file transfers between nodes.

#Cinder Volume Backend on Ceph RBD

The primary Cinder backend in production is RBD:

openstack volume service list -f table
openstack volume type list -f value -c Name

The output of openstack volume service list shows the backend state. Cinder creates volumes as RBD images in the Ceph pool and attaches them to instances without iSCSI — the instance maps RBD directly through librbd.

#Nova Ephemeral Disk on Ceph RBD

Instances using Ceph no longer need a large local disk — the instance's ephemeral disk lives on RBD. Migrating instances between hosts becomes easy because the disk can be opened anywhere:

nova_ceph_pool: "vms"
nova_images_type: "rbd"
cinder_backend_ceph: "yes"
cinder_ceph_pool: "volumes"

nova_images_type: "rbd" makes the instance's ephemeral disk live in the vms pool. Live migration and resizing between hosts run smoothly because there's no physical disk transfer.

#Ceph Cluster Sizing & Deployment

#Cluster Components

A production Ceph cluster minimally consists of:

Verify cluster status from a Ceph node:

ceph -s
ceph osd tree

The output of ceph -s shows a cluster health summary — the HEALTH_OK column is the target. The ceph osd tree command shows each OSD and its state.

#Sizing Principles

Ceph sizing rules of thumb:

• Replication: at least 3 replicas, or 2 replicas plus erasure coding for space efficiency.
• Capacity: total usable capacity is roughly a third of raw capacity at 3x replication.
• Network: separate the cluster network (OSD-to-OSD replication) from the public network (client traffic) — the same dual-NIC pattern as episode 15.

#Deployment via Kolla-Ansible

Ceph can be deployed as part of the Kolla-Ansible stack itself:

enable_ceph: "yes" > /etc/kolla/globals.yml
kolla-ansible -i multinode deploy

With enable_ceph: "yes", Kolla-Ansible deploys monitors, managers, and OSDs as containers, then configures Glance, Cinder, and Nova to use the RBD pools — integrating the entire storage layer in one deployment process.

#Summary

Episode 16 brings production storage together in a single architecture: Ceph as unified storage providing RBD for block, RGW for object, and CephFS for shared filesystem in one cluster, integrated with Glance, Cinder, and Nova on RBD pools, plus a minimum three-node cluster of OSDs, monitors, and managers.

Key takeaways:

• Ceph provides block, object, and filesystem in one cluster.
• Glance, Cinder, and Nova use RBD for images, volumes, and disks.
• RBD eliminates large disk transfers and eases migrations.
• Minimum cluster: 3 monitors, 2-3 managers, 3+ OSDs.
• ceph -s should show HEALTH_OK.
• Separate the cluster network from the public network for replication performance.

In episode 17, we'll cover Security Hardening, TLS & RBAC Policies — securing API endpoints with TLS and HAProxy SSL termination, separating internal and public APIs, customizing policy.yaml for granular RBAC, and protecting the metadata service and preventing spoofing with port security.