Learn Ceph - RADOS & Pool Management

#Introduction

Welcome to episode 4 of the Learn Ceph series! Your cluster is up and healthy. Now it's time to understand the part you'll touch most in daily operations: RADOS pools. Almost every operation — creating RBD images, CephFS filesystems, or RGW buckets — ultimately comes down to pools.

In this episode we'll cover how to create and configure pools, set replication levels, distinguish replicated and erasure coded pools, tune placement groups, and manage the OSD lifecycle: adding, deactivating, and removing.

Good pool management is the key to reliability and performance. Decisions like replica size, PG count, and failure domain will determine how your cluster survives failures.

#Creating and Configuring Pools

#Creating a Replicated Pool

The basic command to create a pool:

ceph osd pool create rbd-pool 128

ceph osd pool create rbd-pool 128 creates a pool named rbd-pool with 128 PGs. The basic syntax takes the PG count as the second argument. For a new pool, also consider setting an application:

ceph osd pool application enable rbd-pool rbd

Setting the pool application helps enable the appropriate features, for example rbd, cephfs, or rgw. It also prevents the pool from being accidentally used by other services.

#Setting the Replication Level

The replication level is controlled by the size parameter (number of copies) and min_size (minimum number of copies for the pool to keep accepting writes):

ceph osd pool set rbd-pool size 3
ceph osd pool set rbd-pool min_size 2

size 3 means every object has three copies. min_size 2 means the pool still accepts writes even if one copy is lost, so the cluster stays usable during recovery. Lowering min_size to 1 risks data loss.

#Pool Types: Replicated vs Erasure Coded

#Replicated Pools

A replicated pool copies every object in full to size OSDs. Its advantages: simple, fast reads and writes, and all operations like snapshots work fully. Its drawback: three times the disk usage for size 3.

Replicated pools are the default choice for RBD and generally for CephFS. If you're not sure what to pick, start with replicated.

#Erasure Coded Pools

An erasure coded (EC) pool splits data into k data chunks and m parity chunks. Only k chunks are needed to read, so disk usage is much more efficient — for example a k=4, m=2 configuration uses only 1.5x capacity.

ceph osd erasure-code-profile set ec42 \
  k=4 m=2 crush-failure-domain=host
ceph osd pool create ec-pool 128 erasure ec42

ceph osd pool create ec-pool 128 erasure ec42 creates an erasure coded pool with the ec42 profile. Note that EC pools have limitations: they don't fully support partial read/partial write operations and aren't yet compatible with all RBD features.

#PG Tuning and Failure Domain

#The Right Number of Placement Groups

The number of PGs determines the granularity of distribution and rebalancing. A practical guideline: for a cluster with total capacity T GB and a target of about 100 PGs per OSD, the total cluster PGs are around T / OSD_size * 100. Modern Ceph has a pg autoscaler that adjusts automatically:

ceph config set global osd_pool_default_pg_autoscale_mode on
ceph osd pool autoscale-status

ceph osd pool autoscale-status shows the estimated ideal PG count per pool and its autoscale mode. With the autoscaler enabled, Ceph automatically increases or decreases a pool's PGs.

#Failure Domain on Pools

The failure domain determines how far apart replicas are placed — host, rack, or osd. For a replicated pool of size 3 with the host failure domain, the three copies will never be on the same host.

ceph osd getcrushmap -o /tmp/map.bin
ceph osd pool get rbd-pool crush_rule

Changing the failure domain is done by creating a new CRUSH rule and assigning it to the pool. We'll dissect CRUSH rules fully in episode 15.

#OSD Lifecycle

#Deactivating and Reactivating OSDs

For maintenance work on an OSD — for example replacing a disk — deactivate the OSD in a controlled way so data is moved out first:

ceph osd out osd.3
systemctl restart ceph-osd@3

ceph osd out osd.3 marks the OSD so its data is gradually moved to other OSDs. This process is called backfill. Once the OSD is healthy again, bring it back with ceph osd in osd.3.

#Removing an OSD

If an OSD is truly broken and must be replaced, remove it from the cluster:

ceph osd out osd.5
ceph osd destroy osd.5 --yes-i-really-mean-it
ceph osd rm osd.5
ceph osd crush remove osd.5
ceph auth del osd.5

This sequence deactivates, destroys, removes from the OSD map and CRUSH, then deletes its key. With cephadm, the simpler approach is ceph orch daemon rm osd.5, which handles all these steps at once.

#Adding a New OSD

Adding a new OSD to replace a broken one or to add capacity:

ceph orch daemon add osd node2:/dev/sdc
ceph osd tree

Once the new OSD is in, Ceph will automatically start rebalancing data across the entire cluster, including the new OSD.

#Conclusion

In this episode you've mastered RADOS pool management: creating replicated and erasure coded pools, setting size and min_size, tuning the number of placement groups with the autoscaler, assigning failure domains, and managing the OSD lifecycle from adding to removing.

The key takeaways:

• A pool is the logical partition where all RADOS objects are stored.
• size determines the number of copies, min_size maintains availability during recovery.
• Replicated pools are simple; erasure coded pools save disk at the cost of CPU overhead.
• The PG autoscaler adjusts the PG count automatically.
• The host failure domain prevents replicas from being on the same node.
• OSDs are managed with out/in, destroy, and ceph orch daemon rm.

In the next episode, episode 5, we'll cover block storage with RBD — creating RBD images, mapping them as block devices, integrating with OpenStack Cinder and Kubernetes CSI, and snapshots, clones, and thin provisioning. Let's turn your pools into storage that's really usable!