Learn Ceph - Advanced Pool & CRUSH Strategies

#Introduction

Welcome to episode 15 of the Learn Ceph series! By now you've mastered basic pools and performance tuning. Now we move into more precise strategies: advanced pool & CRUSH strategies — the ability to make use of heterogeneous hardware, isolate workloads, and manage data migration with full control.

Not all storage is created equal. In one cluster, you might have large HDDs for cold data, SSDs for metadata, and NVMe for hot workloads. The CRUSH map is the tool that decides which data may occupy which devices, so each type of hardware is used to its strengths.

By the end of this episode you'll understand how to use device classes, build CRUSH map rules, create tiering and cache pools, isolate performance between workloads, and migrate pools and rebalance safely. Let's get started.

#Device Classes and CRUSH Map Rules

#What Is a Device Class

A device class is a label automatically attached to an OSD based on the device type: hdd, ssd, or nvme. This label becomes the basis for CRUSH rules that place data only on a specific device class:

ceph osd crush class ls
ceph osd tree

ceph osd crush class ls shows the available classes. When a new OSD is created, Ceph detects the device and marks its class automatically. For heterogeneous clusters, make sure every OSD has the correct class.

#Creating Class-Based CRUSH Rules

A CRUSH rule determines how data is selected from the hierarchy. To place a pool only on SSDs:

ceph osd crush rule create-replicated ssd-fast \
  default host ssd
ceph osd pool set pool-ssd crush_rule ssd-fast

ceph osd crush rule create-replicated creates a rule named ssd-fast that uses the host failure domain and the ssd class. Then ceph osd pool set crush_rule assigns that rule to the pool.

#Clusters with Heterogeneous Hardware

With device classes, a single cluster can hold several device types at once without interfering with each other. Metadata pools go on NVMe, hot data pools on SSD, and archive pools on HDD. This is far more efficient than forcing all data to use the same device class.

#Tiering and Cache Pools

#The Cache Tier Concept

Cache tiering places hot data on fast pools and moves it to slow pools once it cools down. Although the classic tiering feature in Ceph is deprecated, the concept remains relevant and can be re-implemented manually with device classes and data-movement policies.

ceph osd pool create hot 128
ceph osd pool set hot crush_rule nvme-fast
ceph osd pool create cold 128
ceph osd pool set cold crush_rule hdd-archive

ceph osd pool set crush_rule points the hot pool to the NVMe rule and the cold pool to the HDD rule. With this scheme, applications can write to the appropriate pool or move data between pools as its temperature changes.

#Manual Tiering Strategy

Manual tiering is done with a routine process: frequently accessed objects are moved to the hot pool, rarely accessed ones to the cold pool. RGW even supports lifecycle rules with StorageClass to move objects between pools automatically — we'll cover this in episode 17.

#Performance Isolation for Different Workloads

#Separate Pools per Workload

Workloads with different characteristics shouldn't share a pool. Databases need consistent latency, while backups tolerate low throughput but eat up capacity. Separating pools gives you control over distinct tuning: replica count, PGs, and device classes for each.

ceph osd pool set pool-db crush_rule nvme-fast
ceph osd pool set pool-backup crush_rule hdd-archive
ceph osd pool set pool-backup size 2

ceph osd pool set size can even differ between pools. The database pool uses full replication and NVMe, while the backup pool uses HDD and fewer replicas to save capacity.

#Monitoring Isolation

To confirm isolation is working, monitor per-pool metrics like latency and IOPS. If database pool latency rises at the same time as backup activity, device isolation isn't strict enough — likely because the CRUSH rule still places both pools on the same devices.

#Pool Migration and Rebalancing

#Adding or Changing a Rule

When hardware is added or removed, you may need to change a pool's rule. This can be done without downtime because Ceph moves data gradually:

ceph osd pool set pool-arsip crush_rule hdd-archive
ceph status

After crush_rule is changed, Ceph starts moving PGs to OSDs matching the new rule. Monitor ceph status until all PGs are active+clean and no backfill is pending.

#Rebalancing After Adding OSDs

When new OSDs join, the cluster rebalances data so all OSDs are used evenly. The rebalance speed can be controlled:

ceph config set osd osd_max_backfills 4
ceph config set osd osd_recovery_max_active 8

osd_max_backfills and osd_recovery_max_active control how aggressively Ceph moves data. For a busy cluster, lower these values during working hours and raise them during a maintenance window.

#Conclusion

In this episode you've understood advanced pool and CRUSH strategies: using device classes to place data on the right hardware, building class-based CRUSH map rules, creating tiering and cache pools for hybrid architectures, isolating performance between workloads with separate pools, and migrating pools and rebalancing with full control.

The key takeaways:

• Device classes (hdd, ssd, nvme) mark OSDs automatically.
• Class-based CRUSH rules direct data to a specific device class.
• Manual tiering moves hot data to fast pools and cold data to slow pools.
• Separate pools per workload enable different tuning and replication.
• Changing crush_rule triggers gradual data migration without downtime.
• Rebalance speed is controlled with osd_max_backfills and recovery limits.

In the next episode, episode 16, we'll cover Ceph in Kubernetes & Cloud Native — Ceph CSI for RBD and CephFS, Rook integration, OpenStack Cinder, and the cloud native ecosystem, comparing Ceph deployed in Kubernetes vs. external Ceph, and storage orchestration patterns with operators. Time to bridge Ceph with the cloud native era!