Learn Ceph - Performance Tuning & Data Placement

#Introduction

Welcome to episode 9 of the Learn Ceph series! Your cluster is running with clients connected. Now the question shifts from "does it work" to "how fast and efficient is it". This episode covers performance tuning & data placement — the art of making a Ceph cluster serve workloads faster without adding hardware.

Ceph performance is determined by many layers at once: how OSDs store data on disk, network configuration between nodes, the number of placement groups, and how data is placed based on device classes. Optimizing just one layer is rarely enough; what matters is finding the bottleneck that limits you most.

By the end of this episode you'll understand how to optimize BlueStore storage, separate public and cluster network traffic, tune PGs and device classes, and read throughput and latency metrics to prove the improvements. Let's get started.

#OSD Optimization with BlueStore

#BlueStore Structure and DB/WAL

BlueStore is Ceph's default storage backend, writing data directly to raw devices. Internally, BlueStore uses three areas: data (the actual data blocks), DB (metadata like the object map and omap), and WAL (the write-ahead log for durability). Separating DB/WAL onto faster devices, such as NVMe SSDs, gives big latency improvements.

ceph device ls
ceph osd tree

ceph device ls shows the registered physical devices. In cephadm deployments, DB/WAL separation can be configured at OSD creation time by pointing to a companion device via ceph orch daemon add osd.

#Configuring BlueStore Parameters

Some BlueStore parameters often tuned are the cache size and cache mode:

ceph config set osd bluestore_cache_size_hint 4G
ceph config set osd bluestore_cache_mode write_around

bluestore_cache_size_hint allocates RAM cache per OSD. Values that are too large can trigger OOM because cache is allocated per OSD on a node. Start with 1-4 GB per OSD and monitor memory pressure.

#Practical Storage Tips

A common rule of thumb used by operators: put DB and WAL on SSD/NVMe, data on large HDDs, and make sure no DB device is shared by too many OSDs. For latency-sensitive workloads, consider an all-flash cluster.

#Network Tuning and Public/Private Separation

#Public vs Cluster Network

Ceph distinguishes the public network (clients to MON/OSD/RGW) and the cluster network (internal traffic: replication, heartbeat, recovery, backfill). This separation prevents large backfills from competing with client traffic on the same NIC.

ceph config set mon public_network 192.168.100.0/24
ceph config set mon cluster_network 10.0.0.0/24

public_network and cluster_network can be several comma-separated subnets. After changing them, daemons use the appropriate NIC for each type of traffic.

#Jumbo Frames and Offload

On supporting hardware, enable jumbo frames (MTU 9000) across the entire network path and make sure NIC offload isn't disabled:

ip link set dev enp3s0 mtu 9000

MTU 9000 reduces packet overhead and increases throughput for large transfers. Verify the MTU is consistent across all nodes, because mismatched MTUs actually cause fragmentation and hurt performance.

#Pool Placement Groups, CRUSH Tuning, and Device Classes

#The Number of Placement Groups

The right number of PGs balances rebalance granularity against CPU load. Too few PGs make distribution uneven; too many overload the MON and OSDs during peering.

ceph osd pool autoscale-status
ceph pg stat

ceph osd pool autoscale-status shows the current PG count and the target the autoscaler recommends. For stable workloads, set the pool's pg_autoscale_mode to on and let Ceph adjust.

#Device Classes for Heterogeneous Storage

Device classes let Ceph distinguish HDD, SSD, and NVMe in data placement. This is useful for putting latency-sensitive metadata on SSDs while keeping large data on HDDs:

ceph osd crush class ls
ceph osd crush class list

ceph osd crush class ls shows the existing classes. When an OSD is created, Ceph automatically tags the class based on the device. To place a specific pool only on SSDs, create a new CRUSH rule based on the ssd class and assign it to the pool — details in episode 15.

#Monitoring Throughput, Latency, and Backfill

#Performance Metrics with ceph perf

Ceph provides tools to view daemon performance counters in real time:

ceph osd perf
ceph daemon osd.0 perf dump

ceph osd perf shows commit latency per OSD, while ceph daemon osd.0 perf dump outputs all counters such as osd_op_latency and osd_op_w_process_latency. Both values are early indicators of bottlenecks.

#Throughput with rados bench

For benchmarking, Ceph has the rados bench tool, which writes, reads, and deletes objects in a pool:

rados -p bench-pool bench 60 write --no-cleanup
rados -p bench-pool bench 60 seq

rados bench measures IOPS and bandwidth on the selected pool. Run it while the cluster is idle to get a baseline, then compare after tuning. Don't forget to clean up the benchmark data if it's not needed.

#Observing Backfill Behavior

Backfill is the process of moving data when an OSD enters or leaves. Backfill that's too aggressive can hurt normal performance:

ceph config set osd osd_max_backfills 2
ceph config set osd osd_backfill_full_ratio 0.9

osd_max_backfills limits the number of parallel backfills per OSD. The default of 1 is already conservative; raise it when the cluster is idle and lower it during heavy workloads. Monitor backfill progress with ceph pg dump | grep backfill.

#Conclusion

In this episode you've understood how to tune Ceph cluster performance: optimizing BlueStore with DB/WAL separation and cache, separating public and cluster networks with jumbo frame MTU, adjusting PGs and device classes for heterogeneous storage, and monitoring throughput, latency, and backfill behavior with measurable metrics.

The key takeaways:

• BlueStore separates data, DB, and WAL; put DB/WAL on SSDs for lower latency.
• Separate public and cluster networks so backfill doesn't compete with client traffic.
• The PG autoscaler helps keep placement group counts ideal.
• Device classes allow different data placement between HDD and SSD.
• ceph osd perf and rados bench are the first performance measurement tools.
• Limit osd_max_backfills so recovery doesn't choke normal workloads.

In the next episode, episode 10, we'll cover security & access control — CephX authentication and key management, RBAC for RGW users and capabilities, network security with firewalls and TLS, and secure deployment practices for multi-tenant storage. Get your security-first mindset ready!