Learn Ceph - High Availability & Fault Tolerance

#Introduction

Welcome to episode 12 of the Learn Ceph series! After observability in episode 11, it's time to make sure your cluster can survive when something bad happens. This episode covers high availability & fault tolerance — the design and operations that keep Ceph serving even when a component fails.

Ceph's core philosophy is that no single component is a point of failure. MON, MGR, MDS, and RGW are all designed for redundancy, and data is replicated across different failure domains. But high availability isn't just about design — it's also about how you respond to failures when they happen.

By the end of this episode you'll understand how to make components redundant, set up CRUSH failure domains correctly, handle OSD failure and disk replacement, and understand heal, backfill, and scrub operations. Let's get started.

#Component Redundancy

#MON Redundancy

The MON maintains cluster state and needs a majority quorum. With three MONs, the cluster keeps working as long as at most one MON is down. Redundancy is arranged by placing MONs on different hosts:

ceph orch apply mon node1,node2,node3
ceph mon stat

ceph orch apply mon places one MON per host. ceph mon stat shows the MON list and whether a quorum has formed. Never put two MONs on the same host, because one dead host could immediately destroy the quorum.

#MGR, MDS, and RGW Redundancy

The MGR runs in active-standby mode: one active MGR serves the dashboard and modules while the rest stand by, ready to take over. Multiple active MDSs can run at once for CephFS (active-active, see episode 6). RGW is stateless, so it's easy to scale and place behind a load balancer.

ceph orch ps
ceph mgr stat
ceph fs status myfs

ceph orch ps shows all daemons along with their hosts and status. Make sure there's at least one standby MGR, a standby MDS matching the rank, and more than one RGW if object storage traffic matters.

#CRUSH Failure Domains and Isolation

#Failure Domains at the Host Level

The failure domain defines the minimum distance between replicas. Ceph's default uses host: two replicas will never occupy the same host. For larger scale, raise it to rack or even datacenter:

ceph osd crush rule dump replicated_rule
ceph osd tree

ceph osd crush rule dump shows the rule structure including type host. ceph osd tree shows the actual hierarchy. If all the cluster's OSDs are in a single rack, a rack failure domain provides no extra protection.

#Organizing the CRUSH Map Hierarchy

To make use of failure domains at a higher level, build the right hierarchy in the CRUSH map:

ceph osd crush add-bucket rack-a rack
ceph osd crush move node1 rack=rack-a
ceph osd crush move node2 rack=rack-a

ceph osd crush add-bucket creates a bucket of type rack, and ceph osd crush move attaches hosts to it. Once the hierarchy is tidy, create a new rule that uses the rack failure domain so replicas spread across racks.

#Device Isolation

Isolation also applies at the device level: make sure replicas don't land on disks that physically share the same enclosure. On sensitive hardware, combine the host hierarchy with additional buckets like chassis so two disks in one chassis never carry the same replica.

#Handling OSD Failure and Disk Replacement

#Early Detection and Response

When an OSD goes down, Ceph automatically marks it and starts moving data to maintain the replica count. The first response is understanding the cause: check the logs, check the disk, and check the hardware:

ceph osd tree
ceph osd perf
journalctl -u ceph-osd@3 --since "30 min ago"

ceph osd tree shows which OSDs are down. Read the relevant OSD log to see whether this is a disk, network, or process issue. Don't remove an OSD before knowing the cause.

#Replacing Disks Safely

If a disk is broken, replace it with a controlled procedure. For an OSD that can still be decommissioned gracefully:

ceph osd out osd.3
ceph osd wait-for-active-clean --timeout 1800

ceph osd out marks the OSD so its data is moved. Wait until the cluster is active+clean again before shutting the OSD down. For an OSD that's completely unreachable, go straight to the destroy process and replace the disk in the hardware.

#Adding a Replacement OSD

Once the new disk is installed, add a new OSD. cephadm detects new devices and can deploy automatically if --all-available-devices is active, or manually:

ceph orch daemon add osd node2:/dev/sdd
ceph osd tree

The new OSD joins with a new ID and the cluster starts rebalancing. No operation requires stopping service — that's the benefit of well-designed failure domains and replication.

#Heal, Backfill, and Scrub Operations

#Heal and Self-Healing

The self-healing concept is at the core of Ceph: when an OSD fails, the cluster detects the replica shortfall and immediately starts recovery without human intervention. The process runs gradually so it doesn't overwhelm the cluster, and you can monitor it through PG status:

ceph pg stat
ceph -w

ceph -w shows real-time events including backfill and recovery progress. Note the terms recovering, backfilling, and degraded — all normal while the cluster is healing itself.

#Backfill vs. Recovery

Backfill is refilling data onto OSDs that are new or far behind, while recovery is synchronizing inconsistent objects on existing OSDs. Both share bandwidth, so settings like osd_max_backfills from episode 9 remain relevant.

#Scrub and Deep Scrub

Scrub is a periodic verification that compares metadata and data across replicas to detect silent corruption. Deep scrub reads all physical data to verify checksums. Schedule scrubs regularly:

ceph config set osd osd_scrub_begin_hour 2
ceph config set osd osd_scrub_end_hour 5
ceph osd deep-scrub all

osd_scrub_begin_hour and osd_scrub_end_hour limit scrubbing to quiet hours. ceph osd deep-scrub all triggers a manual deep scrub when you suspect data corruption.

#Conclusion

In this episode you've understood how to make a Ceph cluster resistant to failure: ensuring MON, MGR, MDS, and RGW redundancy, setting up CRUSH failure domains at the host and rack level, handling OSD failure with controlled disk replacement, and understanding heal, backfill, and scrub operations to maintain data integrity.

The key takeaways:

• The MON needs a majority quorum; place them on different hosts.
• The MGR is active-standby, MDS is active-active, and RGW is stateless and easy to scale.
• The failure domain defines the minimum distance between replicas.
• Take an OSD out with ceph osd out and wait for active+clean before replacing the disk.
• Backfill fills new OSDs, recovery synchronizes inconsistent objects.
• Scrub and deep scrub protect data integrity against silent corruption.

In the next episode, episode 13, we'll cover multi-site & geo-replication — RGW multisite architecture and data replication, configuring zonegroups, routes, and failover, use cases for global object storage, and consistency and cross-site synchronization models. Get ready to take your cluster to the geographic level!