Learn Ceph - Data Protection & Disaster Recovery

#Introduction

Welcome to episode 14 of the Learn Ceph series! Replication keeps the cluster alive when one node fails, but replication doesn't protect against human error, accidental deletion, or a disaster that destroys an entire site. This episode covers data protection & disaster recovery — the last layer that ensures data can be restored in the worst-case scenario.

Here's the key difference: replication prevents downtime, while backups prevent data loss. A good strategy combines both — replication for availability, snapshots and backups for recovery, and a tested disaster recovery plan for every Ceph interface.

By the end of this episode you'll be able to build a realistic backup strategy, use snapshots and clones correctly, plan disaster recovery for CephFS, RBD, and RGW, and test failover through recovery drills. Let's get started.

#Backup and Restore Strategies

#Healthy Backup Principles

A good backup is defined by three numbers: RPO (Recovery Point Objective), how much data loss you can tolerate; RTO (Recovery Time Objective), how fast you need to recover; and retention, how long backups are kept. Start by setting these three numbers, because they determine your tooling choices.

ceph status
radosgw-admin bucket list

Before building backups, know the assets you're protecting: RGW buckets, RBD images, CephFS filesystems, and the cluster configuration itself. This catalog becomes the basis of the backup list you'll run regularly.

#Backup Tooling Options

There are several common approaches to backing up Ceph:

• Backup at the object level: copy objects from RGW to other storage using rclone or aws s3 sync.
• Backup at the pool level: export pool data using rbd export for RBD images.
• Backup via RBD mirroring: asynchronous replication of RBD images to another cluster or location.
• CephFS backup: use snapshots or rsync to external storage.

rbd export rbd-pool/vol-data /backup/vol-data-20260810.img

rbd export converts an RBD image into a file that can be moved to other storage. Restore is done with rbd import.

#Snapshot and Clone Workflows

#Snapshots as Point-in-Time Backups

Snapshots are the foundation of cheap point-in-time backups. For RBD, snapshots are created per image; for CephFS, per directory through the .snap mechanism; for RGW, use bucket versioning to keep object versions.

rbd snap create rbd-pool/vol-data@backup-harian
mkdir /mnt/myfs/.snap/backup-harian

rbd snap create creates an RBD snapshot, and mkdir .snap/backup-harian creates a CephFS snapshot. Both are nearly free in capacity because they only store changes since the previous snapshot.

#Clones for Restore and Testing

Cloning from a snapshot lets you restore without touching the original data — very useful for testing before actually restoring:

rbd snap protect rbd-pool/vol-data@backup-harian
rbd clone rbd-pool/vol-data@backup-harian rbd-pool/vol-verifikasi
rbd map rbd-pool/vol-verifikasi

rbd clone creates a new image that shares data with the snapshot. Mount it and verify its contents, then discard it once confirmed good. This is the simplest drill proving your backups can be used.

#Recovery from Snapshots

For a direct restore, roll the image back to the snapshot state:

rbd snap rollback rbd-pool/vol-data@backup-harian

rbd snap rollback returns the image to the snapshot state. Remember that a rollback overwrites the current state, so only do it once you're sure that's what you want, or work from a clone first.

#Disaster Recovery for CephFS, RBD, and RGW

#DR Plans per Interface

Each interface has a different recovery path:

• CephFS: restore from snapshots to a directory, or recover the filesystem from intact data and metadata pools.
• RBD: import the exported file, or activate the image on a secondary site if you use RBD mirroring.
• RGW: synchronize from a secondary site, or restore objects from bucket versions or external backups.

rbd import /backup/vol-data-20260810.img rbd-pool/vol-data

rbd import restores an image from the file produced by rbd export. Store backup files in a location physically separate from the primary cluster so they survive a disaster too.

#Prioritizing the Recovery Order

When a disaster happens, recover in sequence: first the cluster configuration and MONs, then data and metadata, then client services. Don't rush to restore services before the data foundation is confirmed healthy — a hasty restore often produces corrupt data you only notice later.

#Testing Failover and Recovery Drills

#Regular Restore Drills

A DR plan that's never tested is just a document. Schedule drills regularly — for example quarterly — with realistic scenarios: restore one RBD image, one CephFS filesystem, and one RGW bucket into a test environment, then verify data integrity.

rbd info rbd-pool/vol-data
radosgw-admin bucket stats --bucket=bucket-pertama

rbd info and radosgw-admin bucket stats verify that the restore result has the right size and object count. Record the results of each drill and fix any procedure that fails.

#Failover Tests for Multisite

If you use multisite from episode 13, test failover by actually moving the endpoint to the secondary site for a few hours. Observe whether writes work, whether data synchronizes after failback, and how much lag occurs. These test results give you a realistic RTO.

#Conclusion

In this episode you've understood how to protect Ceph data across various scenarios: building a backup strategy with clear RPO and RTO, using snapshots and clones for point-in-time recovery, planning disaster recovery for CephFS, RBD, and RGW, and testing failover through regular recovery drills.

The key takeaways:

• Replication prevents downtime; backups prevent data loss.
• Set your RPO, RTO, and retention before choosing backup tooling.
• Snapshots are nearly free in capacity and form the basis of daily backups.
• Cloning from a snapshot lets you verify restores without touching the original data.
• The DR plan must cover CephFS, RBD, and RGW with a clear recovery order.
• Regular recovery drills are the only way to prove backups actually work.

In the next episode, episode 15, we'll cover advanced pool & CRUSH strategies — device classes and CRUSH map rules for heterogeneous clusters, tiering and cache pools for hybrid HDD/SSD architectures, performance isolation for different workloads, and pool migration and rebalancing. Time to move up to the most precise tuning level!