Learn Ceph - Multi-site & Geo-replication

#Introduction

Welcome to episode 13 of the Learn Ceph series! Everything we've covered so far runs in a single physical location. But what if data has to be available in two data centers, or even on two continents? This episode covers multi-site & geo-replication — Ceph's ability to synchronize data across geographically separated sites.

RGW supports a full multisite configuration: multiple Ceph clusters in different locations asynchronously replicate object storage data between them. This enables failover to another site when one location suffers a disaster, while also bringing data closer to global users.

By the end of this episode you'll understand RGW multisite architecture, how to configure zonegroups and zones, failover and route logic, global object storage use cases, and the consistency models that apply. Let's get started.

#RGW Multisite Architecture

#The Realm, Zonegroup, and Zone Hierarchy

Multisite uses the hierarchy you already know from episode 7, but now with clearer roles. A realm unites all sites under one global namespace. A zonegroup is a collection of replicated zones, and a zone is the physical unit where data is actually stored.

realm: global
  zonegroup: primary
    zone: site-a (aktif untuk write)
    zone: site-b (replikasi, read-only)

In the configuration above, site-a accepts writes and site-b receives copies of the data. This structure is the active-passive model, the most common way to start.

#Metadata and Data Sync

Multisite synchronization is split in two: data sync copies bucket contents between zones, while metadata sync synchronizes users, buckets, and policies. Both run continuously and can be monitored with radosgw-admin sync status.

radosgw-admin sync status
radosgw-admin bucket sync status --bucket=bucket-pertama

radosgw-admin sync status shows the data and metadata synchronization progress between zones. The lag on these numbers indicates how far the data copy trails the primary site.

#Configuring Zonegroups and Zones

#Creating a Realm and Zonegroup

The first step in building a multisite configuration is creating a realm that unites both sites:

radosgw-admin realm create --rgw-realm=global --default
radosgw-admin zonegroup create --rgw-zonegroup=primary \
  --master --default

radosgw-admin zonegroup create creates a zonegroup with the master and default roles. The master zonegroup holds the source-of-truth metadata for the global namespace.

#Setting Up Zones in Each Site

In each site, create a zone and commit a period to synchronize the configuration:

radosgw-admin zone create --rgw-zonegroup=primary \
  --rgw-zone=site-a --master --default
radosgw-admin period update --commit

radosgw-admin zone create defines a zone within the zonegroup. radosgw-admin period update --commit publishes the latest configuration to all connected sites.

#Connecting the Second Site

The second site is created using the configuration from the first, usually by exporting and importing the period. Once both zones know each other, RGW on each site is restarted so it uses the new configuration. This process is detailed, so it's best to follow the official guide — the order in which realms, zones, and periods are created determines success.

#Routes, Failover, and Global Use Cases

#Endpoints and Routing

Each zone has its own S3 endpoint. Clients use the endpoint matching their region or location. For failover, point DNS at the secondary site's endpoint when the primary site has problems — for example, with a health check that moves the record to another site.

radosgw-admin zone placement modify \
  --rgw-zone=site-a --placement-id=default-placement \
  --endpoints=https://rgw-a.example.com

radosgw-admin zone placement modify sets the public endpoint for a zone. Consistency between the endpoint in the configuration and DNS must be maintained so S3 redirects work correctly.

#Global Object Storage Use Cases

Multisite configuration suits services that need global availability: media streaming that serves objects from the nearest site, backups that copy data to another site, and data lakes that must be reachable from multiple regions. Most importantly, multisite protects against a disaster in one physical location.

#Active-Active Mode

For workloads that need writes in more than one site, Ceph supports an active-active model where conflicts are resolved deterministically based on timestamps. This mode is more complex because it requires versioning and conflict resolution handling, so it's only recommended once you truly understand your data's characteristics.

#Consistency Models and Cross-Site Synchronization

#Eventual Consistency

Multisite synchronization is asynchronous and eventually consistent: a write on site-a may not be immediately visible on site-b. There's lag that depends on bandwidth between sites and workload. For many object storage use cases this lag is acceptable because data is immutable or rarely changes.

radosgw-admin replication log-status
radosgw-admin data sync status

radosgw-admin replication log-status shows the replication log status per shard. If lag keeps growing, check the bandwidth between sites and the replication log sharding configuration.

#Trade-offs to Understand

Eventual consistency means applications must be ready to read data that's slightly stale on the secondary site. For data that needs strong consistency, write to the primary site and read from the same site, or accept a latency trade-off for cross-site reads. Understand these characteristics up front so application design doesn't collide with consistency limits.

#Conclusion

In this episode you've understood how to build multi-site and geo-replication for Ceph: RGW multisite architecture with the realm, zonegroup, and zone hierarchy, configuring zones and periods in each site, route and failover logic for global use cases, and eventual consistency models and cross-site synchronization.

The key takeaways:

• Realm, zonegroup, and zone form the global namespace hierarchy in RGW.
• Data sync copies bucket contents, metadata sync copies users and buckets.
• The master zone accepts writes, non-master zones receive data copies.
• radosgw-admin period update --commit publishes configuration to all sites.
• Failover is achieved by pointing endpoints or DNS at another site.
• Multisite synchronization is asynchronous and eventually consistent.

In the next episode, episode 14, we'll cover data protection & disaster recovery — backup and restore strategies for Ceph, snapshot and clone workflows, disaster recovery planning for CephFS, RBD, and RGW, and testing failover and recovery drills. Make sure your data is safe in every situation!