Learn Apache Kafka - Disaster Recovery & Data Migration

#Introduction

No cluster is immune to disaster: datacenter damage, configuration errors, or accidental deletion can cripple production. Disaster recovery is the plan that answers the questions: how fast can you get back to operating, and how much data are you willing to lose?

Episode 25 covers backup strategies, MirrorMaker 2.0 for cross-cluster replication, the RPO and RTO concepts, active-active and active-passive patterns, and migrating data between clusters without downtime.

#Backup Strategies

#Snapshots and Configuration Backups

Within-cluster replication protects against broker failures, but not against human error or logical damage. For that you need restorable backups:

• Data backup: snapshot or export records from important topics to object storage.
• Configuration backup: save topic definitions (partitions, retention, cleanup policy) as code — easy to restore with a script.
• Offset backup: consumer group offsets are stored in __consumer_offsets; for cross-cluster recovery, export offset positions too.

#Backup Principles

Backups must be tested, not just created. Periodic restore drills ensure you know how to restore and how long it takes. Combine several layers: replication for broker failure, backups for human error, and MirrorMaker for datacenter disasters.

#MirrorMaker 2.0

#Cross-Cluster Replication

MirrorMaker 2.0 is the official tool for replicating data between clusters. It uses Kafka Connect (episode 12): one connector per source topic, with workers reading and writing across clusters:

clusters = primary, backup
primary.bootstrap.servers = primary.example.com:9092
backup.bootstrap.servers = backup.example.com:9092
 
primary->backup.enabled = true
primary->backup.topics = orders, user-events
replication.factor = 3

primary->backup.enabled = true enables one-way replication from primary to backup. Topics are replicated with the same name; offsets are translated during the process so consumer positions stay meaningful on the destination side.

#Offset Translation and Checkpointing

The trickiest part of replication: offsets in the source cluster aren't the same as offsets in the destination cluster. MirrorMaker 2.0 handles this with offset translation — mapping source offsets to destination offsets — and checkpointing, periodically recording that mapping into an internal topic. Consumers on the backup side can resume from roughly the same position as on the primary side.

#Heartbeats

MirrorMaker sends a heartbeat topic (heartbeats) to monitor the health of replication connections. With heartbeats, you know how far replication has fallen behind and whether the inter-cluster connection works — important metrics for assessing failover readiness.

#Disaster Recovery Patterns

#RPO and RTO

Two numbers define your recovery targets:

• RPO (Recovery Point Objective): the maximum amount of data you can afford to lose in a disaster. The smaller the RPO, the more frequent and synchronous replication needs to be.
• RTO (Recovery Time Objective): the maximum time to get back to operating. The smaller the RTO, the more automated and ready the failover procedure must be.

RPO and RTO are determined by the business, not by technology — they translate the cost of downtime and data loss into architectural design.

#Active-Passive and Active-Active

• Active-passive: one cluster actively accepts writes, another stands by receiving replication. Failover swaps roles — simple and safe, but RTO is larger because a client cutover is needed.
• Active-active: both clusters accept writes and replicate to each other (bidirectional). Small RTO, but prone to data conflicts — offsets and mappings must be managed very carefully.

#Failover Procedures

A good failover procedure: verify the backup cluster's health, stop writes on the primary, confirm the last synchronization, point clients to the backup, then enable the reverse-direction replicator. All steps must be tested in periodic drills — a procedure never tested will fail in a crisis.

#Data Migration

#Migration Between Clusters

MirrorMaker also serves as a migration tool: create a new cluster, replicate from the old one, then move clients one by one. The key to success is preserving the correct order for topics whose consumers must resume from a certain position.

#Zero-Downtime Migration

A downtime-free migration flow:

1. Deploy the new cluster and start replication from the old cluster.
2. Let the data catch up, monitor replication lag.
3. Move producers to the new cluster (new clients use the new bootstrap).
4. Move consumers last; make sure nobody reads from the old cluster anymore.
5. Stop replication and shut down the old cluster after an observation period.

Key point: move producers first, then consumers — new consumers read new data from the destination cluster, while old clients still run on the source cluster.

#Replication Lag and Rollback

During migration, monitor replication lag continuously. Prepare a rollback plan: if a problem is found, point clients back to the old cluster — as long as bidirectional replication or checkpoints still allow it. After the old cluster is shut down, rollback is much harder; keep a sufficient observation period before decommissioning.

#Closing

In this episode 25 you've understood backup strategies, MirrorMaker 2.0 with offset translation and checkpointing, the RPO and RTO concepts, active-active and active-passive patterns, and zero-downtime migration procedures between clusters.

The key takeaways:

• Backups protect against human error; MirrorMaker protects against datacenter disasters.
• Offset translation and checkpointing preserve consumer positions across clusters.
• RPO determines the data you can lose; RTO determines recovery speed.
• Active-passive is safe for failover; active-active is fast but prone to conflicts.
• Zero-downtime migration: move producers first, then consumers.
• Always test failover and migration procedures periodically.

In the next episode 26 we'll discuss Kafka in microservices architectures — event-driven architecture, the saga pattern, CQRS, and outbox. You'll learn choreography, domain events, and the transactional outbox with Debezium.