Learn Apache Kafka - Replication & High Availability

#Introduction

Replication is the reason Kafka stays available when a broker dies. Every partition is copied to several brokers, and when its leader fails, a replacement is promoted. But behind that simplicity lie complex decisions: who gets to be the leader, and when the system is willing to sacrifice availability for consistency.

Episode 24 dissects replication in depth: the roles of leaders and followers, In-Sync Replicas, the high watermark mechanism, the risk of unclean leader election, the role of min.insync.replicas, and rack awareness for multi-AZ deployment.

#Deep Dive into Replication

#Leader and Follower Replicas

Every partition has one leader and several followers. The leader serves all writes and reads; followers only copy data from the leader. When the leader dies, the controller picks the best follower as the new leader. If a follower stops receiving data or fails, it's marked out-of-sync and removed from the ISR.

#In-Sync Replicas (ISR)

The ISR is the list of followers still in sync with the leader. A follower leaves the ISR if it doesn't send fetch requests or lags past replica.lag.time.max.ms (default 30 seconds). A follower that catches up rejoins the ISR. The ISR determines failure tolerance: with an ISR of 3 and a replication factor of 3, two brokers can die without data loss.

Check ISR health with describe:

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

The output shows per partition the Leader, Replicas, and Isr columns. Healthy state: all Replicas appear in Isr, and Leaders are evenly spread. If a replica is missing from the Isr, investigate why the follower is lagging.

#Replica Fetching and the High Watermark

Followers pull data from the leader via periodic fetch requests. Every partition tracks two positions:

• Log-end-offset (LEO): the last offset accepted locally.
• High watermark: the highest offset replicated to all ISR replicas.

Consumers can only read records below the high watermark. This prevents consumers from seeing data that hasn't been replicated — if the leader dies before replication finishes, data above the high watermark isn't considered stored.

#Unclean Leader Election

#Data Loss Risk

When all ISR replicas are dead, Kafka faces a choice: wait for an ISR replica to recover (unavailable, but safe) or promote an out-of-sync follower to leader (available, but data may be lost). This policy is governed by unclean.leader.election.enable:

• false (default): a new leader is only chosen from the ISR; data isn't lost, but partitions may be unavailable.
• true: an out-of-sync follower can be promoted; availability is maintained at the risk of losing records that weren't replicated.

unclean.leader.election.enable=false

unclean.leader.election.enable=false is the recommended production setting for data that must not be lost. For non-critical workloads that prioritize availability, consider true with full awareness of the consequences.

#Availability vs Consistency

This is the classic CAP trade-off in concrete form. Choose based on your data: for financial transactions and critical events, consistency wins (false); for telemetry where loss is acceptable, availability wins (true). Monitor unclean-leader-elections-per-sec to detect whether the active policy is being forced.

The consequences of choosing true need to be fully understood: consumers reading before the election can see records that then "disappear" after an out-of-sync follower is promoted — because records above the high watermark are discarded. Always account for this impact on consuming applications before changing the setting.

#Min In-Sync Replicas

#Configuration and Interaction with acks

min.insync.replicas determines the minimum number of ISR replicas required for a write to be accepted:

min.insync.replicas=2

The combination that guarantees high durability: min.insync.replicas=2 on the broker with acks=all on the producer. With a replication factor of 3, brokers accept writes only if at least 2 replicas are in sync — once the ISR drops to 1, writes are rejected, preventing writes to a lone leader that might lose data.

#Guaranteeing Durability

min.insync.replicas is the most important safety net. It ensures acks=all means all current ISRs receive the data, and the ISR can't shrink below the threshold. Note: the ISR is a dynamic condition, not a fixed size — if a replica is slow, the ISR can drop and block writes. Design the replication factor and threshold accounting for how many brokers can be down.

#Rack Awareness

#broker.rack Configuration

Kafka can place replicas in different failure domains via broker.rack:

broker.rack=us-east-1a

broker.rack=us-east-1a tells Kafka where a broker's failure domain is. When creating a topic, Kafka tries to place leader and follower replicas in different racks, so one rack failure doesn't bring down all replicas of a partition.

#Multi-AZ Deployment

With rack awareness, multi-AZ deployment becomes safe: a replication factor of 3 with brokers spread across three AZs. An AZ failure only drops one replica per partition — the ISR stays at 2 of 3, and writes continue. Combine with min.insync.replicas=2 so one AZ being down neither blocks writes nor sacrifices durability.

#Closing

In this episode 24 you've understood replication in depth: leaders and followers, ISR, the high watermark, the risk of unclean leader election, the role of min.insync.replicas, and rack awareness for multi-AZ deployment.

The key takeaways:

• The ISR determines failure tolerance; out-of-sync followers are removed from the ISR.
• Consumers only read below the high watermark for consistency.
• unclean.leader.election.enable=false prevents data loss at the cost of availability.
• min.insync.replicas=2 plus acks=all guarantees high durability.
• Rack awareness spreads replicas across different failure domains.
• A healthy multi-AZ setup requires spread brokers and active rack awareness.

In the next episode 25 we'll discuss disaster recovery and data migration — MirrorMaker 2.0 for cross-cluster replication, active-active and active-passive patterns, RPO/RTO considerations, and zero-downtime migration between clusters.