Learn Elasticsearch - High Availability & Disaster Recovery

#Introduction

Up to episode 28 you built an automated, documented Elasticsearch. But one final question remains unanswered: what happens if a node dies, a rack burns, or a region is hit by a disaster? The answer depends on two concepts: high availability (HA), which keeps the cluster serving when components fail, and disaster recovery (DR), which ensures data and services can be recovered after a major failure.

Episode 29 covers multi-node cluster design for HA (master quorum anti-split-brain, data node redundancy, zone-aware replica allocation), multi-region deployment strategies, RPO and RTO planning, failover procedures, data consistency, and how to test DR periodically.

#Multi-Node Cluster Design for HA

#Master Quorum and Anti Split-Brain

The master node manages cluster state: shard placement decisions, index creation, and node membership. For a healthy cluster, the master is elected by a majority of master-eligible nodes. The quorum rule: master-eligible nodes needed = floor(total master-eligible / 2) + 1. If the majority is lost, the cluster stops processing — deliberately, to prevent split-brain (two living masters corrupting state simultaneously).

Best practice: use dedicated master nodes in odd numbers — three nodes to withstand one failure, five to withstand two:

node.name: master-01
node.roles: [ master ]
discovery.seed_hosts: ["master-01", "master-02", "master-03"]

Don't put data on master nodes. This role separation makes the cluster more stable — the master keeps working even when data nodes are full or reindexing.

#Data Node Redundancy

Data is spread across shards, and shards have replicas (episode 2). If every index has at least one replica, every shard has a copy on another node — one dead data node doesn't lose data, and the cluster keeps serving searches.

{
  "index": { "number_of_replicas": 1 }
}

Make sure the data node count is enough to hold primary and replicas separately. For one index with one replica, you need at least two data nodes in different zones.

#Zone-Aware Replica Allocation

A replica is only useful if it's on different infrastructure from the primary. Allocation awareness forces Elasticsearch to spread shards based on attributes — usually availability zones:

node.attr.zone: az-a

{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": ["zone"]
  }
}

With this configuration, primary and replicas will never be in the same zone. If one zone dies, replicas in another zone are ready to be promoted. If you're unsure where a shard lives, use GET /_cluster/allocation/explain for diagnosis.

#Multi-Region Deployment Strategies

HA keeps a cluster alive, but doesn't protect against a disaster that kills one cluster. For that, run separate clusters in more than one region. The most common patterns:

• Active-passive: the primary cluster serves production; a standby cluster is on backup. Synchronization via cross-cluster replication (CCR) (episode 22) or snapshots.
• Active-active: both clusters serve traffic and replicate to each other — more complex because conflict handling is needed.

CCR uses the leader-follower model: an index on the primary cluster becomes the leader, an index in the backup region follows it asynchronously. Replication only needs the leader index on the remote cluster.

PUT /produk-copy/_ccr/follow

Reads in the backup region can be directed to the follower; writes stay on the leader. For cross-cluster reads at once, cross-cluster search (episode 22) still applies.

#RPO and RTO Planning

These two numbers are the compass for every DR decision:

RPO determines synchronization frequency. CCR with async replication gives a small RPO (minutes); SLM snapshots (episode 20) give a large RPO (hours). RTO determines failover speed — how quickly the follower is promoted, clients switched, and indexes unfollowed.

Architecture choices follow these numbers: a logging service that may lose an hour of data is fine with daily snapshots; a payment platform with a minute-level RPO must use CCR and always-standby infrastructure. Document both numbers and make sure the targets are realistic — an untested RPO/RTO is just numbers on paper.

#Failover Procedures

When disaster strikes, failover must follow a practiced runbook, not improvisation. The general order for the active-passive pattern with CCR:

1. Confirm the primary region is truly dead and traffic has stopped.
2. In the backup region, check that replication lag is under control and the follower is intact.
3. Promote the follower to primary with POST /produk-copy/_ccr/unfollow then direct writes there.
4. Move clients, DNS, or load balancers to the backup region.
5. Verify health and indexes, then monitor continuously.

#Testing Disaster Recovery

A DR that's never been tested will fail at the moment it's most needed. Test periodically with real failure simulations:

• Chaos testing: kill one data node, one master node, one zone.
• Region-down simulation: shift traffic to the backup region and run load there.
• Restore drill: recover data from a snapshot on a new cluster and compare integrity.
• Measure targets: record actual failover time and data loss, compare against RPO/RTO.

Make these tests a scheduled agenda (for example quarterly), with results documented and fixes followed up. The more often you kill clusters in the lab, the calmer you'll be facing real failures.

#Conclusion

In episode 29 you mastered two layers of resilience. On the HA layer: multi-node cluster design with an odd number of dedicated masters, master quorum as a split-brain shield, data node redundancy through replicas, and zone-aware replica allocation so primaries and replicas never share a zone. On the DR layer: multi-region clusters with CCR leader-follower, RPO and RTO planning, failover runbooks, eventual data consistency, and periodic DR testing.

Key takeaways:

• Quorum anti-split-brain: an odd number of dedicated masters; a healthy majority is required.
• Replicas are redundancy, but only meaningful if on different infrastructure (zone-aware).
• RPO/RTO determine the architecture: CCR for small RPO, snapshots for long-term retention.
• Failover must be a practiced runbook, not improvisation during a crisis.
• DR is tested periodically — chaos testing, restore drills, and target measurement.

Your infrastructure is now strong and resilient. In the final episode, episode 30, we bring it all together: the production deployment checklist and best practices — capacity planning, security hardening, monitoring and alerting, backup restore testing, change management, rolling upgrades, troubleshooting, performance baselines, incident response, cost optimization, and the advanced Elasticsearch 8.x features ready to use. See you there!