Learn Keepalived - Disaster Recovery & Backup Strategies

#Introduction

High availability isn't just about preventing failures — it's also about rising again after one. Episode 18 prepares you for the worst day: backing up configuration, restoring nodes quickly, deciding when to return the VIP to the original node, and facing the most dangerous scenario — split-brain.

Split-brain happens when two nodes both believe they are MASTER. No HA technology is entirely free of it, but with the right design its impact can be contained. This episode gives you strategies to prevent, detect, and recover from that situation.

#Backing Up Configuration and State

#Backing Up the Configuration

keepalived.conf is your most valuable asset. Back it up regularly:

sudo tar czf /var/backups/keepalived-$(date +%F).tar.gz \
  /etc/keepalived \
  /etc/sysconfig/keepalived

The tar czf /var/backups/keepalived-$(date +%F).tar.gz command packages the configuration directory along with the service environment files. Store the archive off-node — in object storage or a backup server — so it survives a total node failure.

#Restoring Quickly

For recovery, just restore the archive and restart:

sudo tar xzf /var/backups/keepalived-2026-08-10.tar.gz -C /
sudo keepalived -t -f /etc/keepalived/keepalived.conf
sudo systemctl restart keepalived

The keepalived -t then restart sequence makes sure the restored configuration is valid before being activated. This procedure must be tested, not just written in documentation.

#Backing Up LVS State

The IPVS table disappears when a node dies. If you need to restore the structure without waiting for health checks, back it up with ipvsadm:

sudo ipvsadm -S > /var/backups/ipvs-rules.txt

The ipvsadm -S output produces rules that can be reloaded with ipvsadm -R. This is useful for reconstructing the IPVS table on a replacement node.

#Failback Policy and Timing

#Automatic Failback with Preemption

By default, an original MASTER node that returns with a higher priority will grab the VIP back after preempt_delay. This automatic failback is convenient but risky if the service on the original node isn't ready yet.

#Controlled Failback with nopreempt

If you prefer manual, controlled failback, use nopreempt and promote the node by intervention:

vrrp_instance VI_1 {
  state BACKUP
  interface eth0
  virtual_router_id 51
  priority 90
  nopreempt
  virtual_ipaddress {
    192.168.1.100/24 dev eth0
  }
}

nopreempt keeps the original node from grabbing the VIP while the current MASTER is still healthy. Failback happens when the MASTER is deliberately shut down, or by restarting Keepalived after confirming the service is ready.

#A Safe Failback Sequence

The recommended failback procedure:

• Make sure the service on the original node is healthy.
• Stop Keepalived on the standby node so the VIP moves automatically.
• Verify the VIP and traffic are back to normal.
• Re-enable Keepalived on the standby node as BACKUP.

This manual procedure is slower but far safer than letting preemption grab the VIP while the service isn't warmed up.

#Split-brain and Partitioned Networks

#What Is Split-brain

If the link between nodes fails but both stay alive, each will believe it is the only MASTER. Both nodes install the same VIP, and traffic gets split inconsistently. This is the most dangerous scenario in VRRP.

#Mitigation by Design

• Put VRRP on a redundant path (two or more links) so a single link failure doesn't isolate a node.
• Use a third-party health check as an external source of truth.
• Enable nopreempt and tracking so VIP contention doesn't run wild.

#Detecting Split-brain

Detect it early through logs and monitoring:

sudo journalctl -u keepalived --no-pager | grep "Entering MASTER STATE"

If the journalctl -u keepalived log shows two different nodes entering MASTER STATE at almost the same time, suspect split-brain. Confirm by checking the VIP on both nodes:

ssh lb02 "ip -brief addr show | grep 192.168.1.100"

If VIP 192.168.1.100 appears on two nodes at once, split-brain has occurred. Immediately shut down one node or stop its service to restore consistency.

#Closing

Episode 18 prepares you for the worst: backups that can be restored quickly, a controlled failback policy, and strategies to handle split-brain from design to detection. Your HA now has a recovery plan, not just failure tolerance.

Key takeaways:

• Back up keepalived.conf regularly to a location off-node.
• Restore with tar, validate with keepalived -t, then restart.
• nopreempt gives you full control over failback timing.
• Split-brain occurs when both nodes claim to be MASTER.
• Use redundant VRRP paths to reduce the risk of isolation.
• Detect split-brain by comparing VIPs and logs across nodes.

In episode 19 next, we cover CI/CD and configuration validation — validating Keepalived configuration in the CI pipeline, automated deployment from versioned configuration, and preflight checks for HA failover readiness before releasing to production.