Learn Keepalived - Health Checks & Failover Policies

#Introduction

High availability is useless if the MASTER node stays up while its service is already dead. Episode 5 answers exactly this problem: health checks and failover policies. You'll write a vrrp_script that monitors a service, connect it to an instance via track_script, and respond automatically when the service goes down.

We'll also cover notify hooks for responding to state changes, and failover policies like nopreempt, preempt_delay, and garp_master_delay that keep VIP transitions under control. By the end of this episode, a MASTER node whose service is down won't survive — the VIP will move on its own.

#vrrp_script and track_script

#Writing vrrp_script

vrrp_script is a health check definition that can be reused by many instances:

vrrp_script chk_haproxy {
  script "/usr/bin/killall -0 haproxy"
  interval 2
  weight -20
  rise 2
  fall 3
}

• script: the command that gets executed; exit code 0 means healthy.
• interval: the gap between checks in seconds.
• weight: how much priority drops when the script fails.
• rise: number of consecutive successes required to declare the service healthy again.
• fall: number of consecutive failures required to declare it failed.

weight -20 means every failure drops the priority by 20. If the MASTER priority is 100 and drops to 80, a BACKUP with priority 90 will win the election and take over.

#Connecting to an Instance with track_script

Defining it isn't enough; the instance must monitor it:

vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 51
  priority 100
  track_script {
    chk_haproxy
  }
  virtual_ipaddress {
    192.168.1.100/24 dev eth0
  }
}

The track_script block connects the chk_haproxy health check to instance VI_1. As soon as haproxy dies, the priority drops and the other node takes over the VIP.

#Notify Hooks

#Types of Notify

Keepalived can call a script whenever a state transition occurs. There are four hooks:

• notify_master: called when the node becomes MASTER.
• notify_backup: called when the node becomes BACKUP.
• notify_fault: called when the node is in FAULT state.
• notify: called on all transitions, with the state as an argument.

Add the hooks to the instance:

vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 51
  priority 100
  notify_master "/etc/keepalived/notify.sh MASTER"
  notify_backup "/etc/keepalived/notify.sh BACKUP"
  notify_fault "/etc/keepalived/notify.sh FAULT"
  virtual_ipaddress {
    192.168.1.100/24 dev eth0
  }
}

The notify_master "/etc/keepalived/notify.sh MASTER" command executes the script with the MASTER argument every time the node's state goes up.

#Example Notification Script

A simple notify script that writes a log entry and can be extended into alerting:

#!/usr/bin/env bash
STATE="$1"
echo "$(date -Is) keepalived -> $STATE" >> /var/log/keepalived-notify.log
if [ "$STATE" = "MASTER" ]; then
  /usr/local/bin/advertise-vip.sh
fi

Remember to make it executable with chmod +x /etc/keepalived/notify.sh; a script without the execute permission will never be called.

#Health Checks from Multiple Sources

#Custom Scripts

Besides killall, you can use more complex scripts, for example to check an HTTP response or disk conditions:

vrrp_script chk_api {
  script "/etc/keepalived/checks/check-api.sh"
  interval 3
  weight -30
  rise 2
  fall 3
}

The check-api.sh script is free to do anything: curl, check a lock file, or measure CPU load. An exit code of 0 is considered healthy; anything else is considered a failure.

For LVS architectures, verify the backend status directly from the kernel with ipvsadm -L -n. Its output shows the registered real servers along with the health check results from the LVS subsystem, forming a layered detection alongside vrrp_script.

#Failover Policies

#nopreempt

By default preemption is active: a node that comes back with a higher priority grabs the VIP. For manual failback scenarios, set nopreempt on all nodes in that instance:

vrrp_instance VI_1 {
  state BACKUP
  interface eth0
  virtual_router_id 51
  priority 90
  nopreempt
  virtual_ipaddress {
    192.168.1.100/24 dev eth0
  }
}

nopreempt keeps the instance as BACKUP as long as the current MASTER is still alive, whatever its priority. Applying it on all nodes is a requirement for consistent behavior.

#preempt_delay

If preemption stays active but you want it delayed, use preempt_delay as in episode 4. This value in seconds gives the service time to warm up before the VIP is taken back.

#garp_master_delay and garp_master_repeat

Gratuitous ARP is controlled by these two parameters:

vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 51
  priority 100
  garp_master_delay 5
  garp_master_repeat 5
  virtual_ipaddress {
    192.168.1.100/24 dev eth0
  }
}

garp_master_delay 5 delays GARP delivery for 5 seconds after becoming MASTER, and garp_master_repeat 5 repeats GARP 5 times. This combination updates the ARP tables of all switches and clients reliably.

#Closing

Key takeaways:

• vrrp_script defines a health check; track_script connects it to an instance.
• weight determines how far the priority drops when a check fails.
• Notify hooks bridge Keepalived to logging, monitoring, and alerting.
• Notify and check scripts must be given the execute permission.
• nopreempt for manual failback; preempt_delay for delayed preemption.
• garp_master_delay and garp_master_repeat control GARP after takeover.

In episode 6 next, we move into LVS integration and load balancing mode — configuring virtual_server, choosing a scheduler and NAT, DR, or TUN mode, setting up persistence, and wiring health checks for each backend server.