Learn Keepalived - Health Check Advanced Patterns

#Introduction

A simple health check like killall -0 only answers one question: is the process alive? Episode 16 levels up: health checks that truly understand the service, failure detection that is tiered and not hasty, and integration with restarts and alerting so operators know before clients complain.

This is the layer that separates HA that works in a demo from HA that survives in production. We'll use Keepalived's full health check types — TCP_CHECK, HTTP_GET, SSL_GET, MISC_CHECK — and build patterns that balance detection speed with resilience against flapping.

#Health Checks for External Services

#HTTP_GET with Body Check

HTTP_GET can verify more than just a status code, for example the response content:

real_server 10.0.0.11 8080 {
  weight 1
  HTTP_GET {
    url {
      path /healthz
      status_code 200
    }
    connect_timeout 3
    nb_get_retry 3
    delay_before_retry 1
  }
}

The HTTP_GET block checks that /healthz returns 200, with nb_get_retry 3 to retry before declaring failure.

#SSL_GET for HTTPS Services

For HTTPS backends, use SSL_GET. Its syntax is the same as HTTP_GET, and Keepalived uses OpenSSL for the handshake:

real_server 10.0.0.12 8443 {
  weight 1
  SSL_GET {
    url {
      path /healthz
      status_code 200
    }
    connect_timeout 3
    nb_get_retry 2
  }
}

SSL_GET makes sure the service doesn't just open a port, but actually completes a TLS handshake before being judged healthy.

#Tiered Failure Detection with rise and fall

#Hysteresis in vrrp_script

At the instance level, rise and fall prevent overly reactive status changes:

vrrp_script chk_app {
  script "/etc/keepalived/checks/check-app.sh"
  interval 2
  weight -30
  rise 2
  fall 4
}

fall 4 means the script must fail 4 times in a row before the priority drops, and rise 2 means it must succeed twice before the status is declared recovered. This pattern absorbs momentary hiccups like GC pauses or brief restarts without moving the VIP.

#Layered Weights

For tiered penalties, run several scripts with different weights:

vrrp_script chk_port {
  script "/usr/bin/killall -0 nginx"
  interval 2
  weight -10
}
 
vrrp_script chk_api {
  script "/etc/keepalived/checks/check-api.sh"
  interval 2
  weight -30
}

A chk_port failure drops the priority by 10, while chk_api drops it by 30. The deeper the damage, the bigger the penalty — until another node eventually takes over.

#Automatic Service Restart Patterns

#Restart Before Failover

Before handing the VIP to another node, try to recover the service in place first. A health check script can handle this:

#!/usr/bin/env bash
if ! systemctl is-active --quiet myapp; then
  systemctl restart myapp
  sleep 3
fi
systemctl is-active --quiet myapp

The systemctl is-active --quiet myapp script returns exit code 0 only if the service is active. If not, it restarts first, then rechecks. Keepalived judges health from the script's final exit code.

#Distinguishing Total Outage from Degradation

Make sure the script distinguishes fatal conditions from recoverable ones: use different non-zero exit codes, then set a larger negative weight for total failure in vrrp_script. That way, light degradation only lowers the priority a little without triggering an unnecessary failover.

#Integration with Monitoring Alerts

#Notify into Alerting

Connect state transitions to your alerting system via notify hooks:

#!/usr/bin/env bash
STATE="${1:-}"
curl -fsS -X POST http://alertmanager:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d "[{\"labels\":{\"alertname\":\"KeepalivedFailover\",\"node\":\"$(hostname)\",\"state\":\"$STATE\"}}]" || true

The curl -fsS -X POST http://alertmanager:9093/api/v2/alerts command sends an alert to Alertmanager every time the state changes. Install this script as notify_master and notify_backup so every failover is immediately visible.

#Verifying from the Health Check Side

To check whether health is being judged correctly, test directly from the command line:

curl -fsS http://10.0.0.11:8080/healthz
echo $?

A successful curl -fsS http://10.0.0.11:8080/healthz output means the backend is healthy. Compare this result with Keepalived's assessment in the logs to make sure your health checks reflect reality.

#Closing

Episode 16 perfects your detection radar: health checks that understand the service via HTTP_GET, SSL_GET, and MISC_CHECK, tiered detection with rise and fall, auto-restart patterns that reduce unnecessary failover, and alerting integration that keeps operators informed first.

Key takeaways:

• HTTP_GET and SSL_GET check the application, not just the port.
• rise and fall absorb momentary hiccups without moving the VIP.
• Layered weights make penalties proportional to severity.
• Health check scripts can auto-restart before failover.
• Notify hooks connect state transitions to Alertmanager.
• Test health checks with curl and compare with the Keepalived logs.

In episode 17 next, we cover Keepalived in container and virtual environments — running Keepalived in Docker with host networking, the DaemonSet pattern in Kubernetes, network namespace and VIP concerns, and a container-based HAProxy plus Keepalived stack example.