Learn HAProxy - High Availability & Clustering

#Introduction

A single HAProxy is a point of failure. If its process dies or its machine breaks, the entire service stops. Episode 13 eliminates that risk by building a highly available HAProxy cluster.

You'll set up two HAProxy nodes in an active/passive configuration using VRRP and Keepalived, integrate service discovery so backends stay up to date, then test failover with a safe procedure.

#Active/Passive Architecture with VRRP

#The VRRP and Virtual IP Concept

VRRP (Virtual Router Redundancy Protocol) lets two nodes share a single virtual IP (VIP). The active node holds the VIP; the passive node stands by. When the active node dies, the passive node takes over the VIP within seconds, without changing the address clients use.

The standard setup:

• Node A: active HAProxy, holds the VIP 10.0.0.10.
• Node B: passive HAProxy, ready to take the VIP when needed.
• The VIP is the only address exposed to clients.

sudo apt install -y keepalived

Install Keepalived on both nodes, then write its configuration at /etc/keepalived/keepalived.conf.

#Active Node Keepalived Configuration

vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    fall 2
    rise 2
}
 
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass sup3rs3cr3t
    }
    virtual_ipaddress {
        10.0.0.10/24
    }
    track_script {
        chk_haproxy
    }
}

vrrp_script chk_haproxy monitors the HAProxy process every 2 seconds. If the process dies, Keepalived lowers the node's priority so the passive node takes over the VIP.

#Passive Node Keepalived Configuration

The passive node is identical, with only two values differing:

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    virtual_ipaddress {
        10.0.0.10/24
    }
    track_script {
        chk_haproxy
    }
}

state BACKUP and priority 100 make this node passive: lower than the active node's priority of 150. virtual_router_id must be the same on both nodes.

#Running and Verifying

sudo systemctl start keepalived
ip addr show eth0 | grep 10.0.0.10

On the active node, sudo systemctl start keepalived followed by an ip addr check should show the VIP 10.0.0.10 attached. On the passive node, the VIP doesn't appear until failover happens.

#Service Discovery Integration

#Following Dynamic Backends via DNS

In modern deployments, backend IPs change constantly. HAProxy can follow DNS resolution periodically:

backend api_back
    balance roundrobin
    server svc api-svc.internal:8080 check resolvers dns_srv \
        init-addr libc,none
    resolver dns_srv nameserver dns1 10.0.0.53:53 \
        resolve_retries 3 timeout retry 1s hold valid 10s

The resolvers dns_srv directive defines the DNS server used to resolve the name api-svc.internal. hold valid 10s determines how long a resolution result is considered valid.

#Consul and Templates

If you use Consul, the server list can be generated through templating. The common pattern:

consul-template \
  -template "/etc/haproxy/consul.tmpl:/etc/haproxy/haproxy.cfg" \
  -exec "systemctl reload haproxy"

consul-template rewrites haproxy.cfg every time the service list changes, then triggers a reload. The template file contains a loop that generates a server line for each service instance.

#Testing Failover and Recovery

#A Safe Test Scenario

A testing procedure you should master:

1. Note the current VIP address with ip addr show.
2. Stop HAProxy on the active node: sudo systemctl stop haproxy.
3. Watch the VIP move to the passive node within seconds.
4. Recover the active node and watch the VIP return after the passive node gives it up.

sudo systemctl stop haproxy
ip addr show eth0 | grep 10.0.0.10

On the passive node, ip addr show eth0 | grep 10.0.0.10 should start showing the VIP once the advert_int period elapses. That's proof failover works.

#Recovery Checklist

When the active node recovers:

• Make sure HAProxy starts and backend health checks return to normal.
• Check the Keepalived logs: journalctl -u keepalived.
• Confirm the VIP isn't split between the two nodes (split-brain).
• Test requests through the VIP to make sure the service is intact.

curl -s -o /dev/null -w "%{http_code}\n" http://10.0.0.10/

curl -s -o /dev/null -w "%{http_code}\n" tests the service through the VIP. During failover and after recovery, the status should stay healthy.

#Closing

Episode 13 removes HAProxy's single point of failure: two nodes sharing a VIP with Keepalived, backends that follow service discovery, and a test procedure that turns failover from a nerve-racking moment into a routine.

Key takeaways:

• VRRP moves the virtual IP between nodes on failure.
• track_script makes Keepalived assess HAProxy's health.
• The active node has higher priority; the passive node stands ready to take over.
• DNS resolvers and Consul keep backends up to date.
• Test failover regularly and check logs to detect split-brain.

In the next episode we'll cover security hardening — ACL and timeout best practices, prevention of protocol downgrades and injections, and safe default configuration for operations.