Learn Keepalived - Production Hardening & Best Practices

#Introduction

Congratulations, you've reached the final episode! Episode 22 distills the whole journey into one thing: production readiness. All the concepts you've learned — VRRP, health checks, LVS, security, monitoring, SLOs — are brought together in a hardening checklist and ongoing operational procedures.

This episode also stresses the two habits most often forgotten: practice and documentation. Even the best infrastructure will wobble if it's never tested in disaster scenarios and never documented for the people who will take over. Let's close this series with a foundation that doesn't easily rot.

#Security Hardening Checklist

#Secure Configuration

Start from the things we built in episodes 9 and 13:

• Use auth_type AH or strict network isolation; not auth_type PASS.
• Put VRRP on a dedicated VLAN or use unicast VRRP.
• Enable rp_filter and arp_filter on all nodes.
• Restrict health check endpoints to HA nodes only.
• Set chmod 600 on all files containing secrets.

Complete it with a quick verification:

sudo ss -tlnp | grep -E "keepalived|:9093|:80"
sudo sysctl net.ipv4.conf.all.rp_filter

The ss -tlnp | grep keepalived output shows the opened ports, and sysctl net.ipv4.conf.all.rp_filter confirms reverse path filtering is active.

#Safe Operational Practices

• Give root access only to those who need it.
• Monitor configuration changes through Git and review.
• Avoid credentials in health check or notify scripts.
• Rotate secrets periodically and document the rotation.

#Availability Hardening Checklist

#Layered Redundancy

Availability starts with design:

• At least two nodes in every virtual router.
• Redundant uplinks and correct track_interface.
• A VRRP path that doesn't depend on a single switch or a single link.
• More than one backend, protected by health checks.

#Operational Readiness

• Versioned configuration that can be rolled back with a tag.
• keepalived.conf and IPVS rules backed up off-node.
• Tested alerting for state transitions and lost VIPs.
• Incident runbooks practiced in scheduled game days.

#Disaster Recovery Drills and Upgrades

#Routine Game Days

Schedule recovery drills at least quarterly. The scenarios that must be tested:

• Shut down the MASTER node completely and measure failover time.
• Shut down the BACKUP node and make sure there's no impact.
• Bring down one uplink and make sure the VIP moves.
• Recover the node and observe failback according to policy.

Record every result and compare with the SLOs set in episode 21.

#Safe Keepalived Upgrades

Upgrading Keepalived across versions must be gradual and staged:

sudo apt update
sudo apt install -y keepalived
sudo keepalived -t -f /etc/keepalived/keepalived.conf
sudo systemctl restart keepalived

Upgrade one node at a time: start with the BACKUP, verify, then move to the MASTER. Run keepalived -t after the upgrade, because features like auth_type PASS can change between versions (remember episode 9).

#Verify After Upgrade

After both nodes are upgraded, run a measured failover to make sure behavior is unchanged, then watch the logs for a few days. Record the version running on each node so documentation stays accurate.

#Documenting Topology and Support Boundaries

#Documentation That Must Exist

A Keepalived topology should be documented minimally with:

• A diagram: nodes, VIPs, virtual_router_id, and traffic direction.
• An ownership table: which VIP belongs to which instance on which node.
• Policies: preemption, failback, and SLOs.
• Runbooks and on-call contacts.

Documentation isn't a luxury; it's the safety net when an incident happens at 3 AM.

#Setting Support Boundaries

Define the support scope clearly: what HA guarantees (VIP, failover, L4 load balancing) and what is not Keepalived's responsibility (application state, database replication, consensus). With clear boundaries, team and management expectations don't drift.

#An Ideal Final Configuration

A closing configuration that reflects all the best practices:

global_defs {
  router_id LB-01
  log_facility 5
}
 
vrrp_script chk_haproxy {
  script "/usr/bin/killall -0 haproxy"
  interval 2
  weight -30
  rise 2
  fall 3
}
 
vrrp_instance VI_1 {
  state MASTER
  interface eth0
  virtual_router_id 51
  priority 100
  advert_int 100cs
  preempt_delay 30
  track_interface {
    eth1
  }
  track_script {
    chk_haproxy
  }
  virtual_ipaddress {
    192.168.1.100/24 dev eth0
  }
}

This vrrp_instance VI_1 block sums up the lessons: advert_int 100cs for fast failover, preempt_delay 30 for controlled failback, track_interface and track_script for automatic priority drops.

#Closing

Episode 22 closes the series the right way: verified hardening, scheduled recovery drills, staged upgrades, and documentation that will save your team at 3 AM. You are no longer just a Keepalived user — you are its operator.

Key takeaways:

• Hardening covers security and availability at the same time.
• Scheduled recovery drills keep runbooks from becoming fiction.
• Alternating upgrades from BACKUP to MASTER preserve availability.
• Re-verify with keepalived -t after every upgrade.
• Document the topology, policies, and support boundaries.
• Make these habits ongoing, not a one-time project.

The Learn Keepalived series is complete! From the pre-requisites in episode 0 to production hardening in episode 22, you've completed the full journey: understanding VRRP, building failover, integrating LVS, securing traffic, monitoring status, and managing operations. Apply, practice, and document — may your infrastructure always stay alive, just like the daemon you manage.