Learn Keepalived - Logging & Troubleshooting

#Introduction

No matter how good your configuration is, incidents happen. What separates a seasoned operator is the ability to read logs and dissect packets. Episode 7 gives you those skills: Keepalived log levels, syslog integration, how to debug VRRP state changes, and a map of the most common failure modes.

We'll also capture VRRP advertisements straight from the network with tcpdump, so you can see with your own eyes why two nodes never become one virtual router. By the end of this episode, you'll have a systematic procedure for solving Keepalived problems.

#Log Levels and Syslog Integration

#Foreground Mode for Debugging

When troubleshooting, run Keepalived in the foreground with full logging:

sudo keepalived --dont-fork --log-console --log-detail --vrrp

The combination of --dont-fork, --log-console, and --log-detail makes the daemon print all VRRP details straight to the terminal. This mode is ideal for labs and debug sessions; in production, rely on systemd and syslog instead.

#Directing Logs to a Specific Facility

Keepalived sends logs to syslog with the default LOG_DAEMON facility. You can redirect it to make it easier to separate:

global_defs {
  router_id LB-01
  log_facility 5
}

With log_facility 5, logs go to the local1 facility (value 5 in the configuration). Configure rsyslog to write this facility to a separate file:

echo "local1.* /var/log/keepalived.log" | sudo tee /etc/rsyslog.d/30-keepalived.conf
sudo systemctl restart rsyslog

The echo ... | sudo tee /etc/rsyslog.d/30-keepalived.conf command creates the rsyslog rule file, and systemctl restart rsyslog applies it. From now on, Keepalived logs have a home of their own.

#Debugging VRRP State Changes

#Reading Transition Logs

State transitions are clearly recorded in the logs. A common pattern to watch for:

sudo journalctl -u keepalived -f

Look for lines like Entering MASTER STATE and Entering BACKUP STATE. A flapping pattern, where a node alternates MASTER within a short time, usually points to multicast problems, authentication, or differences in advert_int.

#Capturing Advertisements with tcpdump

The strongest evidence comes from the network. Capture VRRP packets:

sudo tcpdump -i eth0 vrrp -vv

The tcpdump -i eth0 vrrp -vv output shows incoming and outgoing VRRP advertisements, including the priority being sent. If you don't see any packets at all, multicast is likely blocked on the switch, firewall, or the wrong interface.

Each advertisement carries information like virtual_router_id, priority, and adver_int. Two nodes that disagree on these values will never form a coherent virtual router.

#Common Failure Modes and Solutions

#Authentication Mismatch

If one node uses authentication and another doesn't, or the passwords differ, advertisements get rejected. Symptom: the node keeps sitting in BACKUP or FAULT, and the log shows dropped packets. Make sure the authentication block is identical on all nodes, or remove it entirely.

#Conflicting Virtual Router ID

Two different VIPs using the same virtual_router_id on the same network overwrite each other. Symptom: the VIP suddenly disappears or moves for no reason. Solution: make sure every virtual router has a unique ID, and check the sent virtual_router_id value with tcpdump.

#Blocked Multicast

VRRP depends on multicast 224.0.0.18. Both host firewalls and switches that don't allow multicast will break communication. Quick verification:

sudo iptables -L -n | grep -i multicast

iptables -L -n filtered by multicast shows whether any rule is blocking it. For a lab, temporarily flush the rule and test again.

#Desynchronized Advert Interval

A difference in advert_int between nodes makes the BACKUP miscalculate master_down_interval. Symptom: failover is late or happens abruptly. Always use the same advert_int on all nodes in the same instance.

#A Systematic Troubleshooting Procedure

An inspection order you can make a habit:

sudo systemctl status keepalived --no-pager
sudo keepalived -t -f /etc/keepalived/keepalived.conf
sudo journalctl -u keepalived --no-pager | tail -50
sudo ip -brief addr show
sudo tcpdump -i eth0 vrrp -c 20

• Make sure the service is active (running).
• Make sure the configuration passes keepalived -t.
• Read the last 50 log lines to see transitions and errors.
• Check that the VIP is actually installed on the right node.
• Capture VRRP packets to confirm the nodes hear each other.

This sequence resolves most problems before you suspect deeper issues like kernel or routing problems.

#Closing

Episode 7 turns you from an operator who panics when the VIP disappears into a systematic one: understanding log levels, dissecting advertisements with tcpdump, and mapping failure modes such as authentication mismatch, conflicting IDs, and blocked multicast.

Key takeaways:

• --dont-fork --log-console --log-detail for foreground debugging.
• The syslog facility makes Keepalived logs easy to separate and analyze.
• tcpdump -i eth0 vrrp is the strongest evidence of inter-node communication.
• Authentication mismatch is the leading cause of rejected advertisements.
• A conflicting virtual_router_id makes VIPs overwrite each other.
• Follow the diagnosis order: status, syntax, logs, VIP, then packets.

In episode 8 next, we cover advanced keepalived configuration — synchronizing VRRP groups with vrrp_sync_group, configuring multiple virtual routers and VIPs at once, and tracking interfaces, routes, and scripts together for larger HA architectures.