Learn MetalLB - High Availability & Failover

#Introduction

A good cluster must not only work — it must keep working when a component dies. Episode 15 covers MetalLB high availability & failover: what happens to announcements when a node dies, how quickly Services recover, and how the different designs (Layer 2 versus BGP) handle failures.

Context to keep in mind: MetalLB doesn't handle traffic itself. It only directs the network to the right nodes. So "failover" here means keeping announcements accurate when the topology changes. Understanding these mechanisms determines how tight an SLA you can promise.

#Leader Election and Layer 2 Failover

#How the Leader Is Chosen

In Layer 2 mode, for each Service IP, the speakers elect one leader through a distributed election process. The result can be seen in the speaker logs. That leader answers ARP/NDP for the IP.

kubectl get pods -n metallb-system -o wide
kubectl logs -n metallb-system -l component=speaker --tail=50 | grep -i announcing

kubectl logs -n metallb-system -l component=speaker --tail=50 | grep -i announcing shows lines like Announcing 192.168.1.200 from node worker-1 — that's the node acting as leader for the IP.

#Failover When the Leader Node Dies

When the leader node dies, other speakers detect that it's no longer responding to the election, and a new node is elected as leader. The new node immediately sends gratuitous ARP to update the caches of devices on the network.

kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
kubectl get pods -n metallb-system -o wide
kubectl logs -n metallb-system -l component=speaker --tail=20

kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data drains the node to simulate maintenance. After that, the speaker logs show the announcement moving to another node. Layer 2 failover usually takes a few seconds — failure detection plus the new ARP announcement.

#ARP Behavior During Failover

Client devices store IP-to-MAC mappings in their ARP caches. When the leader changes, the old cache still points at the dead node's MAC. The new leader's gratuitous ARP updates the cache, but clients with aggressive caching can hold onto the old address for a few seconds. As a result, some short connections may drop during this window — consider this when designing applications with long-lived sessions.

#BGP High Availability: Multi-Node ECMP

#All Nodes Advertise the Same Prefix

Unlike Layer 2, in BGP mode every node advertises the same prefix to the routers. When a node dies, its BGP session drops and the router removes that path from its routing table, then forwards traffic through the other surviving nodes — no election process needed.

show ip bgp 192.168.1.200
show ip route 192.168.1.200

show ip bgp 192.168.1.200 on the router shows multiple paths to the same prefix — proof that ECMP is active. When one path dies, the router recalculates automatically without manual intervention.

#BGP Advantages for HA

• Faster and cleaner failover — routers communicate directly via BGP.
• No dependence on client ARP cache updates.
• Traffic load spreads across many nodes as long as all paths are healthy.

#Path Monitoring and Peer Failure

#How Routers Assess Health

BGP maintains sessions via the hold time: if the router doesn't receive a keepalive from the peer within the limit, the session is considered dead and the path is dropped. The holdTime value in BGPPeer controls the aggressiveness of this detection:

apiVersion: metallb.io/v1beta1
kind: BGPPeer
metadata:
  name: core-a
  namespace: metallb-system
spec:
  myASN: 64500
  peerASN: 64512
  peerAddress: 192.168.1.1
  holdTime: 30s

holdTime: 30s detects peer failures faster than the default 90 seconds. The trade-off: too-aggressive a hold time can cause flapping if the network is unstable. Start from the default value, then lower it gradually while monitoring stability.

#Testing Peer Failure

To test how the cluster responds to a peer failure, shut down one of the peer routers:

kubectl get bgppeer
kubectl logs -n metallb-system -l component=speaker --tail=30 | grep -i bgp

kubectl get bgppeer shows the peer status; kubectl logs -n metallb-system -l component=speaker --tail=30 | grep -i bgp shows the closed session logs. As long as one peer remains alive, routes still reach the network — that's the value of the multi-peer configuration from episode 13.

#Practice: Setting Up Correct HA

#High Availability Checklist

To achieve real high availability, make sure:

• There is more than one node that can announce IPs.
• In L2 mode, the cluster has at least two nodes and the speaker runs on all of them.
• In BGP mode, all nodes are peered to at least two routers.
• externalTrafficPolicy: Local is set and pod replicas are spread across many nodes (remember episode 9).
• Alerts are in place for BGP session failures and speaker count reductions.

#Routine Test Simulations

Failover can't be tested once and forgotten. Schedule simulations: drain a node, observe the announcement shift, and record the recovery time. This also drills the runbook we'll cover in episode 21.

#Conclusion

Episode 15 completes high availability & failover: leader election and Layer 2 failover with its ARP cache limitations, plus BGP's advantages with multi-node ECMP and automatic path monitoring.

Key takeaways:

• Layer 2 elects one leader per IP; a new node is announced via gratuitous ARP when the leader dies.
• Layer 2 failover takes a few seconds and depends on client ARP caches.
• BGP advertises prefixes from all nodes; routers recalculate paths when a node dies.
• holdTime controls peer failure detection speed.
• Multi-peer (episode 13) is the key to BGP path redundancy.
• Routine failover simulations are the best way to stay operationally ready.

In the next episode, episode 16, we'll discuss advanced BGP — using communities to tag routes, local preference, prefix aggregation, multi-hop peering to external routers, and the route reflector topology for larger networks.