Learn Keepalived - Monitoring & Alerting

#Introduction

HA that isn't observed is HA that can't be accounted for. Episode 20 builds complete observability: monitoring Keepalived status with a Prometheus exporter, drafting alerting for state changes and lost VIPs, and designing a dashboard that shows HA health at a glance.

The goal is simple: incidents must be detected earlier than user complaints. With the right metrics and well-targeted alerts, your team knows exactly when the MASTER moved, when a VIP is missing, and when a backend is unhealthy.

#Monitoring Keepalived Status

#Exporting Metrics to Prometheus

Keepalived doesn't ship a built-in Prometheus metrics endpoint, but the community provides keepalived_exporter, which reads the daemon's status:

services:
  keepalived-exporter:
    image: ghcr.io/philips-software/keepalived-exporter:latest
    network_mode: host
    container_name: keepalived-exporter
    restart: unless-stopped

The keepalived-exporter service runs with network_mode: host so it can interact with the Keepalived status on the host. This exporter exposes metrics like the VRRP instance state and the number of state transitions.

#Meaningful Metrics

The most useful metrics to monitor:

• VRRP instance state: whether a node is MASTER or BACKUP.
• Active VIP: whether the virtual IP is actually installed on the interface.
• Health check script: the last script result, success or failure.
• LVS real servers: the availability of each backend.

All these metrics should be exported with instance and node labels so dashboards can group them easily.

#Alerting on State Changes

#MASTER Transition Alert

A transition to MASTER isn't always bad — sometimes it's exactly right. But it must be visible. Send an alert every time the state changes:

groups:
  - name: keepalived
    rules:
      - alert: KeepalivedMasterChanged
        expr: changes(vrrp_state_master{instance="lb01"}[5m]) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "MASTER berubah di lb01"

The rule changes(vrrp_state_master[5m]) > 0 fires an alert when the MASTER metric changes within 5 minutes. This catches failovers that happen without being scheduled.

#Lost VIP Alert

The most critical scenario: no node holds the VIP. This alert must be fast and high priority:

      - alert: KeepalivedVipLost
        expr: vrrp_vip_active == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Tidak ada node yang memegang VIP"

vrrp_vip_active == 0 means the VIP isn't installed on any exposed node. The critical alert with for: 30s ensures no false positives from transition delays.

#Alerting for Unhealthy Backends

#Relying on LVS Metrics

An unhealthy backend should be visible even before VIP failover happens:

      - alert: LvsBackendDown
        expr: ipvs_healthy_real_server == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Semua backend pada {{ $labels.vip }} tidak sehat"

ipvs_healthy_real_server == 0 shows there are no healthy real servers on a VIP. A warning alert gives the team time to act before the service actually dies.

#Avoiding Alert Noise

Too many alerts make a team numb. Use a reasonable for, tiered severities, and combine several conditions. One meaningful alert is better than ten ignored ones.

#Operational Dashboard for HA

#Must-Have Panels

A Grafana dashboard for HA should show:

• A MASTER/BACKUP state map per node and per instance.
• State change history over 24 hours.
• The number of failover transitions per period.
• Health check status and backend availability.
• Failover latency from routine measurements.

With these panels, a single glance answers the questions: is HA working, and did anything just move?

#Combining with Logs

Metrics give trends; logs give detail. Connect both in one place:

sudo systemctl status keepalived --no-pager | grep "active"
sudo journalctl -u keepalived --no-pager | tail -20

The journalctl -u keepalived output can be streamed into Loki or another centralized logging system. When an alert fires, the operator opens the logs directly to understand the transition context.

#Closing

Episode 20 makes your HA transparent: metrics from keepalived_exporter flow into Prometheus, alerting catches state changes, lost VIPs, and unhealthy backends, and the dashboard shows HA health on one screen.

Key takeaways:

• Use keepalived_exporter to expose VRRP and IPVS status metrics.
• A MASTER transition alert catches unscheduled failovers.
• A lost VIP alert is top priority, with a short for.
• An unhealthy backend alert prevents surprises before the service dies.
• The dashboard shows state, transitions, health checks, and failover latency.
• Combine metrics with logs for complete incident context.

In episode 21 next, we cover SLOs, runbooks, and operational readiness — defining VIP availability and failover time targets, drafting runbooks for incidents, and ensuring operational readiness for maintenance.