Learn Keepalived - SLOs, Runbooks & Operational Readiness

#Introduction

At this point you've built an HA that works. Episode 21 answers the harder question: how do you know the HA is working as promised? The answer is operationalization — measurable SLOs, tested runbooks, and readiness for maintenance.

This episode turns a technical project into a system managed with discipline. You'll set targets for VIP availability and failover time, write runbooks that can be followed under panic, and design maintenance procedures that don't sacrifice availability.

#Defining SLOs for the VIP

#Choosing Realistic Targets

An SLO is a promise that can be measured. For HA, two key metrics:

• VIP availability: the percentage of time the VIP answers traffic, for example 99.95% per month.
• Failover time: the window between a node dying and the VIP becoming active on another node, for example under 5 seconds.

First calculate your system's capability from the measurements in episode 15, then set targets below that capability so there's always headroom.

#Measuring SLOs with Blackbox

Measure availability from the outside, not from inside the nodes:

prober:
  - name: vip_tcp
    prober: tcp
  - name: vip_http
    prober: http
 
targets:
  - url: "tcp://192.168.1.100:80"
    probe: vip_tcp
  - url: "http://192.168.1.100/healthz"
    probe: vip_http

The blackbox exporter configuration probes the VIP over TCP and HTTP. Uptime metrics from these probes become the basis for real SLO calculation, not just numbers on a document.

#Computing SLOs in Prometheus

Convert probes into availability numbers:

availability = sukses / total_probe * 100
sli_failover = waktu_saat_vip_pindah - waktu_saat_node_mati

A simple error budget: with a 99.95% target, maximum downtime is about 21.9 minutes per month. Track error budget consumption on the dashboard so the team knows where they stand against the promise.

#Runbooks for Failover Incidents

#An Effective Runbook Structure

A good runbook answers three questions: what happened, who acts, and what are the steps. An example structure:

Judul: VIP 192.168.1.100 tidak menjawab
1. Konfirmasi gejala: ping VIP gagal dari beberapa sumber.
2. Cek siapa MASTER: journalctl -u keepalived di semua node.
3. Cek VIP terpasang: ip -brief addr show di node MASTER.
4. Cek layanan backend: curl /healthz pada real server.
5. Jika MASTER tidak memegang VIP: restart keepalived.
6. Jika layanan backend mati: pulihkan backend terlebih dahulu.
7. Jika perlu: failover manual dengan mematikan node MASTER.
8. Verifikasi pemulihan dan catat timeline.

A runbook with clearly numbered steps reduces panic during an incident. Keep it in the same repository as the configuration so it always stays in sync with topology changes.

#Runbook Drills

A runbook that was never tested is fiction. Schedule a game day: shut down a node deliberately in staging, follow the runbook step by step, and record the obstacles that appear. Improve the runbook based on the findings so that when a real incident happens, the team is already used to it.

#Operational Readiness for Maintenance

#A Safe Maintenance Window

Changes to HA nodes must have a clear rollback procedure. Before touching a node:

sudo keepalived -t -f /etc/keepalived/keepalived.conf
sudo systemctl is-active keepalived
ping -c 1 -W 1 192.168.1.100 > /dev/null && echo "VIP OK"

The pre-flight keepalived -t ensures the configuration to be installed is valid, and the VIP check ensures HA is healthy before maintenance starts.

#Alternating Maintenance

For zero downtime, maintain nodes one at a time:

• Maintain the BACKUP node, test it, then bring it back.
• Move the VIP to another node in a controlled way, then maintain the MASTER.
• After finishing, verify both nodes and record the results.

With this sequence, there is never a moment when no node is ready to take over.

#Operational Readiness Checklist

A summary of what should always be available:

• A backup configuration that can be restored within 15 minutes.
• An incident runbook tested at a game day.
• SLOs and error budget monitored on the dashboard.
• Documented on-call contacts and escalation paths.
• Alternating maintenance procedures already carried out regularly.

#Closing

Episode 21 completes the operational side of HA: measurable SLOs for VIP and failover, structured and practiced runbooks, and maintenance readiness that doesn't sacrifice availability. Your Keepalived is now managed to the same standard as any other production service.

Key takeaways:

• Set VIP availability SLOs and failover time targets.
• Measure SLOs from outside the nodes with blackbox probes.
• Runbooks must have numbered steps and live with the configuration.
• Test runbooks in scheduled game days.
• Alternating maintenance enables zero downtime.
• Monitor the error budget so availability promises stay realistic.

In episode 22, the final episode, we cover production hardening and best practices — security and availability hardening checklists, disaster recovery drills and upgrade strategies, and documenting the Keepalived topology and its support boundaries.