Learn HAProxy - SLOs, SLIs & Operational Metrics

#Introduction

Metrics without targets are just numbers. Episode 21 gives those numbers meaning through SLOs (Service Level Objectives) and SLIs (Service Level Indicators), then connects them to alerting and runbooks.

You'll define SLIs for HAProxy traffic, set up alert rules for error rate, latency, and saturation, and write runbooks that let traffic incidents be handled calmly.

#Defining SLIs and SLOs

#Choosing the Right SLIs

An SLI is a chosen quality measurement. For HTTP traffic through HAProxy, the most common SLIs:

• Availability: the percentage of requests that succeed (non-5xx status).
• Latency: the percentage of requests that finish within a given time limit.
• Throughput: the requests per second served.

The availability formula in PromQL:

1 - (
  sum(rate(haproxy_frontend_http_responses_total{code=~"5.."}[5m]))
  /
  sum(rate(haproxy_frontend_http_requests_total[5m]))
)

The formula above computes the proportion of 5xx responses to total requests over 5 minutes. The code=~"5.." code uses a label matcher to grab all 5xx responses.

#Setting SLOs

SLOs set targets on SLIs:

• Availability SLO: 99.9 percent of requests successful per month.
• Latency SLO: 95 percent of requests complete under 200 milliseconds.
• Error budget: the remaining failures allowed, for example 0.1 percent.

The principle: SLOs must be realistic and agreed with the team and the service owner. Don't set targets that make the team embarrassed to acknowledge them.

#Measuring with a Dashboard

Visualize SLOs with a Grafana panel: a time chart for error rate and a line for the SLO target. When the curve approaches the line, you're burning your error budget.

#Alerting Based on Operational Metrics

#Alerts for Error Rate

Alerts should only wake a human when truly necessary:

groups:
  - name: haproxy.rules
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(haproxy_frontend_http_responses_total{code=~"5.."}[5m]))
          / sum(rate(haproxy_frontend_http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5 percent for 10 minutes"

for: 10m ensures the alert only fires if the condition persists for 10 minutes, avoiding false alarms from momentary spikes. Critical severity means immediate action.

#Alerts for Latency and Saturation

The next two categories:

groups:
  - name: haproxy.latency
    rules:
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            rate(haproxy_frontend_http_request_duration_seconds_bucket[5m])) > 0.5
        for: 15m
        labels:
          severity: warning
 
      - alert: ConnSaturation
        expr: haproxy_frontend_current_sessions / haproxy_frontend_max_sessions > 0.9
        for: 10m
        labels:
          severity: warning

histogram_quantile(0.95, ...) > 0.5 fires an alert when 95 percent of requests are slower than half a second. ConnSaturation fires when connection usage approaches the maxconn limit.

#Distinguishing Pages and Issues

Not every warning must wake someone:

• Page: 5xx error rate and connection saturation — a human immediately.
• Issue: latency rising without errors — a ticket, not a call.
• Info: metrics drifting from baseline — a dashboard, not an alert.

This division keeps the team responsive without alert fatigue.

#Runbooks for Traffic Incidents

#Node Failover Runbook

A runbook is a tested sequence of steps. Example for a dead HAProxy node:

1. Confirm the node is dead: check journalctl -u haproxy and systemctl status haproxy.
2. Verify the VIP moved to the passive node: ip addr show.
3. Test the service through the VIP with curl.
4. Begin repairing the active node, bring it back, and watch the VIP.

systemctl status haproxy --no-pager
ip addr show eth0 | grep 10.0.0.10
curl -s -o /dev/null -w "%{http_code}\n" http://10.0.0.10/

systemctl status haproxy --no-pager confirms the process state before acting, and curl ... http://10.0.0.10/ confirms the service is still alive through the VIP.

#High Traffic Runbook

When the error rate spikes because of traffic:

1. Identify the direction: all backends or one backend? Look at the per-server dashboard.
2. Check capacity: connections vs maxconn, backend CPU.
3. Emergency action: add backend servers, or add HAProxy nodes.
4. If needed, enable the stricter rate limiting from episode 10 to dampen the load.
5. Document every step after the incident.

#DNS and Service Discovery Runbook

When a backend disappears because of service discovery:

1. Check name resolution: dig users-svc.internal.
2. Check the server status in HAProxy: show servers state.
3. Make sure resolvers and hold valid match.
4. If resolution is stuck, check the DNS server and cache policy.

#Keeping Runbooks Alive

A runbook that isn't tested is nonsense. Practices that keep runbooks relevant:

• Test every runbook regularly through game days.
• Update runbooks when the configuration changes.
• Store runbooks close to the team: a wiki, repo, or incident system.

curl -s -o /dev/null -w "%{http_code}\n" http://localhost/ 

curl -s -o /dev/null -w "%{http_code}\n" is the fastest way to verify health during a drill or a real incident.

#Closing

Episode 21 turns metrics into a measurable promise: SLIs chosen deliberately, SLOs that are agreed, alerting filtered so it isn't noisy, and runbooks that turn incidents into an executable routine.

Key takeaways:

• SLIs measure; SLOs target; the error budget records the remaining failures.
• Error rate alerts need a for clause to avoid false alarms.
• Latency and saturation are SLIs that complement availability.
• Page for emergencies; issue for tickets; info for dashboards.
• Runbooks must be tested regularly to stay valid.

In the next episode, the final one, we'll cover production hardening & best practices — a production checklist for security, availability, observability, and operational readiness, disaster recovery and configuration backup strategies, and how to document conventions and support boundaries.