Learn HAProxy - Observability & Integration

#Introduction

Metrics on the stats page are good for humans, but not for automation. Episode 18 takes HAProxy into the modern observability world: metrics Prometheus can scrape, logs flowing to Loki and the ELK stack, and request correlation that makes cross-service debugging possible.

You'll build a real observability pipeline — not just theory — from a metrics endpoint all the way to correlation ID patterns.

#Providing Metrics to Prometheus

#Opening the Metrics Endpoint

Since HAProxy 2.x, the Prometheus exporter is available directly without an additional agent:

listen prometheus
    bind *:8405
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
    no server

The line http-request use-service prometheus-exporter if { path /metrics } invokes the internal metric exporter service, and no server marks that this listen has no regular backend.

#Verifying the Endpoint

Make sure metrics come out correctly:

curl -s http://localhost:8405/metrics | head -n 5

curl -s http://localhost:8405/metrics | head -n 5 shows the first lines of metrics, for example haproxy_frontend_requests_total. If it appears, the exporter configuration is correct.

#Prometheus Configuration

Add this target in prometheus.yml:

scrape_configs:
  - job_name: haproxy
    static_configs:
      - targets: ["haproxy1:8405", "haproxy2:8405"]

The target haproxy1:8405 tells Prometheus which endpoint to scrape. Make sure the network between Prometheus and HAProxy is open on port 8405.

#Dashboard and Log Integration

#Grafana with a Standard Dashboard

Prometheus metrics can be mapped directly in Grafana. The most important metrics:

• haproxy_frontend_http_requests_total: total requests per frontend.
• haproxy_server_current_sessions: active connections per server.
• haproxy_server_health_failures_total: number of health check failures.
• haproxy_frontend_http_response_time_average: average response time.

Build a dashboard with a rate panel: rate(haproxy_frontend_http_requests_total[5m]) to see the request rate per 5 minutes.

#Streaming Logs to Loki

Loki pulls logs from syslog. Configure HAProxy to write logs to remote syslog:

global
    log 10.0.0.50:1514 local0
    log 10.0.0.50:1514 local1 notice
 
defaults
    mode http
    option httplog
    log global

The log 10.0.0.50:1514 local0 directive sends logs to the syslog server at 10.0.0.50. Loki with a syslog receiver then processes them, and Grafana can display the logs alongside the metrics.

#The ELK Stack

For ELK, the same pattern applies: logs are sent to syslog, collected by Logstash via a syslog or filebeat input, stored in Elasticsearch, and visualized in Kibana. The main Logstash filters usually extract the timestamp, frontend, backend, and status code from the HTTP log lines.

#Tracing and Correlation IDs

#Adding a Correlation ID

When a single request passes through many services, you need the same ID across the whole trail. HAProxy can generate and propagate it:

frontend api_front
    bind *:443 ssl crt /etc/haproxy/certs/fullchain.pem
    mode http
    option httplog
 
    http-request set-var(txn.request_id) \
        %[uuid(),lower]
    http-request set-header X-Request-ID %[var(txn.request_id)]
 
    log-format "%ci %[var(txn.request_id)] %ST %B %tr"
    default_backend api_back

http-request set-var(txn.request_id) %[uuid(),lower] generates a UUID, then set-header X-Request-ID forwards it to the backend. The log-format directive includes this ID in every log line.

#Tracing Requests Across Services

With X-Request-ID in every service's logs, you can trace a single request:

curl -s -I https://api.example.com/orders/1 -k | grep -i x-request-id

Note the X-Request-ID value from curl -s -I ... | grep -i x-request-id, then use that value to search HAProxy log lines and application logs. This is the basic distributed tracing pattern without extra libraries.

#Connecting to External Tracing

If you already use OpenTelemetry, this correlation ID can be used as the traceparent:

http-request set-header traceparent \
    "00-%[var(txn.request_id)]-0000000000000000-01"

The line above forms a simple traceparent header from the same UUID. Backends using OpenTelemetry will connect directly to the same trace.

#Designing Healthy Observability

#Metrics, Logs, and Traces

A balanced combination:

• Metrics: request rate, error rate, latency, connections — for alerts and capacity.
• Logs: details of every request — for debugging.
• Traces: the journey of a single request across services — for finding bottlenecks.

All three complement each other; don't rely on just one.

#Keeping Costs Reasonable

Full logs are expensive. Apply sampling and tiered retention policies: full logs for a few days, summaries for longer. Store Prometheus metrics with retention matched to your alerting needs.

#Closing

Episode 18 makes HAProxy part of the observability ecosystem: metrics for Prometheus and Grafana, logs for Loki and the ELK stack, and correlation IDs for tracing requests across services.

Key takeaways:

• prometheus-exporter opens a metrics endpoint without an extra agent.
• Key metrics: total requests, active connections, health failures, and latency.
• Send logs to remote syslog for Loki or Logstash.
• uuid() and X-Request-ID build cross-service correlation IDs.
• Metrics, logs, and traces are three complementary pillars.

In the next episode we'll cover HAProxy on Kubernetes & container environments — DaemonSet, Deployment, and sidecar deployment patterns, ingress controller versus standalone HAProxy, and configuration management with ConfigMaps and Secrets.