Learn HAProxy - Production Hardening & Best Practices

#Introduction

Episode 22 is the climax of the series. All the skills you've built — routing, security, performance, observability, automation — now come together into one practice: running HAProxy in production calmly.

You'll walk away with a complete production checklist, backup and disaster recovery strategies, a safe upgrade procedure, and the habit of documenting conventions so the team can operate HAProxy without relying on a single person.

#Production Checklist

#Security

Check before go-live:

• TLS locked down: no-sslv3, no-tlsv10, no-tlsv11, modern ciphers.
• HSTS active and HTTP redirected to HTTPS.
• Admin endpoints hidden, stats password-protected.
• Configuration file with strict permissions: chmod 600.
• Rate limiting and deny conditions in place on the frontend.

#Availability

Make sure there's no point of failure:

• At least two HAProxy nodes with VRRP and a virtual IP.
• Active health checks for all backends.
• Backup servers as a safety net.
• Failover tested regularly, not just during incidents.

#Observability

You must be able to see before you can predict:

• Prometheus metrics active and already scraped by Grafana.
• HTTP logs flowing to a log storage system.
• Correlation IDs wired across all services.
• Dashboards and SLO alerts agreed upon.

#Operational Readiness

Repetitive processes must be documented:

• Failover, high traffic, and DNS runbooks stored.
• Upgrade procedure written and tested on staging.
• Scheduled configuration backups running.
• Only authorized people can change production configuration.

#Disaster Recovery and Backup

#Backing Up the Configuration

The configuration is your most valuable asset. Back up the whole configuration directory:

tar czf /backup/haproxy-$(date +%F).tar.gz \
  /etc/haproxy/ /run/haproxy-admin.sock 2>/dev/null || true
ls -lh /backup/haproxy-*.tar.gz | tail -n 3

tar czf /backup/haproxy-$(date +%F).tar.gz /etc/haproxy/ packages the configuration, MAPs, and certificates with the date in the filename. Store backups in storage outside the machine, for example object storage.

#Restore Drills

A backup without a restore test isn't a backup. Regular drills:

1. Prepare a clean machine with the same HAProxy version.
2. Extract the archive into /etc/haproxy/.
3. Run haproxy -c to validate.
4. Start the service and verify traffic.

tar xzf /backup/haproxy-2026-08-10.tar.gz -C /
haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl restart haproxy

The tar xzf ... -C / sequence followed by haproxy -c ensures the files are recovered and valid before the service is started.

#The Disaster Recovery Plan

A concise DR document must answer:

• What's the maximum recovery time (RTO)?
• How much data loss is acceptable (RPO)?
• Who can run the restore outside working hours?
• Where are the backups stored and how do you access them?

Clear answers make decisions easy during a panic.

#A Safe Upgrade Strategy

#Know Your Version and Changes

Before upgrading, record the current and target versions:

haproxy -v
apt-cache policy haproxy | head -n 5

apt-cache policy haproxy shows the installed and available versions. Read the changelog between versions: removed or changed directives can invalidate old configurations.

#A Gradual Upgrade Procedure

Don't upgrade all nodes at once:

1. Validate the configuration with the new binary version on staging.
2. Upgrade one node in drain mode, starting with the least important node.
3. Watch metrics and logs for a few hours.
4. Move on to other nodes once everything is stable.

echo "set server web_back/node1 state drain" | socat stdio /run/haproxy.sock
sudo apt upgrade -y haproxy
haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl restart haproxy
echo "set server web_back/node1 state ready" | socat stdio /run/haproxy.sock

The drain, upgrade, validate, restart, ready pattern means an upgrade never disrupts running traffic.

#A Clear Rollback

Prepare a way back before you start:

• Keep the old binary version, or at least record its version number.
• Back up the configuration before upgrading.
• If metrics worsen, restore the version and reload.

#Documenting Conventions and Support Boundaries

#Conventions Worth Writing Down

Small documentation saves a lot of time:

• Frontend and backend naming rules.
• Comment format for every configuration section.
• The approval flow for configuration changes.
• Backup locations and restore procedures.

# owner: platform-team
# contact: #infra on slack
# last-reviewed: 2026-08-10
# configs under version control: yes

Elements like these at the top of haproxy.cfg communicate ownership and status at a glance to anyone who opens it.

#Realistic Support Boundaries

Be honest about what is supported:

• How many HAProxy nodes can be changed without approval?
• Which features are managed by the platform team versus the application team?
• Which HAProxy versions are supported and when must you upgrade?
• What isn't within the team's responsibility: this avoids wrong expectations.

#Closing the Series with a Reflection

Look back at the journey: you started with TCP and HTTP concepts in episode 0, and closed it in episode 22 with a production checklist. From a simple installation to SLOs, HAProxy is no longer a black box but a tool you fully master.

#Closing

Episode 22 wraps everything into one: a checklist that turns configuration into mature production, backup and DR that make disasters less frightening, gradual upgrades, and documentation that puts the team on the same footing.

Key takeaways:

• The production checklist covers security, availability, observability, and operations.
• A backup without a restore test isn't a backup.
• Upgrade one node at a time with the drain, restart, ready pattern.
• Document conventions and support boundaries for the whole team.
• HAProxy is a tool: configuration and operational discipline determine the outcome.

Congratulations on finishing the Learn HAProxy series! You now have a complete map from prerequisites to production. The next step is applying it in a real environment — start small, validate every change, and build your own experience. The series may be over, but your journey with HAProxy has just begun.