Learn Zabbix - High Availability (HA) & Disaster Recovery

#Introduction

Up to episode 15, all our deployments used a single Zabbix server. That works well until the server dies — and then all monitoring dies with it, exactly when it's needed most. Episode 16 covers two answers to that problem: HA clusters for continuous availability, and disaster recovery to make sure data and configuration can be restored.

HA and DR are complementary partners. HA keeps the system alive when one node fails. DR preserves the ability to come back from zero — in another location, from backups. Both are requirements for Zabbix to be worth using on critical infrastructure.

#Zabbix Server HA Cluster

#The Active and Standby Node Concept

A Zabbix HA cluster consists of two or more server nodes sharing one database. One active node processes all collection, while standby nodes monitor the active node's health. If the active node fails, one of the standbys takes over.

The advantage of this approach: no shared storage or dedicated load balancer is needed. All you need is an extra node and a database accessible to both nodes.

#HAClusterNodes Configuration

HA is configured in zabbix_server.conf on every node:

HANodeName=zabbix-server-01
HAClusterNodes=zabbix-server-01,zabbix-server-02

• HANodeName: this node's unique name, which must differ on each server.
• HAClusterNodes: the comma-separated list of names of all nodes in the cluster.

The parameters HANodeName and HAClusterNodes tell the server who it is and who its cluster peers are. Once both nodes are running, the cluster forms automatically.

#Failover Behavior

A standby node monitors the health of the active node and the database. When the active node fails, the standby waits a moment to confirm the failure is real, then takes over the active role and starts processing collection. This process is automatic, with no manual intervention.

zabbix_server -R ha_status

The command zabbix_server -R ha_status shows the status of each node in the cluster. Monitor this output periodically — HA status is also available as an internal item for Zabbix to monitor itself.

#Disaster Recovery

#Database Backup

The database is the heart of Zabbix — all configuration, history, and trends live there. Regular backups are the first priority. For MySQL:

mysqldump -uzabbix -p --single-transaction --routines --triggers zabbix | gzip > zabbix-backup.sql.gz

The --single-transaction option produces a consistent backup without locking tables. For PostgreSQL, use pg_dump; if you use TimescaleDB from episode 12, include a dump in a format that supports hypertable restore.

Store backups in a different location from the server — ideally outside the same location — and test the restore periodically.

#Configuration Backup

Besides the database, back up the configuration that isn't stored in the database:

• The zabbix_server.conf and zabbix_proxy.conf files.
• The zabbix_agent2.conf file and PSK keys on each host.
• The reverse proxy configuration and TLS certificates.

Templates can also be exported as YAML files and kept in version control — an effective way to keep templates documented and restorable at any time.

#Restore Procedure

Restoring means turning a backup back into a running system:

1. Install Zabbix with the same version as when the backup was made.
2. Create a new database and import the backup.
3. Adjust zabbix_server.conf to point at the restored database.
4. Start the server and verify the frontend shows the same data.

gunzip -c zabbix-backup.sql.gz | mysql -uzabbix -p zabbix

The command gunzip -c zabbix-backup.sql.gz | mysql -uzabbix -p zabbix restores the backup contents to the database. Always document this procedure and practice it in a test environment before it's actually needed.

#Designing an Availability Strategy

Architecture decisions can be staged:

• Standalone + scheduled backups: enough for dev or non-critical environments.
• HA cluster: for environments that need continuity without downtime.
• HA + DR across locations: for critical infrastructure with strict RPO and RTO.

Node A (active) ─┐
                 ├── Database ── daily backup → off-site location
Node B (standby)─┘

Define your RPO target (how much data loss is acceptable) and RTO target (how fast you must recover) from the start. These two numbers determine the backup frequency and the architectural complexity required.

#Closing

Episode 16 prepared Zabbix for worst-case scenarios: an HA cluster with active and standby nodes keeps collection running when one node fails, and database and configuration backups with a restore procedure ensure the system can be rebuilt from zero.

Key takeaways:

• An HA cluster shares one database; a standby node takes over when the active node fails.
• HANodeName and HAClusterNodes are the two core HA configuration parameters.
• Database backups cover configuration, history, and trends.
• Store server configuration and PSK keys outside the database.
• Restore must be tested periodically, not just documented.

In the next episode 17 we'll discuss performance tuning and capacity planning — adjusting caches and worker counts, database tuning, calculating NVP for sizing, and monitoring Zabbix's own health with internal checks.