Learn Wazuh - Monitoring Wazuh Itself & Backup

#Introduction

In episode 19 we tuned performance and calculated capacity. But the Wazuh we use to monitor the environment is useless if it itself has problems we don't notice. In episode 20 we turn the lens around: monitor Wazuh itself, then set up proper backup and recovery.

Many new admins start with big installations and advanced features, but forget that the security tool should be the most closely watched target. If the indexer turns red in the middle of an incident, the whole team loses visibility exactly when they need it most. The habit of checking Wazuh health daily is part of operations, not just an extra.

We'll cover four monitoring areas: indexer status, manager logs, agent health, and API metrics. Then we move to configuration backup, indexer snapshots, and a disaster recovery strategy for cluster architectures. All of this forms a safety net before a real incident arrives.

#Indexer Status: Green, Yellow, Red

OpenSearch reports cluster health in three colors. Green means all primary and replica shards are available. Yellow means not all replica shards are available yet, but primary data can still be read. Red means a primary shard is missing, and some data can't be accessed.

The fastest way to read this status is calling the health API:

curl -sk https://localhost:9200/_cluster/health?v
curl -sk https://localhost:9200/_cat/shards?v

Red status must be handled immediately: check whether a node is dead, whether the disk is full, and whether the last snapshot is available. Yellow status can run temporarily, but shouldn't be left dragging on because it's a sign something is abnormal with replication.

#Reading the Manager Logs

wazuh-manager writes its activity trail to several log files. The main file is ossec.log, which records everything the manager does, including errors while loading rules or configuration.

These simple commands help find problems:

tail -n 50 /var/ossec/logs/ossec.log
grep -i error /var/ossec/logs/ossec.log | tail -n 20

Beyond that file, there's api.log for API activity, cluster.log for inter-node cluster communication, and integrations.log for integration results like TheHive and Shuffle. The habit of reading ossec.log before changing configuration will save a lot of time when something isn't working as expected.

#Agent Health

A healthy agent is the foundation of all Wazuh data. In the dashboard, the agents tab shows each endpoint's status: Active, Disconnected, or Never connected. Agents that disconnect often need checking, because losing an agent means losing a detection window.

On the agent side, the log file at /var/ossec/logs/ossec.log records local problems like a full buffer or failure to send events. Meanwhile on the manager side, we can check when the manager last received an event from an agent. Agent status that keeps flipping between Active and Disconnected is usually related to network, resources, or communication timeouts.

#Metrics via the Wazuh API

Wazuh provides an API that can be used to fetch metrics without opening the dashboard. For example, the agent list by status, or event statistics per daemon.

An API call with basic authentication looks like this:

curl -sk -u admin:PAKAI_PASSWORD_API https://localhost:55000/agents?status=active

From this API, you can build your own monitoring scripts: check the number of disconnected agents, observe daemon statistics, and send notifications when anomalies occur. The same API is also used by the dashboard, so understanding its responses is useful for automation.

#Backing Up the Manager Configuration

Wazuh configuration is far easier to back up than its data, and it almost never changes as fast as alert data. The most important thing to save is the /var/ossec/etc directory containing ossec.conf, agent keys, and internal settings.

A simple tar command is enough for a baseline:

tar -czf /backup/wazuh-config.tar.gz /var/ossec/etc /var/ossec/var/db

Copy that archive to a place separate from the Wazuh server. Remember, a backup stored on the same server doesn't help when that server is lost. Also test its recovery, not just make sure the file is stored.

#Indexer Snapshots

Alert data in the indexer is backed up through the snapshot mechanism. Snapshots can be sent to a repository in the form of a filesystem or object storage. Make sure the repository path is accessible to the indexer nodes, then register that repository.

curl -sk -X PUT https://localhost:9200/_snapshot/wazuh-backup -H "Content-Type: application/json" -d '{"type": "fs", "settings": {"location": "/backup/snapshots"}}'

curl -sk -X PUT https://localhost:9200/_snapshot/wazuh-backup/snapshot-1?wait_for_completion=true

Snapshots are point-in-time: they capture the data condition at the moment they run. Schedule snapshots regularly, for example every day outside peak hours, and keep several generations. That way, when data corruption or accidental deletion occurs, we can return to the last healthy point.

#DR Strategy for a Manager Cluster

Disaster recovery isn't just about having a data copy, but a tested process to get back on our feet. For a cluster architecture, start with the most crucial thing: agent keys and identities in client.keys. Without this copy, all agents must be re-enrolled one by one after recovery.

Document the topology, versions, and step-by-step recovery procedures. Set reasonable recovery targets: how long the system may be down, and how much data may be lost. From this you know how often backups must run and how fast the restore procedure must work.

Finally, run recovery drills regularly. These drills find holes that aren't visible when reading documentation, such as wrong file permissions or an unmounted repository path. A team that has restored Wazuh from scratch will be far calmer when a real incident occurs.

#Conclusion

In episode 20 you practiced caring for Wazuh:

• Indexer cluster status is read as green, yellow, or red.
• Manager logs hold the error trail that's most useful when troubleshooting.
• Agent health is monitored from the dashboard and logs on both sides.
• The Wazuh API opens the way to automated monitoring.
• Manager configuration is backed up with archives stored separately.
• Indexer snapshots save alert data from loss.
• A DR strategy is only complete after its procedure is truly tested.

Key takeaways:

• Wazuh itself must be on the daily monitoring priority list.
• Red indexer status is an emergency that must be handled fast.
• A backup without a recovery test is just an illusion of security.
• Agent keys are the most crucial asset to restore.
• Schedule snapshots and keep several generations.
• Practice recovery regularly before it's really needed.

In episode 21 we look to the future: modern features in the latest stable version and the Wazuh 5.0 roadmap. See you there!