Learn Wazuh - Performance Tuning & Capacity Planning

#Introduction

In episode 18 we connected Wazuh to TheHive, Shuffle, and notification paths. The more that's sent outward, the greater the load on the manager and indexer. In episode 19 we step back to care for the foundation: how to keep Wazuh responsive as the number of agents and event volume grows.

We'll cover two things that are often mixed up. Performance tuning is action so existing components work efficiently, like adjusting the indexer JVM heap or enlarging the queue. Capacity planning is forward-looking calculation to make sure resources are sufficient, like how much CPU, RAM, and disk for 100, 1,000, or 10,000 agents.

The golden rule is simple: don't add resources before removing waste. Always measure first, then tune, then decide to add hardware. We start with the unit most often used to measure load, events per second, or EPS.

#EPS as the Primary Measurement Unit

EPS is the number of events entering Wazuh per second. All components, from agent, manager, to indexer, are measured with this unit. You can estimate total EPS with a simple formula:

Total EPS equals the number of agents times the average events per agent per second. One normal agent produces about 1 to 5 EPS depending on the sources monitored. A Windows agent with many event channels tends to be busier than a Linux agent that only reads a few log files.

From this EPS we derive two important needs. First, the compute need to process events on time. Second, the storage need, because every incoming event is stored as a document in the indexer. So before talking about hardware, get used to calculating EPS first.

#Measuring a Baseline Before Tuning

Tuning without data is just guessing. Before changing anything, gather actual numbers from the dashboard and API. A few commands are useful for reading cluster condition:

curl -sk https://localhost:9200/_cat/health?v
curl -sk https://localhost:9200/_cat/indices/wazuh-alerts*?h=index,docs.count,store.size

For the manager side, read the state file that records queues and discarded events:

cat /var/ossec/var/run/wazuh-remoted.state

Pay attention to the discarded_count field. If this number keeps rising, events are starting to be dropped before being processed. That's a sign the queue or workers need tuning, not a sign to immediately add RAM.

#Indexer Tuning: JVM Heap

The indexer based on OpenSearch runs on Java, and the JVM heap is the key to its performance. A heap that's too small makes garbage collection run too often, while a heap that's too large can leave the operating system short of memory for page cache.

The heap settings are in the indexer's jvm.options file:

-Xms4g
-Xmx4g

As a rule of thumb, set the minimum and maximum equal so Java doesn't keep resizing the heap. A general guideline is around half of the indexer node's total RAM, with a reasonable cap of 4 GB for small deployments and 8 to 16 GB for large ones.

#Retention with Index Lifecycle

Wazuh alert data grows without stopping, and storage is always the first to give up. Index lifecycle manages the index life cycle: when to rollover, move, and finally delete. Wazuh uses this policy through the Index State Management mechanism on the indexer.

A simple example policy: alert indices stay active for one day, then are deleted after 90 days.

{
  "policy": {
    "description": "Retensi alert Wazuh selama 90 hari",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          { "rollover": { "min_index_age": "1d" } }
        ],
        "transitions": [
          { "state_name": "delete" }
        ]
      },
      {
        "name": "delete",
        "actions": [
          { "delete": {} }
        ]
      }
    ]
  }
}

Choose the retention period according to compliance needs. Regulations like PCI DSS require certain storage periods, so adjust to the obligations in force, not just how much disk remains. Remember, deleted data can't be recovered unless there's a separate backup.

#Reasonable Shard Sizing

Each index is divided into shards, and shards that are too small or too numerous are equally troublesome. Too small creates many empty shards that only add overhead. Too large makes recovery and redistribution operations slow.

A general guideline is one shard holding about 5 to 20 GB of data. With daily rollover, the daily index size can be controlled to stay in that range. The actual shard size can be estimated from EPS times rollover duration, then times the average size of one alert.

Don't be tempted to add shards just to look scalable. Notice whether dashboard queries are already slow, then adjust the shard count and indexer node scale.

#Queue and Workers on the Manager

The manager receives events from thousands of agents via wazuh-remoted, then processes them in wazuh-analysisd. These two points have queues and workers that can be tuned when spikes occur.

<remote>
  <connection>secure</connection>
  <port>1514</port>
  <protocol>tcp</protocol>
  <queue_size>196608</queue_size>
</remote>

remoted.worker_pool=8
analysisd.event_threads=4
analysisd.winevt_threads=8
analysisd.rule_matching_threads=8

Enlarge the queue only if queues pile up during short spikes. Add workers only to the event categories that are truly the bottleneck. Adding all workers to maximum values instead triggers lots of context switching and can slow everything down.

#Capacity Planning: 100, 1,000, and 10,000 Agents

After tuning, we calculate resources. These numbers are reasonable starting points, not fixed prices:

• 100 agents. One all-in-one server is still reasonable: 4 vCPU, 8 GB RAM, and a 4 GB indexer heap. 500 GB of disk is enough for 90 days of retention if the average EPS isn't wild.
• 1,000 agents. Start separating roles. A manager with 8 vCPU and 16 GB RAM, a single-node indexer with 16 GB RAM and an 8 GB heap, and a separate dashboard with 4 vCPU and 8 GB RAM. EPS at this level usually reaches thousands, so watch the indexer most closely.
• 10,000 agents. Enter distributed architecture. A manager cluster with several worker nodes divides the agent load, and a multi-node indexer cluster divides indexing and search. Add indexer nodes every time EPS needs grow by roughly 1,000 to 2,000.

Disk follows with a simple formula: total EPS times 86,400 seconds times the retention length, then times the average alert size. Because this number can balloon quickly, use that formula to challenge your own retention policy.

#Conclusion

In episode 19 you learned how to keep Wazuh agile:

• EPS is the primary unit for measuring load and planning capacity.
• Measure a baseline first, then tune; don't guess.
• The indexer JVM heap is balanced between Java needs and page cache.
• Index lifecycle controls retention and prevents full disks.
• Shard sizes are kept from being too small or too large.
• Queues and workers are tuned per bottleneck, not all at once.
• The capacity for 100, 1,000, and 10,000 agents needs different architectures.

Key takeaways:

• Always measure before changing configuration or adding hardware.
• Total EPS is the basis for CPU, RAM, and disk calculations.
• Raise heap and workers gradually while monitoring the impact.
• The retention policy determines long-term storage costs.
• Disk is often the first point to give up, so design retention from the start.
• When one node is no longer enough, start thinking about a cluster architecture.

In episode 20 we cover the side most often forgotten: monitoring Wazuh itself and setting up proper backup and recovery. See you there!