Learn Zabbix - Triggers, Events & Problem Detection

#Introduction

Items collect data, preprocessing cleans it. But so far Zabbix only "sees" without "understanding". Episode 7 gives Zabbix the ability that makes it valuable: triggers — conditions that assess whether an item value has deviated from normal. Without triggers, Zabbix is just a numbers warehouse; with triggers, it becomes a problem detection system.

This is also an episode about events — objects that mark status changes. The event lifecycle, severity, and correlation are the topics that separate a good monitoring operator from someone who merely copies trigger expressions. Let's dissect it from the foundation.

#Triggers and the Expression Builder

#Trigger Expression Anatomy

A trigger evaluates one or more items using aggregation functions. The basic syntax of a Zabbix trigger expression:

{host:key.function(param)} > value

A real example: average user CPU usage over one minute above 90 percent.

{host-target-01:system.cpu.util[,user].avg(1m)} > 90

The expression {host:key.avg(1m)} > 90 reads: the average key value over one minute exceeds 90. In the frontend, you write this via the expression builder, which provides functions and a real-time value preview — very helpful for avoiding syntax errors.

#Frequently Used Functions

A few functions you must know:

• last(): the item's latest value.
• avg(): the average over a time range.
• change(): the difference from the previous value.
• count(): the number of values meeting a condition in a range.

{host-target-01:system.uptime.change()} < 0

The trigger above fires if uptime drops — indicating a host reboot. Examples like this change() pattern are a classic way to detect unexpected restarts.

#Severity and Dependencies

#Five Severity Levels

Zabbix defines five problem severities, from lowest:

• Information: informational, not always a problem.
• Warning: needs attention.
• Average: a problem starting to cause disruption.
• High: a serious problem.
• Disaster: critical, needs immediate action.

Severity is not just a label. It's the foundation for filtering in actions — for example escalation only for High and above — and for display priority on the problem dashboard.

#Dependencies: Suppressing Cascading Alarms

When one server dies, every item on it becomes unavailable and dozens of triggers can fire at once. A dependency lets one trigger "shadow" another: if the parent trigger fires, dependent child triggers are automatically marked as not dependent and produce no notifications.

Host down (parent)
    └── Disk space full (not sent if parent fires)
    └── Service X down (not sent)

Setting up dependencies correctly is one of the most effective ways to reduce alarm noise. The hierarchy above lets an operator receive just one notification for a single root cause.

#Events: The Problem Lifecycle

#Problem, Recovery, and Update

When a trigger expression evaluates to true, Zabbix creates a problem event. When the expression goes false again, a recovery event is created and the problem is closed. In between, a problem can receive updates — comments, assignment, or severity changes — all recorded as event history.

normal → problem → (update/acknowledge) → recovery

Problem events carry important metadata: the trigger value, severity, host, and tags. All of these are available to actions and dashboard views.

#Event Tags and Correlation

Event tags are key-value pairs attached to an event, for example service:web or dc:jakarta. Tags unlock correlation: merging several events into one coherent problem, or grouping events from the same host. Correlation is an advanced weapon for large environments prone to duplicate alarms.

#Building Reliable Triggers

#Avoiding False Positives

A reliable trigger is one that doesn't easily fire falsely. Practical principles:

• Use avg() instead of last() for fluctuating metrics — for example CPU usage.
• Provide a recovery expression that separates the fire and recover thresholds, so a problem "sticks" until it truly recovers.
• Test expressions in Latest data and Problems before rolling out to many hosts. To check the raw value that will be evaluated, run zabbix_get -s 192.168.1.20 -k system.cpu.util[,user] from the server.

#Recovery Expression

By default, a trigger recovers when the expression evaluates to false. For noisy environments, set a separate recovery expression. For example: fire above 90 percent, only recover below 70 percent. This prevents the trigger from flickering around the threshold.

#Closing

Episode 7 gave Zabbix the ability to think: triggers with the expression builder and the last, avg, change, and count functions, five severity levels, dependencies to suppress cascading alarms, and the event lifecycle from problem to recovery with tags and correlation.

Key takeaways:

• A trigger evaluates item values with an expression and produces events.
• last(), avg(), change(), and count() are the four core functions.
• Severity determines priority and is the basis for action filters.
• Dependencies suppress chains of alarms from a single root cause.
• Events have a problem, update, recovery lifecycle — with tags for correlation.

In the next episode 8 we'll discuss actions, media, and alerting — configuring actions that respond to events, connecting media types like email, webhook, and Telegram, and designing escalations that don't flood operators.