Learning Cron Job - Notifications & Alerting

#Introduction

In episode 11 we spread the load so all jobs finish on time. But one question remains unanswered: how do you know a job failed? A job that fails silently is one of the biggest dangers in automation — the system appears to run normally, backups never happen, and you only find out when data is lost. This episode builds the right notification and alerting layer: informative, targeted, and not noisy.

#MAILTO: Email Notifications

From episode 5, cron job output can be sent via email using MAILTO:

MAILTO=ops@example.com
 
30 2 * * * /usr/local/bin/backup.sh

#Per-Job MAILTO Variations

MAILTO can be set per line by placing it before the command:

SHELL=/bin/bash
MAILTO=ops@example.com
 
30 2 * * * /usr/local/bin/backup.sh
15 3 * * * MAILTO=dba@example.com /usr/local/bin/pg_dump.sh

This pattern is useful when different jobs are handled by different teams.

#Notifications to Slack/Telegram via curl

#Slack Webhooks

Slack has Incoming Webhooks — a URL that accepts JSON and posts it to a channel:

#!/bin/bash
WEBHOOK_URL="https://hooks.slack.com/services/T000/B000/XXX"
curl -fsS -X POST -H 'Content-type: application/json' \
  --data '{"text":"GAGAL: backup tanggal '"$(date +%F)"'"}' \
  "$WEBHOOK_URL"

#Telegram Bots

Telegram uses the Bot API with a bot token and chat-id:

#!/bin/bash
TOKEN="123456:ABC-DEF..."
CHAT_ID="-100123456789"
curl -fsS "https://api.telegram.org/bot${TOKEN}/sendMessage" \
  --data-urlencode "chat_id=${CHAT_ID}" \
  --data-urlencode "text=GAGAL: backup $(date +%F)"

#On-Failure Alerts: Only Interrupt When It Fails

Good alerting only fires when the exit code != 0. If you send a notification on every success, your team will lose sensitivity to messages — "alert fatigue". Only send success for jobs that genuinely need visual confirmation:

#!/bin/bash
set -uo pipefail
 
LOG=/var/log/backup.log
echo "=== Backup $(date) ===" >> "$LOG"
 
if /usr/local/bin/backup.sh >> "$LOG" 2>&1; then
    echo "OK" >> "$LOG"
else
    echo "GAGAL (exit $?)" >> "$LOG"
    curl -fsS "$WEBHOOK_URL" -H 'Content-type: application/json' \
        --data '{"text":"GAGAL backup: '"$(date)"'"}' || true
    exit 1
fi

Note the || true on curl — a failed alert must not change the job's status. With this pattern, an alert is a pure failure signal:

• Exit 0 → success, no alert.
• Exit != 0 → failed, send an alert.
• Internal retry that eventually succeeds → exit 0, no alert needed.

#Monitoring Integration: Zabbix and Prometheus

#Zabbix: Trapper Items

Zabbix supports sender — hosts push metric values, and alerts trigger when a value exceeds a threshold (e.g. duration > 3600 seconds):

zabbix_sender -z zabbix.example.com -s "host-db" -k "backup.duration" -o 42

#Prometheus: Pushgateway

Prometheus is normally pull-based (scrape), but for one-shot jobs it uses the Pushgateway:

cat <<EOF | curl -fsS --data-binary @- http://pushgw:9091/metrics/job/backup/instance/host-db
# TYPE backup_success gauge
backup_success 0
EOF

A Prometheus alert rule can detect backup_success == 0 or a metric that doesn't appear at all (missed job — covered in episode 20).

#Notification Channel Summary

#Closing

Key takeaways:

• MAILTO sends job output via email; it can be set per job.
• Slack/Telegram via curl gives real-time notifications.
• On-failure alerts: send only when exit code != 0, preventing alert fatigue.
• Guard alerts with || true so a failed notification doesn't corrupt the status.
• Integrate Zabbix sender / Prometheus Pushgateway for metrics + alerting.

In episode 13 we enter the security phase: crontab security — allow/deny — how /etc/cron.allow and /etc/cron.deny control who may create schedules, when to use default deny, and why jobs shouldn't run as root without reason!