Learning Cron Job - Job Monitoring & Observability

#Introduction

In episode 19 we managed distributed scheduling. Now we answer the nagging question: how do you know a job isn't running — before it's too late?

In episode 12 we built alerts for when a job fails (exit code != 0). But there's a sneakier failure: a job that never appears at all — a host is down, a crontab was deleted, a schedule was missed due to DST, or a cluster controller misbehaves. No exit code, no email, nothing. Detecting this kind of failure requires observability and auditing.

#Exporting Metrics to Prometheus

#The node_exporter Textfile Collector

The simplest pattern for job metrics from a host: the node_exporter textfile collector. The job script writes metrics to a .prom file, and node_exporter reads it when scraped.

#!/bin/bash
set -uo pipefail
 
DIR=/var/lib/node_exporter/textfile
START=$(date +%s)
STATUS=0
 
if /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1; then
    STATUS=1
else
    STATUS=0
fi
END=$(date +%s)
 
printf '# HELP cron_backup_success 1 jika backup sukses\n' > "$DIR/backup.prom"
printf '# TYPE cron_backup_success gauge\n' >> "$DIR/backup.prom"
printf 'cron_backup_success 1\n' >> "$DIR/backup.prom"   # status 1
 
printf '# TYPE cron_backup_duration_seconds gauge\n' >> "$DIR/backup.prom"
printf 'cron_backup_duration_seconds %d\n' "$((END - START))" >> "$DIR/backup.prom"

The resulting metrics:

# TYPE cron_backup_success gauge
cron_backup_success 1
# TYPE cron_backup_duration_seconds gauge
cron_backup_duration_seconds 42

#On-Miss Alerting: Detecting Jobs That Never Appear

The cron_backup_success metric only exists if the job finished writing it. If the job never runs, the metric isn't updated — and this is where on-miss alerts show their power:

groups:
  - name: cronjob
    rules:
      - alert: CronJobMissed
        expr: time() - cron_backup_success > 90000
        annotations:
          summary: Backup tidak muncul selama 25 jam

      - alert: CronJobMissing
        expr: absent(cron_backup_success)
        for: 24h
        annotations:
          summary: Metrik backup hilang — job mungkin tidak pernah jalan

The key difference: on-failure alerts (episode 12) catch jobs that failed; on-miss alerts catch jobs that never started. You need both.

#Metrics in a Cluster: Job Metrics and CronJob Status

Status CronJob di cluster

kubectl get cronjobs -A -o wide
kubectl get jobs -A

Also set up a Blackbox/Probe for services that a job is supposed to update — e.g. a version file on an HTTP endpoint that a periodic backup refreshes; if its contents go stale, there's a problem.

#Audit: Who Changed the Crontab

#Execution Audit Log

From episode 5, /var/log/cron records every execution. Add auditing at the user level:

sudo auditctl -w /var/spool/cron/ -p wa -k cron-change
sudo ausearch -k cron-change --start today

This answers "who changed the crontab and when" — important when schedules change without the team knowing.

#Periodic Reviews

Observability without process is idle data. Adopt a ritual:

• Monthly: audit all crontabs (crontab -l per user) — is every job still needed?
• On change: every crontab edit goes through review (e.g. crontab file + PR, from episode 4).
• Documentation: every job has an owner and a reason; an ownerless job is a risk.

for u in $(cut -d: -f1 /etc/passwd); do
    sudo crontab -l -u "$u" 2>/dev/null && echo "--- $u ---"
done

#Observability Layer Summary

#Closing

Key takeaways:

• The node_exporter textfile collector exports job duration and status to Prometheus.
• On-miss alerts catch jobs that never run — not just those that fail.
• Audit crontab with auditd to answer "who changed the schedule".
• Periodic reviews clean up stale jobs and establish ownership.
• Start observability from critical jobs, then expand.

In episode 21 we'll cover roadmap and community — where cronie is headed, the migration trends toward systemd timers and K8s CronJob, and the learning ecosystem: crontab.guru, man 5 crontab, and distro documentation!