Learning Cron Job - Production Patterns: Lock, Retry & Idempotency

#Introduction

In episode 8 we understood anacron and system directories. Now we go up a level: from "job runs" to "job is reliable". The difference shows when something unexpected happens — a job runs two minutes too long, a script crashes halfway, or a server is overloaded.

There are three pillars of reliability we'll build: lock (prevent overlap), retry (handle transient failures), and idempotency (safe to rerun), plus timeout as a safety net.

#Lock: Preventing Overlapping Jobs

#The Problem

Imagine a backup that usually finishes in 40 minutes, but one day runs for 70 minutes. With an hourly schedule, the next instance starts before the first one finishes — two backup processes run at once, overwrite each other's files, and corrupt both.

Cron doesn't prevent this by default. You have to do it yourself.

#flock: An Elegant File Lock

flock locks a file and only runs the command if the lock is available:

0 * * * * flock -n /var/lock/backup.lock /usr/local/bin/backup.sh

• -n (non-blocking): if another process already holds the lock, the command doesn't run — it doesn't wait.
• If you want to wait for the lock to be released before starting (queueing), drop the -n.

The result: an overlapping second job exits immediately without damaging anything.

#A More Complete Locking Pattern

For full control and clear logs, use flock inside the script:

#!/bin/bash
set -euo pipefail
 
exec 9>/var/lock/backup.lock
if ! flock -n 9; then
    echo "Job sebelumnya masih berjalan, keluar." >> /var/log/backup.log
    exit 1
fi
 
rsync -a /data /backup >> /var/log/backup.log 2>&1

#Alternative: mkdir as a Lock

An old trick that's still useful — directory creation is atomic:

if ! mkdir /var/lock/backup.run 2>/dev/null; then
    echo "Lock sudah ada."; exit 1
fi
trap 'rmdir /var/lock/backup.run' EXIT

A failed mkdir means the lock is already held. trap ... EXIT ensures the lock is cleaned up even if the script errors.

#Retry: Handling Transient Failures

A momentary network drop, a database restart — many transient failures recover with a retry. A simple retry with a loop:

#!/bin/bash
MAX=3
for i in $(seq 1 $MAX); do
    if curl -fsS https://api.example.com/sync; then
        exit 0
    fi
    echo "Percobaan $i gagal, menunggu..." >> /var/log/sync.log
    sleep $((i * 30))
done
echo "Gagal setelah $MAX percobaan" >> /var/log/sync.log
exit 1

A backoff pattern (sleep grows longer each attempt) avoids hammering a service that's already struggling. Don't retry forever at a fixed interval — set a limit, then fail with a clear exit code.

#Idempotency: Safe to Rerun

Idempotent means running an operation many times produces the same result as running it once. A full backup that overwrites with the latest data, a sync that produces the same state — both are idempotent.

rsync -a --delete /data/ /backup/          # hasil akhir selalu sama
pg_dump -Fc db > /backup/db.dump           # menimpa, bukan menumpuk

The opposite of idempotent: a script that accumulates — e.g. pg_dump >> file or creating a new file each time without cleaning up old ones. A job rerun after a retry will produce duplicates.

#Timeout: The Last Safety Net

A script can hang — waiting on the network, on a lock that never releases. timeout cuts off the process after a given duration:

0 * * * * timeout 30m /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

timeout 30m kills the process (SIGTERM by default) if it exceeds 30 minutes. Without this, a hung job holds the lock forever and blocks every subsequent execution.

#Pattern Summary

#Closing

Key takeaways:

• A job that runs longer than its interval can overlap — prevent it with flock -n.
• Handle lock-acquisition failure explicitly: log + alert.
• Retry with backoff and a clear limit; don't retry forever.
• Make scripts idempotent so they're safe to rerun.
• timeout stops a hung job from blocking everything.

In episode 10 we'll apply all of this to a real case: backup and maintenance automation — scheduling rsync, restic, and pg_dump, verifying backup results, plus logrotate, tmp cleanup, and careful package updates!