Learning Cron Job - Randomization & Avoiding Peak Load

#Introduction

In episode 10 we scheduled backups. Now consider a bigger scenario: you manage a hundred servers, and all of them run a backup at 0 2 * * *. At 02.00, a hundred simultaneous backup processes hit the destination server — bandwidth is drained, storage is overloaded, and a "correct" schedule ends up creating a new problem.

This is the peak load problem you must think about from the start. The cause isn't a wrong schedule — it's the natural synchronization when everyone picks a "reasonable" hour. This episode covers how to spread the load.

#Randomization: Spreading Load Naturally

#sleep $((RANDOM % 300))

The simplest and highly effective trick: let the job wait for a random amount of time before starting its work.

30 2 * * * sleep $((RANDOM % 300)) && /usr/local/bin/backup.sh

• $((RANDOM % 300)) produces a random number from 0-299.
• The job is "scheduled to start" at 02.30, but actually works between 02.30 and 02.35.

With a hundred servers, that 5-minute window spreads a hundred backups across a wider time range — the load peak becomes much gentler.

#Hash-Based Intervals: A Deterministic Alternative

Pure randomness is unpredictable — sometimes good, sometimes awkward for auditing. A deterministic alternative: derive the delay from a hash of the hostname.

30 2 * * * sleep $((0x$(hostname | md5sum | cut -c1-4) % 300)) && /usr/local/bin/backup.sh

Each host's delay is constant across executions (easy to audit) but different between hosts (load is spread).

#RANDOM_DELAY in anacron

For cron.daily jobs managed by anacron, use RANDOM_DELAY (from episode 8):

RANDOM_DELAY=45

Anacron delays each job by a random amount of up to 45 minutes after the machine becomes active, spreading the boot load.

#Choosing Quiet Hours

#The 02.00-04.00 Window

Heavy backup needs ideally run during quiet hours — typically 02.00-04.00 local time, when user traffic is minimal and resource contention is low.

15 2 * * * /usr/local/bin/backup.sh

#Considerations Before Choosing the Hour

• User timezone: quiet hours in one region may be busy hours in another — align with where your workloads live (episode 16).
• Other backup schedules: avoid the same hour as your cloud provider's maintenance windows.
• Window headroom: make sure the job finishes before busy hours begin.

#Real Case: Many Hosts Backing Up Simultaneously

Imagine 100 hosts running 0 2 * * * rsync ... nas:/backup. Without mitigation:

1. At 02.00: 100 simultaneous rsync connections to the NAS.
2. NAS bandwidth is drained, queues grow long, throughput drops drastically.
3. Several jobs exceed their timeout and fail — then retry together, making things worse.

With random delay, the load curve changes from a sharp spike to a gentle slope — all jobs finish across a longer window, and none fail due to resource contention.

#The Complete Combined Pattern

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
 
30 2 * * * sleep $((RANDOM % 300)) && flock -n /var/lock/backup.lock timeout 50m /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

The combination: random delay (spread load) + flock (anti-overlap) + timeout (anti-hang) + log (trace).

#Strategy Comparison

#Closing

Key takeaways:

• Jobs scheduled at the same hour across many hosts create artificial peak load.
• sleep $((RANDOM % 300)) spreads load simply and effectively.
• Scale the random window with the number of hosts and job duration.
• Choose quiet hours (02.00-04.00) per your workload's timezone.
• Combine randomization with flock and timeout for maximum effect.

In episode 12 we'll cover notifications and alerting — MAILTO, sending messages to Slack/Telegram via curl, Zabbix/Prometheus monitoring integration, and an on-failure alert pattern that only bothers you when something is genuinely wrong!