Learn Wget - Timestamping & Incremental Optimization

#Introduction

In episode 14 you secured credentials and verified every file that came in. Now it's efficiency's turn: are you still re-downloading files that haven't changed? This habit costs more than you think — cloud bandwidth is paid per byte, waiting time accumulates, and the same servers are loaded repeatedly for identical data. Episode 15 teaches wget to be smarter: download only what changed, and do it politely.

#The Problem: Always Re-downloading, Paying Twice

Imagine mirroring 2 GB of documentation every morning. On day two, 99 percent of the content is identical to yesterday — but a naively configured wget will download all 2 GB again, paying for bandwidth, loading the server, and wasting time. In the cloud, this isn't just inefficiency — it's a bill. The solution is called timestamping.

#Timestamping with -N

The -N option (alias --timestamping) compares the local modification date with the modification date reported by the server:

wget -N https://example.com/data/daily-report.csv

The working logic is simple and decisive: if the local file is already as new as the server version, wget skips it. Run the same command again, and the result:

File 'daily-report.csv' not modified on server. Omitting download.

On the first run, all files are downloaded. On the second run, only changed files touch the network. It's like a receptionist asking "is this letter different from yesterday's?" before reprinting everything.

#-N for Incremental Updates

The power of -N really shows when combined with recursive mode. Instead of re-mirroring an entire site, wget only updates what changed:

wget -N -r -np https://docs.example.com/

-r for recursion, -np so it doesn't ascend to the parent directory, and -N ensures only changed files are re-downloaded. This is the incremental mirror pattern — the local replica stays complete, but the daily maintenance cost is much smaller.

#--no-if-modified-since

In -N mode, wget uses the If-Modified-Since header to ask the server: "send it only if it changed since this date". Compliant servers answer 304 Not Modified and wget skips it. The --no-if-modified-since option switches this strategy:

wget -N --no-if-modified-since https://example.com/data/file.bin

With this option, wget sends a HEAD request to check the modification date first, then downloads only if it actually changed. Use it when the server ignores the If-Modified-Since header — for example a backend behind a load balancer without shared state that answers 304 incorrectly. One extra request in exchange for accuracy.

#--limit-rate: The Speed Brake

A download that monopolizes all bandwidth is a bad neighbor. The --limit-rate option installs a brake:

wget --limit-rate=500k https://cdn.example.com/backup-2026.db

Values are accepted in units like 500k (kilobytes) or 2m (megabytes). It's like a water tap: opened fully, the whole house runs dry; set a little, everyone still showers. On shared networks, --limit-rate keeps large downloads from starving others — and on some ISPs, it actually makes large downloads more stable.

#--wait and --random-wait: The Polite Neighbor

In recursive mode, wget sends requests to the same server in succession. --wait inserts a pause between requests:

wget -r --wait=5 https://docs.example.com/

This gives the server room to breathe — important when mirroring small sites whose servers weren't designed for spikes. But an always-identical pause can read as a pattern, and patterns are easy to block. --random-wait breaks it up:

wget -r --wait=5 --random-wait https://docs.example.com/

The pause becomes random within a range around the --wait value. Imagine two guests: one knocks on the door exactly every 5 seconds, the other with an unpredictable pause — the second is far less annoying.

#--waitretry: Gradual Backoff on Failure

Failures aren't always permanent — a server may be busy now and recover a few seconds later. --waitretry gives wget a measured backoff strategy:

wget --waitretry=10 https://example.com/large-file.bin

Each retry waits longer, up to the specified maximum — instead of hammering a server that's down. This is the same backoff principle used across the industry: a smart network doesn't attack, it waits patiently.

#Complete Script: Automatic Mirror Update

Assemble everything into a self-contained script — the pattern for a mirror maintained every night without supervision:

#!/usr/bin/env bash
set -euo pipefail
 
MIRROR_DIR="$HOME/mirror/docs.example.com"
LOG="$HOME/mirror/mirror.log"
 
mkdir -p "$MIRROR_DIR"
 
wget --recursive --no-parent --timestamping \
  --directory-prefix="$MIRROR_DIR" \
  --limit-rate=500k --wait=2 --random-wait \
  --append-output="$LOG" \
  https://docs.example.com/
 
echo "Selesai: $(date -u +%FT%TZ)" >> "$LOG"

Breaking down the logic: --timestamping guarantees only changed files are downloaded, --limit-rate protects the shared network, --wait with --random-wait keeps the server comfortable, and --append-output writes an auditable trail of every run to a log. Scripts like this are what we'll schedule with cron in episode 16.

#Closing

Episode 15 turned wget from a naive downloader into a thinking one: timestamping with -N to skip unchanged files, --no-if-modified-since for handling non-compliant servers, --limit-rate for speed control, --wait and --random-wait for being a polite neighbor, and --waitretry for gradual backoff on failure.

The key thing to remember: efficiency and politeness are two sides of the same coin — both reduce load, and both are appreciated by your cloud bill and the servers you access alike.

In episode 16, all these skills become automated: scripting and automation — exit codes, logging, cron scheduling, and integrating wget into CI/CD pipelines. See you there!