Learn Wget - Scripting & Automation

#Introduction

In episode 15 you made wget efficient — only what changed gets downloaded, and everything is done politely. Now it's time for wget to work without supervision. Scripts don't read screens, don't judge results with eyes, and will never wake up at night to check whether a download succeeded. Episode 16 teaches the language machines actually understand: exit codes, log-friendly output modes, cron scheduling, and CI/CD pipeline integration.

#Exit Codes: The Language Between wget and Scripts

Every time wget finishes, it exits with an exit code — a number that tells how the process ended. This is the only thing a script can reliably read, via the $? variable:

wget -q https://example.com/file.tar.gz
echo "wget keluar dengan kode: $?"

The most important codes for script logic:

• 0 — success, no problems.
• 1 — generic error.
• 2 — parse error (options, .wgetrc, or .netrc).
• 3 — file I/O error.
• 4 — network failure.
• 5 — SSL verification failed.
• 6 — user/password authentication failed.
• 7 — protocol error.
• 8 — the server responded with an error (4xx/5xx).

Note the two most common: 4 for network and 5 for certificates — both carry diagnostic information far more useful than a plain "failed".

#Output Modes: -q, -nv, -o, and -a

On screen, wget's progress bar looks alive. In logs, a progress bar is garbage. Four options control output behavior:

• -q (alias --quiet) — turns off all output except errors.
• -nv (alias --no-verbose) — concise mode: still shows URLs and important messages, without the progress meter.
• -o logfile — writes messages to a file, overwriting the old contents.
• -a logfile (alias --append-output) — writes messages to a file, appending to the old contents.

wget -nv -a /var/log/mirror.log https://docs.example.com/

The -nv -a combination is the healthiest pairing for automation: output doesn't flood the terminal, and every run adds to an auditable file — exactly the pattern we used in episode 15.

#The if wget ...; then Pattern

Once wget talks via exit codes, it can enter conditional logic. The most common pattern is if wget ...; then:

if wget -q --timestamping --directory-prefix="$HOME/data" \
  "https://api.example.com/dump.zip"; then
  echo "unduhan sukses"
else
  echo "unduhan gagal" >&2
  exit 1
fi

This pattern makes a wget failure change the script's flow — not just print an error on a screen nobody may ever read. In scripts that don't use this pattern, a failed download is treated as a success simply because the next command still runs.

#Scheduled Download Script

Assemble everything into a self-contained script — representative of what you'll encounter in the real world:

#!/usr/bin/env bash
set -euo pipefail
 
URL="${1:-https://example.com/report/daily.csv}"
DEST="${2:-$HOME/data}"
LOG="$DEST/download.log"
 
mkdir -p "$DEST"
 
if wget --quiet --timestamping --directory-prefix="$DEST" "$URL"; then
  echo "$(date -u +%FT%TZ) OK $URL" >> "$LOG"
else
  code=$?
  echo "$(date -u +%FT%TZ) GAGAL kode=$code $URL" >> "$LOG"
  exit "$code"
fi

Breaking down the logic: set -euo pipefail stops the script on unhandled errors; --timestamping prevents re-downloading the same file; the exit code is captured and logged with a timestamp; and failures are forwarded as the exit code — the final signal for cron or CI.

#Cron: The System Scheduler

cron is the system's built-in scheduler that runs commands at specific times. The script above just needs to be registered:

0 3 * * * /home/budi/bin/unduh-harian.sh >> /home/budi/data/cron.log 2>&1

This line runs the script every day at 03:00 — an hour when the network is quiet and the destination server is free. Two things often trip up beginners: cron runs with a minimal environment, so use absolute paths inside the script; and standard output is visible to no one, so redirect it to a log with >> ... 2>&1 or rely on wget's own -a.

#CI/CD: wget in the Pipeline

CI/CD pipelines are wget's second home: fetching artifacts, dependencies, or database dumps on every build, and verifying URLs before release. Two examples on the most common platforms.

#GitHub Actions: Fetching Artifacts

name: Download Dump
on:
  schedule:
    - cron: '0 4 * * *'
jobs:
  fetch:
    runs-on: ubuntu-latest
    steps:
      - name: Ambil dump database
        run: |
          wget --quiet --timestamping \
            --header="Authorization: Bearer $TOKEN" \
            -P data/ "$DUMP_URL"
        env:
          DUMP_URL: ${{ secrets.DUMP_URL }}
          TOKEN: ${{ secrets.DUMP_TOKEN }}

Notice the details that keep this job healthy: the URL and token are never literal in the file — both come from the secret store; --timestamping avoids unnecessary downloads; and -P data/ places the file in a directory the next step can use.

#GitLab CI: Smoke Testing with --spider

--spider is a special wget mode that only checks URL existence without downloading the content — perfect for smoke tests:

smoke-test:
  stage: test
  image: ubuntu:24.04
  script:
    - apt-get update && apt-get install -y wget
    - wget --spider --quiet https://staging.example.com/health
    - wget --spider --quiet https://staging.example.com/api/v1/status

If a URL responds with a 4xx/5xx error, wget exits with code 8 and the step fails — and a failed step fails the pipeline. With --spider, you get the first alarm before users complain, without loading the server with a full download.

#Closing

Episode 16 turned wget into an automation machine: understanding exit codes from 0 to 8 as the language of failure, controlling output with -q, -nv, -o, and -a, using the if wget ...; then pattern for script logic, scheduling daily downloads with cron, and integrating wget into GitHub Actions and GitLab CI for fetching artifacts and smoke testing with --spider.

The key thing to remember: a script doesn't read the screen — it reads the exit code. Teach wget to speak machine language, and you can hand it any job without supervision.

In episode 17, we take wget to a different level: WARC and web archiving — recording every request and response in the standard web archive format, combining it with recursive crawls, and analyzing the results. See you there!