Learn Wget - Production Readiness & Security Hardening

#Introduction

In episode 18 you learned to read failures, and in episode 20 you saw wget working inside Docker, provisioning scripts, and CI pipelines. Now comes the most important question: are your downloads safe for production? On a laptop, a careless flag only breaks your own download. On a production server — or inside a pipeline serving hundreds of users — the same mistake can become a real security hole.

The key concept: every wget flag is a security decision. Wget's defaults are safe enough, but many "let's get it done fast" habits — turning off certificate verification, skipping checksum verification, unlimited crawling — actually create holes. Episode 21 is a hardening checklist: mandatory habits, correct techniques, and traps to avoid.

#HTTPS Always Verified

Since wget 1.10, certificate verification is the default. That means every successful HTTPS download happens only if the server certificate is valid and signed by a CA the system trusts. If verification fails, wget exits with code 5 — and in episode 18 we already agreed to never turn off verification.

# JANGAN lakukan ini di produksi
wget --no-check-certificate https://example.com/file.zip

Instead of turning off verification, fix what's wrong. If the CA isn't known to the system, point to its bundle explicitly:

wget --ca-certificate=/etc/ssl/certs/ca-certificates.crt \
  https://example.com/file.zip

Also check the system date — a common mistake that makes "valid" certificates look expired. And for crawls, use --https-only so wget never descends to plain HTTP when it finds mixed links.

#Checksum and Signature Verification

A technically successful download isn't necessarily safe. Files can be corrupted along the way, or — more dangerously — tampered with by an attacker who controls the network. That's why production never stops at the download: always verify.

wget https://example.com/app-1.2.3.tar.gz
wget https://example.com/SHA256SUMS
sha256sum -c SHA256SUMS --ignore-missing
gpg --verify app-1.2.3.tar.gz.sig app-1.2.3.tar.gz

sha256sum -c compares the file's checksum against the official list, and gpg --verify authenticates the release's digital signature. The checksum proves the file wasn't changed; the GPG signature proves the file came from the legitimate source. Those two layers are the gold standard of software distribution — and both are easy to set up alongside wget.

#Restricting Excessive Recursive Crawls

An unbounded crawl in production is a recipe for disaster: your server can be seen as an attacker, the target site can go down, and bandwidth is wasted on unwanted data. Limit every dimension — depth, scope, and speed:

wget -r -l 3 -np --max-redirect 5 \
  --wait=2 --random-wait --limit-rate=500k \
  https://docs.example.com/guide/

Let's break down the decision behind each flag:

• -l 3 — stop after three link levels; without this, a crawl could traverse the internet endlessly.
• -np — don't ascend to the parent directory; the crawl stays within the allowed area.
• --max-redirect 5 — limit redirects so requests aren't carried in circles.
• --wait=2 --random-wait — pauses between requests with a random pattern, respecting server load.
• --limit-rate=500k — fence the bandwidth, don't exhaust the production connection.

#Security Checklist for Automated Downloads

Here are the habits that must become reflexes when wget runs automatically — in cron, in CI, or in server scripts.

#Manage .netrc with Strict Permissions

wget reads ~/.netrc to grab FTP and HTTP credentials automatically. It's convenient, but that file stores passwords in plain text — so its permissions must be guarded:

machine ftp.example.com
login arman
password rahasia

chmod 600 ~/.netrc

Permission 600 ensures only its owner can read it. Never commit this file to a repository, and if needed, generate its contents from a secret manager at deploy time — not by storing it as a static file in an image.

#Credentials Never in Command Arguments

The most common mistake in the real world is writing credentials directly on the command line. The problem isn't just shell history — a process's argument contents are visible to all users via ps while the process runs. In CI, the command line is even recorded in pipeline logs.

# Password muncul di ps, riwayat shell, dan log CI
wget --http-user=arman --http-password=sandirahasia \
  https://example.com/file.zip

Use .netrc (with 600 permissions), or if you need to run an external program to fetch the password, use --use-askpass — wget calls that command to obtain credentials without touching the arguments.

#Always on the Latest Stable Version

The journey from 1.24.5 to 1.25.0 we discussed in episode 19 closed real security holes, and that security maintenance continues. wget doesn't update itself — there's no auto-update. The version installed on your server is the version responsible for your downloads' security.

wget --version

Make version checking part of your routine: apt update && apt upgrade wget on Debian/Ubuntu distributions, or dnf update wget on RHEL/Fedora. These small habits are what close holes before they're exploited.

#Monitor Automated Download Logs

Automated downloads running without supervision are debt. Give them an auditable trail with -o, then handle failures explicitly:

wget -o /var/log/mirror.log -r -l 2 -np \
  https://example.com/
if [ $? -ne 0 ]; then
  echo "mirror gagal" | mail -s "wget alert" admin@example.com
fi

This log answers the question "what happened at 02:00 yesterday?" in one second — far more valuable than guessing from memory. Also check the exit code explicitly: 4 for network, 5 for SSL, 8 for server errors. A good alarm is one that knows the problem's category from the start.

#Closing

Episode 21 closed the security side with five pillars: HTTPS with uncompromising certificate verification, checksum and GPG verification after downloads, restricting excessive recursive crawls, a security checklist for automated downloads — .netrc with 600 permissions, credentials that never touch arguments, always-updated versions, and always-monitored logs.

The key takeaway: wget security isn't about features, it's about habits. Its defaults are already safe; your job is to not break them for convenience, and to assert its limits explicitly — in every line of production scripts.

In episode 22 — the final episode — we close the entire journey: recap, best practices & final reflections from the 23 episodes of Learn Wget. See you there!