Learn Wget - Troubleshooting & Debugging

#Introduction

In episode 17 you saw wget work as a web archive recorder via WARC — a tool that seems "invisible" until a download suddenly fails on a production server at midnight. Well, that's when debugging becomes what separates practitioners who guess from practitioners who read evidence.

The problem is, wget is quiet. When a download fails, it just exits with an exit code and maybe one line of error message — without explaining exactly where the failure occurred. Like a car stalling mid-road with no indicator lights: you know something is wrong, but not in which system. Episode 18 installs a "full dashboard" on wget: debug mode, server response header display, a status code and exit code dictionary, and solutions for the most common errors in the real world.

#Debug -d: Seeing Everything That Happens

The -d (debug) option makes wget open the machine's door and narrate every step it takes — including the request sent and the response received, complete with all headers. This is the first option to turn on whenever something feels strange.

wget -d https://example.com/

The output is long and noisy, but the information is structured. Here's an example of its shape:

---request begin---
GET / HTTP/1.1
User-Agent: Wget/1.25.0 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: example.com
Connection: Keep-Alive
---request end---
HTTP/1.1 200 OK
...

The key to reading it lies in two markers: ---request begin--- through ---request end--- shows exactly what wget sent to the server, and the lines after are the responses coming back. With this pattern you immediately know the failure point: if there are no response lines at all, the connection failed. If response headers appear but the file never completes, the problem is mid-transfer — not at connection start.

#-S: Seeing the Server's Response Headers

-d shows everything, but sometimes you only need one thing: the headers the server sent back. The -S (server-response) option prints the status line along with all response headers before the body. Combine it with -O - to write the body to the terminal, so you see both on one screen:

wget -S -O - https://example.com/

The first line out is the status line, for example HTTP/1.1 200 OK. That's where you start reading the server's health. Here's a map of the most common status codes:

#Exit Codes: The Language Between wget and Scripts

When wget finishes, it exits with an exit code. This number is the language scripts understand — not just "failed", but the failure's category. In your scripts you can check it explicitly:

wget -q https://example.com/file.iso
echo "exit code: $?"

One rule worth memorizing: except for codes 0 and 1, smaller-numbered exit codes take priority over larger-numbered ones when several error types occur together. That means if a download fails due to connection (4) and certificate (5) simultaneously, wget reports 4 — not 5.

#Common Errors and Their Solutions

Now let's map the most frequent errors to their root causes. Remember the principle from episode 9: don't guess, look at the evidence.

#robots.txt Blocking the Crawl

When wget works recursively, it honors the Robots Exclusion Standard — the server is asked to send robots.txt, and paths denied there won't be downloaded. The symptom: the crawl runs, but many URLs are silently skipped, or a Loading robots.txt; please ignore errors. line appears in the log. This isn't a bug — it's the server's policy being obeyed by wget.

wget -r -l 2 -e robots=off https://example.com/docs/

If you have permission to fetch the blocked paths (for example, a site you own), disable it with -e robots=off. Note: this isn't a "cheat", it's an ethical decision. A server forbids something for a reason; forcing it without permission is a violation.

#403 Forbidden

Status 403 means the server denies access — and the most common cause for wget is User-Agent blocking. Many servers, WAFs, and CDNs differentiate client identities. wget's default User-Agent (Wget/1.25.0) is easy to recognize and often blocked by sites that want to reject crawlers.

wget --user-agent="Mozilla/5.0 (X11; Linux x86_64)" \
  https://example.com/resource/file.zip

If 403 persists after replacing the User-Agent, check authorization — some resources need credentials (episode 14). Wget marks 4xx/5xx responses with exit code 8.

#SSL Certificate Problem

This error appears when TLS verification fails, and wget exits with code 5. The causes vary: expired certificates, hostname mismatch, a CA the system doesn't know, or a wrong system date. Oddly, a skewed machine clock is the most often overlooked cause — certificates have validity periods, and if your local clock is far from the real time, verification fails immediately.

date

If the date is correct, try pointing wget at an explicit CA bundle — sometimes the system default hasn't been updated:

wget --ca-certificate=/etc/ssl/certs/ca-certificates.crt \
  https://example.com/file.zip

#Connection Refused

The message failed: Connection refused means the server rejected the TCP connection — usually because no service is listening on that port, or a firewall explicitly refuses. The exit code is 4. If a service is restarting or was just deployed, wget can be told to retry:

wget --retry-connrefused --tries=10 --wait=5 \
  https://example.com/file.zip

Also check that you're targeting the correct host and port. A correct host with a wrong port (for example 80 for HTTPS) is the most confusing cause — it always looks like "the server is broken" when it's actually the URL that's wrong.

#No Such File or Directory (FTP)

Over FTP, this error appears when the requested file or directory isn't found — either a typo in the path, or a successful login but pointing at the wrong directory. Unlike HTTP, wget over FTP must go through a change directory command before fetching the file; if the directory doesn't exist, this is the message that comes out.

wget --spider ftp://ftp.example.com/pub/archive/file.tar.gz

Use --spider to check a file's existence without downloading its contents. Also make sure the path is relative to the FTP account's home directory, and mind the connection mode: wget uses passive mode by default; if your FTP server has problems with it, try enabling active mode with --no-passive-ftp.

#Rate Limiting

Repeated 429 or 503 statuses mean you're too fast or the server is overwhelmed. Forcing onward only makes it worse — and risks a permanent IP block. The solution is planned self-throttling: add pauses between requests, limit speed, and cap the attempt count.

wget --wait=5 --random-wait --limit-rate=100k --tries=5 \
  --retry-on-http-error=429,503 -r -l 2 https://example.com/

--wait=5 pauses five seconds between requests, --random-wait randomizes it so no pattern forms (rigid patterns are easily recognized as bots), and --limit-rate=100k caps bandwidth. --retry-on-http-error=429,503 makes wget treat 429 and 503 as temporary errors worth retrying — not final failures.

#The Correct Diagnosis Order

The key to wget debugging is building a consistent ladder, from fastest to deepest:

1. Check the exit code — what's the problem category: network (4), SSL (5), or server (8)?
2. Look at the status code with -S — is the server answering, and how?
3. Open the -d log — at which phase did the conversation stop?
4. Fix the root cause — don't cover it with permissive flags.

This order prevents wasted time: if it failed on SSL (code 5), don't fiddle with the User-Agent. If the server returns 403 (code 8), don't panic and change ports. Each evidence layer points to the next.

#Closing

Episode 18 equipped you with a complete diagnostic toolkit: reading -d logs to see raw requests and responses, displaying server headers with -S, translating HTTP status codes and wget exit codes from 1 through 8, and mapping common errors — robots.txt, 403, certificates, connection refused, FTP, and rate limiting — along with their root causes.

The most valuable thing isn't the tools, but the habit: debugging isn't guessing, it's reading evidence in order. Start with the exit code, descend to the status code, then open the debug log.

In episode 19, we glance into the future: Wget vs Wget2 & modern features — getting to know wget's libwget-based successor, comparing CLI options, and the cutting-edge features already present in stable Wget 1.x. See you there!