Learn Wget - Core Concepts & Main Architecture

#Introduction

After understanding wget's history in episode 1 — born as Geturl in 1996, becoming GNU Wget in 1998, with two lineages, 1.x and Wget2 — it's time to dissect how wget works from the inside. This episode is the bridge between "why wget exists" and "how to use it", so pay close attention to the flow.

Many wget users use it like a photocopier: give it an address, wait for the result. Yet behind a single wget https://example.com line there's a regular, predictable sequence of events. Understanding this sequence — like opening the hood before driving — will make you far more confident when the result isn't what you expected.

#The Non-Interactive Model: Wget's Soul

One word defines wget: non-interactive. Wget doesn't pop up dialogs, doesn't wait for keyboard input, doesn't need a graphical interface. It works like an automatic machine: you give it a URL and options, wget does everything on its own, then exits.

That's what makes wget perfect for scripts, cron, and headless servers — something browsers can't do. On a production server, no one sits waiting for a download to finish; there are scheduled jobs that download at midnight and record their results. Wget was designed for exactly that world from the start.

#The Workflow Behind a Single Command

Every time wget downloads, it goes through the same six stages. Think of it as the process of sending a package through a courier:

Parse URL & options — wget reads the URL and options you provide, breaking them into scheme, host, port, and path. It's like writing the full address on a package label.
Resolve DNS — wget translates the domain name into an IP address by asking a DNS server. It's like looking up the destination postal code in an address book.
Connect (TLS) — wget opens a TCP connection to the destination IP and port, then performs a TLS handshake if the scheme is https. It's like checking the door and agreeing on a sealed envelope before the goods go out.
Send request — wget sends the request line and headers to the server. It's like handing over the package along with its delivery note.
Receive response — wget receives the status code, headers, and body from the server. It's like accepting a receipt from the courier.
Write file — wget writes the body to a file on disk, complete with overwrite and resume handling if the file already exists. It's like putting the delivered goods into the warehouse.

You'll see this six-stage pattern with your own eyes in episode 3 when wget displays its log — Resolving, Connecting, HTTP request sent, Saving to. Learn which stage succeeds and which fails, because that's where your debugging direction lies.

#wgetrc: Wget's Configuration Center

Wget provides a global configuration file called wgetrc. There are two locations read in sequence:

System wgetrc — /etc/wgetrc (Linux/macOS), applies to all users.
User wgetrc — ~/.wgetrc, applies only to your account.

This file holds default values for almost every wget option — like an application's settings file. You can place the same preferences there for every session:

~/.wgetrc - example minimal configuration

# ~/.wgetrc - contoh konfigurasi minimal
# Berlaku untuk semua sesi wget pengguna ini
tries = 5
timeout = 30
continue = on
quiet = off
wait = 2
robots = off
user_agent = belajar-wget/1.0
connect_timeout = 10

Values in wgetrc are defaults: you can still override them with command-line options on each invocation. The principle: wgetrc for consistent habits, command-line options for per-download decisions.

Tip

Don't flood your wgetrc right away. Start with just tries and timeout, then add as needed. An overly aggressive wgetrc can actually make wget behave unpredictably in situations that need default behavior — for example, a robots = off you didn't realize was active in all sessions.

#robots.txt & Download Etiquette

Before downloading a site, wget also pays attention to robots.txt — the web etiquette standard that tells crawlers which pages may or may not be fetched. By default, wget respects robots.txt. This matters for avoiding downloads that are impolite to other people's servers.

If you really need to bypass it — for example, on a site you own — use wget -e robots=off URL. But remember the etiquette: disabling robots means you take full responsibility for the load you put on the server.

#Determining the Local File Name from the URL

When writing files, wget determines the local file name from the path part of the URL. For example, URL https://example.com/files/logo.png produces the local file logo.png. If the URL ends with / or contains no file name — like https://example.com/ — wget uses the default name index.html.

This rule can be overridden in three ways:

Situation	Solution
Default name not wanted	`-O FILE` option for a custom name
Server sends a name via header	`--content-disposition` option
Want to keep the path structure	`-x` option (covered in episode 5)

This understanding is important because many beginners are confused when they see the downloaded file named index.html even though the URL ends with /. The answer is simple: wget has no other name to use as a reference.

#Exit Codes: Wget's Silent Language

Every time wget finishes, it "speaks" through an exit code — a number returned to the shell. This is wget's primary signaling system for scripting, and its rules are very simple: 0 means success, anything other than 0 means there's a problem.

Check exit code after a download

wget -q https://example.com
echo $?

Example when a failure occurs

wget -q https://domain-yang-tidak-ada-xyz.example
echo $?

In the second example, you'll see exit code 4 — meaning wget experienced a network failure. Each number has a specific meaning documented in man wget under the EXIT STATUS section:

Exit Code	Meaning
0	Success, no problems
1	Generic error
2	Parse error (example: wrong option or configuration file)
3	File I/O error
4	Network failure
5	SSL verification failure
6	Username/password authentication failure
7	Protocol error
8	Server sent an error response

Important

Exit codes are wget's main language for scripts. When you write if wget ...; then or wget ... || exit 1, you're reading this language. Later in the scripting episode, the habit of checking exit codes will separate reliable scripts from "just okay" ones.

#Logs & Output: Controlling the Noise

By default, wget displays a log with the download progress in the terminal. To control this noise, wget provides several options:

Option	Function
`-q`	Silences all output except fatal errors
`-nv`	Turns off verbose output, still shows important info
`-a FILE`	Appends the log to FILE
`-o FILE`	Writes the log to FILE
`-d`	Debug mode: the most complete detail for troubleshooting
`-S`	Shows the response headers sent by the server

There's one interesting detail about the progress bar: when wget runs in a terminal, it shows a progress bar complete with percentages; but when output is redirected to a file or isn't a terminal, wget switches to a dotted display. Don't be surprised to see lines of dots in script output — that's normal, not an error.

#Closing

In episode 2, you've dissected wget's architecture from the inside: the non-interactive model that is its soul, the six transfer stages (parse URL, resolve DNS, connect TLS, send request, receive response, write file), the role of wgetrc, the robots.txt and file-naming rules, the exit code language, and how to control logs and output.

Key takeaways:

Wget is non-interactive: once it runs, it finishes — perfect for scripts, cron, and servers.
Six transfer stages happen behind a single command — know where the failure occurred.
wgetrc (system + user) stores defaults; command-line options always win.
Exit code 0 = success; memorize important codes like 4 (network) and 8 (server error).
-q, -nv, -a, -o, -d, -S are your log control buttons.

In episode 3, we start the real hands-on work: basic file downloads — running your first download to https://example.com, understanding file-name determination and the -O option, streaming output to stdout with -O -, and mastering the progress bar and log output. See you in episode 3!