Learn Wget - Proxies & Networking

#Introduction

In episode 11 you mirrored an entire site — wget followed links, copied the directory structure, and produced a complete local replica. But throughout that episode there was one assumption we never questioned: the path wget takes always goes straight through to the server. In the real world, that assumption is often wrong. Corporate offices, campuses, and even some ISPs force all traffic through a gateway called a proxy — and if you don't know how to talk to it, your downloads will hang in silence.

Episode 12 opens the hood on the network layer: how wget finds its way to the server, with or without a proxy, over IPv4 or IPv6, from which address it leaves, and how DNS shapes your download experience.

#Proxy: The Receptionist at the Gate

Imagine an office with a single receptionist. All packages in and out must pass through the reception desk; the receptionist decides which packages may leave, where they go, and sometimes keeps copies so they don't need to be ordered twice. A proxy works exactly like that: an intermediary that receives your requests, forwards them to the destination server, then relays the replies back. It's used for filtering (blocking certain domains), caching (keeping copies to save bandwidth), and audit (logging all access).

Because wget is non-interactive and often runs automatically, understanding how to configure a proxy isn't a complement — it's a prerequisite. Without proper configuration, all downloads behind a corporate firewall will fail with an unhelpful message.

#Enabling Proxy with Environment Variables

wget doesn't read proxies from a special configuration file — it reads environment variables, just like most network tools do. The three most important: http_proxy for HTTP traffic, https_proxy for HTTPS, and no_proxy for the list of hosts that must be accessed directly.

export http_proxy="http://proxy.contoh.local:8080"
export https_proxy="http://proxy.contoh.local:8080"
export no_proxy="localhost,127.0.0.1,.internal.example.com"
 
wget https://docs.example.com/

Once the variables are defined, wget automatically routes its connections through the proxy — no extra flags needed. Note the no_proxy pattern: hosts are comma-separated, and a leading dot means "including all subdomains". This keeps requests to internal servers from looping through a gateway that doesn't know the way.

#Bypassing the Proxy: --proxy=off and --no-proxy

Sometimes you want to disable the proxy for just one command — for example because the proxy is having issues, or because you know the local network doesn't need it. Two options distinguish the cases:

wget --proxy=off https://docs.example.com/

--proxy=off turns off the proxy completely for this invocation, without touching the environment variables. It's like taking a side street to avoid a congested toll gate — once, without changing the default route.

wget --no-proxy='localhost,127.0.0.1' https://internal.example.com/

Conversely, --no-proxy=host-list still uses the proxy in general, but excludes specific hosts — functioning like the no_proxy variable, yet applied per-invocation. Use it when only a handful of hosts must be accessed directly.

#Proxy Authentication

Many corporate proxies don't let just anyone through — they ask for proof of identity. When a proxy demands authentication, it responds with status 407 Proxy Authentication Required, and wget will ask for credentials. There are two ways to provide them:

wget --proxy-user=budi --proxy-password=rahasia \
  https://cdn.example.com/file.tar.gz

wget "http://budi:rahasia@proxy.contoh.local:8080" \
  https://cdn.example.com/file.tar.gz

The second way slips credentials directly into the proxy URL. Both are valid, but the second has two downsides: the credentials are visible on the command line (and stored in shell history), and you must retype them every time. For scripts that run automatically, --proxy-user and --proxy-password read from the environment are far cleaner — and never write a literal --proxy-password into a file that goes into git. We'll cover proper credential storage in episode 14.

#Forcing IPv4 or IPv6

The internet now runs on two addresses at once: IPv4 and IPv6. wget tries to pick the most sensible one automatically, but sometimes that choice is wrong — for example your network has partially broken IPv6, or the destination server only has an IPv4 address. Two options give you full control:

wget -4 https://example.com/

wget -6 https://example.com/

-4 (alias --inet4-only) and -6 (alias --inet6-only) lock in the protocol used. It's like choosing which toll lane to enter when there are two gates: one lane may be closer, but the other is more reliable. If a download frequently fails with a stalled connection on one network only, try forcing -4 — a quarter of mysterious connection problems end here.

#Bind Address and Bind Interface

On machines with multiple IP addresses — servers, workstations with several connections — wget usually picks the egress address automatically. The --bind-address option sets the local address used:

wget --bind-address=10.0.1.20 https://cdn.example.com/file.tar.gz

Its practical use: routing egress through a specific interface — for example forcing downloads through the network with the largest bandwidth, or through an IP registered in the destination server's allowlist. On newer wget versions, --bind-interface=eth0 does the same thing by naming the interface. These options are rarely used on a laptop, but they become distinguishing factors in production environments with complex routing.

#DNS: The Resolver That Determines Direction

Before a single byte moves, wget must translate the domain name into an IP address — the job of DNS (Domain Name System). This stage determines speed and reliability more than you might think: a slow resolver means every new connection waits; failed resolution means the download ends before it starts, with the message "Unable to resolve host address".

Two wget behaviors worth knowing:

• Internal DNS cache — wget caches resolution results while running. --no-dns-cache turns it off, useful when IP addresses change often and you need fresh resolution for each request.
• Resolution timeout — --dns-timeout=10 limits how long the resolver may take. Without a limit, a hanging resolution can hold up scripts for hours.

wget --no-dns-cache --dns-timeout=10 https://example.com/

#Closing

Episode 12 dissected the network layer we've been taking for granted: going through a proxy via the http_proxy, https_proxy, and no_proxy variables, disabling it per-invocation with --proxy=off and --no-proxy, authenticating to a proxy with --proxy-user and --proxy-password, locking the protocol with -4 or -6, choosing the egress address with --bind-address, and understanding DNS's role in speed and reliability.

The key thing to remember: the network path isn't fate — it's configuration. Every layer between wget and the server can be controlled, and every uncontrolled layer is an invisible point of failure.

In episode 13, we go one layer up: HTTPS, TLS, and certificates — how wget verifies server identity, why --no-check-certificate is an expensive decision, and how client certificates and HSTS strengthen trust. See you there!