Learn Wget - Configuration & wgetrc

#Introduction

In episode 7 you set headers, User-Agent, cookies, and credentials one by one in each command. That's tedious and error-prone — especially when the same options must be repeated across dozens of commands. The solution is wgetrc: a configuration file wget reads automatically every time it runs, where you store consistent defaults.

Think of wgetrc like a tidy workbench: instead of arranging your tools every time you start working, everything is already in place and you just use it. Once you understand wgetrc, your wget commands become much shorter because the repetitive parts have been moved to a single place.

#Two Configuration Locations

Wget reads configuration from two files, with different scopes:

• /etc/wgetrc — global configuration, applies to all users on the machine. Usually managed by system admins to set organizational policy.
• ~/.wgetrc — per-user configuration, applies only to the relevant user. This is the file you'll touch most often.

Both are read when wget starts, and values from ~/.wgetrc override values from /etc/wgetrc for the same option. This pattern is like layered configuration: base policy in global, personal adjustments in per-user.

#wgetrc Syntax

The wgetrc file format is simple: one option per line in the option = value pattern. Option names are the same as long options on the command line, but without the leading dashes — and wget is case-insensitive and ignores underscores and hyphens, so user_agent and user-agent are both valid.

# Identitas dan kebijakan akses
user_agent = "devvnull-bot/1.0"
tries = 5
timeout = 30
 
# Proxy korporat - nilai default semua request
http_proxy = "http://proxy.corp:8080"
https_proxy = "http://proxy.corp:8080"
no_proxy = "localhost,127.0.0.1"
 
# Perilaku download
continue = on
dot_style = binary

Lines starting with # are comments. Boolean values are written on or off — for example continue = on is equivalent to the -c flag on the command line, and continue = off reverts it. With the file above, every wget command automatically uses a 30-second timeout, retries five times, and resumes interrupted downloads — without typing any options.

#Option Priority

So what happens when the same option appears in several places? Wget applies a strict priority order, strongest first:

1. Command line — options you type directly always win.
2. -e / --execute — configuration commands on the command line, executed after wgetrc so they override the file.
3. ~/.wgetrc — per-user configuration.
4. /etc/wgetrc — global configuration.
5. Built-in defaults — used when nothing else sets the value.

The meaning is simple: wgetrc is a safety net, not a prison. If ~/.wgetrc sets tries = 5, writing --tries 10 on the command line overrides it for that command only — wget --tries 10 URL remains valid and the wgetrc value doesn't need to change. Keep this rule in mind whenever you debug why wget is behaving "strangely" — often the cause is a config file value you can't see.

#Dynamic Execution with -e

Not all configuration deserves to be stored permanently. To change a single value for just one command — without touching any file — use the -e option, which executes a configuration command like a wgetrc line:

wget -e robots=off -e tries=3 -r http://docs.example.com/guide/

Each -e accepts one option = value command. Because these commands run after wgetrc is read, -e is the right tool for one-off overrides: changing the verbose level, turning off the proxy for a single download — for example wget -e use_proxy=off URL — or raising the retry limit without changing permanent config. This pattern is also useful in scripts where you don't want to pollute the user's configuration files.

#Default Policies in wgetrc

Here's the practical application of wgetrc: a place to set consistent default policies for a work machine or server. The three most commonly stored here are proxy, User-Agent, and retry count.

# Kebijakan akses jaringan
use_proxy = on
http_proxy = "http://proxy.corp:8080"
https_proxy = "http://proxy.corp:8080"
no_proxy = "localhost,127.0.0.1,*.internal.example.com"
 
# Identitas bot - sertakan kontak untuk admin server
user_agent = "devvnull-bot/1.0 (+https://example.com/contact)"
 
# Keandalan download
tries = 8
waitretry = 5
retry_connrefused = on
continue = on

With the policy above, all wget instances on that machine automatically go through the corporate proxy, introduce themselves as a contactable bot, and behave patiently toward unstable connections. Values that vary between environments still go on the command line; values that are permanent move here.

#Cleaning Up and Debugging Configuration

Options that accept lists, like -X (exclude directories), are cumulative: values from wgetrc stay active until you clear them. Wget provides an explicit way to empty a value directly from the command line — by writing an empty list:

wget -X "" -X /cgi-bin,/~nobody http://example.com/

The -X "" line discards all exclude values inherited from wgetrc, and the next -X sets a new list. The same pattern applies to -A, -R, and -I — useful when global configuration sets an exclusion you don't want for a specific command.

When wget behaves unexpectedly, the fastest debug step is to isolate the variable. With the priority hierarchy from the previous section, you just trace where the value comes from: command line, -e, or a config file. To see the details of the negotiation happening, the -d (debug) option shows request headers, server responses, and the decisions wget makes.

#Closing

Episode 8 taught you to tidy up wget: the two configuration locations (/etc/wgetrc global and ~/.wgetrc per-user), the option = value syntax with on/off booleans, the priority hierarchy from command line down to built-in defaults, dynamic execution with -e for one-off overrides, and patterns for storing default policies like proxy, User-Agent, and retries in wgetrc.

The most important takeaway: wgetrc removes repeated flags, and the command line remains the highest authority. The two work together — config files for defaults, command line for exceptions.

In episode 9, we enter wget's most iconic and most dangerous feature: recursive download for crawling inside a site, controlling depth, respecting robots.txt, and spider mode for checking link availability without downloading content. See you there!