Learn Wget - Recursive Download & Spider

#Introduction

In episode 8 you tidied up wget via configuration files. Now it's time for the most iconic and most dangerous feature: recursive download. With a single -r option, wget changes from a simple download tool into a crawler — it follows links from page to page, downloads everything, then follows the links from the newly downloaded pages again.

This power must be balanced with control. Without clear limits, -r can download a site far larger than you imagine, burn bandwidth, and overload servers. This episode teaches you how to crawl properly: depth, directory limits, robots.txt etiquette, and spider mode that saves all of it.

#Basic Recursion: -r

The -r (recursive) option enables crawling. Wget fetches the page you give it, parses all the links in it, downloads the relevant ones, then repeats the process from the newly downloaded pages:

wget -r http://docs.example.com/guide/

This mechanism is like following a chain of directions: each page tells wget where to go next. By default, wget limits itself to the same host as the initial URL (no jumping to other domains), and the built-in maximum depth is 5 levels. These defaults are deliberately conservative to protect you from big mistakes.

#Depth Limit: -l N

The depth level determines how deep wget descends into the link structure. Think of a folder tree: level 1 is the initial URL, level 2 is the pages the initial URL references, level 3 is the pages those pages reference, and so on.

wget -r -l 2 http://docs.example.com/guide/

wget -r -l inf http://docs.example.com/guide/

Setting -l 2 restricts wget to the starting page and the pages it directly references — very reasonable for documentation that only needs one layer of sub-pages. The value -l 0 or -l inf means unlimited; use it only when you truly want the whole site, usually together with the directory limits below.

#Don't Ascend to Parent: -np

The classic recursion problem: many pages contain relative links like ../ that point to directories above. Without a safeguard, wget can "climb" to the parent directory and unknowingly download the entire site from the top — far beyond the part you wanted.

wget -r -np http://docs.example.com/guide/

The -np (no-parent) option ensures wget never ascends to a directory above the starting point. This is the most commonly used safeguard in practice: you target guide/, and wget stops there — no matter how many ../ links it encounters. Make a habit of writing -np whenever you recurse a site subdirectory.

#Respecting robots.txt

Wget has built-in etiquette: in recursive mode, it fetches and obeys the site's robots.txt file. If that file states a path must not be accessed by bots, wget skips it.

User-agent: *
Disallow: /admin/
Disallow: /search

A bypass is still possible via -e robots=off, because robots is a variable that can be changed at runtime. But this is a weapon to be used with great care — only for content you're legitimately entitled to fetch, and with full awareness that you're ignoring the site owner's request.

#Waiting Politely: -w

Recursion produces many requests in a short time — and a flood of requests is the fastest way to get your IP blocked. The -w (wait) option forces wget to pause a number of seconds before every request:

wget -r -np -w 2 http://docs.example.com/guide/

The command above pauses two seconds between every request. For recursion reaching across a large site, -w isn't just politeness — it keeps you welcome on the server and reduces the risk of rate limiting. When crawling other people's sites, -w 1 to -w 5 is good common practice; for your own sites, feel free.

#Spider Mode: Checking Links Without Downloading

Not all wget work ends with files on disk. Sometimes all you need is an answer: is this URL still alive? For that, there's spider mode:

wget --spider http://example.com/halaman

In spider mode, wget only sends a request and reads the response without saving any content. Standard output shows Remote file exists. if the page is found, or an error message for 404 if not. This feature turns wget into a link checker and lightweight monitoring tool — without flooding the disk.

To check many links at once, combine it with file input and -nv for clean output:

wget --spider -nv -i daftar-url.txt

Each line in daftar-url.txt is checked one by one. This is a suitable pattern for periodic audits: gather all links from a profile page or sitemap, save them to a file, then run the spider to find dead links.

#Practice: Repeated Link Audits

The spider pattern is most useful when run repeatedly — for example every night via cron. Keep a list of important URLs (landing pages, product pages, status endpoints), then run an automated audit:

URLS=( https://example.com/ https://example.com/produk https://example.com/status )
for url in "${URLS[@]}"; do
  if wget --spider -q "$url"; then
    echo "OK $url"
  else
    echo "FAIL $url"
  fi
done

The script above prints each URL's status and flags the failures. Scheduled via cron, it becomes an early warning system: if an important page starts returning 404, you know within hours — not after customers complain. This snippet uses bash array syntax; if your shell isn't bash, a simpler version processing the URL list one line at a time is enough.

#Closing

Episode 9 equips you with controlled crawling: basic recursion with -r, depth limits with -l N, the -np safeguard against ascending to the parent, respect for robots.txt with responsible bypass options, politeness between requests with -w, and spider mode for checking URL availability without downloading content.

The key takeaway: recursion gives wget power, and control gives you peace of mind. Always set -np and a sensible depth, respect robots.txt, and use --spider for jobs that only need a yes-or-no answer.

In episode 10, all these elements are assembled into wget's flagship feature: website mirroring — a single command to copy an entire site to disk, complete with pages, assets, and links that can be opened offline. See you there!