Learn Wget - Batch Download & Multiple Files

#Introduction

Up to episode 4, all our downloads handled a single URL per command. In the real world, work is rarely that tidy: downloading dozens of assets, fetching all config files from several servers, or backing up a collection of documents — these are all batch downloads. This episode is about working in bulk: one command for many files, with full control over where and what the results are named.

It's like the difference between mailing letters one at a time and dropping all packages off at one courier with a single destination list. Wget reads that destination list, plans its delivery routes, and reports the results — without you retyping a single URL.

#Input File List: `-i`

The -i option (or --input-file) tells wget to read a URL list from a file — one URL per line. Start by creating a destination list:

urls.txt - list of URLs, one per line

https://example.com/
https://example.com/robots.txt
https://example.org/
https://example.org/robots.txt
https://example.net/
https://example.net/robots.txt
https://www.example.com/
https://www.example.com/robots.txt
https://example.edu/
https://example.edu/robots.txt

Then run the batch download:

Batch download from a URL list

wget -i urls.txt

Wget reads each line, downloads one at a time, and saves the results following the naming rules we've learned. Blank lines are skipped automatically. Wget also accepts input from stdin with -i - — useful when the URL list is generated by another command:

URL list comes from another command

cat urls.txt | wget -i -

Note

One thing you should know: wget does not treat lines starting with # as comments — it will try to download that line and fail with a parse error. Keep your URL lists free of comment lines, or put those lists in a separate file.

#Combining `-i` with Other Options

The power of -i shows when combined with the options we've learned — -c for resume, -P for directories, -a for logs. All options apply to every URL in the list:

Batch download with resume and log

wget -c -i urls.txt -a batch.log

Here -c ensures interrupted files are resumed, and -a batch.log records the entire history in one file. This is the standard pattern for batch jobs that run repeatedly — for example, daily asset synchronization.

#Directory Prefix: `-P`

By default, wget downloads to the current working directory. To gather batch results into a dedicated directory, use -P (or --directory-prefix):

Save results to a specific directory

wget -P /tmp/downloads https://example.com/robots.txt

Wget creates that directory if it doesn't exist, then places robots.txt inside it. It's like designating the "destination warehouse" before shipping — keeping download results from mixing with other working files. Combine it with -i to route an entire batch into one folder:

Batch download to a destination directory

wget -i urls.txt -P /tmp/downloads

#File Name from Header: `--content-disposition`

There's one case where the URL-based naming rule fails: URLs like https://example.com/download?id=42 don't contain a file name. Without help, wget names the result download?id=42 — ugly for humans to read and troublesome to use as a file.

Many good servers send a suggested name via the Content-Disposition: attachment; filename="laporan.pdf" header. Wget can use it with the --content-disposition option:

Use the file name from the server header

wget --content-disposition 'https://example.com/download?id=42'

Instead of download?id=42, the file is saved as laporan.pdf — exactly the name the server suggested. This is very helpful for automated downloads from endpoints that generate dynamic files.

Tip

Check first whether the server sends that header: wget -S URL will show all response headers. If there's a Content-Disposition line, --content-disposition will work; if not, wget falls back to URL-based naming.

#Directory Structure: `-x`

By default, wget downloads files with their name only — the directory structure from the URL is discarded. The -x option (or --force-directories) changes this behavior: wget recreates the directory structure from the URL path, including the host name:

Keep the URL directory structure

wget -x https://example.com/assets/img/logo.png

As a result, the file is saved at example.com/assets/img/logo.png — exactly following the URL structure. This is the foundation of mirroring, which we'll explore in depth in the recursive download episodes. With -x, download results can become a direct replica of the original server's structure.

The practical difference:

Situation	Without `-x`	With `-x`
A single file	`logo.png` in the working directory	`example.com/assets/img/logo.png`
Many files from one host	Scattered in one directory	Neatly following the URL structure

#Frequently Used Combinations

Here's a summary of the batch download patterns most commonly used in production:

Complete batch download - resume, directory, log

wget -c -i urls.txt -P /backup/files -a batch.log

Reading it: resume interrupted files (-c), fetch all URLs from the list (-i urls.txt), save to /backup/files (-P), and record everything in batch.log (-a). Four simple options that together form a reliable download job ready to be scheduled in cron.

Warning

Avoid combining -O with batch. -O only makes sense for one URL — in a batch, the results will be written to the same file and overwrite each other. For batches, use -P (directory) and -x (structure) to manage locations, not -O.

#Common Pitfalls

Putting comment lines in urls.txt. Wget tries to download them and fails. Keep the URL list clean, without #.
Combining -O with many URLs. All results overwrite the same file. Use -P or -x for batches.
Forgetting -P, leaving batch results mixed in the working directory. Designate the destination warehouse from the start.
Expecting --content-disposition to work on servers that don't send that header. This option only works if the header exists — check with wget -S URL.

#Closing

In episode 5, you've mastered mass downloads: reading a URL list from a file with -i (including from stdin), combining it with -c and -a for reliable batch jobs, setting the destination directory with -P, using the file name from server headers with --content-disposition, and preserving the URL structure with -x.

Key takeaways:

-i FILE downloads many URLs from a single list; -i - reads from stdin.
-P DIR sets the destination directory; -x preserves the URL structure.
--content-disposition uses the file name from server headers.
-O is only for a single URL; for batches use -P and -x.
Keep URL lists free of comment lines — wget doesn't treat them as comments.

In episode 6, we'll cover FTP & file transfer — downloading and uploading over the FTP protocol, understanding how it differs from HTTP, and safe practices for transferring files between servers. See you in episode 6!