Learn Wget - Filtering & Directory Structure

#Introduction

In episode 10 you could copy an entire site. But copying an entire site is rarely what you actually want — sometimes you only need the images, only the documentation pages, or only files from one directory while skipping the others. This episode covers filtering: choosing what gets downloaded, and directory structure: deciding where files are stored.

The two go hand in hand. Filters determine the contents, structure determines the shape. You can download exactly what you target and place the results somewhere tidy — without manual cleanup after the download finishes.

#Accept and Reject Extensions: -A and -R

The most direct way to narrow downloads is via extension lists. -A (accept) only downloads files with the listed extensions, while -R (reject) downloads everything except the listed ones.

wget -r -A jpg,png http://gallery.example.com/

wget -r -R mp4,avi,mkv http://media.example.com/

Lists are comma-separated. Besides extensions, wget also accepts wildcard patterns — -R '*.tmp' for example rejects all files ending in .tmp. If a list element contains the wildcard characters *, ?, [, or ], wget treats it as a pattern and you must quote it so the shell doesn't expand it too.

#Regex-Based Filters: --accept-regex and --reject-regex

Extensions aren't always enough — sometimes the decision must be made from the whole URL. For that, wget provides --accept-regex and --reject-regex, which match a regex against the complete URL:

wget -r --accept-regex '.*\.(jpg|png)$' http://gallery.example.com/

wget -r --reject-regex '.*/archives/' http://blog.example.com/

Regex gives a precision that extension lists can't reach: matching path segments, combining extensions with directories, even version patterns. Use regex when extension lists start to feel rigid — for example filtering reports with a specific year format, --accept-regex '.*report-2026.*\.pdf$'.

#Include and Exclude Directories: -I and -X

Sometimes the decision isn't about file type, but about location on the site. -I (include) restricts recursion to specific directories, while -X (exclude) skips directories from the recursion journey:

wget -r -I /docs,/assets http://docs.example.com/

wget -r -X /admin,/cgi-bin http://example.com/

Both lists accept wildcards, so -X '/old/*' can exclude all old subdirectories at once. This is very useful for selective mirrors: download only the documentation, or avoid areas you don't need — without guessing from page contents.

#Local Directory Structure

Now the second question: where are files stored? By default, recursion saves files under a directory named after the host — http://docs.example.com/guide/intro.html becomes docs.example.com/guide/intro.html on disk. Three options give you control over this structure:

wget -r -nH http://docs.example.com/

-nH (no-host-directories) removes the host-name layer, so the result is guide/intro.html. If there's still a path layer you want dropped, --cut-dirs trims it:

wget -r -nH --cut-dirs=2 http://docs.example.com/pub/release/

For content downloaded outside recursion, -x (force-directories) forces full directory structure creation even for a single file, while -nd (no-directories) does the opposite: all files are dropped into one folder without hierarchy.

wget -x http://docs.example.com/robots.txt

#File Name Portability: --restrict-file-names

Finally, a detail often forgotten until it causes problems: file naming rules differ between operating systems. Characters like :, ?, *, or " are valid on Unix but problematic on Windows or ASCII filesystems. The --restrict-file-names option controls which characters wget may use when creating local file names:

wget -r --restrict-file-names=windows http://docs.example.com/

Selectable values include unix, windows, ascii, and nocontrol, and can be combined with commas — for example --restrict-file-names=unix,nocontrol combines Unix naming rules with a ban on control characters. Wget replaces forbidden characters with %HH notation (for example %3F for a question mark), keeping download results legal on the destination filesystem.

#Combined Practice: Documentation Site Selection

Filters and structure are rarely used on their own. A real scenario: downloading a project's documentation to read offline, only the English parts and code samples, without build outputs and without the server folders.

wget -r -m -nH \
     -I /docs/en \
     -X /docs/fr,/server \
     -A html,md,png \
     --cut-dirs=1 \
     http://docs.example.com/

One command combines all of this episode's capabilities: full recursion with -m, local structure without the host name via -nH and without directory prefixes via --cut-dirs=1, only the English directory with -I, excluding translations and server folders with -X, and filtering to useful extensions with -A. The result: a clean, light documentation folder ready to open offline.

#Closing

Episode 11 gives you full control over the contents and shape of download results: accept and reject extensions with -A and -R, precise filters with --accept-regex and --reject-regex, directory restrictions with -I and -X, directory structure control via -nH, --cut-dirs, -x, and -nd, and file name portability with --restrict-file-names.

The key takeaway: filters determine what gets downloaded, structure determines where files are stored — and both must be designed together from the first command, not cleaned up afterward.

In episode 12, we enter the network layer: HTTP and SOCKS proxies, proxy authentication, DNS manipulation, and connection troubleshooting so wget keeps running in complex corporate environments. See you there!