Learn Wget - Website Mirroring

#Introduction

In episode 9 you learned to crawl with control: depth, directory limits, and spider mode. Now all those elements are assembled into the feature that made wget famous: website mirroring — copying an entire site to local disk so it can be opened without a connection, whenever needed.

Mirroring isn't just downloading HTML pages. Modern sites are built from many layers: stylesheets, images, JavaScript, pages with URLs that don't end in .html, and absolute links pointing at the original domain. A correct mirror must handle all of them, and wget provides specific options for each layer.

Mirroring has diverse uses, too: archiving internal documentation so it stays readable when the production site is down, compliance copies for audit purposes, or simply content you want to read on a device without a connection. Once this structure is understood, a single wget command is enough for all of it.

#Mirror in One Command: -m

The heart of mirroring is the -m (mirror) option, which activates a set of options all at once in a single command:

wget -m http://docs.example.com/

Internally, -m is equivalent to -r -l inf -N --no-remove-listing — recursive, unlimited depth, timestamping enabled, and FTP directory listings preserved. Timestamping (-N) is the key: when the mirror is rerun, wget only downloads files that changed on the server, instead of copying everything again. As a result, a mirror can be scheduled as a periodic synchronization process.

For tidy results, practical mirroring usually adds -np and -nH: -np prevents wget from ascending to the parent directory, and -nH removes the domain-named folder from the result structure. The combination wget -m -np -nH http://docs.example.com/ gives you a clean, self-contained copy of the site.

When to use -m instead of just -r? The answer is guided by your goal: -r is enough for fetching part of the content at a given depth, while -m is designed for copying an entire site and keeping it in sync via timestamping. If you think you'll load the same results again later, use -m from the start.

#Page Assets: -p

Plain recursion only follows links between pages. It doesn't automatically download the files that make a page look the way it should: inline images, stylesheets, and scripts. The -p (page-requisites) option closes that gap:

wget -p http://docs.example.com/index.html

Without -p, a downloaded page looks "bare" — the text is there, but the styles and images are missing. With -p, wget downloads every file needed to render the page completely. In a mirroring context, -p almost always comes along: wget -m -p URL means "copy the site along with its appearance".

One important note: -p works per page, but in recursive mode it also traverses cross-page assets — stylesheets and images shared by many pages are downloaded only once, not repeated for each page. This makes -p seem "expensive" on the surface, but it's actually efficient because wget tracks which files it has already downloaded.

#Correct Extensions: -E

Many sites serve HTML from URLs that don't end in .html — for example artikel.php, produk.cgi, or /beranda. Wget saves raw content with the URL-based name, so a local browser doesn't recognize it as HTML and opens it as text or a download.

wget -m -E http://docs.example.com/

The -E (adjust-extension) option adds the .html extension to HTML files and .css to CSS files that were saved without the proper extension. As a result, mirrored pages can be opened directly in a local browser or served by an Apache server without extra configuration.

Note one nuance: -E adds extensions based on the Content-Type the server sends. If the URL artikel.php returns Content-Type: text/html, wget saves it as artikel.php.html. That's why -E makes the most sense combined with -k — the converted links still point at the correct file names after the extensions are added.

#Convert Links: Offline Browsing with -k

The last problem: links inside HTML pages still point at absolute URLs, for example http://docs.example.com/css/style.css. When opened offline, the browser will try to reach the internet — and the page stops working. The -k (convert-links) option fixes this after the download finishes:

wget -m -p -k -E -nH http://docs.example.com/

-k rewrites the links in every document to point at the corresponding local files — turning absolute URLs into ones relative to the mirror structure. After this process, opening index.html with a local browser is enough; all pages, images, and stylesheets are interconnected without the internet. If you want to keep the original copies before they're changed, add -K for backups ending in .orig.

#Periodic Updates: The Power of Timestamping

A mirror isn't a one-time job. Sites grow every day, and a mirror that isn't updated quickly becomes stale. This is where the -N (timestamping) option hidden inside -m comes in: wget compares the local file modification time against the server version, then downloads only the files that are newer.

wget -m -N -p -k http://docs.example.com/

The command above refreshes the mirror without re-downloading unchanged files — saving time and bandwidth, especially for sites with many static assets. Think of it as folder synchronization: -N makes wget ask "who is newer?" before deciding to copy, rather than copying everything every time.

#Verifying Mirror Results

A mirror isn't done until it's verified. The first simple step: make sure the main page exists and the directory structure is formed:

find docs.example.com -type f | head -20

If the structure looks reasonable, run the real test: open index.html in a local browser. The page should render with the correct styles, images should appear, and navigation between pages should work without an internet connection. This is the most convincing proof that -p and -k are working as expected.

Two additional indicators worth checking. First, the total size of the mirror result — wget reports it at the end of the process. A documentation site mirror is usually tens of megabytes, a media site can be tens of gigabytes; a number far from expectations signals a wrong filter or recursion escaping its bounds. Second, the presence of .orig files if you used -K — it indicates link conversion really happened, not skipped.

#When -p Can Be Left Out

Every option has a cost. -p downloads all page assets — great for a complete archive, but wasteful if you only need the text and structure. For large, asset-heavy sites, consider a mirror without -p if your goal is content searching, not copying the appearance.

wget -m -np -A html,htm -E http://docs.example.com/

By restricting extensions to HTML and enabling -E, you get a light, easily searchable copy of all pages. When one day you need the full version with the original appearance, rerun with -p. Choosing options to match your goal is part of wget maturity — not always using the heaviest combination.

#Closing

Episode 10 equips you with full mirroring capabilities: -m as a single command for unlimited recursion plus timestamping, -p for page assets, -E for correct extensions, -k for link conversion so pages open offline, and the habit of verifying mirror results before use.

The key takeaway: a good mirror is one that opens offline and updates incrementally. The full command wget -m -p -k -E -nH URL is the combination that turns wget into a reliable web archive tool.

In episode 11, we learn to narrow things down: filtering downloaded files by extension and regex, restricting or excluding directories, and controlling the directory structure of download results to match your wishes. See you there!