Learn Wget - WARC & Web Archiving

#Introduction

In episode 16 you turned wget into an automation machine — scripts, cron, and CI pipelines that work without supervision. Now we take wget to a different level: not just downloading files to disk, but recording evidence of interactions — what request was sent, what response came back, what headers accompanied them, and when it all happened. Episode 17 covers WARC, the standard web archive format used by the Internet Archive, and how wget can produce legitimate archives for audit and documentation.

#What Is WARC

WARC (Web ARChive) is an ISO 28500 standard format for storing web interactions — not just page contents, but the complete metadata of every exchange. A single WARC file can contain several record types:

• warcinfo — metadata about the crawl itself: wget version, date, and operator information.
• request — the HTTP request sent.
• response — the HTTP response received, complete with headers.
• metadata — additional information.

Think of an airplane's black box: it doesn't record just the final outcome, but the entire communication journey. WARC is the black box for the web — it's what the Internet Archive uses for the Wayback Machine and what Common Crawl uses for petabyte-scale research datasets.

#Producing Your First WARC

The --warc-file option stores the entire crawl result in a WARC file:

wget --warc-file=arsip https://example.com/

wget produces arsip.warc.gz (gzip-compressed by default — --no-warc-compression turns it off). While the process runs, wget writes to arsip.open.warc.gz; when finished, the file is renamed and closed cleanly. At file opening, the warcinfo record notes who ran the crawl and when — a useful trail when the archive must be accounted for.

#Combining with Recursive Crawls

The power of WARC shows when combined with the recursive modes from episodes 10 and 11. A single command can mirror a site and record it entirely into an archive:

wget --recursive --level=2 --no-parent \
  --warc-file=arsip-docs \
  https://docs.example.com/

The result is two things at once: a file structure on disk for direct use, and a WARC archive as evidence. For stricter audit needs, embed additional metadata into the warcinfo record:

wget --recursive --warc-file=arsip-audit \
  --warc-header="Operator: Arman Dwi Pangestu" \
  --warc-header="Tujuan: Audit kepatuhan" \
  https://docs.example.com/

Each --warc-header line becomes one metadata line in the opening record — a way of writing "who, why, and when" directly into the archive itself.

#CDX: Quick Indexing

WARC files can be large, and finding a single URL inside one is like searching for a needle in a haystack. The --warc-cdx option writes a CDX index — a concise list of every URL along with its record's location in the file:

wget --recursive --warc-file=arsip \
  --warc-cdx=arsip.cdx \
  https://docs.example.com/

CDX works like a book's table of contents: without reading the entire contents, you can know which page holds a particular URL. For archives checked routinely, CDX saves a lot of time.

#Deduplication: --warc-dedup

Repeated crawls of the same site produce many duplicates. --warc-dedup uses an existing CDX file as the comparison base — if a response was already stored with identical content, wget doesn't store it again:

wget --recursive --warc-file=arsip-baru \
  --warc-dedup=arsip.cdx \
  https://docs.example.com/

wget compares each response's content digest (fingerprint) against the old CDX. It's analogous to a librarian who doesn't copy the same book twice — they simply point to the existing copy. Huge efficiency for large archives updated routinely.

#Use Cases: Archives and Audits

Why does web archiving matter beyond the research domain? Three most tangible cases:

• Reproducibility — documents cited in reports can change or disappear. WARC freezes what was actually seen on a given date, so claims can be re-verified anytime.
• Compliance and litigation — when content becomes the subject of a dispute, what's needed isn't a summary but evidence: who sent what, and what the server answered.
• Preservation — projects like the Internet Archive keep web pages alive after their original sites die. With WARC, you can become the archivist for your own domain.

This is like a digital notary: not just knowing that a document exists, but being able to show when and how it arrived.

#Analyzing WARC

WARC is just a format; to read its contents you need tooling. The fastest way is warcio — the standard Python library for processing WARC:

python3 -m pip install warcio

python3 -m warcio list arsip.warc.gz

The command above prints each record's type. To dig deeper — for example, showing the URL along with each response's HTTP status:

analisa-warc.py

from warcio.archiveiterator import ArchiveIterator
 
with open("arsip.warc.gz", "rb") as fh:
    for record in ArchiveIterator(fh):
        if record.rec_type == "response":
            url = record.rec_headers.get_header("WARC-Target-URI")
            status = record.http_headers.status
            print(status, url)

Other tooling worth knowing: warcat for manipulating and merging WARC files, and pywb for bringing an archive back to life as a browsable site in the style of the Wayback Machine. WARC is an open format — as long as the format is standard, your archives aren't tied to any one tool.

#Closing

Episode 17 took wget from downloader to archivist: understanding the WARC format and its record types, producing archives with --warc-file, combining them with recursive crawls, embedding audit metadata with --warc-header, writing quick indexes with --warc-cdx, saving space with --warc-dedup, and analyzing the results with warcio and supporting tooling.

The key thing to remember: downloading is taking; archiving is proving. With WARC, wget doesn't just bring files home — it records an interaction trail that can be accounted for later.

In episode 18, we flip roles: troubleshooting and debugging — reading wget logs when something fails, understanding common error messages, and strategies for breaking down download problems from the most common to the strangest. See you there!