Learn Wget - Resume, Retry & Download Control

#Introduction

Up to episode 3, all our downloads went smoothly. In the real world, it's not always that rosy: connections drop mid-way, servers get busy, bandwidth is narrow, and downloading a 2GB file can't wait for you to stare at the screen. This episode is about resilience — how to make wget survive failures and control its behavior with precision.

Imagine sending an expedition truck on a long journey. What distinguishes a professional courier from an amateur isn't the truck — it's how they handle rough roads: resuming from the last position, retrying alternate routes, managing speed, and reporting results even when unobserved. That's exactly what we'll build into wget in this episode.

#Resume Download: -c

The most common scenario: a download is interrupted mid-way — connection drops, laptop closes, or the server responds slowly. Wget has already left a partial file on disk. Instead of starting from zero, wget can resume from the last position with the -c option (or --continue):

wget -c https://speed.hetzner.de/100MB.bin

Here's how it works: wget checks the existing partial file, then sends a request with the Range: bytes=... header to the server. Servers that support resume respond with status 206 Partial Content and send only the remaining portion. Servers that don't support it respond with 200 OK and the full content — and wget is forced to start over.

To confirm a server supports resume, you can check its response headers with wget -S URL — if there's an Accept-Ranges: bytes header, the server is ready to serve resumes.

#Retry: --tries

Networks aren't always stable, and servers can be busy. Wget has a built-in retry mechanism — by default wget tries up to 20 times before giving up. You can control this with the --tries option (or -t):

wget --tries=5 https://example.com/big-file.iso

wget --tries=inf https://example.com/big-file.iso

Considerations: the default of 20 is enough for most cases, but in batch scripts that download hundreds of files, 20 attempts per file can make the job run for a very long time. --tries=3 to --tries=5 are the values commonly used in production. Between attempts, wget also inserts a short pause — controlled with --wait if you want to manage it.

#Timeout: Time Control

A timeout is the time limit that stops a download if there's no progress — wget won't wait forever. Wget distinguishes several types of timeouts:

--read-timeout is the one you most often need to pay attention to: if the server sends data very slowly, wget stops after this limit without new data. An example of realistic settings:

wget --timeout=60 --dns-timeout=10 https://example.com/big-file.iso

The pattern above gives 60 seconds of leeway for connection and reading, with the exception of DNS, which is accelerated to 10 seconds. Timeout values that are too small make downloads fail pointlessly; too large, and wget sits silent for hours while a server is stuck.

#Limit Speed: --limit-rate

Downloading large files can eat up all your bandwidth — and disrupt other services on the same server. Wget provides the --limit-rate option to cap transfer speed:

wget --limit-rate=500k https://speed.hetzner.de/100MB.bin

The suffixes k, m, and g mean kilo, mega, and giga bytes per second respectively — so 500k equals 500 KB/second, and 2m equals 2 MB/second. It's like setting a safe following distance on the highway: the download still runs, but it doesn't endanger other road users.

#Background Download: -b

Large file downloads take a long time — and you don't need to wait for them in the terminal. The -b option (or --background) moves wget to the background right away:

wget -b -o download.log https://speed.hetzner.de/100MB.bin

With -b, wget returns to the shell prompt immediately and continues the download behind the scenes. Without -o, wget writes its log to a file named wget-log in the working directory. To monitor progress, open the log file:

tail -f download.log

This is a favorite pattern for long downloads: start it in the morning, let it work, and check the log in the afternoon. Wget also records its PID at the start of the log, so you can stop it with kill if needed.

#Common Pitfalls

1. Forgetting -c on large files. An interrupted download rerun without -c means starting from zero. Get into the habit of adding -c from the start.

2. Expecting resume on servers without Range support. If the server answers 200 instead of 206, no resume happens — that's a server limitation, not a wget bug.

3. --tries too large in batch jobs. 20 attempts per file over hundreds of files extends a job by hours. Use small values in batch scenarios.

4. -b without -o creates accumulating wget-log files. Give clear log names with -o so they're easy to track.

#Closing

In episode 4, you've equipped wget with resilience: resuming interrupted downloads with -c, configuring retries with --tries, controlling wait times with the --timeout family, limiting speed with --limit-rate, and running downloads in the background with -b while monitoring their logs.

Key takeaways:

• -c resumes a download from the last position; it needs Range support from the server (206 Partial Content).
• --tries=N limits retries; --tries=inf tries endlessly.
• --timeout sets the time limit; --dns-timeout and --read-timeout give per-stage control.
• --limit-rate=500k keeps bandwidth for other services.
• -b moves a download to the background; monitor it via the log file.

In episode 5, we'll cover batch download & multiple files — downloading dozens of URLs at once from a file list with -i, setting the destination directory with -P and -x, and using the file name from server headers with --content-disposition. See you in episode 5!