Learn Wget - FTP & File Transfer

#Introduction

In episode 5 you downloaded many files at once using file input and batch download. Now we move to a protocol that is actually part of wget's own historical roots: FTP. Before HTTP dominated the web, FTP was the primary way to distribute files over the internet — and wget itself is a descendant of the Geturl program, which was built from the start to fetch files over FTP. It makes sense that wget remains a full-featured FTP client to this day.

Why is this episode important? Lots of infrastructure still lives on FTP: software archive mirrors (GNU, Linux distros), public institutional data, and corporate internal NAS boxes. If you work with legacy servers or automate fetching data from FTP sources, wget is the fastest tool to tame them — without needing to install a separate FTP client.

#FTP URL Schemes in Wget

Wget talks FTP simply by changing the URL scheme: ftp:// for plain FTP, and ftps:// for FTP over TLS (FTPS). Otherwise, the command is identical to HTTP — wget needs no special handling:

wget ftp://files.example.com/laporan-2026.pdf

FTP is like an unencrypted telephone: anyone controlling the network path can eavesdrop on commands, credentials, and file contents passing through. FTPS is the version wrapped in TLS — data and credentials are sent encrypted. The rule of thumb: as long as the server offers ftps://, use it. Plain FTP should only be used on networks you fully trust.

#FTP Authentication: --user and --password

Many FTP servers request credentials before allowing access to certain directories. Wget provides two options for this: --user for the user name and --password for the password.

wget --user arman --password 's3cret!' ftp://files.example.com/laporan-2026.pdf

A detail that has saved many people: if you include --user but don't write --password, wget will prompt for the password interactively in the terminal when needed. That way there's no excuse for pasting a password into the command history — just write --user and let wget ask.

There's also a way to put credentials inside the URL, for example ftp://arman:rahasia@files.example.com/laporan-2026.pdf. This pattern is the most leak-prone because URLs often end up in logs, scripts, and history. Avoid it except for one-off use on a personal machine.

Not all FTP servers require an account. Many public mirrors serve anonymous access: without --user, wget automatically logs in as the anonymous user — wget ftp://ftp.example.com/pub/file works directly against public archives like GNU mirrors or Linux distro package mirrors. This is why wget remains the tool of choice for fetching packages from public mirrors without setting up any account.

For cases where FTP credentials differ from HTTP credentials on a single URL, wget also provides --ftp-user and --ftp-password, which override the generic --user and --password values specifically for FTP connections. Details like these are rarely used, but they save you when a single command must handle multiple hosts with different identities.

#Recursive Download: Pulling an Entire FTP Directory

One of wget's strengths over FTP is pulling an entire directory tree in a single command. The -r (recursive) option makes wget descend into all subdirectories and download every file it finds:

wget -r ftp://files.example.com/backup/

The command above pulls the contents of backup/ along with all its subfolders, then replicates that directory structure to local disk. This is very useful for one-way synchronization: grab all the data from FTP and make it a local copy in one command. While the process runs, wget creates a .listing file as a directory listing record for recursion purposes — this temporary file is removed automatically when finished.

#FTP Behind a Firewall: Passive and Active Modes

FTP has a protocol quirk: it uses two connections at once — one for commands (port 21) and another for data transfer. How that data connection is opened determines whether a download succeeds behind firewalls and NAT.

By default wget uses passive mode: the client opens the data connection, exactly like the direction of an HTTP connection. This mode works in almost all environments, including behind office NAT. Some old firewalls only allow active mode, where instead the server calls the client to open the data connection. If an FTP download hangs or fails at the transfer stage, try enabling active mode:

wget --no-passive-ftp ftp://files.example.com/laporan-2026.pdf

The analogy: passive mode is like calling back — the client initiates, so it's safe behind NAT. Active mode is like asking the server to call your home — it needs a door deliberately opened. In modern practice, passive mode is almost always the right choice, and you only need --no-passive-ftp when dealing with genuinely legacy environments.

#FTP Globbing: Time-Saving Wildcard Patterns

When facing a directory full of files with similar names — for example dozens of daily log files — downloading them one by one makes no sense. Over FTP, wget supports globbing: the wildcard patterns *, ?, and [] to select many files at once.

wget 'ftp://files.example.com/logs/*.txt'

Here's the interesting part: wget doesn't send that wildcard pattern to the server. It fetches the directory listing, matches the glob pattern locally, then downloads every matching file. That's why globbing only works with Unix-style FTP servers (or those mimicking the Unix ls listing format).

#FTP Globbing vs HTTP

This is the part that confuses beginners the most: the same wildcard patterns don't work over HTTP.

A helpful analogy: FTP is like an archive cabinet you can open and pick files directly by their labels. HTTP is like a delivery service — you give a full address, and there's no concept of "grab everything whose name starts with X". That's why wget 'ftp://host/dir/*.txt' downloads many files, while a similar command with an HTTP URL almost certainly fails.

#Combined Practice: Recursive + Authentication + Selection

The capabilities in this episode are rarely used in isolation. A real-world scenario that often shows up in the field: pulling an entire report archive from internal FTP, only PDF files, complete with authentication — in a single command:

wget -r --user arman \
     --accept pdf \
     ftp://files.example.com/arsip/

One command handles three things at once: descend into all subdirectories, log in with the identity arman, and only keep PDF files. For sensitive archives, just switch the scheme to ftps:// — wget -r ftps://files.example.com/arsip/ — and the entire transfer runs encrypted. Combinations like this are why wget remains a go-to tool for admins doing mass data transfers between servers.

#Closing

Episode 6 equips you with file transfer capabilities over FTP: ftp:// and ftps:// URLs, authentication with --user and --password along with the interactive prompt, recursive directory downloads with -r, wildcard globbing to select many files at once, and an understanding of why globbing doesn't apply over HTTP.

The key takeaway: FTP in wget isn't just an alternative to HTTP — it's a protocol with unique behaviors whose differences you need to understand, from directory structure replication and globbing to credential security. And always choose FTPS for sensitive data.

In episode 7, we return to HTTP for finer matters: sending requests with custom headers, replacing the User-Agent, sending POST data, and managing cookies and Basic Auth. See you in episode 7!