Learn Wget - HTTP Requests, Headers & Cookies

#Introduction

In episode 6 you transacted with FTP servers. Now we return to HTTP — the protocol behind most web traffic — and look at the "face" of a request. By default, wget sends a plain request: the User-Agent Wget/1.21, no cookies, no special headers. That's fine for public pages, but many servers behave differently toward requests that "look" like bots.

The ability to shape requests is key to three common scenarios: accessing content that's only open to certain browsers, sending data to a server (login forms, simple APIs), and maintaining a login session while mirroring a protected site. This episode gives you the tools for all three.

#Custom Headers with --header

Every HTTP request consists of a method line, a target URL, and a set of headers. Wget sends minimal built-in headers — and when you need more, the --header option is the doorway.

wget --header="Accept-Language: id,en" \
     --header="X-Client: devvnull-bot" \
     https://api.example.com/data.json

The --header option can be repeated as many times as you like; each line adds one header to all requests in that session. Custom headers serve a wide range of needs: choosing content language, labeling your application's identity, sending Authorization tokens for APIs, right up to setting Content-Type on POST requests. Note the format: name and value are separated by a colon, exactly like a real HTTP header.

#Replacing the User-Agent

User-Agent is the header that identifies the software sending the request. By default wget sends Wget/1.21 — honest, but not always welcome: some servers serve different page versions to browsers vs bots, and others reject unrecognized agents. The --user-agent option lets you change it:

wget --user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36" \
     https://example.com/halaman-browser-saja

Hold two attitudes. First, replacing the User-Agent is a legitimate practice for accessing content intended for browsers, and many official docs recommend including your contact in the agent so server admins can reach the bot's owner. Second, never use a fake agent to bypass access restrictions you should respect — that violates ethics and may violate the site's terms of service.

#Sending POST Data: --post-data and --post-file

Wget was born as a downloader, but it can also send data to servers. The --post-data option sends a string as the POST request body, while --post-file takes the body from a file's contents:

wget --post-data="user=arman&role=devops" \
     https://httpbin.org/post

wget --post-file=payload.txt https://httpbin.org/post

Both options change the method to POST and send the data as-is in the request body. The expected format is key=value pairs separated by & (form-url-encoded), exactly like an HTML form submission. Note that wget does not support multipart for file uploads — file uploads are still the domain of curl or other clients.

#Cookie Jar: Maintaining a Login Session

Cookies are how HTTP remembers state between requests — and they're the foundation of almost every web login system. Wget manages cookies through a paired set of options: --save-cookies to save cookies to a file, and --load-cookies to load them back on a later session. The saved file uses the Netscape cookies.txt format, a text format that can be read and edited.

The usage pattern has two steps. First, log in to the site once — usually via a POST request to the auth page — while saving the cookies:

wget --save-cookies cookies.txt \
     --post-data="user=arman&password=rahasia" \
     https://example.com/login

Second, use the saved cookies for subsequent requests that need the session:

wget --load-cookies cookies.txt \
     -p https://example.com/area-pribadi/artikel.php

#Basic Auth: --user, --password, and --auth-no-challenge

Besides cookies, the simplest way to secure HTTP is Basic Auth: credentials sent as an Authorization header in encoded form. Wget supports it via --user and --password, the same as FTP:

wget --user arman --password rahasia \
     https://api.example.com/private/data.json

Wget's default behavior is to wait for the challenge: it sends a plain request, and only sends credentials after the server responds with a 401 code requesting them. The --auth-no-challenge option changes this behavior — wget includes Basic credentials in every request immediately without waiting for the challenge, for the rare servers that never send 401.

#Inspecting Your Own Requests

All the options above work behind the scenes — and when the result isn't what you expected, you need to see what was actually sent. Sites like httpbin.org provide endpoints that return an echo of your request: the headers, body, and method you sent, without needing your own server.

wget -qO- --post-data="user=arman" \
     --header="X-Custom: tes" \
     https://httpbin.org/post

The response contains a JSON object with a headers field showing all your request headers (including custom ones) and a form field showing the POST data. This is the fastest way to validate that --header is written correctly, --user-agent actually got replaced, and the POST data arrives in the expected form — before you aim at a production server that doesn't forgive mistakes.

#Closing

Episode 7 gives you full control over the shape of HTTP requests: repeatable custom headers with --header, replaceable identity via --user-agent, POST data with --post-data and --post-file, login session management via the cookie jar with --save-cookies and --load-cookies, and Basic Auth with --auth-no-challenge for servers that don't present a challenge.

The key takeaway: the same request, a different face, a different result. Servers decide behavior based on what you send — and wget gives you control over that. Use it ethically: don't imitate an agent to break restrictions, and always protect your credentials.

In episode 8, we'll tidy up all these flags: storing default policies like User-Agent, proxy, and retry counts in the wgetrc configuration file, including its priority over the command line. See you there!