Learn Curl - Parallel Transfer & Performance

#Introduction

In episode 14, you saw how HTTP/2 and HTTP/3 reduce waiting time at the protocol level. Now the question shifts: how do you use those capabilities to make an entire batch of work finish faster? The answer lives in two places: parallelism — running many transfers at the same time — and transfer optimization — making sure every byte travels as efficiently as possible, then proving it with numbers. Episode 15 is about speed that's measured, not felt.

#Parallel: --parallel and --parallel-max

The naive way to fetch many URLs is a loop: one request finishes, then the next request departs. Total time is the sum of all request times — and while one request waits for the server's response, the connection and CPU sit idle.

--parallel (alias -Z) changes that: curl runs all URLs concurrently in a single process, with connections capped by --parallel-max (default 50):

curl -Z -o /dev/null \
  https://api.example.com/users \
  https://api.example.com/posts \
  https://api.example.com/comments \
  https://api.example.com/orders \
  https://api.example.com/products

curl -Z --parallel-max 10 -o /dev/null \
  https://api.example.com/users \
  https://api.example.com/posts \
  https://api.example.com/comments

Imagine one cashier serving five queues at once instead of one by one. As long as those requests are independent — none depends on another's result — parallelism cuts execution time drastically without writing a single line of threading logic.

#HTTP/2 Multiplexing in One Process

The parallelism above works even with HTTP/1.1 — curl opens many connections. But with HTTP/2, there's a second layer: all URLs to the same host share one TCP connection via multiplexing. Each URL becomes a stream inside the same connection, running in parallel without opening new connections.

curl -Z https://api.example.com/users https://api.example.com/posts

Why does this matter? Every new TCP connection means a new handshake — especially the expensive TLS handshake from episode 12. HTTP/2 multiplexing eliminates that cost for batches heading to the same host. This is the most efficient combination: -Z for parallelism across URLs, HTTP/2 for sharing one connection behind it.

#Connection Reuse and Keep-Alive

From episode 11, you know curl reuses connections for same-host URLs within one command. This works through HTTP keep-alive: a connection isn't closed after one request, but kept for the next request.

A habit that often throws away this benefit: calling curl repeatedly, one process per request. Each call is a new process startup, a new connection, and a new TLS handshake. If your script calls curl ten times to the same host, you pay ten times the handshake cost — when once would have been enough.

# Better: one process, one connection, all URLs share keep-alive
curl -o a.json https://api.example.com/a \
     -o b.json https://api.example.com/b \
     -o c.json https://api.example.com/c

If a script must keep calling curl repeatedly, at minimum combine into one call with multiple -os — or hand it over to --parallel, which handles connection management for you.

#Output Buffering: When to Stream

By default, curl buffers data written to a terminal or pipe — useful for keeping output intact. But for streaming — for example following real-time logs or piping a long response to another process — buffering makes data stall and arrive in bursts.

The -N option (alias --no-buffer) turns off buffering so data flows as soon as it arrives:

curl -N https://api.example.com/events

curl -N https://api.example.com/events | while read -r line; do
  echo "[event] $line"
done

Rule of thumb: for normal downloads leave the default buffering; for streaming and pipelines that need immediate data, use -N.

#Compression: --compressed

HTTP responses are often text — JSON, HTML, or logs — which compresses extremely well. Without compression, you download more bytes than necessary. --compressed asks the server to send a compressed version and decompresses it automatically:

curl --compressed https://api.example.com/reports/big.json

curl sends the Accept-Encoding: deflate, gzip header and — if the build supports it — br (brotli) and zstd join the list too. Servers that support it send a compressed response, and curl decompresses it before writing to output.

curl --version

Check the words brotli and zstd in the Features line to see which algorithms your build supports. For APIs returning large JSON, compression often cuts transfer size by 70-80% — a number immediately visible in -w.

#Measuring Performance with -w

None of the optimizations above mean anything without proof. The -w (write-out) option from episodes 6 and 18 isn't just for debugging — it's a one-line benchmark that can be compared across configurations. Collect the key variables in one format:

curl -s -o /dev/null -w "HTTP %{http_code} | DNS %{time_namelookup}s | Koneksi %{time_connect}s | TLS %{time_appconnect}s | Byte pertama %{time_starttransfer}s | Total %{time_total}s | %{size_download} bytes | %{speed_download} B/s\n" \
  https://api.example.com/health

Important variables for performance measurement:

Note the diagnosis pattern: a high time_appconnect points to TLS (episode 12), a high time_starttransfer points to a slow-responding server, and a size_download that shrinks drastically with --compressed is proof compression is working.

#Benchmark: Before and After

Let's assemble everything into a repeatable experiment: compare five sequential requests against five parallel requests, to the same host.

curl -s -o /dev/null -w "%{time_total}\n" \
  https://api.example.com/users https://api.example.com/posts \
  https://api.example.com/comments https://api.example.com/orders \
  https://api.example.com/products

curl -Z -s -o /dev/null -w "%{time_total}\n" \
  https://api.example.com/users https://api.example.com/posts \
  https://api.example.com/comments https://api.example.com/orders \
  https://api.example.com/products

Run both several times and compare. For a batch to the same host, HTTP/2 multiplexing makes the parallel version feel dramatically faster — and the difference is a number you can bring to a meeting when explaining why a script must be reworked. Optimization without measurement is just opinion; with -w, you have data.

#Closing

Episode 15 equips you with a complete performance kit: parallelism with -Z and --parallel-max, HTTP/2 multiplexing sharing one connection, connection reuse and keep-alive eliminating repeated handshakes, buffering with -N for streaming, compression with --compressed (gzip, brotli, zstd), and performance measurement with -w proving every optimization.

The core thing to remember: speed isn't about typing faster, but about eliminating wasted time. Repeated handshakes, uncompressed bytes, and requests waiting their turn are measurable waste — and now you know how to eliminate them and prove it.

In the next episode 16, we'll turn all these capabilities into a machine: scripting and automation — exit codes, parseable output, curl pipelines with jq, and integrating curl into CI/CD. See you!