Learn Curl - Timeouts, Retries & Rate Limiting

#Introduction

In episode 10, you won the authentication battle — the server recognizes you. But there's a subtler enemy: time and network instability. Requests that hang indefinitely, servers that occasionally time out, connections that suddenly drop, or bandwidth devoured by giant transfers. This episode is about resilience: making curl never hang, not giving up on temporary failures, and not wrecking the network with greed.

These are the skills that separate experimental scripts from production scripts. Monitoring that calls an endpoint without a timeout will freeze along with the network; a CI pipeline without retry will fail because of a single blip; a download without limit-rate will slow down the whole office. Let's dissect each one.

#Timeouts

#--connect-timeout: Connection Phase Limit

The connection phase is the most prone to hanging — slow DNS, a server not responding to SYN, or a silently filtered port. --connect-timeout limits only this phase:

curl --connect-timeout 10 https://api.example.com/health

If the connection doesn't succeed within 10 seconds, curl gives up with an error — instead of hanging for hours. This value should be small (seconds, not minutes): a healthy TCP connection almost always establishes within seconds. If it takes longer, something is almost certainly wrong with the network.

#--max-time: Entire Transfer Limit

Unlike --connect-timeout which only covers the connection phase, --max-time limits the total operation duration — from connection until the transfer completes. This is the most important safety net for scripts:

curl --max-time 60 -O https://example.com/downloads/backup-2026.db

Think of the difference like this: --connect-timeout is the time limit for queuing at the door, --max-time is the time limit for the entire journey. For large file downloads, --max-time prevents a transfer from stalling indefinitely due to a half-dead connection. In cron and CI scripts, always set --max-time on requests that call external services.

#Retries

The real network is noisy: connections occasionally drop, servers restart, load balancers switch over. --retry makes curl automatically repeat the request when facing temporary failures:

curl --retry 3 --retry-delay 2 --retry-all-errors \
  https://api.example.com/health

The three options work together:

• --retry 3 — a maximum of three retry attempts.
• --retry-delay 2 — a two-second pause between attempts (giving the server time to breathe).
• --retry-all-errors — retries on all types of failures, not just those considered temporary.

Without --retry-all-errors, curl only retries failures that are transient — for example timeouts, connection resets, or HTTP 5xx codes — and gives up on other failures like syntax errors or 400. This option is the difference between "script failed because the network blinked" and "script waits for the network to stabilize then continues".

#Rate Limiting: --limit-rate

A full-speed download can drain office bandwidth or anger a server. --limit-rate reins in the transfer speed:

curl --limit-rate 200K -O https://example.com/downloads/backup-2026.db

The value can be a plain number (bytes/second) or with a suffix — 200K for 200 kilobytes, 5M for 5 megabytes per second. This is useful when downloading many files at once during work hours: limiting the speed keeps bandwidth fair for everyone and makes requests look more polite to the server. Imagine a water pipe — you adjust the tap so it doesn't drain the entire supply.

#Partial Downloads: --range

Sometimes you only need part of a file, not all of it — the first few bytes for inspection, or a particular section of a large log. The -r option (alias --range) requests a specific byte range:

curl -r 0-1023 -O https://example.com/logs/app.log
curl -r 500- -O https://example.com/logs/app.log

The range 0-1023 fetches the first 1024 bytes; 500- fetches from byte 500 to the end. Servers that support HTTP Range respond with 206 Partial Content. This feature works two ways with -C - from episode 7 — resume download is essentially a range request automatically assembled by curl.

#Keep-Alive and Connection Reuse

HTTP Keep-Alive allows multiple requests to share one TCP connection, avoiding repeated handshake costs. This is active by default in curl — every time you send several URLs in one command, curl tries to reuse the same connection:

curl -o a.json -o b.json \
  https://api.example.com/a \
  https://api.example.com/b

With -v, you can see the Connection: keep-alive header and the connection reused for the second request. Grouping requests to the same host in a single command is a simple yet impactful optimization: reducing handshake latency and load on the server. In scripts, note that calling curl multiple times separately (one process per request) throws away this benefit — combine same-host URLs when possible.

#The Complete Production Package

All the features above are most useful when combined. Here's an example command that absorbs all of them — firm timeouts, patient retry, and controlled speed:

curl --connect-timeout 10 --max-time 120 \
  --retry 4 --retry-delay 5 --retry-all-errors \
  --limit-rate 1M -C - -O \
  https://mirror.example.com/app-2026.07.iso

Let's dissect it: --connect-timeout 10 limits the connection phase, --max-time 120 limits the total transfer, --retry 4 --retry-delay 5 --retry-all-errors gives four retry chances with five-second pauses for almost any failure, --limit-rate 1M keeps bandwidth in check, and -C - ensures that if everything fails midway, the next run resumes from the last point. This is the face of curl ready for cron, CI, or production installers.

#Closing

Episode 11 makes curl resilient: --connect-timeout and --max-time to limit time, --retry with --retry-delay and --retry-all-errors to face temporary failures, --limit-rate to keep speed in check, --range for partial file retrieval, and an understanding of keep-alive and connection reuse.

The core thing to remember: timeouts keep you from hanging, retries keep you from giving up, and rate limits keep you from being greedy. The three are the coating of production-ready scripts — ready to face a network that's never perfect.

In the next episode 12, we'll enter the deepest security layer: TLS and certificates — how curl verifies certificates, what a certificate chain is, and when it's safe to relax verification. See you!