Learn Curl - Scripting & Automation

#Introduction

So far curl has always been run manually — typing a command, reading the result, judging it yourself. In episode 16, you'll turn curl into a machine: commands run without supervision, whose results are read by scripts, and that become part of a CI pipeline protecting production. The difference isn't in the curl command itself — it's in how you treat it: as a function with a return value, not as a command that merely prints text.

#Exit Codes: The Language Between curl and Scripts

Every time curl finishes, it exits with an exit code — a number that tells how the request ended. This number is the only thing a script can reliably read. Scripts check that code through $?:

curl -sS https://api.example.com/health
echo "curl keluar dengan kode: $?"

A far more important rule for scripts: code 0 means success, whatever the body contains. If the server neatly answers 404 Not Found, curl still considers it a success — the body was delivered, the error is only an HTTP status. For scripts, this is the biggest trap. The solution is --fail-with-body: curl exits non-zero (code 22) when the server answers 4xx/5xx, while still showing the body so debugging stays easy:

curl -sS --fail-with-body https://api.example.com/users/99999

if curl -sS --fail-with-body --connect-timeout 5 --max-time 30 \
  https://api.example.com/health > /dev/null; then
  echo "Layanan sehat"
else
  echo "Layanan bermasalah" >&2
  exit 1
fi

Use the if curl ...; then pattern every time a request failure must change the script flow — not just waiting for curl to print an error on a screen nobody may ever read.

#Machine-Readable Output: -w

Scripts aren't good at reading free text — they're good at reading structure. The -w (write-out) option from episodes 6, 15, and 18 turns curl's output into a parseable line: status code, timing, and size in one format you define.

curl -sS -o /dev/null \
  -w "%{http_code} %{time_total}s %{size_download}bytes %{url_effective}\n" \
  https://api.example.com/health

Three things make this line suitable for logs: one line per request (easy to grep), deterministic variables (not changeable text), and it can be stamped with a date via date +%F if needed. Writing results to a log file with >> creates an auditable trail — exactly what we recommended in episode 13.

#The curl | jq Pipeline: Automatic Parsing

Humans read JSON with their eyes; scripts read it with jq — filters that extract specific fields from a JSON document. The curl | jq pipeline is the most productive pairing in the entire curl ecosystem:

curl -sS https://api.github.com/repos/curl/curl/releases/latest \
  | jq -r '.tag_name'

curl -sS https://api.example.com/orders \
  | jq -r '.orders[] | select(.status == "paid") | .id'

• -r makes the output "raw" without quotes — the result is ready to use as an argument or store in a variable.
• Filters like select(.status == "paid") choose data based on conditions — an ability that turns raw JSON into a concise loopable list.

for id in $(curl -sS https://api.example.com/orders \
            | jq -r '.orders[].id'); do
  echo "Memeriksa order $id"
  curl -sS --fail-with-body "https://api.example.com/orders/$id" \
    | jq -r '.total'
done

Note that the loop above calls curl once for the list, then once per item — a reasonable pattern for automation. From episode 15 you know: if a batch of requests to the same host can run in parallel, consider --parallel for a faster version.

#Idempotency and Reproducibility

Scripts running in CI must give the same result every time they run — reproducible. Some habits break this:

• Random timestamps in the body — makes requests never identical and hard to compare.
• Inexplicit headers — results that depend on default values which can change between versions.
• Inline-written payloads — hard to review and prone to typos.

The solution is simple: make requests idempotent (running twice gives the same effect) and deterministic (identical content each run). Store payloads in files committed to the repository, and request explicitly:

curl -sS --fail-with-body --json @payloads/create-user.json \
  https://api.example.com/users

The create-user.json file is committed along with the script, reviewed in the same pull request — exactly the request library pattern we'll use in episode 22. A repeatable request is a request you can stand behind.

#curl in CI/CD

This is where all the skills come together. A CI pipeline doesn't care how beautiful the command is — it cares about the exit code. Here's curl on the two most common platforms.

#Health Check in GitHub Actions

name: Health Check
on:
  schedule:
    - cron: '*/5 * * * *'
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - name: Cek kesehatan API
        run: curl --fail-with-body --silent --connect-timeout 5 --max-time 30 \
          https://api.example.com/health

If the service isn't healthy, --fail-with-body makes this step fail — and a failed step fails the job. This scheduled health check is the first alarm before users complain.

#Smoke Test in GitLab CI

smoke-test:
  stage: test
  image: curlimages/curl:8.12.0
  variables:
    API_BASE: https://staging.example.com
  script:
    - curl --fail-with-body --silent "$API_BASE/health"
    - curl --fail-with-body --silent "$API_BASE/api/v1/users" \
        -H "Authorization: Bearer $API_TOKEN"

Notice the details that make this job repeatable: the curl image version is pinned (curlimages/curl:8.12.0) so behavior doesn't drift between builds, the base URL and token come through the environment, and every step uses --fail-with-body so an HTTP failure counts as a pipeline failure.

#Complete Script: Deploy Gate

Let's assemble everything into one standalone script — representative of what you'll meet in the working world: check health, wait for the service to stabilize, then mark deploy readiness.

#!/usr/bin/env bash
set -euo pipefail
 
ENDPOINT="${1:-https://api.example.com/health}"
ATTEMPTS=5
DELAY=3
 
for i in $(seq 1 "$ATTEMPTS"); do
  code=$(curl -sS -o /dev/null -w "%{http_code}" \
    --connect-timeout 5 --max-time 30 "$ENDPOINT" || true)
  echo "Percobaan $i/$ATTEMPTS: HTTP $code"
 
  if [ "$code" = "200" ]; then
    echo "Layanan sehat, deploy boleh jalan"
    exit 0
  fi
  sleep "$DELAY"
done
 
echo "Layanan belum sehat setelah $ATTEMPTS percobaan" >&2
exit 1

Let's dissect the logic: set -euo pipefail makes the script stop at the first unhandled error; || true prevents a failing curl from immediately stopping the loop; -w captures the status through the http_code variable in a comparable form; and the exit code at the end becomes the final signal for the pipeline. Scripts like this are what separate thoughtful automation from automation that just runs.

#Closing

Episode 16 turns curl into an automation machine: understanding exit codes and --fail-with-body as the failure language, producing structured output with -w for parseable logs, building the curl | jq pipeline for automatic JSON parsing, making requests idempotent and reproducible, and deploying curl in GitHub Actions and GitLab CI as health checks and smoke tests.

The core thing to remember: scripts don't read the screen — they read the exit code. All the beauty of terminal output means nothing if a script can't judge success or failure. Teach curl to speak the script language, and you can hand it any job without supervision.

In the next episode 17, we'll look at how curl interacts with its surrounding ecosystem: API testing tools — comparison with wget, httpie, and Postman, and how curl becomes the foundation of modern testing tools. See you!