Learn BASH Scripting - Interaction with System Tools & External APIs

#Introduction

In episode 22 we covered integration with sed, awk, and text processing — how scripts process text, files, and logs inside the server — and in this episode we open the window outward. We'll connect BASH scripts to the outside world: HTTP APIs, notification services, and databases.

Imagine your script until now as a diligent worker inside a single closed room: tidying files, cleaning logs, processing data. This episode turns it into a courier communicating with the world: it can ask a weather service, report status to the team's Slack channel, send a Telegram message, and execute database queries. This is a big leap, because most high-value DevOps automation — monitoring, alerting, deployment pipelines, backup reporting — is built on the ability of "scripts talking to other systems".

Also imagine a real situation: three in the morning, the backup script fails. Without API integration, you'd only find out the next morning when reading logs. With integration, your Telegram pings at 03:00: "BACKUP FAILED on the production server". The difference between "waiting for the problem to be found" and "the problem finding you" is what makes integration a mandatory DevOps skill.

In this episode we'll dissect curl as an HTTP client, jq as a JSON parser, notifications to Slack and Telegram, non-interactive database queries, then combine them in one end-to-end case study. Brace yourself — this episode produces scripts that are genuinely useful in the working world.

#Main Discussion

#curl: The HTTP Client That Accompanies Every Script

curl is an HTTP/HTTPS client that's almost always available on Linux systems (check with curl --version). It's your script's "long arm": making requests, receiving responses, and storing them.

The simplest request is GET:

curl -s https://api.example.com/users

Without -s, curl shows a progress meter to stderr — useful in an interactive terminal, but annoying inside a script. Use -s (silent) to hide it. However, -s also hides errors, so in production scripts it's almost always paired with -f (fail on error): curl will exit with a non-zero status if the server replies 4xx/5xx, so failures can be detected inside the script.

if curl -sf https://api.example.com/health > /dev/null; then
    echo "API sehat"
else
    echo "API bermasalah"
fi

To send data, we use -X POST together with -d (data). Example of sending a form:

curl -s -X POST -d "title=Belajar%20BASH" https://api.example.com/posts

Note that -X POST and -d change the method from GET (default) to POST. Custom headers can be added with -H, for example an auth token or JSON content type:

curl -s -X POST \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"name":"Arman","role":"devops"}' https://api.example.com/users

API output is often long JSON. Rather than polluting the screen, save it to a file with -o, or use a filename followed by -O to fetch content:

curl -s -o response.json https://api.example.com/report
curl -s -w "HTTP %{http_code} dalam %{time_total}s\n" -o /dev/null https://api.example.com/health

The -w (write-out) option is your best ally for measuring performance: %{http_code} gives the status code, %{time_total} gives the total request time. This is very useful for health checks that also record latency.

#jq: The JSON Parser That Saves Scripts

BASH has no built-in way to parse JSON — trying to extract a value from JSON with sed/regex is a recipe for disaster, because JSON structure can change at any moment. That's where jq comes in: a dedicated JSON parser that reads input and extracts whatever you ask for. If it isn't installed, install it with your package manager (apt install jq, brew install jq, etc.).

The most basic use: take one field:

curl -s api.example.com/weather | jq '.temperature'

If the response is nested (objects inside objects), use dots to descend levels:

{
  "location": {
    "city": "Jakarta",
    "country": "ID"
  },
  "temperature": 29.5
}

curl -s api.example.com/weather | jq '.location.city'
curl -s api.example.com/forecast | jq '.days[0].temp_max'

For arrays, .[0] takes the first element, while .[] iterates all elements. The combination .days[] | select(.rain > 0) filters elements by condition — equivalent to grep, but aware of JSON structure:

curl -s api.example.com/forecast | jq '.days[] | select(.rain > 0) | .date'

One more crucial option: -r (raw). Without -r, jq's output is always wrapped in string quotes; with -r, the output is pure text — so it can be captured directly as a BASH variable:

kota=$(curl -s api.example.com/weather | jq -r '.location.city')
echo "Cuaca di $kota"

Remember the pattern from episode 22: echo "$json" | jq '.name' works because jq reads from stdin — and the curl | jq pipeline is the most common pairing in DevOps scripts. With jq, you can process API responses into ready-to-use data structures, not just raw text.

#Notifications: Scripts That Talk to Slack & Telegram

Automation scripts often run at night, far from human eyes. To get results to you, scripts need to send notifications. The two most common targets in Indonesia: Slack (team channels) and Telegram (personal devices).

Slack uses Incoming Webhooks: you create a unique URL in Slack, then simply POST JSON containing text to that URL:

curl -s -X POST -H 'Content-type: application/json' \
     -d '{"text":"Deploy selesai: versi 2.4.1 live di produksi 🚀"}' \
     "$SLACK_WEBHOOK_URL"

Telegram uses the Bot API. Create a bot via BotFather to get a token, then send a message to sendMessage:

curl -s "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage" \
     --data-urlencode "chat_id=$TELEGRAM_CHAT_ID" \
     --data-urlencode "text=Backup server berhasil diselesaikan"

--data-urlencode is critical here: it ensures spaces, special characters, and URL symbols inside the message are encoded correctly — writing -d "text=..." raw will break if the message contains spaces or &.

#Non-Interactive Database CLIs

Automation often needs to read or write data to a database. Fortunately, the mysql and psql CLI clients can run non-interactively — without an interactive terminal session — so they fit inside scripts.

For MySQL/MariaDB, query directly with -e, or use a heredoc for long queries:

mysql -h "$DB_HOST" -u "$DB_USER" -p"$DB_PASS" myapp <<SQL
SELECT COUNT(*) AS total_users FROM users WHERE status = 'active';
SQL

For PostgreSQL, psql -c does the same, and pg_dump extracts backups:

psql -h "$DB_HOST" -U "$DB_USER" -d myapp -c \
     "SELECT COUNT(*) FROM orders WHERE created_at >= NOW() - interval '1 day';"
 
pg_dump -h "$DB_HOST" -U "$DB_USER" -d myapp -Fc -f /backup/myapp.dump

#Practice: Fetch API Data, Process with jq, Send a Notification

Time to combine it all. We'll build a script that fetches a weather forecast, checks whether tomorrow will see heavy rain, and notifies the team on Slack if so. This is a real template for weather-based monitoring scripts — for example for teams managing outdoor events or logistics distribution.

#!/usr/bin/env bash
set -euo pipefail
 
SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL:?SLACK_WEBHOOK_URL wajib diisi}"
CITY="${1:-Jakarta}"
 
json=$(curl -sf "https://api.open-meteo.com/v1/forecast?latitude=-6.2&longitude=106.8&daily=precipitation_sum&timezone=Asia%2FJakarta")
 
rain=$(echo "$json" | jq -r '.daily.precipitation_sum[0]')
 
if [ "$rain" -gt 10 ]; then
    curl -s -X POST -H 'Content-type: application/json' \
         -d "{\"text\":\"⚠️ Hujan deras ($rain mm) besok di $CITY. Antisipasi keterlambatan logistik!\"}" \
         "$SLACK_WEBHOOK_URL" > /dev/null
else
    echo "Aman: curah hujan besok $rain mm"
fi

Discuss this script line by line:

• set -euo pipefail — strict mode (remember episode 13): the script stops if any step fails.
• :? on SLACK_WEBHOOK_URL — if the variable isn't set, the script errors immediately with a clear message. This pattern closes the door on "script runs but silently has no credentials".
• curl -sf — silent + fail on error: network failures aren't passed on to jq.
• jq -r '.daily.precipitation_sum[0]' — take the first day's rain value as pure text.
• [ "$rain" -gt 10 ] — arithmetic comparison (remember episode 5). Careful: > inside [...] is redirection, which is why we use -gt.
• The notification is only sent when the condition is met — this is the heart of conditional alerting.

This kind of script is the basic architecture of all API-based automation: fetch → process → compare → act. Once you understand this pattern, almost every integration — deploy status, stock prices, uptime monitoring, new orders — is just a variation of the same template.

#Common Integration Pitfalls

1. -s hides errors until it's too late. Without -f or checking $?/%{http_code}, an error response (e.g. 500) is treated as success and processed further. Always pair -sf and verify the status before processing output.

2. jq isn't installed on the server. A script depending on jq fails silently. Detect it early with a clear message:

if ! command -v jq > /dev/null; then
    echo "ERROR: jq tidak terpasang. Instal dengan 'apt install jq'" >&2
    exit 1
fi

3. Missing quotes around the JSON payload. The Slack payload {"text":"..."} contains many double quotes — if it isn't wrapped in single quotes correctly, the shell breaks it apart. For dynamic values, build JSON with jq -n or printf and escape double quotes explicitly, as in the practice script above.

4. Hitting API rate limits. Public APIs limit requests per minute. A script calling an API in a loop without pausing will be rejected (429). Add sleep between calls, and for retries use exponential backoff (sleep $((2 ** retry))).

5. Logging credentials. Never print tokens or passwords to logs. Just log the result ("notification sent"), not a payload containing secrets.

#Conclusion

In this episode 23, you've opened your BASH script to the outside world. We learned curl as an HTTP client: -s for silent, -f to fail on errors, -o to save output, -w to measure latency, and -X POST with -d/-H to send data and headers. Then jq for JSON processing: taking fields with .key, accessing arrays with .[] and indices, filtering with select, and capturing clean output with -r. We also built notifications to Slack via webhook and Telegram via Bot API, executed non-interactive MySQL/PostgreSQL queries with heredocs, and assembled everything into a weather monitoring script that sends an alert only when the condition is met.

The key takeaways:

• The curl -sf pair plus status verification is the minimum standard for requests inside scripts.
• jq is the only reliable way to process JSON in BASH; combine it with -r for pure values.
• The fetch → process → compare → act pattern is the foundation of all API-based automation.
• Credentials always come from environment variables, never hardcoded or logged.
• Use --data-urlencode for notification messages containing spaces and symbols.

With the ability to talk to the outside world, your scripts are now alive and stand-alone. But a living script needs to tell its story — in episode 24 we'll cover logging, colorizing output, and terminal UX: recording traces with timestamps and log levels, coloring output to guide the eye, and adding spinners and progress for a professional terminal experience. See you in the next episode!