Learn BASH Scripting - Integration with Sed, Awk & Text Processing

#Introduction

In episode 21 we covered building professional CLIs with getopts — how your script reads arguments, manages options, and presents tidy help — and in this episode we shift from input to data. The question we'll answer is no longer "how does the script receive commands", but "how does the script process text, files, and logs quickly and reliably".

On a production server, almost all data isn't a database — it's text. Application logs, config files, user lists in CSV format, output from other commands — all of it takes the shape of lines and columns that must be cut, transformed, joined, and counted. Pure BASH can actually do it with for and while loops, but that's like tearing down a car engine with just a screwdriver: possible, but slow, bug-prone, and inelegant. That's where sed and awk come in as two specialized tools that do the job in a single line.

Consider yourself a records clerk at an office. Sed is the intelligent cut-and-paste machine: it can find specific lines and replace them without opening a file in an editor. Awk is the spreadsheet calculator in the terminal: it can read a table, pick columns, and sum their values in one pass. The combination of both — plus BASH pipelines — lets your script process hundreds of thousands of lines in seconds, something impossible to do by hand.

In this episode we'll dissect sed first, then awk, then CSV/TSV parsing in pure BASH, and close with a real practice analyzing a web server's access.log plus the common pitfalls that trip people up most often.

#Main Discussion

#Why Text Processing Is the DevOps Main Weapon

Before diving into syntax, understand why this skill is so valuable. The principle comes from the Unix philosophy: each program does one thing well, and pipelines join them together. Instead of writing 50 lines of loops in BASH to process a file, you just chain a few small filters that throw data at each other through |.

Imagine this scenario: there are 500 thousand lines in access.log and a manager asks, "who are the 10 IP addresses visiting our site most this morning?". Answering with a text editor or Excel takes hours. With awk, sort, and uniq, the answer comes out in a split second. This is the ability that separates an admin who operates a server from an admin who understands a server.

There are three levels of text processing we'll build:

1. Sed — transforming text lines (search and replace, delete, print specific lines).
2. Awk — reading columns and computing aggregations.
3. Pure BASH — line-by-line parsing for logic that needs finer control.

Each has its place. The key in this episode is knowing when to use which.

#Sed: A Non-Interactive Editor Over a Data Stream

The name sed stands for stream editor. Unlike vi or nano, which wait for human interaction, sed takes text, processes it line by line according to commands, then prints the result — perfect for pipelines. The most-used operation is substitution:

echo "hello world, hello bash" | sed 's/world/bash/'
# hello bash, hello bash

By default, s/.../.../ only replaces the first occurrence on each line. Add the g (global) flag to replace all:

echo "hello world, hello bash" | sed 's/hello/hai/g'
# hai world, hai bash

What makes sed truly powerful is its support for regex (recall the regex patterns from episode 6) and capture groups. With the -E option (extended regex), we can capture part of a pattern and rearrange the result using \1, \2, and so on:

echo "user=arman-putra" | sed -E 's/user=(.*)-(.*)/user=\1 \2/'
# user=arman putra

The pattern (.*)-(.*) captures the two parts before and after the hyphen, then \1 \2 rearranges them with a space. This is very useful for data normalization: reformatting dates, tidying filenames, or splitting merged columns.

To modify a file directly, use -i (in-place). The most classic example in the DevOps world is editing config files from inside a script:

sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config

Imagine how dangerous it would be without sed: you'd have to open sshd_config, find the line, change it, save — manually, on dozens of servers. With sed, one script changes them all at once, and the result is consistent.

Sed also supports address ranges — selecting specific lines to process. The syntax is start,end + command:

sed -n '2,4p' file.txt       # cetak baris 2 sampai 4 saja
sed '/^#/d' file.conf        # hapus semua baris komentar (diawali #)
sed '/^DEBUG/,/^END/p' log   # cetak rentang antara DEBUG dan END

The -n option suppresses default output so only the requested lines (p) are printed — equivalent to a "line filter". And d deletes matching lines. The combination -n + /pattern/p is sed's way of mimicking a grep function with more control.

#Awk: Column Processor and Aggregator

If sed operates on lines, awk operates on columns. Awk works with a pattern { action } model: for each line matching the pattern, run the action. By default awk splits each line into columns based on spaces or tabs and stores them in variables $1, $2, and so on. $0 is the whole line.

awk '{ print $1, $3 }' access.log

Some built-in variables you must memorize:

Awk's real power shows in the BEGIN and END blocks, which enable aggregation. The classic example: summing response sizes (column 10) in an access log — the standard Apache/Nginx log format:

awk '{ total += $10 } END { print "Total bytes:", total }' access.log

The line total += $10 runs for every line; once done, END prints the final result. This is the "spreadsheet calculator" mentioned earlier — and it runs over millions of lines without running out of memory, because awk processes one line at a time.

Awk can also print with controlled formatting via printf, and capture values inside a script. The result of an awk execution can be captured as a BASH variable:

total_404=$(awk '$9 == 404 { c++ } END { print c+0 }' access.log)
echo "Jumlah 404: $total_404"

Notice the expression $9 == 404 — it acts as a pattern: only lines whose 9th column equals 404 are processed. That's awk's way of doing filter and count at once, one pipeline without an extra grep.

#Parsing CSV/TSV in Pure BASH

Not all text processing can be handled by sed/awk. Sometimes you need to read a CSV/TSV file line by line and run BASH logic for each record — for example calling an API for each line, or creating a directory for each user. For that, pure BASH with read is the answer:

while IFS=',' read -r nama email role; do
    echo "Membuat akun: $nama ($role)"
    useradd "$nama" -m
done < users.csv

There are three important parts that are often misunderstood:

1. IFS=',' — sets the comma as the column separator for this read only, without changing the script's global IFS.
2. -r — prevents backslashes in the file from being interpreted as escapes; without it, a Windows path like C:\data could become C:data.
3. < users.csv — file redirection into the loop; this keeps the loop running in the same subshell-free context as the script.

The same pattern applies to TSV (tab-separated): just swap IFS=',' for IFS=$'\t'.

#Practice: Analyzing access.log in One Pipeline

Time to combine it all. Suppose we have an access.log in the standard format:

203.0.113.10 - - [02/Aug/2026:08:15:22 +0000] "GET /products HTTP/1.1" 200 4321
198.51.100.4 - - [02/Aug/2026:08:15:31 +0000] "GET /products HTTP/1.1" 200 4321
192.0.2.77  - - [02/Aug/2026:08:16:05 +0000] "POST /api/orders HTTP/1.1" 500 214
203.0.113.10 - - [02/Aug/2026:08:16:44 +0000] "GET /assets/app.css HTTP/1.1" 200 83102

The columns: IP (1), ident (2), user (3), timestamp (4-5), request (6-8), status (9), bytes (10). Now answer four business questions at once:

total_req=$(wc -l < access.log)
unique_ip=$(awk '{ print $1 }' access.log | sort -u | wc -l)
total_bytes=$(awk '{ s += $10 } END { print s }' access.log)
echo "Total request : $total_req"
echo "IP unik       : $unique_ip"
echo "Total bytes   : $total_bytes"
 
echo "Top 10 IP tersibuk:"
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -10

Notice two important patterns:

• awk '{print $1}' | sort | uniq -c — sort first, then uniq. Unlike sort -u, uniq -c also gives the occurrence count, so we can sort again with sort -rn to get a ranking.
• wc -l < file uses redirection, not an argument — this prevents the filename from being printed in the output.

For 500 thousand lines, all the commands above finish in seconds. Try to imagine doing that manually in a text editor.

#When to Use Pure BASH, Sed, or Awk

The question that comes up most: "when do I use which?". The rule of thumb is simple:

The mental model: lines → sed, columns → awk, logic → pure BASH. Once a task involves looping with conditions and calling other commands, move it to a BASH loop. Once a task is only filtering and reshaping data, let sed/awk handle it in one line.

#Common Pitfalls

1. sed -i that isn't portable. In GNU sed (Linux), sed -i works directly. In BSD sed (macOS), sed -i requires a backup argument, so sed -i 's/x/y/' errors on a Mac. The portable solution for both: sed -i.bak 's/x/y/' — giving a backup at the same time. Delete the .bak file if it isn't needed.

2. Awk FS vs BASH IFS. Both are "column separators", but they live in different worlds. Awk reads FS (default space, set with -F or the FS variable), while read in BASH reads IFS. Writing IFS=',' inside a script won't change awk's behavior, and -F, won't change read. Don't mix up the contexts.

3. CRLF lines silently corrupting the last column. Files made on Windows end with \r\n. When parsed, the \r sticks to the end of the last column — for example the value 4321 becomes 4321\r, and awk's summing can produce weird numbers. Clean it first at the start of the pipeline:

cat data.csv | while IFS=',' read -r a b c; do ...; done
sed 's/\r$//' data.csv | while IFS=',' read -r a b c; do ...; done

4. Forgetting that uniq needs sorted input. uniq only removes duplicates that are adjacent. Using uniq without sort produces wrong counts. Always sort first, or use sort -u if you only need a unique list without counts.

5. Testing sed directly against a production file. Always test on a copy first without -i, or use sed -i.bak. One wrong regex pattern can destroy a config file that hundreds of users depend on.

#Conclusion

In this episode 22, you've mastered the foundations of text processing in BASH. We started with sed: substitution with s/.../.../, capture groups with \1 to rearrange patterns, address ranges to select specific lines, up to sed -i to edit config files directly from a script. Then awk: reading columns with $1, NF, and NR, summing aggregations in the END block, and filtering with expressions like $9 == 404. We also learned CSV/TSV parsing with a while IFS=',' read -r loop, then practiced everything to analyze an access.log — total requests, unique IPs, total bytes, and the 10 busiest IPs — in a single set of pipelines.

The key takeaways:

• Lines → sed, columns → awk, logic → pure BASH. Choose the tool by the shape of the task, not by habit.
• Capture groups (-E + \1) turn sed from a "text replacer" into a "structure processor".
• Awk aggregation in BEGIN/END handles millions of lines without wasting memory.
• Always -r on read, always stay aware of pure BASH's limits with complex CSV.
• Clean \r (CRLF) and test without -i before touching production files.

This data-processing ability from files becomes the bridge to a wider world: in episode 23 we'll cover interaction with system tools & external APIs — using curl to call HTTP APIs, jq to process JSON, sending notifications to Slack and Telegram, and running non-interactive database queries. Keep the fire burning!