Learn Linux - Text Processing & Filtering Commands

#Introduction

After episode 5 where we covered pipelines, redirection, and I/O streams — how to direct command output and chain it with | — in this episode we'll fill in the "fuel" for those pipelines: text processing commands. A pipeline without filters is like an engine without fuel: it can be connected, but it produces nothing.

Why is this topic crucial? Because in Linux, almost everything is text. Application logs are text, configuration is text, command output is text, even the system's internal APIs are exposed as text. The ability to filter (grep), sort (sort), summarize (uniq -c), cut columns (cut), and extract patterns (sed, awk) from text is what separates the admin who "types commands" from the admin who "solves problems".

In this episode we'll cover: first, grep as the text search engine; second, sort and uniq for sorting and summarizing; third, cut, wc, and tr for cutting and measuring; fourth, sed and awk as the streaming editor and column reporter; fifth, find, locate, and xargs for finding and executing; finally, a real case study parsing a web server's access log.

#Main Discussion

#`grep`: The Text Search Engine

grep (global regular expression print) reads input and displays only the lines that match a pattern. It's the most basic and most used filter. Let's start with the most important options:

Option	Function
`-i`	Ignore case differences
`-v`	Invert: show lines that don't match
`-n`	Show line numbers
`-r`	Search recursively into directories
`-E`	Enable extended regex (advanced patterns)
`-c`	Count matching lines
`-w`	Match whole words, not partial

grep basics

grep -i error /var/log/syslog
grep -v "^#" /etc/ssh/sshd_config
grep -rn "Listen" /etc/nginx/

The examples above illustrate three common usage patterns: searching for errors in logs ignoring capitalization (-i), discarding comment lines from a config file (-v "^#"), and searching for the word Listen across the entire NGINX directory while showing file names and line numbers (-rn).

Note

grep -v is the inverse of grep — it's a discard filter. Very useful for cleaning noise: for example excluding comments (grep -v "^#") and empty lines (grep -v "^$") from config files so only the active configuration remains. The combination grep -v -e '^#' -e '^$' is a classic sysadmin idiom.

#Extended Regex with `-E`

Basic grep patterns already support regex, but for alternation (|) and several other metacharacters, you need -E. Notice the diff below — before and after using extended regex:

From plain grep to grep -E

grep -E "error|fatal|panic" /var/log/app.log
grep -E "^(error|fatal|panic)" /var/log/app.log

The first line searches for lines containing any of the three words in any position; the second line only matches lines that start with one of the three words. Note that egrep is an old alias for grep -E — don't get used to egrep, as it's considered deprecated and isn't available on every system.

#`sort` and `uniq`: Sorting and Summarizing

sort sorts text lines. By default it sorts lexicographically (like a dictionary), so 10 appears before 9. For numeric data, use -n:

`sort` option	Function
`-n`	Sort numerically (not dictionary order)
`-r`	Sort in reverse (descending)
`-k2`	Sort by the 2nd column/field
`-u`	Discard duplicate lines after sorting

uniq removes adjacent identical lines. The key word: adjacent. If duplicates aren't next to each other, uniq won't remove them — that's why the standard pattern is always sort | uniq, not uniq alone.

sort + uniq -c

cat kata.txt
sort kata.txt | uniq -c
sort kata.txt | uniq -c | sort -rn

The third line is an extremely valuable idiom: sort | uniq -c | sort -rn produces the frequency of each line, sorted from most frequent. uniq -c prepends the occurrence count, then sort -rn sorts numerically in reverse so the largest numbers are on top. We'll use this exact idiom in the log case study later.

#`cut`, `wc`, and `tr`: Cutting, Measuring, and Transforming

cut cuts a specific part of each line. The key concepts: delimiter (-d) is the column separator character, and field (-f) is the column taken.

Cutting columns with cut

cut -d: -f1 /etc/passwd
cut -d: -f1,7 /etc/passwd

The first example shows user names (column 1 of /etc/passwd, which is colon-separated); the second example shows columns 1 and 7 (users along with their default shell).

wc (word count) counts things in the input — lines (-l), words (-w), or characters (-c):

Measuring text

wc -l /etc/passwd
echo "one two three" | wc -w

tr (translate) replaces or deletes characters. It works per-character, not per-word:

Transforming characters with tr

cut -d: -f1,7 /etc/passwd | tr ':' ' '

Here tr ':' ' ' replaces every colon with a space — a small technique that turns delimiter-separated output into pleasant readable columns.

#`sed`: The Streaming Editor for Find & Replace

sed (stream editor) reads line by line and performs transformations. The most popular operation is substitution s/pattern/replacement/options:

Substitution with sed

sed 's/localhost/127.0.0.1/' config.txt
sed 's/error/ERROR/g' app.log

Without the g (global) option, sed only replaces the first occurrence on each line; with g, all occurrences are replaced. This pattern resembles a text editor's find & replace — except it works on a data stream and can go into a pipeline.

sed showing specific lines

sed -n '5,10p' config.txt

-n suppresses default output, and 5,10p prints lines 5 through 10 — a quick way to "view part of a file" without opening it entirely.

Caution

sed -i edits files directly in place without a backup. One typo in the pattern can alter an entire file beyond recovery. Always run sed without -i first to see the result, then repeat with -i. For important files, use sed -i.bak which automatically creates a file.bak copy before modifying.

#`awk`: Column Processing & Reporting

awk is a small programming language designed specifically for processing columnar text. For admins, three core abilities suffice: selecting columns ($1, $2, ...), conditions (if), and accumulation (counter variables).

Basic awk

awk '{print $1}' /etc/passwd
awk -F: '{print $1, $7}' /etc/passwd

Note: awk '{print $1}' takes the first field separated by spaces; -F: changes the delimiter to a colon, just like cut -d:. The key difference from cut: awk can do calculations and conditionals inside it:

Awk with conditions

awk '{ if (NF > 3) print NR": "$0 }' file.txt

This example prints lines that have more than 3 fields (NF = number of fields), along with their line numbers (NR). This is what makes awk the "go-to weapon" when filtering conditions are already too complex for grep.

#`find`, `locate`, and `xargs`: Finding & Executing

find searches for files by attribute — name, size, modification time — and can directly execute commands on each result:

Finding files

find /var/log -name "*.log" -mtime -7
find /home -type f -size +100M

The first line finds .log files modified in the last 7 days; the second line finds files larger than 100 MB under /home. Compare with locate, which searches through an index database (not by traversing directories), so it's far faster — but that database is updated by updatedb, so newly created files may not show up yet.

xargs turns standard output into arguments for another command. It's the bridge between "text output" and "command parameters":

xargs connecting output to arguments

find /var/log -name "*.log" | xargs grep "ERROR"

find locates all log files, then xargs assembles those names as arguments for grep "ERROR" — the result is every log file scanned for the word ERROR without typing them one by one.

Tip

When files found by find contain spaces or strange characters, default xargs will split them incorrectly. Use find ... -print0 and xargs -0 which use null as the separator — safe against all characters, including spaces and newlines. This becomes a habit that saves you in unpredictable production environments.

#Case Study: Parsing a Web Server Access Log

Now let's combine everything in a real problem every web admin faces: finding the 10 IPs that access the site most often from the NGINX/Apache access.log.

The standard combined log format has each line starting with the IP address, like this:

First step: extract the IP column — the first column of each line, so awk '{print $1}':

Step 1: extract the IP column

awk '{print $1}' access.log

Second step: sort so identical IPs are adjacent (sort), then count each IP's occurrences (uniq -c):

Step 2: count each IP's frequency

awk '{print $1}' access.log | sort | uniq -c

Third step: sort from most frequent (sort -rn), then take the top 10 (head):

Step 3: top 10 IPs

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

Output: top 10 IPs

   1247 203.0.113.7
    982 198.51.100.24
    613 192.0.2.15
     48 10.0.0.33

This result directly answers the question "who accesses the server the most?" — the first piece of information needed to detect attacks, plan caching, or simply understand traffic. Notice how one simple pipeline completes an analysis that in Excel would take several columns of formulas.

Important

The order of the pipeline above must not be changed: sort must come before uniq (because uniq only removes adjacent duplicates), and sort -rn for frequency must come after uniq -c. If sort -rn is placed at the start, you'll only sort IPs lexicographically, and the count results will be a mess. Remember the order: extract → sort → uniq → sort by frequency → head.

#Common Pitfalls

Mistake	Symptom	Solution
`uniq` without `sort` first	Duplicates not removed	Always `sort \| uniq`
Deprecated `egrep`	Not on every system, considered legacy	Use `grep -E`
`sed -i` without viewing the result first	File permanently changed incorrectly	Test without `-i`, then `sed -i.bak`
`sort` without `-n` for numbers	Wrong order (10 before 9)	Add `-n`
Case-sensitive `grep` on logs	Results missed due to capitalization	Use `-i` if needed
`xargs` without `-0` on names with spaces	Arguments incorrectly truncated	Use `-print0` + `xargs -0`
Searching for new files with `locate`	File not found	Run `updatedb` or use `find`

#Conclusion

In this episode 6, we've filled in the text processing toolkit you'll use every day: grep for filtering, sort and uniq for sorting and summarizing, cut and tr for cutting columns, wc for measuring, sed for streaming substitution, awk for column reporting, and find, locate, and xargs for finding and executing.

Key takeaways:

All data in Linux is text — and each tool above is a scalpel for cutting it.
The idiom sort | uniq -c | sort -rn is the standard way to count frequencies from any column.
sed without -i is safe, sed -i without a backup is dangerous — always test first.
awk goes beyond cut when you need conditions and calculations.
xargs bridges output and arguments — pair it with -0 for safety.

Now you can read and process text at scale. In the next episode 7, we'll discuss Symbolic Links, Archiving & Compression — understanding the difference between hard links and symlinks, creating and extracting tar archives, comparing gzip, bzip2, and xz, through to building a safe /etc backup routine. Stay motivated, because these backup skills will save you from disasters on production servers!