Learn Linux - Pipelines, Redirection & I/O Streams

#Introduction

After episode 4 where we covered how to read, view, and edit text files — from cat, less, head, tail -f for real-time log monitoring, to the CLI editors nano and vim — in this episode we'll discuss one of the concepts that sets Linux apart from other operating systems: I/O streams, redirection, and pipelines.

Why is this concept important? Because almost all Linux administration work — from reading logs, saving command results, to chaining several commands into one pipeline — rests on your ability to direct the flow of data. Without understanding this, you'll only type commands one at a time and copy output manually. By understanding it, you can build automated, efficient workflows.

In this episode we'll cover four things: first, the three standard streams that always accompany every process; second, redirection operators for saving and directing output; third, the pipeline (|) for combining commands and tee for displaying and saving at once; fourth, a real case study chaining all the concepts to build a system report.

#Main Discussion

#The Three Standard Streams: stdin, stdout, stderr

Every program running on Linux has three default "pipes" connecting it to its surroundings. These three pipes are called standard streams, and each has a file descriptor (fd) in the form of a number:

Stream	File Descriptor	Direction	Function
`stdin`	0	In	Input read by the program (usually the keyboard)
`stdout`	1	Out	The program's normal output (the desired result)
`stderr`	2	Out	Error / diagnostic output (error messages)

Here's a restaurant analogy: the customer orders through the counter (stdin), the kitchen sends out dishes through the service door (stdout), and if the chef has complaints, those messages go out through a separate route (stderr). These two separate output paths aren't without reason — you often want to save successful results to a file while errors are displayed on screen, and with this separation, both can be directed independently.

Let's see the difference in practice. The ls command below works normally, but the second command is deliberately made to fail:

stdout vs stderr

ls /home
ls /tidak_ada

Output

/home
ls: cannot access '/tidak_ada': No such file or directory

Notice: the first line is stdout (the real result), while the second line is stderr (the error message). Both appear in the terminal because the terminal shows both by default — but technically they travel different paths, and this is what we'll exploit through redirection.

Note

Why isn't stderr simply merged into stdout? Because in automation, you often want to "discard" errors or separate them into their own log file. Imagine a script producing 1000 lines of output and 10 lines of errors — if the two are mixed, filtering the errors becomes much harder. Path separation is a deliberate design, not a coincidence.

#Basic Redirection Operators: `>` and `>>`

The > operator redirects stdout to a file. If the file already exists, its contents are overwritten; if it doesn't exist, the file is created. The >> operator does the same, but appends to the end of the file.

Overwrite vs Append

echo "first line" > log.txt
echo "second line" > log.txt
cat log.txt
echo "third line" >> log.txt

Output of cat log.txt

second line
third line

Notice the sequence: the first > writes first line; the second > overwrites with second line; then >> appends third line. The final result: first line is gone forever.

Caution

The > operator is destructive without confirmation. Typing ls /var/log > log.txt intending to "save", when you actually meant to append, will destroy the previous contents of log.txt. Always ask yourself: do I want to overwrite or append? If in doubt, use >>. All too often after this accident happens, you realize that bash has no undo button.

#Redirecting stderr and Combining Both Streams

Besides stdout, we can explicitly redirect stderr with 2>. The syntax is always the file descriptor number followed by the operator with no space:

Operator	Meaning
`1>`	Redirect stdout to a file (same as `>`)
`2>`	Redirect stderr to a file
`&>`	Redirect stdout and stderr to a file
`2>&1`	Direct stderr to the same destination as stdout
`<`	Take input from a file, not the keyboard

A practical example of separating the two streams:

Separating stdout and stderr

ls /home /tidak_ada 1> hasil.txt 2> error.txt
cat hasil.txt
cat error.txt

Contents of hasil.txt and error.txt

/home
ls: cannot access '/tidak_ada': No such file or directory

Here hasil.txt contains only the successful output, while error.txt contains only the error messages. This is the pattern scripts use to record every failure in a separate audit log.

Then there's the 2>&1 operator — note that its order is very important. The expression 1> file 2>&1 means: direct stdout to file, then direct stderr to "stdout's current destination" (which is file). The result is both streams mixed into one file, similar to &>:

Combining stdout and stderr into one file

ls /home /tidak_ada > gabungan.txt 2>&1
cat gabungan.txt

Contents of gabungan.txt

/home
ls: cannot access '/tidak_ada': No such file or directory

Important

The order of 2>&1 determines where stderr goes. Write cmd > file 2>&1: stdout and stderr both go into file. But if you write cmd 2>&1 > file, the order is reversed: stderr is directed to the terminal (stdout's current location), and only then is stdout directed to file. The result is that errors still appear on screen, not in the file. The golden rule: 2>&1 must be placed after the stdout redirection you want it to follow.

#Input Redirection: `<`

The < symbol reverses the direction: instead of a command's output being sent to a file, the file's contents are sent into the command as stdin. Many commands accept file input as a direct argument (for example wc -l log.txt), so < can feel redundant — but there are cases where it's mandatory, especially when a command is designed to read stdin:

Input redirection

wc -l < laporan.txt
sort < laporan.txt

The most useful example: feeding input to an interactive script non-interactively. If login.sh asks for a username via stdin, you can supply the answer from the file nama.txt with ./login.sh < nama.txt. This way, a script that should be interactive can run in an automated pipeline.

#The Pipeline `|`: Chaining Commands

Redirection connects commands to files; the pipeline connects commands to commands. The | operator takes the stdout of the command on the left and connects it to the stdin of the command on the right. This is UNIX-style function composition: A | B means "run A, send its output as input to B".

Basic pipeline

ls -l | wc -l

The pipeline ls -l | wc -l counts the number of lines of ls -l output — not counting files directly, but counting printed lines (note that this line count is one more than the number of files because the total is shown in the total header). The key: every command in a pipeline runs simultaneously, and data flows like water between them.

Pipelines become very powerful when combined with filters. A real example from a later episode (which we'll cover in depth in episode 6): finding a specific process:

Pipeline with a filter

ps aux | grep nginx

This reads the entire process list (ps aux), then filters the lines containing the word nginx (grep). The pattern ps aux | grep <name> is one of the most used pipelines by Linux administrators in the world.

#`tee`: Display and Save at Once

There are situations where you want the output to stay visible in the terminal while also being saved to a file. The > operator forces you to choose one. The solution: the tee command — its name is the analogy: like the letter T splitting one flow into two. tee reads stdin, writes it to a file, and forwards it to stdout:

tee: display and save

ls -l | tee hasil-dir.txt

The ls -l output still appears on screen, while a full copy goes into hasil-dir.txt. Add -a for append mode:

`tee` option	Function
(no option)	Overwrites the file
`-a`	Appends to the end of the file
`-i`	Ignores interrupt signals

tee is most useful for logging: when you run a long process and want to observe its progress live while still recording all output for audit.

#Case Study: Building a Report & Handling sudo Permissions

Now let's chain all the concepts in one real scenario: building a disk usage report and separating errors into a log.

Disk report with stream separation

df -h > /var/log/disk-report.txt 2>> /var/log/disk-error.log && echo "Report saved" >> /var/log/disk-report.txt

df -h (disk free) writes a disk usage summary to the report file, while if there are errors (for example because a directory can't be accessed), those errors are appended to a separate error log — not mixed into the clean report. Notice the use of &&: the command after it only runs if df -h succeeds (exit status 0).

For a more complex pipeline case, for example taking the five most memory-hungry processes and saving them:

Top 5 busiest processes

ps aux --sort=-%mem | head -5 | tee top5-proses.txt

Here three concepts meet: the pipeline (ps aux | head), a filter (head), and tee to save the result.

#Common Trap: `sudo` and Redirection Order

This is one of the most frequent mistakes that confuses admins. Try running:

Wrong and right with sudo

sudo echo "new content" > /etc/hosts
sudo tee /etc/hosts > /dev/null <<< "new content"

The first command fails with Permission denied — why? Because the redirection > /etc/hosts is done by your shell, not by sudo. The sequence of events: the shell opens the file /etc/hosts for overwriting before sudo is run, and since your shell runs as a regular user, access is denied. sudo only applies to the echo command, not to the shell's opening of the file.

The solution is to reverse the responsibility: use sudo tee, where tee (running as root) is the one that opens and writes the file:

Solution: sudo tee

echo "new content" | sudo tee /etc/hosts

Tip

The rule of thumb for writing root-required files: sudo tee. The pattern command | sudo tee file or echo "..." | sudo tee -a file for appending. This way, the process that opens the file is tee, running with root privileges — and you can still use a pipeline on the left. This is a pattern professional Linux admins use every day.

#Common Pitfalls

Mistake	Symptom	Solution
Overwriting an important file with `>`	Old file contents lost permanently	Always think about `>>`; back up with `cp` if needed
`cmd 2>&1 > file` (wrong order)	Errors still appear on screen, not in the file	Write `> file 2>&1` — `2>&1` after the stdout redirection
`sudo echo ... > /etc/...`	`Permission denied` despite using sudo	Use `sudo tee` (see callout above)
Forgetting that `>` overwrites a file containing old logs	Audit logs lost, traces erased	Use `>>` for logs, or rotate with `logrotate`
Assuming all terminal output is stdout	Errors mixed into a "clean" report	Understand stderr (fd 2) and separate it with `2>>`
Space in `2 > file`	Bash interprets it as an argument, not redirection	Write `2>` with no space between the fd number and the operator

#Conclusion

In this episode 5, we've unpacked the foundation of interprocess communication in Linux: the three standard streams (stdin fd 0, stdout fd 1, stderr fd 2), redirection operators (>, >>, 2>, &>, 2>&1, <), the | pipeline for chaining commands, and tee for displaying and saving output at the same time.

Key takeaways:

The three standard streams give you granular control over where output goes — and the stdout/stderr separation is a feature, not a bug.
Redirection connects commands to files; pipelines connect commands to commands.
2>&1 must be placed after the stdout redirection you want it to follow.
sudo doesn't apply to shell redirection — use sudo tee when writing root files.
> overwrites without confirmation — always consider >> for logs.

The power of chaining commands only becomes fully felt when combined with text processing commands. In the next episode 6, we'll discuss Text Processing & Filtering Commands — mastering grep, sort, uniq, cut, wc, tr, sed, and awk to filter, transform, and extract data from text, including a real case study: parsing a web server's access log to find the IPs that visit your site most often. Stay motivated, because after this episode you'll be able to "read" log files of any size!