Learn Linux - System Performance Tuning & Troubleshooting

#Introduction

After episode 23 where we covered system logging, rotation & auditing — how to read a system's condition from /var/log/, keep logs from bloating, and make logs an investigation tool — in this episode we'll use that understanding to face the most nerve-wracking situation for an admin: a server that feels slow, or even down.

Every admin has been there. Monitoring fires an alert at 3 AM: the application refuses connections, or response times spike. As soon as you log in, the terminal feels heavy, and one question pops into your head: "where do I start?" This is where the difference between a panicked admin and an experienced one shows. Not in memorized commands, but in method: the experienced know that troubleshooting is a structured diagnosis process, not guesswork.

Imagine you're a doctor receiving a patient with a high fever. Fever medicine isn't the first answer — the doctor first finds out the cause: viral infection, bacterial, or inflammation? A slow server is the same. "Sluggish" is just a symptom; the cause could be CPU running out of power, RAM full so the system relies on swap, a slow and full disk, or a congested network. Giving the wrong medicine — e.g. adding RAM when the problem is the disk — is like giving fever medicine to a patient with a broken bone.

In this episode we'll build a systematic diagnosis approach. We start with how to read load average correctly, then analyze CPU, memory, and disk I/O with top, vmstat, and iostat. We'll cover the OOM killer that attacks silently, swap management as RAM's "breathing room", and close with a step-by-step CPU spike diagnosis case study along with common mistakes.

#Main Discussion

#The Troubleshooting Approach: Find the Bottleneck, Not the Symptom

The first principle to hold on to: measure before fixing. Never "fix" something not proven to be the cause. A simple method you can repeat every time:

1. Measure — collect data from all components (CPU, memory, disk, network) over the same time period.
2. Identify the bottleneck — which component is closest to its limit (100% utilization or long queues)?
3. Hypothesize — state explicitly what you believe is the cause.
4. Test — make one change, measure again, and compare.
5. Document — record what was done and the results, so it can be reused in the next incident.

There's a very fitting metaphor: a queue at the supermarket checkout. A healthy system is like a store with enough cashiers — short queues, everyone moves. An overloaded system is like a store on a holiday with one cashier: people queue up long, and the cashier works as hard as possible (CPU at 100%) but still can't keep up. The question you must answer isn't "why are people queuing?", but "which component is that single cashier?" — that's the bottleneck.

#Load Average & CPU: Reading uptime Correctly

The first command usually run is uptime, and it's also the most misread:

 09:41:02 up 12 days,  3:22,  1 user,  load average: 3.52, 2.90, 2.61

The three numbers at the end are the load average for the last 1, 5, and 15 minutes. This number represents the count of processes currently running plus those waiting for CPU time (runnable + uninterruptible) — not a CPU usage percentage. One thing you must understand: load average can't be interpreted without knowing the number of CPU cores.

The most common mistake here: seeing load average: 3.52 and panicking that the CPU is overloaded. But if the server has 8 cores, a load of 3.52 means the system is using less than half its capacity. The rule of thumb: divide the load average by the core count. Below 1.0 per core = still healthy; around 1.0 per core = full but normal; far above 1.0 per core and sustained = queues piling up, CPU is the bottleneck.

nproc
uptime

Also pay attention to the trend of the three numbers, not a single value. 1.5, 3.0, 4.5 means the load is increasing — the problem is getting worse. Conversely 4.5, 3.0, 1.5 means the load is decreasing — the system may have passed its peak. The 15-minute number is a "historical snapshot", not the current condition.

To see which processes are eating the CPU, top is the essential tool:

top -o %CPU

The %CPU column shows the percentage of a single core. A process showing 100 uses one full core; on a 4-core server, 400 means using the entire CPU. From here you can quickly see who the "culprit" is — a wild database query, uncontrolled application workers, or an out-of-control script.

#Memory: RAM, Swap, and the OOM Killer

After CPU, the second component is memory. The fastest command is free -h:

              total        used        free      shared  buff/cache   available
Mem:           15Gi       7.1Gi       1.2Gi       456Mi       6.7Gi       7.3Gi
Swap:         2.0Gi       300Mi       1.7Gi

Two numbers to watch: available (not free!) and the Swap column. The free column is misleading because Linux uses free memory for page cache — that's actually healthy. The available column estimates the memory applications can genuinely use without triggering swap. If available runs thin (e.g. below 10% of the total) while swap is in use, the system is short on memory.

When RAM runs out, Linux does two things. First, it uses swap — moving rarely-used memory pages to disk. Swap is like extra storage space in a warehouse: much slower than the workbench (RAM), but it prevents the system from collapsing immediately. Second, when swap is also exhausted, the kernel calls the OOM killer — a guard that "chooses" a process to kill to save the system.

The OOM killer is the failure cause that most often makes applications vanish without a trace: running normally one minute, gone the next, and only journalctl tells the truth:

sudo dmesg -T | grep -iE "out of memory|killed process"
journalctl -k -b | grep -i "oom"

A line like Out of memory: Killed process 1234 (mysqld) is indisputable evidence: memory ran out, and the kernel made a sad decision. If you see this pattern, the answer isn't blindly adding swap, but finding the cause of the memory leak or adding RAM — which we'll discuss in the swap section.

#Disk I/O: iostat and the iowait Story

The third most-forgotten component is disk I/O. A server can have abundant CPU and RAM yet still be slow if the disk is the bottleneck. To see it, use iostat from the sysstat package:

iostat -xz 1

Interpreting the options above — -x for extended statistics, -z to show only active devices, and 1 for a 1-second interval. The two most important columns:

• %util — the percentage of time the device is busy serving requests. Consistently above 60–70% indicates the disk is overloaded; near 100% means I/O queues are piling up.
• await — the average time (in milliseconds) a request takes to complete. Small numbers mean fast; bloated numbers mean a slow disk or long queues.

The relationship between the two matters: if %util is high and await is high, the disk is truly busy — the solution could be moving the workload or upgrading the disk. But if %util is high while await is low, that means many small requests are served quickly — the problem might be too many I/O threads, not a slow disk.

You'll also often see the term iowait in top or uptime output — the percentage of time the CPU waits for I/O operations to finish. High iowait means the CPU is willing to be idle because it's waiting for the disk — and this is a strong clue that the disk is the bottleneck, not the CPU. Often an old HDD on a server "seems healthy" when it's actually the never-noticed source of slowness, because all CPU metrics look normal.

#Network: The Fourth Component Not to Forget

When CPU, memory, and disk all look healthy but the app is still slow, check the network. Common scenarios: LAN bandwidth maxed out by uncontrolled backups or data syncs, or TCP connections piling up without finishing (timeouts) so the application's connection pool is exhausted.

To check open ports and connection states, ss is the modern replacement for netstat:

ss -tunap

If you see thousands of piling SYN_SENT or TIME_WAIT connections, that's an indication of a network-side problem — not the application. To see per-interface network throughput, sar -n DEV (from the same sysstat package as iostat) gives a historical summary:

sar -n DEV

The principle stays the same as with other components: find the component closest to its limit. A 100% utilized network is clearly the bottleneck — before blaming the slow app, make sure the bandwidth isn't being "eaten" by another process.

#Swap Management: Adding Breathing Room Without Adding RAM

When RAM is limited and processes keep requesting memory, one legitimate short-term solution is adding swap. Swap gives the system breathing room: rarely-used processes can be "parked" in swap so RAM frees up for active processes. Remember, swap is an emergency lifesaver, not a RAM replacement — but adding a swap file can save a server from OOM in minutes, without waiting for hardware additions.

The steps are simple. First, check the current swap condition with swapon --show and free -h:

swapon --show
free -h

Second, create a 2 GiB swap file, format it as a swap area, then activate it:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Note the chmod 600: a swap file contains memory data that could be sensitive, so don't let other users read it. After swapon, verify with swapon --show — the swap file appears as an active device.

fallocate works well on modern filesystems like ext4 and XFS. On filesystems that don't support it (or if you have doubts), use dd as an alternative:

sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress

Third, so swap survives a reboot, add an entry to /etc/fstab:

/swapfile none swap sw 0 0

Finally, set vm.swappiness — how aggressively the kernel uses swap. The default value on many distros is 60, meaning the kernel starts moving pages to swap relatively quickly. For a server with sufficient RAM, a value of 10 is much better — the kernel is more patient using RAM first before touching swap:

sysctl vm.swappiness
# Temporary change (lost at reboot)
sudo sysctl -w vm.swappiness=10
# Persistent: write to /etc/sysctl.d/99-swap.conf
echo "vm.swappiness = 10" | sudo tee /etc/sysctl.d/99-swap.conf

The two lines marked -- show the approach that only works temporarily — changes with sysctl -w disappear at reboot. The two lines marked ++ are the correct way: writing to /etc/sysctl.d/ makes the parameter persist. This /etc/sysctl.d/ file pattern we already know from episode 16 — numbered files execute in order.

#Practice: Diagnosing a CPU Spike Scenario Step by Step

Time to combine all the tools into one diagnosis flow. Take the scenario: monitoring fires an alert, "CPU utilization 100% since 10 minutes ago, the application is responding very slowly."

Step 1 — Get the big picture. Log in and immediately run uptime and nproc:

uptime
nproc

Suppose the result is load average: 15.20, 12.80, 9.10 on a 4-core machine. Divide by the cores: 15.2 / 4 ≈ 3.8 per core — far above 1.0, and the numbers rise from 9.1 → 12.8 → 15.2. Initial conclusion: the load is rising sharply and the CPU is the bottleneck. A reasonable hypothesis: some process is consuming a lot of CPU.

Step 2 — Find the culprit. Run top -o %CPU and look at the top five rows:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 1234 mysql     20   0 2.402g 1.1g  100m S  380.0  7.3  182:32.91 mysqld
 1010 www-data  20   0  312m  34m   18m S    2.0  0.2    2:10.11 php-fpm
  899 www-data  20   0  312m  32m   18m S    1.0  0.2    1:55.77 php-fpm

mysqld with %CPU 380 on a 4-core machine means the database is using almost the entire CPU. The large 182:32 time shows this isn't a new condition — the SQL machine is consistently consuming CPU.

Step 3 — Check whether this is a hidden memory or I/O problem. Run vmstat 1 for a few seconds:

vmstat 1 5

If the si/so columns are heavily active, the system is memory thrashing — the real problem is actually RAM, not CPU. If not, continue to the next step.

Step 4 — Verify with iostat. Check whether the disk is also burdened (often the cause of high database CPU is queries forcing the disk to work hard):

iostat -xz 1

If %util is low and await is reasonable, the disk isn't the cause. The conclusion narrows: the database itself is eating CPU.

Step 5 — Find the cause on the application side. Log into MySQL and see the running processes:

SHOW FULL PROCESSLIST;

The result shows many SELECT ... FROM orders WHERE ... queries with large Time values and no indexes. The root cause is identified: queries without indexes making the database repeatedly do full table scans. The solution is on the database side (adding indexes, increasing the query cache), not a hardware upgrade.

This five-step pattern — big picture, find the culprit, check memory, check disk, then go into the application — is a recipe you can reuse again and again. The order matters: from the fastest and cheapest (uptime) toward the most specific (database queries), so you never waste time in the wrong place.

#Common Mistakes (Common Pitfalls)

1. Misreading load average on multicore. Assuming a load of 3 on a 2-core machine is the same as a load of 3 on an 8-core machine. Remember: always divide by nproc. This is the most expensive interpretation mistake in the tuning world.

2. Swap on a slow disk. Putting swap on an old HDD or the same disk as an intensive workload makes the system "bleed" slowly. Check with iostat — if si/so are active and swap %util is high, consider moving swap or adding RAM.

3. Ignoring iowait. Seeing the CPU "idle" in top (e.g. 80% idle) then concluding the system is healthy, when 40% iowait signals the CPU is waiting on the disk. Idle isn't healthy if the disk is becoming the jam.

4. Assuming the OOM killer is an application bug. A suddenly missing process is often blamed on the app, when the actual cause is dmesg saying "Out of memory: Killed process". Always check dmesg/journal before blaming code.

5. Panicking without data. Adding RAM or CPU based only on intuition, not measurement. First measure which component is truly the bottleneck — most often the answer is actually configuration (queries, cache, indexes) that can be fixed for free.

#Conclusion

In this episode 24, you've built a systematic approach to handling slow or down servers. We started from the principle measure before fixing and bottleneck identification, then dissected every component: load average that must always be read against the core count, memory with free -h and the available column, the OOM killer hiding in dmesg, disk I/O with iostat -xz 1 and the %util/await/iowait story, and network with ss. We also learned to manage swap with fallocate, mkswap, swapon, and tune vm.swappiness, then closed with the five-step CPU spike diagnosis case study.

Key points to take with you:

• Troubleshooting is a structured diagnosis process, not guesswork: measure, identify the bottleneck, hypothesize, test, document.
• Load average must always be compared to the core count (nproc); the three-number trend matters more than a single value.
• Read the available column in free -h, not free — and always check dmesg for OOM killer traces.
• iostat -xz 1 is the window into the disk: high %util + high await = busy disk; high iowait = CPU waiting on the disk.
• Swap is an emergency lifesaver, not a RAM replacement; put it on a separate disk and control it with vm.swappiness.

Now you know how to keep a server alive and fast. In the next episode 25, we'll flip the focus from making it run to keeping it safe from attackers: Linux Hardening & Security Best Practices Production — the principle of shrinking the attack surface, fail2ban to protect SSH, an introduction to SELinux and AppArmor, security auditing with Lynis, and a hardening checklist for a new Ubuntu server. Stay sharp!