Learn Linux Filesystem - Performance Tuning

#Introduction

Two servers with identical hardware can show very different storage performance — entirely because of filesystem and kernel configuration. Episode 16 covers the tunable layers: mount options, allocation sizes, I/O schedulers, and page cache.

But tuning without measurement is just guessing. That's why half of this episode covers benchmarking: fio as the industry standard, dd for quick measurements, and iostat and vmstat for reading system behavior while a workload runs.

The golden rule: change one variable, measure, record, then change the next. Tuning many things at once means you won't know which change worked.

#Mount Options and Recordsize

#noatime, discard, and compress

The three options that have the most impact:

• noatime: removes atime updates — reducing writes on every read.
• discard: sends TRIM commands to SSDs — maintaining long-term write performance.
• compress: transparent compression — reducing the physical data written and read.

sudo mount -o noatime,discard /dev/loop0 /mnt/lab

discard has two modes: discard (runs when a file is deleted, slight overhead) and nodiscard + scheduled trim (more predictable). On SSDs, pick one — not both.

#Recordsize and Block Size

The allocation size determines how files are mapped to blocks:

• ext4 block size is set at mkfs with -b (4096 default).
• XFS uses allocation groups and can be set at mkfs.
• ZFS recordsize is set per dataset.
• btrfs operates with 1GB block groups split internally.

Match it to the I/O pattern: 8K for databases, 1M for large media. Measuring first is far better than guessing:

stat -f -c '%s' /mnt/lab
zfs get recordsize labpool/data

#I/O Schedulers

#mq-deadline and BFQ

The Linux kernel uses I/O schedulers that order requests to devices. Common choices:

• mq-deadline: balances latency with a deadline per request — the default for SSDs.
• bfq: prioritizes fairness between processes — good for desktops and interactive systems.
• none (or noop): passes everything straight to the device — optimal for NVMe that already has its own queue.

View and change the scheduler per device:

cat /sys/block/sda/queue/scheduler
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

Make the change persistent with the kernel parameter elevator=mq-deadline in /etc/default/grub or a udev rule file.

#Queue Depth and Sectors

Three additional rarely seen knobs:

• queue/nr_requests: the number of queued requests — increase for HDDs.
• queue/max_sectors_kb: the maximum I/O size per request.
• queue/rotational: 1 for HDD, 0 for SSD.

Check their values:

cat /sys/block/sda/queue/nr_requests
cat /sys/block/sda/queue/max_sectors_kb

#Page Cache and Writeback Tuning

#vm.dirty_ratio and vm.dirty_background_ratio

The kernel holds writes in the page cache before sending them to disk. Two main parameters:

• vm.dirty_background_ratio: the percentage of memory that triggers background writeback (default 10).
• vm.dirty_ratio: the percentage that forces writing processes to wait (default 20).

sysctl vm.dirty_background_ratio vm.dirty_ratio

Consideration: high values accumulate writes (good for throughput), but make data unsafe on power loss. For workloads relying on fsync (databases), leave the defaults.

#vfs_cache_pressure

vm.vfs_cache_pressure controls how aggressively the kernel reclaims inode/dentry caches (default 100). A lower value keeps metadata caches alive longer:

sudo sysctl -w vm.vfs_cache_pressure=50

This tuning helps servers with many frequently accessed small files.

#Benchmarking with fio

#Measurement Profiles

Before measuring, decide the workload profile you want to test:

• Throughput: large sequential reads/writes (1M).
• Latency: single small operations (4K, queue depth 1).
• IOPS: parallel small operations (4K, queue depth 32).

Run an fio benchmark for each profile:

fio --name=randread --ioengine=libaio --rw=randread \
  --bs=4k --size=1G --numjobs=4 --iodepth=32 \
  --direct=1 --group_reporting

--direct=1 bypasses the page cache so you measure the real device. Change --rw to randwrite, read, or write for other profiles.

#Reading the Results

The fio output shows IOPS, BW (bandwidth), and clat (per-operation latency). Compare results between configurations carefully:

• Change one variable per trial.
• Measure repeatedly (3-5 times) and take the median.
• Record the configuration along with its results in a table.

For quick measurements, dd is still useful:

dd if=/dev/zero of=/mnt/lab/test bs=1M count=1024 oflag=direct

#Reading iostat and vmstat

#iostat

iostat shows utilization and throughput per device:

iostat -x -m 2

Important columns: %util (percentage of time busy), await (average service time), r/s and w/s (read/write IOPS), and rrqm/s (merged requests).

#vmstat

vmstat shows system memory and I/O activity:

vmstat 2

Watch the bi and bo columns (blocks in/out per second), si/so (swap), and wa (percentage of CPU waiting on I/O). High wa is a sign of a storage bottleneck.

#Conclusion

Performance tuning is a never-ending measure-change-measure cycle. By mastering mount options, recordsize, I/O schedulers, and the page cache, you can steer performance in the direction you want. With fio, iostat, and vmstat, you can prove that your tuning actually works.

Key takeaways:

• Change one variable, measure, then compare — don't guess.
• noatime, discard, and compress are three high-impact options.
• I/O schedulers: mq-deadline for servers, bfq for interactive, none for NVMe.
• vm.dirty_ratio controls the balance between throughput and data safety.
• fio --direct=1 measures the real device without the page cache.
• iostat and vmstat are your radar when a bottleneck appears.

In the next episode, episode 17, we cover monitoring and integrity — df, du, iostat, btrfs device stats, zpool status/iostat, SMART with smartctl, Prometheus integration, and scrub schedules and alerting. You'll build comprehensive filesystem supervision.