Learn Rsync - Performance Tuning for Large Datasets

#Introduction

In previous episodes, rsync was used for daily backups that finish quickly. Episode 18 covers a different situation: large datasets — tens of GB to TB — where every percent of speed means hours faster completion and more reasonable bandwidth usage.

Why does this matter? Because a slow rsync isn't just "waiting longer" — it holds locks, uses bandwidth, and blocks other workloads. Correct tuning turns a 40-hour transfer into a 6-hour one. We dissect three areas: transfer strategy, parallelism, and benchmarking.

#--whole-file vs Delta

Rsync chooses between two transfer strategies:

• Delta-transfer (default) — splits files into blocks, compares them, sends only the changed parts. Its goal is minimizing data sent.
• --whole-file — sends whole files, no splitting. Its goal is minimizing CPU and latency.

rsync -avh --whole-file /media/ /backup/media/

#Parallelism: Multiple rsyncs per Shard

Rsync is single-threaded per process: one process uses one connection and one pipeline. For TB datasets, that's the bottleneck — the NIC and disk can be far faster than a single rsync stream.

The solution: split the dataset into shards and run several rsyncs in parallel:

for d in a b c d; do
  rsync -avh --bwlimit=2048 /data/$d/ root@backup:/backup/$d/ &
done
wait

The common pattern: one rsync per top-level directory, or per file range (e.g. split a file list into N parts with split). A reasonable parallel count: 2-4 to start — not 20, because the destination disk can get overwhelmed and end up slower. Combine with --bwlimit to keep total bandwidth under control.

#Benchmarking NIC and Disk

Don't guess the bottleneck — measure it. Two essential tools:

iperf3 -c backup-host                    # maximum TCP throughput
dd if=/dev/zero of=/tmp/test bs=1M count=4096 conv=fdatasync 2>&1 | tail -1
hdparm -t /dev/sdb                       # disk read speed (HDD/SSD)

The numbers from iperf3 and dd/hdparm tell you the physical limits. Then compare them with rsync's actual throughput:

rsync -avh --stats /large/ backup-host:/backup/

If rsync is far below the physical limits, tuning still has a chance: try --whole-file, parallelism, or socket buffers (episode 12). If it's already near the limit, the problem isn't rsync — it's already optimal.

#Case Study: MongoDB/Postgres Data Dir

A database data directory (e.g. /var/lib/postgresql or /var/lib/mongodb) must not be rsynced directly while the database is running — files can change mid-read and the result is corrupt. Two correct patterns:

1. Consistent snapshot — use a filesystem snapshot mechanism (LVM lvcreate --snapshot, btrfs/zfs snapshots), then rsync from the snapshot.
2. Dump + transfer — pg_dump/mongodump, then rsync the dump file (the pattern from episode 11).

For replica seeding (building a secondary in MongoDB/Postgres), the commonly used approach:

rsync -avh --bwlimit=2048 --whole-file \
  /snap/lvm/data/ postgres@replica:/var/lib/postgresql/16/main/

Here rsync only moves already-consistent files (from a snapshot), throttled so it doesn't disturb production.

#Case Study: Media Library

A media library (video, image) is the classic large-dataset case with files that almost never change after being written. The consequences:

• The first backup is expensive, subsequent ones are cheap — delta-transfer works at its best.
• -z compression doesn't help (video/image files are already compressed) — turn it off.
• New files are added more often than old ones are changed — make sure the *.tmp etc. filters remain (episode 6).

rsync -avh --no-compress --bwlimit=2048 /media/ backup:/media/

--no-compress explicitly turns off -z — saving CPU that brings no benefit for media files.

#Case Study: TB-Level Datasets

For tens-of-TB datasets, the complete strategy:

1. Split the dataset into logical shards (per file cluster or range).
2. Parallelize 2-4 rsyncs, each with --bwlimit so the total stays controlled.
3. Schedule in phases: an initial phase during quiet production hours, a short delta final phase (the migration pattern from episode 11).
4. Monitor throughput per shard; stop a problematic one without disturbing the others.
5. Verify with -c spot-checks (episode 16), not the whole dataset — for TB-level, full checksums are too expensive.

One principle to always remember at this scale: measure, plan, then run — not "fire one big rsync and pray".

#Closing

In this episode you've mastered performance tuning for large datasets.

Key takeaways:

• --whole-file for very fast networks; delta for slow networks and rarely changing files.
• Parallelize with multiple rsyncs per shard; start with 2-4 processes, not dozens.
• Benchmark first with iperf3/dd/hdparm, then compare rsync throughput.
• Databases: don't rsync a live data dir — use snapshots or dumps.
• Media library: turn off -z; TB-level: shard + parallel + throttle + spot-check.

In episode 19 we broaden our view: alternatives & the backup ecosystem — comparing rsync (synchronization) with rclone (cloud object storage), restic/borg (dedup + encryption), and tar/dd (image-level), plus the best combinations. See you in episode 19!