Learn Borg Backup - Performance & Large Dataset

#Introduction

Everything we have built so far assumes backups finish within a reasonable time. But when a dataset grows to the TB scale, even night after night, things that once felt instant start to slow down. Episode 18 turns you from a user into a tuner: understanding the three main performance levers — chunker, compression, and caching — then executing with honest benchmarks.

#The Three Main Performance Levers

#1. Chunker Parameters

The chunker determines the chunk size — and this size affects the dedup vs overhead balance:

• Small chunks → finer dedup (small changes are caught), but heavier metadata and CPU.
• Large chunks → lower overhead, but small changes in large files re-store more.

Borg's default parameters are buzhash,19,23,21,4095: an average of 2 MiB per chunk. For certain datasets you can lower the size for more aggressive dedup:

borg create --chunker-params buzhash,19,23,21,4095 --stats \
  /backup/borg::"{hostname}-{now}" /home

Lowering the numbers (e.g. exponent 20 → 1 MiB average) adds dedup granularity at a CPU cost. Measure first — do not change parameters without data.

#2. Compression Level

From episode 6: lz4 is fast, zstd,10 is balanced. On large datasets, the compression level choice has a big impact on backup duration — and this is where the trade-off is most visible. For hosts whose backups collide with busy hours, a lower compression level makes more sense than an extreme level that slows the pipeline.

#3. Caching: The Files Cache

Borg keeps a files cache locally (in ~/.cache/borg) containing file metadata + chunk lists. The next backup compares metadata (mtime, size, ctime, inode) with the cache: unchanged files are skipped without being re-hashed — a huge saving on datasets that are mostly static.

• Do not delete the cache without a reason: deleting it forces re-hashing all files on the next backup (far slower).
• If you suspect cache corruption (episode 15), clear it and accept one slower backup as the cost.
• For datasets with millions of files, a large cache is normal — give it enough disk space.

#Benchmarking Dedup & Speed

#How to Measure Honestly

Do not just use raw time — measure the components separately:

time borg create --compression zstd,10 --stats \
  /backup/borg::bench /var/lib/data
 
# run again (all data already present) to measure dedup speed
time borg create --compression zstd,10 --stats \
  /backup/borg::bench-2 /var/lib/data

The important readings from these two runs:

• First run: read + compression + encryption + writing new chunks (full throughput).
• Second run: pure dedup speed — if it is nearly instant, the files cache is working.

--stats gives you the numbers: New files, Unchanged files, Deduplicated size. A high Unchanged files ratio shows the cache is effective.

#Tracking the Numbers

Record each backup's duration and size in a log; a duration spike without data changes is a sign of a weakening disk or a lost exclusion (episode 12).

#TB-level Backups

#Special Considerations

• Larger chunker for big, rarely-changing data: reduces the number of chunks and metadata.
• Adjusted compression: for incompressible data (media, compressed archives), lz4 avoids wasted CPU.
• Adequate disk cache: the files cache for millions of files needs space and IO — put it on a fast disk when possible.
• Split the dataset: one create over many large directories can be split per directory so progress is visible and failures are isolated.
• A larger buffer: --buffer-size 1G for high throughput on slow networks or distant disks.

#Throttling: ionice and nice

A backup must run without killing production. Lower the I/O and CPU priority:

ionice -c 3 nice -n 10 \
  borg create --stats /backup/borg::"{hostname}-{now}" /home

• ionice -c 3 → I/O idle class: the disk is only used when no other workload needs it.
• nice -n 10 → CPU priority lower than normal processes.

For cgroup (systemd):

[Service]
Nice=10
IOSchedulingClass=idle
IOWeight=1

#Common Pitfalls

• Changing chunker/compression without benchmarking: every change partially erases the historical dedup effectiveness. Measure first.
• Deleting the files cache: the next backup becomes far slower. Delete only when you suspect cache corruption.
• No throttling in production: a backup eating all I/O slows applications during the day — use ionice/nice or schedule outside business hours.
• Measuring a single run: speed only means something after several runs (dedup + cache play a role). Measure the trend, not a snapshot.

#Closing

• Chunker: chunk size determines the dedup granularity vs overhead — change it with data, not guesses.
• Compression: the level determines backup duration on large datasets.
• The files cache avoids re-hashing unchanged files — do not delete it without a reason.
• Benchmarking: first run = full throughput, second run = dedup speed.
• TB-level datasets: adjust chunker, compression, cache, and split the dataset.
• Throttling: ionice -c 3 and nice / IOSchedulingClass=idle protect production.

In episode 19 we widen the ecosystem: BorgWeb & Tools — a GUI to manage repositories, integration with borgmatic and FUSE, connections to classic monitoring (Nagios/Zabbix) via wrappers, and metric exports to Prometheus.