Learning Restic - Performance Tuning & Large Data

#Introduction

So far you have practiced with small datasets. Episode 18 is about scale: when data reaches the TB level, or 50 hosts back up to one server, the rules change. Backup is no longer about "does it work", but "how long and how cheap".

The key is not random tinkering, but understanding where the bottleneck is: CPU, disk, or network — then adjusting the right parameters.

#Measure First: Benchmark the Dedup Rate

Before tuning, measure the baseline. The dedup rate is the most informative number:

restic stats --mode raw-data
restic stats --mode restore-size

The comparison of the two (episode 6) determines the strategy: high-dedup data (containers, logs, code) benefits hugely from chunking; unique data (photos/videos) needs more bandwidth than CPU.

Also measure backup time per data volume:

restic backup /data --json | jq '.total_duration, .data_added_packed'

Make these numbers your baseline; every tuning is measured against it, not by feel.

#--read-concurrency: Read Parallelism

Restore and check --read-data open many blobs at once. --read-concurrency controls the number of parallel reads:

restic restore latest --target /tmp/r --read-concurrency 32

• Local disk (SSD): raise to 16–32 — SSDs can serve many parallel I/Os.
• HDD: fewer (4–8) — many connections just cause head thrashing.
• Network/object storage: match the backend connection; the s3.connections option (episode 16) is related.

#Network Buffers & Latency

For remote backends (S3, rest-server), latency matters more than bandwidth. Every request uses one connection; with small buffers, throughput is limited by round-trips. Common S3 tuning:

restic -o s3.connections=10 \
  -o s3.part-size=64 \
  -o s3.upload-concurrency=8 \
  backup /data

• s3.connections: number of parallel connections.
• s3.part-size (MiB): multipart part size — larger parts = fewer requests.
• s3.upload-concurrency: parallel uploads per file.

Check your backend's docs — every backend has its own -o options you can explore with restic help.

#Caching: --cache-dir

Restic keeps a local index and blob cache to avoid re-downloading:

restic --cache-dir /var/cache/restic backup /data

• The index cache speeds up opening the repository — important for large repos.
• Put the cache on fast I/O (SSD); avoid the same disk as the raw data when possible.
• The cache size grows; manage it with periodic cleanup.

#TB-Level Datasets

Practices for data > 1 TB:

• Split backups per top-level directory — separate snapshots, per-part check/prune, easier resumes.
• Schedule per part — not one giant midnight backup, but several backups with different schedules and tags.
• --read-data-subset (episode 11) for incremental verification.
• Use restic ls/find before restore to avoid wrong-path selective restores.
• Watch the prune time — for TB-level repos, schedule prune weekly in quiet hours, not after every backup.

#Many Hosts to One Server

When many hosts back up to one rest-server/S3, management rules:

• Stagger the schedules: don't let 50 hosts back up at once — randomize the minute ($((RANDOM % 60))) in cron.
• Bandwidth limit per host: --limit-upload (episode 7) so small hosts don't starve.
• Repo per host: one repository per host (episode 15) so one host's prune does not lock others.
• Monitor per host (episode 20): know which host failed without waiting for complaints.

M=$((RANDOM % 60))
echo "$M 2 * * * root /usr/local/bin/backup-restic.sh" > /etc/cron.d/restic-backup

#Conclusion

• Measure the baseline first: dedup rate + throughput — tuning without measurement is guesswork.
• --read-concurrency: match the disk type (high on SSD, low on HDD).
• S3 tuning: s3.connections, s3.part-size, s3.upload-concurrency.
• --cache-dir on SSD separates fast index from large data.
• TB datasets: split backups, --read-data-subset, a separate prune schedule.
• Many hosts: stagger schedules, limit bandwidth, repo per host, monitoring.

In the next episode, episode 19, we explore the ecosystem: rest-server, SDK & wrappers — rest-server as a backup-server product, wrappers like resticprofile and restic-compose, Grafana dashboards, Prometheus metrics, and integrating backups into CI/CD pipelines.