Learning Restic - Deduplication, Storage & Compression

#Introduction

Episode 5 closed the basic backup→restore cycle. But there is one thing that truly sets restic apart from rsync or tar: it stores deltas, not copies. This is why you can back up a server every day for years without the disk exploding.

This episode dissects restic's efficiency engine: chunk-based deduplication, how compression contributes, and how to read the real numbers.

#Content-Defined Chunking: The Heart of Dedup

We already met this concept in episode 2. Now let's see why this method matters practically.

Restic splits files into pieces (chunks) at positions determined by the data content (rolling hash), not every N bytes. Take a growing log file:

Imagine a log file

[2026-08-13 08:00] request OK
[2026-08-13 08:01] request OK
[2026-08-13 08:02] request FAIL

With fixed-size chunking, every chunk shifts whenever a new line is added — every chunk becomes "new". With content-defined chunking, only the pieces that actually changed are stored again; the rest are recognized as identical and not stored again. This is what keeps daily backups of servers with large logs lightweight.

#Cases That Benefit from Dedup

• The same file in many directories (templates, node_modules, build caches).
• Virtual disks (VM images) that rarely change.
• Growing log files — most of their content is the same as yesterday's backup.
• Directory restructuring (moving files into new folders).

#Reading Dedup Numbers with stats

Restic provides two points of view on size:

restic stats --mode restore-size
restic stats --mode raw-data

• restore-size: the total file size if all snapshots were restored — your "original" data.
• raw-data: the physical blob size actually stored in the repository — after dedup and before compression.

The ratio between them is the strength of your dedup. For example: restore-size of 500 GiB with raw-data of 12 GiB means dedup is working extremely well (a container server with many identical images), while a ratio close to 1:1 indicates mostly unique data — normal for photos/videos.

#Compression: zstd Since 0.14

Besides dedup, restic compresses chunks before encrypting them. Since restic 0.14, compression uses zstd and is enabled by default through the auto mode.

#The --compression Modes

restic backup --compression auto /data   # default: saves space, compresses when worthwhile
restic backup --compression max /data    # highest ratio, more CPU
restic backup --compression off /data    # no compression, allocative data

• auto (default): chunks that clearly cannot be compressed (encrypted files, media) are skipped — this saves CPU on already-compressed data such as JPG/MP4.
• max: forces the best ratio; suited for text, JSON, logs — but eats CPU.
• off: allocative data like VM images can be stored uncompressed for faster restores.

#Dedup + Compression: The Big Picture

The flow for each new chunk: chunking → dedup check (already there? skip) → zstd compression → AES-256 encryption → store as a blob. The combination of all three is why restic is called "backups done right": every byte you don't need to store is bandwidth and money saved.

#Measuring Real Savings

To measure the impact on your repo:

restic stats --mode raw-data
du -sh /backup/restic

Compare the raw-data figure against the total restore-size of all snapshots. The difference is the amount of data you avoided storing thanks to dedup — a great number to report to your team.

#Conclusion

• Content-defined chunking makes chunk boundaries follow the data content → delta dedup.
• Log files, container images, and duplicate files are the biggest dedup wins.
• restic stats --mode restore-size = original size; --mode raw-data = physical size.
• Default zstd compression (--compression auto) saves storage without wasting CPU.
• Dedup avoids duplication; compression shrinks unique data — the two layers stack.
• --compression is a repository parameter: choose once, stay consistent.

In the next episode, episode 7, we map out backends: local, SFTP, S3 & REST server — the differences between each storage target, their repository URLs, credential configuration (AWS keys), and bandwidth throttling with --limit-upload/--limit-download. Time for your backups to leave a single machine.