Learn Proxmox Backup Server - Deduplication & Garbage Collection

#Introduction

Restore was tested in episode 5 — now we talk about cost. Unmanaged backups keep growing until the disk is full. PBS solves two problems at once: deduplication ensures data is not stored repeatedly, and garbage collection cleans up leftover chunks that are no longer needed. Episode 6 is the heart of PBS efficiency.

Imagine dedup like a hotel with a central kitchen: all guests (VMs) order the same eggs, but the hotel only keeps one portion of eggs in the kitchen (unique chunks), then records who ate them. When a guest leaves, eggs used only by that guest remain in the kitchen until the staff (GC) confirms nobody needs them anymore.

#Deduplication: Avoiding Duplicate Data

#Content-Defined Chunking and Cross-Backup Dedup

As discussed in episode 2, PBS splits data into chunks based on content-defined chunking (CDC). The direct consequence: one unique chunk is stored only once, across the entire datastore. This means:

• Across backups: today's and yesterday's snapshots share unchanged data chunks — the next backup only sends the delta.
• Across VMs: ten identical Debian VMs store the OS chunks only once.
• Across hosts: file backups from hosts running the same software get deduplicated too.

Dedup happens automatically, transparently, and is calculated in real time by PBS. No special configuration — you just enjoy the efficiency.

#Reading the Dedup Metrics in a Datastore

The PBS web UI shows key statistics per datastore: Size (physical capacity used), Original (the actual data size before dedup/compress), Dedup (unique chunk size after dedup, before compress), and Compressed (the real size after compression). From these, two ratios emerge:

• Dedup ratio = Original / Dedup — how much duplicate storage was avoided.
• Compression ratio = Dedup / Compressed — zstd efficiency.

proxmox-backup-manager datastore list --output-format json

#Garbage Collection

#Why GC Is Needed

When retention (episode 8) deletes a snapshot, PBS does not immediately delete its chunks — because the same chunk may be used by another snapshot. Deleting without checking would corrupt still-living snapshots. Garbage collection answers this: it counts every chunk's references, then deletes those no longer referenced by any snapshot.

The flow has two stages:

1. Prune: delete snapshots based on the retention policy (keep-last, keep-daily, etc.).
2. Garbage collection: after pruning, find and delete orphaned chunks.

This order is mandatory: GC without prune is pointless (all chunks are still referenced), and prune without GC makes the disk balloon (old snapshot chunks stay stored).

#Running GC

proxmox-backup-manager datastore garbage-collection store1

While running, PBS enters the mark phase (marking all referenced chunks — can take a long time for large datastores) then the sweep phase (deleting unmarked ones). The final log shows the number of freed chunks:

Example GC output

TASK OK — GC finished.
Marked 12345 chunks, removed 678 chunks, freed 2.1 GiB

#Scheduling GC

Do not keep running GC manually — schedule it. Ideally during quiet hours and after the prune job finishes. In episode 3 we already created a weekly GC (sun 02:00). Create an explicit job via the CLI:

proxmox-backup-manager gc job create \
  --datastore store1 --schedule "sun 02:00"

proxmox-backup-manager gc job list

#Practice: Prune + GC Together

A healthy production order:

1. Daily backups run according to schedule.
2. A prune job cleans up old snapshots (e.g. daily at 03:00).
3. A GC job runs after it (e.g. weekly Sunday 02:00) to free space.

The result: the datastore grows according to retention, not without limit. Check disk health periodically with df -h and monitor the datastore metrics in the web UI.

#Closing

Key takeaways:

• Content-defined chunking produces dedup across backups, VMs, even hosts.
• Dedup is automatic; read the Size/Original/Dedup/Compressed metrics in the datastore web UI.
• Dedup ratio = Original/Dedup; compression ratio = Dedup/Compressed.
• Prune deletes snapshots according to retention; GC deletes chunks no longer referenced.
• Mandatory order: prune first, GC after.
• Schedule GC in quiet hours (proxmox-backup-manager gc job create) and do not overlap it with backups/sync.

In the next episode, episode 7, we will secure the data: client-side encryption — generating a base64 key on the client side, setting the encryption-key on backup tasks, verifying that the server cannot read the data, and proper key management practices, because without the key, restore is impossible!