Learn pgBackRest - Parallelism & Compression

#Introduction

For a small database, backups run "fast enough" no matter what you do. But when data grows to hundreds of gigabytes or terabytes, backup and restore speed becomes an architectural decision. In episode 7 we open pgBackRest's two main performance levers: parallelism (using many processes to work simultaneously) and compression (shrinking the size in the repository).

These two features aren't just "nice options" — for large databases, they're the difference between backups taking hours and taking minutes. In episode 19 we'll measure them with a real benchmark.

#Parallelism: Many Hands, Less Time

#The Concept of Processes

By default pgBackRest works with one process (process-max=1): one file is copied, compressed, and written to the repository at a time. With a larger process-max, pgBackRest splits the work into many worker processes — each handling a different file in parallel.

/etc/pgbackrest.conf

[global]
process-max = 4

This value can be overridden per command:

Backup with 8 processes

sudo -u postgres pgbackrest --stanza=main --process-max=8 backup --type=diff

Note the terminology: process-max limits the total number of processes. Within it, there's the concept of process per file — one large file (for example a 50 GB relation) can't be split across several processes at once on versions before 2.47; each process handles a whole file. So parallelism is most effective when the database consists of many files.

#What's the Right Value?

There's no magic number. Rules of thumb:

Start with 2-4 for small servers, increase gradually.
Look at the number of CPU cores: a process-max far above the cores doesn't speed things up — it only adds I/O contention.
Watch disk/network bandwidth: if copying to NFS or object storage, the network may be the bottleneck, not the CPU.

Check core count before tuning

nproc

Tip

Parallelism speeds things up, but uses CPU and I/O — on a server serving production load, raise process-max during the backup window (at night), not during peak hours. Repeating values in config vs per-command gives you this flexibility.

#Compression: Shrinking the Repository Size

#Algorithms and Default

pgBackRest supports several compression algorithms: zst (zstd), lz4, gz (gzip), bz2, and none. Since v2.50, the default is zst (zstd) — an excellent balance of compression ratio and speed.

/etc/pgbackrest.conf

[global]
compress-type = zst
compress-level = 3

compress-type = zst — use zstd (default, but being explicit never hurts).
compress-level = 3 — compression level; the zstd default is 3. The higher the level, the smaller the size, the slower the compression.

A quick comparison:

Type	Speed	Ratio	Notes
`lz4`	Very fast	Low	Light compression, fast restore
`zst`	Fast	Good	Default, best balance
`gz`	Medium	Good	Wide compatibility
`bz2`	Slow	Highest	Saves space, expensive on CPU

Note

Restore also decompresses. A very small backup (high level) speeds up network transfer during restore, but slows down decompression on the CPU side. Measure this trade-off with a benchmark (episode 19), not by feel.

#Delta Backup: Only the Changed Parts

Another important feature that works alongside compression and parallelism is delta backup (not to be confused with differential). When the next backup is run, pgBackRest compares file checksums with the previous backup and only stores the blocks/files that changed — plus it leverages deduplication between backups (block incremental functionality since 2.47).

In practice, these three features complement each other:

Parallelism speeds up the process.
Compression shrinks what's written.
Delta/dedupe avoids rewriting data already in the repository.

The result for large databases: the second and subsequent backups can be much faster and smaller than the first full, because only the changes are processed.

#Exercise: Measuring the Effect

Compare duration and size directly:

Compare process-max 1 vs 4

sudo -u postgres pgbackrest --stanza=main --process-max=1 backup --type=diff
sudo -u postgres pgbackrest --stanza=main --process-max=4 backup --type=diff
sudo -u postgres pgbackrest info

Notice the backup size difference in info: with dedupe, the second differential backup should be much smaller than the first. This is real evidence that pgBackRest doesn't recopy unchanged data.

Warning

Don't just raise process-max without measuring. On machines with few cores or slow storage, high parallelism actually slows things down because CPU/disk compete with each other. Always verify with pgbackrest info and note the backup duration (shown in the command output) before and after tuning.

#Conclusion

Key takeaways:

process-max splits backup/restore into many parallel processes; tune it to cores and bandwidth.
compress-type = zst is the modern default (zstd); the level can be raised for a higher ratio.
lz4 for extreme speed, bz2 for extreme space savings — choose based on the trade-off.
Delta backup + block deduplication only writes data that changed between backups.
Always measure the effect with pgbackrest info and actual durations.

In the next episode we'll reverse direction: restore — pgbackrest restore, the --type=immediate type for the latest state, --type=time for PITR, --type=lsn, --type=name, and --delta for syncing an existing cluster. This is the most important moment in the entire backup system: proving that your backups can actually be recovered!