Learn pgBackRest - Troubleshooting & Debug

#Introduction

In episode 14 we installed alarms — now it's time to learn what happens when the alarm rings. pgBackRest is a very communicative tool: it writes detailed logs to files and can display full debug output on the console. The real problem isn't "an error appeared," but reading the error correctly — and in episode 15 we train that with the four most common cases in the field.

The mindset to hold: every pgBackRest error names the file and process involved. Good debugging is following that trail, not guessing.

#Raising the Log Level

#Console vs File Logs

pgBackRest has two log paths: console (stdout/stderr when a command runs) and file (in /var/log/pgbackrest/). Both can have their levels set independently:

sudo -u postgres pgbackrest --stanza=main --log-level-console=debug backup --type=diff

sudo -u postgres pgbackrest --stanza=main --log-level-file=debug backup --type=diff

Available levels: off, error, warn, info, detail, debug, trace. For troubleshooting, debug is usually enough; trace is very verbose and only for extreme cases.

#Reading Logs in /var/log/pgbackrest/

Log files are stored per process with a name containing the timestamp and PID:

ls -lt /var/log/pgbackrest/
sudo tail -50 /var/log/pgbackrest/pgbackrest-20260813-120000.log

The log line format: level, process (P00 = master, P01+ = worker), and message. Note the following patterns on error:

• P00 ... ERROR: [046]: ... — error with a code; [046] etc. are categories documented at pgbackrest.org.
• P01 ... ERROR: ... while processing file ... — indicates the specific file that failed.
• DETAIL and HINT lines — explanation and suggested fix.

#Common Case 1: Failed archive-push

#Symptoms

WAL never reaches the repository; pg_stat_archiver.failed_count rises; check fails at the archive-push part.

#Structured Diagnosis

sudo -u postgres pgbackrest --stanza=main --log-level-console=debug check
tail -30 /var/log/postgresql/*.log | grep -i archive

The most common errors and their fixes:

• archive_command has the wrong binary path: make sure pgbackrest is in the postgres user's PATH, or write an absolute path.
• archive_command isn't enabled: archive_mode=on needs a restart (episode 6).
• The stanza hasn't been created: run stanza-create first.

If the error mentions a code and a WAL file, test archive-push manually with a test file:

sudo -u postgres pgbackrest --stanza=main archive-push /var/lib/postgresql/16/main/pg_wal/0000000100000000000000FF

#Common Case 2: Permission Denied

#Symptoms

permission denied appears in the logs — usually when a backup is first run as the wrong user, or after a directory has been recreated.

#Diagnosis

ls -ld /var/lib/pgbackrest /var/lib/pgbackrest/backup /var/log/pgbackrest
ls -ld /var/lib/postgresql/16/main

The fix:

sudo chown -R postgres:postgres /var/lib/pgbackrest
sudo chown -R postgres:postgres /var/log/pgbackrest

#A Classic Mistake

Running pgbackrest as root for a local backup. When archive-push is run by the server (the postgres user), access to a repository created by root actually fails. Always be consistent: local backup = the postgres user, or a dedicated user with the same repository access.

#Common Case 3: Full Repository

#Symptoms

archive-push starts failing; df -h shows the repository at 100%. Backups fail midway because there's no space.

#Sequential Handling

1. Free up space quickly: remove temp/old backup files, or add storage capacity.
2. Check retention: if WAL is piling up, repo1-retention-archive isn't applied (episode 10).
3. Check for leaks: .tmp files from interrupted backups can clog up space:

sudo find /var/lib/pgbackrest -name "*.tmp" -size +100M

4. Once space is safe, pending archive-push operations are automatically retried by PostgreSQL — failed WAL isn't lost, it's just waiting.

#Common Case 4: Version Mismatch

#Symptoms

Errors like db version ... does not match ... or stanza already exists. These appear when the pgBackRest or PostgreSQL version doesn't match what's recorded in the stanza.

#Causes and Solutions

• PostgreSQL was upgraded: run stanza-upgrade to update the stanza metadata:

sudo -u postgres pgbackrest --stanza=main stanza-upgrade

• pgBackRest was upgraded: generally safe, but verify with check after the upgrade.
• A different cluster with the same stanza name: the stanza is already locked by another cluster's identity. Don't force-delete it; create a new stanza with a different name.

#A Concise Debugging Workflow

When an error comes, follow this sequence without skipping:

1. Read the full error message — not just the first line.
2. Rerun with --log-level-console=debug.
3. Check /var/log/pgbackrest/ for context not visible on the console.
4. Verify the most basic things first: user, permissions, disk space, and version.
5. Test the smallest component (manual archive-push) before blaming the big one.

#Conclusion

Key takeaways:

• --log-level-console=debug for interactive incidents; --log-level-file=debug for long investigations.
• Log files in /var/log/pgbackrest/ contain error codes, DETAIL, and HINT.
• Failed archive-push: check the binary path, archive_mode, and the stanza.
• Permission denied: make sure the user is consistent and ownership is correct.
• Full repository: free up space, check retention, clean up .tmp.
• Version mismatch: stanza-upgrade for a PostgreSQL upgrade; never force-delete a stanza.

In the next episode we'll prove your backups can actually be recovered: restore testing (drills) — doing periodic restore tests to a separate instance, validating data (checksums, row counts), automating the drill with scripts + reports, and documenting the runbook. A backup that was never tested is hope disguised as a plan!