Learn Borg Backup - Troubleshooting

#Introduction

At some point, your backup will fail — that is not an "if", but a "when". What separates an experienced operator is not the ability to avoid problems, but the speed of diagnosing and recovering from them without making things worse. Episode 16 equips you with Borg's debugging toolkit: borg check as the first-line checker, --debug-topic for detailed traces, up to structured logging with BORG_LOGGING_CONF.

#The Debugging Toolkit

#Start with borg check

Before blaming anything, verify the repository's health with borg check /backup/borg. The result points the way: errors in the index/segment indicate a repository problem; errors while rebuilding the cache indicate a local client problem.

#--debug-topic for Detailed Traces

Borg can enable debug logging per topic — far more useful than the noisy full debug:

borg create --debug-topic=archive,chunker --list /backup/borg::test /home

Common topics: archive (archive metadata), chunker (the chunk-splitting process), repository, cache, ssh. Combining --debug + --debug-topic gives maximum detail.

#Structured Logging with BORG_LOGGING_CONF

For long-term debugging in production, use your own logging configuration:

[loggers]
keys = root, repository
 
[handlers]
keys = file, console
 
[formatters]
keys = standard
 
[logger]
level = INFO
 
[logger.repository]
level = DEBUG
 
[handler_file]
class = FileHandler
level = DEBUG
args = ('/var/log/borg.log', 'a', 'UTF-8', 1)
 
[handler_console]
class = StreamHandler
level = WARNING
 
[formatter_standard]
format = %(asctime)s %(levelname)s %(name)s %(message)s

Enable it with export BORG_LOGGING_CONF=/etc/borg/logging.conf then run borg commands as usual. Detailed logs go to /var/log/borg.log while the console stays clean — a healthy pattern for automation.

#Common Cases

#1. Repository Lock

Failed to create/acquire the lock on /backup/borg

Cause: another borg process is running, or an old lock from a crash/power outage. First check processes with ps aux | grep "[b]org". If there is no process but the lock remains, it is a stale lock — never rm lock files manually; use borg break-lock /backup/borg, and only if you are sure it is genuinely stale.

#2. Repository Corruption

Repository or key data seem corrupt

First step: do not panic and do not jump straight to --repair. Diagnose with borg check /backup/borg --debug-topic=repository:

• If the index is damaged but the segments are intact: borg check rebuilds the index from the segments.
• If segments are truly corrupt: consider borg check --repair, with a copy of the repository elsewhere first. Remember the warning from episode 12: --repair deletes data that cannot be repaired.

#3. Version Mismatch

Repository was made with a different security feature level

A repo created by Borg 2.0 cannot be read by 1.4, and vice versa. Quick diagnosis: compare borg --version with the version in the repository config. The fix: align the client and server versions (episode 15), or for 2.0-beta repos do not force 1.x to read them — build a new repo with the correct version.

#4. SSH/Remote Failure

Connection closed by remote host

Test the connection manually with BORG_RSH='ssh -v' borg list borg@backup-host:/srv/borg/backup. -v shows the SSH handshake; failures in the command= of authorized_keys usually appear clearly here.

#Recovery from Damaged Segments

When a segment is truly damaged and holds chunks referenced by an archive:

1. Identify the damaged chunks via borg check --verify-data — which chunks fail.
2. Find the archives depending on those chunks — only those archives are affected.
3. Restore from another copy if one exists (pattern 2 in episode 14 makes off-site act as a safety net).
4. As a last resort, borg check --repair discards the corrupt chunks — the related files are lost from the archive, but other archives stay intact.

The most effective preventive practice: always have a second repo copy and routine borg check. The best recovery is the one never needed.

#Common Pitfalls

• Deleting lock files manually: always borg break-lock, not rm on files under lock.exclusive.
• --repair as a first reflex: diagnose first; some corruption can be resolved without repair.
• Ignoring version warnings: reading a 2.0 repo with 1.4 damages metadata. Respect version boundaries.
• Debugging without a logging config: in production, use BORG_LOGGING_CONF so traces are stored, not lost in the terminal.

#Closing

• Diagnosis starts with borg check, not assumptions.
• --debug-topic gives per-topic traces; BORG_LOGGING_CONF stores logs for production.
• Common cases: repo lock (use borg break-lock), corruption (check → careful repair), version mismatch, and SSH failures.
• Recovery from damaged segments: identify the impact, use another copy, --repair as a last resort.
• A second repo copy + routine checks are the best preventive practices.

In episode 17 we look to the future: Borg 1.4.5 vs Borg 2.0 — why 1.4.5 remains the production choice, what 2.0 overhauls (repo format, hash index, --match-archives, segment management), and when you may move.