Learn Borg Backup - Hardening & Multi-host

#Introduction

In episode 13 your repository became a fortress. Now it is time to apply that pattern at organization scale: many hosts, many teams, and different recovery demands. The biggest temptation here is "one big repository for everything" — it looks practical, but it merges risks, keys, and retention policies that should stay separate. Episode 14 teaches the correct design.

#Multi-host: One Repository per Host/Team

#Why Not One Giant Repository

One repository holding the backups of every server is tempting — one setup, everything goes in. But here is the cost:

• Shared keys: one key for all hosts. One leak exposes all data.
• Mixed retention policies: a host with 2 years of audit data and a host with 7 days of logs become one rule.
• Degraded performance: the repo index grows as archives accumulate, slowing every operation.
• Dangerous prune: a single --prefix mistake can delete another host's backups (episode 8).
• Blast radius: a corrupt repo = every host loses its backups at once.

#The Correct Design

Separate per host or per data group with the same policy:

/srv/borg/
├── web-01/        # web-01 host repo (restrict path per host)
├── web-02/        # web-02 host repo
├── db-primary/    # database cluster repo
└── team-data/     # one team's shared repo

On the SSH side, each repo maps to a different restricted command — and ideally a different key per host:

command="borg serve --restrict-to-repository /srv/borg/web-01",restrict,no-pty ssh-ed25519 AAAA...web01-key
command="borg serve --restrict-to-repository /srv/borg/web-02",restrict,no-pty ssh-ed25519 AAAA...web02-key

#Key Isolation

The principle: one key = one host = one repository. This enables individual revocation without touching other hosts, and limits the damage if one key leaks. Repo passphrases should also be unique per repo — manage them with a password manager or secret management, not one global passphrase.

#Access Monitoring

Backups are a window into all of your server data — access to the repository must be visible:

• SSH audit: monitor /var/log/auth.log on the backup host for borg user connections; alert on logins outside the backup schedule.
• Failed logins: repeated failed SSH attempts signal brute force — set up fail2ban or similar.
• Admin access to the repo: who has shell access to the backup host? Keep it minimal (episode 11: the passwordless borg user).
• borgmatic logs: make sure on_error and check results are recorded and monitored (episode 20).

#Best Practices: Off-site and Restore Tests

#Off-site Backups on a Different Host

The 3-2-1 principle we mentioned in episode 1, now in concrete form:

• 3 copies: the original data, a local backup, and an off-site backup.
• 2 media: e.g. local disk + another host's backup storage (not a partition on the same disk).
• 1 off-site: a host in a different location — not the same room as the production servers.

For off-site, there are two common patterns:

# borgmatic already supports multiple repositories:
repositories:
  - path: /backup/borg
    label: local
  - path: borg@offsite-host:/srv/borg/backup
    label: offsite

# back up locally first, then transfer the repo to the off-site host
borg create /backup/borg::"{hostname}-{now}" /home /etc
borg prune --keep-daily 7 --keep-weekly 4 /backup/borg
borg compact /backup/borg
rsync -a /backup/borg/ borg@offsite-host:/srv/borg/backup/

Pattern 2 is attractive because the off-site host only receives a copy of the repo — no keys are ever live there, and the transfer can be scheduled outside backup hours.

#Scheduled Restore Tests

We stressed this in episode 5; now it is a policy:

• Monthly: restore one file/directory from the newest archive and compare checksums.
• Quarterly: full restore of one critical host into a staging VM.
• Yearly: full restore + data verification (borg check --verify-data) afterwards.

A restore test is not a "hope it works" — it is the only proof that backups actually work.

#Common Pitfalls

• One repo, many hosts, one key: the worst combination. Separate them.
• Off-site in the same room: the same fire/physical attack destroys both.
• Local repo on the same disk as the data: a disk failure wipes the data and the backup together.
• Restore tests only during incidents: too late. Schedule them like backups.
• Not monitoring the backup host: undetected damage for years is just as bad as having no backup.

#Closing

• Multi-host design: one repository per host/team, one key per host.
• Key isolation enables individual revocation and limits the blast radius.
• Access monitoring: SSH audits, alerting, and borgmatic logs.
• Off-site on a different host and location; avoid a single point of failure.
• Scheduled restore tests are the proof that backups work.

In episode 15 we look at a more specific security realm: CVE & security releases — the two CVEs fixed by Borg 1.4.5 (metadata discarding and cache corruption), why you should upgrade immediately, and routine borg check as part of your security posture.