Learn Borg Backup - Core Concepts & Key Architecture

#Introduction

In episode 1 we learned why Borg exists and why dedup matters. Now it is time to understand how Borg works under the hood: what a repository is, what an archive is, and how content-defined chunking produces efficient deduplication. This understanding is not just theory — when your backup misbehaves in episode 16, you will come back to this chapter.

#Architecture: Repository and Archive

#Repository: The Home of All Data

The repository is a storage location (local or remote via SSH) that holds all of Borg's data. Internally, a repository contains:

• config: the repository configuration — format version, encryption mode, and other settings.
• data: segments that store the encrypted data chunks, written in an append-only fashion.
• index & hints: a catalog mapping chunk hashes to their location in segments. The index can be rebuilt from segments if lost — this is the basis of borg check in episode 12.

#Archive: A Dataset Snapshot

An archive is a "snapshot" of one dataset at one point in time — the result of a single borg create command. A single repository can hold hundreds of archives from different days, weeks, and even years. Each archive stores complete file metadata (name, permissions, timestamp, chunk list) and references to the data chunks.

The key insight: two archives that contain the same file will share the same chunks. Adding one file to a second archive does not store that file's data again — that is the source of Borg's efficiency.

#Content-Defined Chunker

To split files into chunks, Borg uses content-defined chunking (CDC) with the buzhash algorithm. Because chunk boundaries are determined by content patterns, not byte positions, shifted data does not change all chunks. By default the average chunk is around 2 MiB, and this parameter can be tuned for specific datasets (episode 18).

#Key Components

#The borg Binary

borg is a single Python binary containing many subcommands. Every operation — from creating a repo to checking integrity — goes through this subcommand. There is no daemon, no external database; just the binary and the filesystem.

#Repository Location

A repo is written as a local path (/backup/borg) or a remote one (user@backup-host:/backup/borg). Borg detects the user@host: pattern and automatically uses SSH — including calling borg serve on the server side, as we build in episode 11.

#Key

Borg's encryption depends on a key. There are two main styles: repokey (the key is stored inside the repository, protected by a passphrase) and keyfile (the key is in a separate file). Without the correct key or passphrase, the repository is unreadable — even by the original owner of the data. Full details in episode 7.

#Subcommand Map

borg init        # create a repository
borg create      # create an archive (backup)
borg list        # list archives in the repo
borg info        # archive/repo details
borg extract     # restore from an archive
borg mount       # mount an archive via FUSE
borg prune       # delete old archives (retention)
borg compact     # reclaim segment space
borg check       # verify repo integrity

The order of this map is the backup lifecycle: init → create (repeated many times) → prune + compact → check. We follow that flow exactly from episode 3 to 12.

#Data Flow: What Happens During borg create

1. Borg reads the source files and splits them into chunks (CDC).
2. Each chunk is hashed; hashes already present in the index are skipped — the data is not stored again.
3. New chunks are compressed, then encrypted with the key.
4. File metadata (name, permissions, timestamp, chunk list) is stored as archive data.
5. Everything is written as append-only segments — old archives are never modified.

An important consequence of step 5: deleting an archive does not immediately free space. Chunks no longer referenced are only truly discarded when borg compact is run — we cover this in episode 8.

#Why This Design Is Good

• Immutable: an archive, once created, never changes, so it is consistent at restore time.
• Efficient: storage scales with unique data, not the total size of the dataset.
• Secure: append-only segments + authenticated encryption form the basis of the anti-ransomware defenses in episode 13.

#Closing

• A repository holds data segments + index; an archive is a point-in-time snapshot.
• CDC (buzhash) determines chunks from content, not position — the key to dedup efficiency.
• Core components: the borg binary, repo location (local/SSH), key, and subcommands.
• The backup lifecycle: init → create → prune/compact → check.
• Archives are immutable; space is truly freed only by borg compact.

In episode 3 we run the first real command: repository initialization with borg init, choosing between the repokey, keyfile, and authenticated encryption modes, and setting up BORG_REPO and BORG_PASSPHRASE/BORG_PASSCOMMAND correctly.