Learn Proxmox Backup Server - Scale: Many Datastores & Sync Architecture

#Introduction

Episode 17 introduced PBS 4.2 and S3 object storage. Now the question changes: what if you manage not one PBS, but a dozen PBS servers for dozens of clients? Episode 18 covers scaling — many datastores per server, namespaces and backup groups to organize clients, hub-spoke architecture via remote sync, and capacity planning before hardware is bought.

The analogy is like expanding one office into a multi-branch company: each branch (site) has its own archive warehouse, one central warehouse receives copies of everything, and before building a new building you calculate how many shelves are needed and how many trucks must pass per day.

#Scaling with Many Datastores

#One Server, Many Datastores

One PBS does not mean one datastore. A datastore is a logical unit that can be created many times on one server, each with its own path, GC schedule, verify schedule, retention, and ACLs:

Tenant/department isolation: team A and team B do not see or interfere with each other's schedules.
Different policies per workload: store-prod keeps 30 days, store-archive keeps 1 year.
Different backends: store1 on local ZFS, store-s3 on object storage (episodes 17 & 19).

Create a second datastore on the same server

proxmox-backup-manager datastore create store-prod \
  --path /mnt/zfs/store-prod \
  --gc-schedule "sun 02:00" \
  --verify-new
proxmox-backup-manager datastore list

#Namespaces: Logical Folders Within a Datastore

Since PBS 3.x, a datastore can be further divided into namespaces — logical folders to separate clients without creating a new physical datastore. Namespaces are very useful for SaaS/MSP scenarios: one large datastore is shared, and each tenant uses its own namespace with its own ACL and sync.

Create and list namespaces

proxmox-backup-manager namespace create store-prod tenant-a
proxmox-backup-manager namespace create store-prod tenant-b
proxmox-backup-manager namespace list store-prod

Every snapshot in PBS actually belongs to one backup group identified by type/id (e.g. vm/100, ct/200, host/web-01). The combination of datastore + namespace + backup group is the full "address" of a backup — and the basis for grouping clients.

#Client Groups & Organization

A healthy pattern: one namespace (or datastore) per client group, with consistent naming conventions.

PVE backup jobs send vm/... to the specified datastore (episode 4); choose the right datastore per job.
Host clients use proxmox-backup-client with explicit --backup-type/--backup-id (episode 11) so they map easily to a namespace.
Per-namespace ACLs (episode 12): the ops@pbs user for namespace tenant-a, and access rights for tenant-b only to their own team.

Tip

Start from one datastore per logical group, then use namespaces when that datastore feels large and mixed up. Namespaces add flexibility without adding new mount points — but write down the naming rules, because namespaces without a convention quickly become a new "junk folder".

#Hub-Spoke Architecture

#The Hub-Spoke Concept

As the number of sites grows, the episode 10 off-site pattern evolves into hub-spoke: each site has a local PBS (spoke) for fast on-site backups, and one central PBS (hub) pulls snapshot copies from all spokes via sync jobs. Sync remains a pull model (episode 10): the hub authenticates to the spokes, not the other way around.

The advantage: fast restore from the local PBS for daily needs, while the hub becomes a centralized off-site guarantee that can later be replicated to S3 (episode 19) without opening access paths into every site.

#Hub-Spoke Practice

On each spoke, prepare a sync user + API token with read rights to the datastore/namespace to be pulled (episode 10). On the hub, create one sync job per spoke:

Hub sync job for spoke site-a

proxmox-backup-manager sync job create \
  --remote site-a.example.com \
  --remote-user sync@pbs \
  --remote-auth token \
  --remote-datastore store1 \
  --datastore hub-site-a \
  --schedule "daily 03:00" \
  --keep-last 7 --keep-daily 14 --keep-monthly 6 \
  --remove-vanished

Data flows spoke → hub; --keep-* retention on the hub limits growth, and --remove-vanished keeps deletions in sync. Snapshots on the spoke and hub are two copies of the same data — exactly the second layer of the 3-2-1 strategy (episode 10).

#Rate Limits & Schedules

WAN links between sites are usually narrower than LAN. PBS sync jobs support rate limiting and worker threads so sync does not eat all the office bandwidth:

Sync with rate limit and workers

proxmox-backup-manager sync job create \
  --remote site-a.example.com \
  --remote-user sync@pbs --remote-auth token \
  --remote-datastore store1 --datastore hub-site-a \
  --schedule "daily 02:00" \
  --rate-in 50M --burst-in 100M \
  --worker-threads 2

--rate-in limits the download speed (data entering the hub), --burst-in allows a temporary burst.
--worker-threads (default 1) speeds up parallel chunk transfer when bandwidth is available.

Schedule sync outside local backup hours, and separate it from verify/GC windows (episodes 6 & 9).

#Capacity Planning

#Calculating Dedup Ratio & Disk Needs

The episode 6 metrics (Size/Original/Dedup/Compressed) are not just reports — they are the raw material for sizing. Rough disk needs:

Example: 10 VMs × 100 GiB = 1 TiB of original data. With dedup 8x and zstd compression 2x: 1 TiB / 8 / 2 = 64 GiB of unique chunks — plus metadata overhead and headroom. The key: measure the ratio from a running datastore (episode 6) before buying new disks, do not guess.

#Bandwidth & 10G Networking

Daily backups must finish within the allotted window. Calculate the need:

The first backup is full (all data); subsequent backups are only deltas (new chunks).
Example: a 2 TiB daily delta in a 4-hour window → requires 2 TiB / 4 hours ≈ 1.2 Gbps continuously — beyond the comfortable range of 1G.

That is why 10G is recommended for the PVE → PBS and PBS → hub paths: it gives room for spikes without extending the window. Use jumbo frames and proxmox-backup-client benchmark (episode 20) to measure the real TLS + zstd throughput before setting the window.

#Disk Sizing

For a local ZFS datastore:

recordsize 1M (episode 15) for 4 MiB chunk efficiency.
raidz2 / mirror for redundancy — a backup on a single disk is not a backup.
Leave 20-30% headroom: GC needs space during the mark/sweep phases (episode 6), and a 100% disk is a primary source of backup failures.
Monitor df -h and project monthly growth; buy before it fills, not after.

Warning

Do not calculate capacity only from the "original" snapshot size. Deduplicated and compressed chunks use far less — but indexes, metadata, and the GC phase also eat space. Calculate two numbers: the theoretical need (formula above) and the monthly growth rate of a real datastore. The latter is more trustworthy.

#Closing

Key takeaways:

One PBS can have many datastores: tenant, policy, and backend isolation on a single server.
Namespaces (PBS 3.x+) organize clients without creating new datastores; the type/id backup group is the full snapshot address.
Hub-spoke: a local PBS per site + one central hub pulling copies via sync (pull), with rate limits and worker threads.
Disk sizing ≈ Original / dedup ratio / compression ratio, plus overhead and a 20-30% headroom.
Bandwidth: calculate the window vs daily delta; 10G gives a safe margin for backup paths.
Measure real ratios from a running datastore — do not guess.

In the next episode, episode 19, we will connect PBS to the outside ecosystem: making the PBS datastore filesystem an NFS/SMB target for other backup agents (Veeam, Bacula) and using PBS 4.x S3 object storage as a cloud off-site destination. Your infrastructure no longer stands alone!