Learn Proxmox Backup Server - Monitoring & Performance

#Introduction

Episode 19 connected PBS to the outside ecosystem. Now there is a question often forgotten: how do you know backups are healthy before it is too late? A backup that "appears successful" can fail silently — a full datastore, delayed verify, or a sync that never runs. Episode 20 answers this with monitoring & performance: monitoring PBS from outside (API, InfluxDB, Prometheus, Zabbix) then measuring and tuning performance (chunk size, compression, parallelism).

The analogy is like a control panel in the archive room: you do not wait for the shelves to collapse, you install load sensors, staff hour tracking, and alarms — then routinely measure how fast archives are moved so the warehouse closing schedule does not slip.

#Monitoring PBS

#Metrics You Must Monitor

• Datastore usage: percentage used and estimated days remaining — the first alarm when the disk runs low.
• Dedup factor (deduplication-factor): the ratio underpinning capacity planning (episode 18); if it drops drastically, something changed in the workload.
• Job status: backup, verify, prune, GC, and sync — which succeeded, errored, or overran.
• Age of the last backup (snapshot age): a backup that never arrives means your RPO has been silently violated.

#Data Sources: Web UI, Task Log, and API

The web UI (port 8007) shows the dashboard and per-datastore task logs — good for manual inspection. For automation, PBS exposes a JSON API on the same port. Create a read-only API token and use it in curl:

curl -sk \
  -H "Authorization: PBSAPIToken=monitor@pbs!metrics:TOKEN" \
  https://pbs.local:8007/api2/json/admin/datastore/store1/status

The response contains used, total, avail, and deduplication-factor — everything alerting needs. Task logs can be fetched from /api2/json/nodes/localhost/tasks. On the PVE side, check backup storage status with pvesm:

pvesm status
pvesm list pbs-backup

#InfluxDB: Native PBS Export

PBS has native metric server support: Configuration → Metric Server → add InfluxDB (HTTP or UDP). Without extra installation, PBS sends the same RRD data as its GUI charts — for example the blockstat series per datastore/object (host, VM) — to InfluxDB, which is then visualized in Grafana. This is the easiest path to a time-series dashboard.

#Prometheus & Zabbix

PBS does not provide a built-in /metrics endpoint. For Prometheus, two common patterns:

1. Community exporter (e.g. pbs-exporter) that translates the JSON API into Prometheus metrics.
2. Textfile collector: a small script runs curl against the API then writes a .prom file that node_exporter reads.

For Zabbix, use an HTTP agent that pulls the JSON API endpoint above with a PBS token, then build a template (metrics + triggers for full disk and failed jobs). Whatever the choice, it all funnels through the same API — so this "read-only token + /status endpoint" pattern, once understood, can be reused anywhere.

#Logs

Task logs are stored in /var/log/proxmox-backup/tasks/ and can be read via the GUI (Administration → Tasks) or journalctl:

journalctl -u proxmox-backup -u proxmox-backup-proxy -f
ls -lt /var/log/proxmox-backup/tasks/ | head

How long task logs are kept is controlled by --task-log-max-days on the node config (episode 16 used this during troubleshooting).

#Performance: Benchmark First

Before tuning, measure. PBS provides a built-in benchmark that measures each stage of the backup pipeline — TLS, SHA256, compression, decompression, AES256/GCM, and verify:

proxmox-backup-client benchmark --repository 10.0.1.5:store1

The result is an MB/s figure per stage. How to read it:

• Low TLS speed → network bottleneck (check the NIC, jumbo frames, rate limits).
• Low SHA256 / AES256-GCM → CPU; verify AES-NI is active (episode 7).
• Run once with --keyfile and once without, to measure the encryption overhead on your hardware.

#Tuning: Chunk Size, Compression & Parallelism

#Chunk Size

PBS splits data into fixed-size chunks. The size is set via --chunk-size (in KiB, 64-4096, default 4096 = 4 MiB, must be a power of two):

• Large chunks → better compression and dedup ratio, more compact index, but need more RAM for the chunk index and coarser dedup granularity.
• Small chunks → finer dedup, but more metadata overhead.

Change it only for new backups; old chunks keep their creation-time size. For mixed workloads, the 4 MiB default is proven balanced — tuning is usually for special cases (e.g. lots of small files).

#zstd Compression

PBS uses zstd as the default — the best ratio with the best speed. In PVE, compression is set per backup storage (compression): none, lzo, zlib, or zstd. For ZFS datastores, watch the compression overlap (episode 15): if ZFS already has compression=on and the data is already PBS-compressed, ZFS compression adds almost nothing — disable one to save CPU.

#Parallelism (io-threads)

Verify parallelism is controlled via datastore tuning — the replacement for the "io-threads" knob in modern PBS:

proxmox-backup-manager datastore update store1 \
  --tuning default-verification-readers=4,default-verification-workers=4

default-verification-readers and default-verification-workers split verification work (episode 9) across several threads — increase if the disk can handle it and verify is the bottleneck, decrease if it triggers contention with backups. Sync jobs also have --worker-threads (episode 18) for parallel chunk transfer.

#The Strengthening Foundation

The tuning above only adds on top of the foundations from episodes 15 & 18: recordsize 1M on ZFS for 4 MiB chunks, 10G networking with jumbo frames for the PVE → PBS path, and backup windows separated from verify/GC windows. Re-benchmark after every hardware change — not once and then forgotten.

#Closing

Key takeaways:

• Monitor datastore usage, dedup factor, job status, and last backup age.
• The JSON API on port 8007 is the single door every monitoring tool needs; create a read-only token.
• InfluxDB: native PBS support (Configuration → Metric Server); Prometheus/Zabbix use a community exporter or HTTP agent.
• proxmox-backup-client benchmark measures each stage — network vs CPU vs storage.
• --chunk-size (default 4 MiB) only affects new backups; zstd is default; parallelism via datastore --tuning.
• Tune one variable, measure, then move to the next.

In the next episode, episode 21, we will look ahead: the PBS roadmap — the S3 object storage direction, dedup/performance improvements, and cloud integration — plus the community map: forum, wiki, mailing list, and official documentation. From "measured" to "ready to grow along with it"!