Learn pgBackRest - Monitoring & Metrics

#Introduction

A backup running smoothly never draws attention. The problem: a backup system fails silently — a missed schedule, WAL stopping to archive, a full repository — and nobody knows until the data is actually needed for a restore. In episode 14 we install eyes and ears: monitoring with Prometheus and alerting that cries out before a disaster becomes fatal.

This is the principle that changes everything: a backup isn't a "task that runs," it's a measured SLA. If you don't monitor the age of the last backup, you don't know whether your recovery guarantee is real.

#Metrics: Reading Status with Scripts

#pgbackrest info --output=json

Before monitoring tools, pgBackRest already provides structured output for scripts: --output=json. Try it yourself:

sudo -u postgres pgbackrest info --output=json

The output contains per stanza: status, age of the last backup (backup/timestamp-stop), type, and WAL range. From here a script can extract important metrics:

sudo -u postgres pgbackrest info --output=json | \
  node -e 'const d=JSON.parse(require("fs").readFileSync(0));const b=d[0].backup[0];console.log("last backup age (s):", Math.round((Date.now()/1000 - b.timestamp.stop)*100)/100)'

This pattern is the basis of every "is backup fresh?" script — and can be run from cron for simple alerts without Prometheus (we cover that in the alerting section).

#Prometheus Integration

#pgbackrest_exporter

pgbackrest_exporter is an exporter that translates pgBackRest output into Prometheus metrics ready to be scraped. Install and run it on the database host:

# download the binary from the pgbackrest_exporter project release
./pgbackrest_exporter --pgbackrest.executable=/usr/bin/pgbackrest \
                      --web.listen-address=:9854

The exporter reads pgbackrest info periodically and exposes metrics at /metrics, for example:

• pgbackrest_info_backup_timestamp_stop — when the last backup finished.
• pgbackrest_info_backup_size — the backup size.
• pgbackrest_info_backup_type — the backup type (full/diff/incr).

#Configuring Prometheus

scrape_configs:
  - job_name: pgbackrest
    static_configs:
      - targets: ["db01.internal:9854"]

With the timestamp stop metric in Prometheus, you can compute the backup age as a derived metric:

time() - pgbackrest_info_backup_timestamp_stop

#Alerting: Alarms Before It's Too Late

#Alert for Failed or Late (Deadline-Missed) Backups

Two conditions that must always be alerted:

• Backup failed — cron/systemd records a non-zero exit code.
• Backup late — a backup hasn't appeared by the deadline. This is more dangerous, because "no news" doesn't mean healthy.

With a simple cron, check the backup age every hour:

crontab -u postgres -e

5 * * * * /usr/local/bin/pgbackrest_age_check.sh main 26

The script reads the JSON and compares the last backup's age with a threshold (for example 26 hours for a daily full+diff schedule):

#!/usr/bin/env bash
set -euo pipefail
stanza=$1
max_age_hours=$2
age_s=$(pgbackrest info --output=json --stanza="$stanza" \
  | node -e 'const d=JSON.parse(require("fs").readFileSync(0));console.log(Math.round((Date.now()/1000-d[0].backup[0].timestamp.stop)*100)/100)')
age_h=$(awk -v s="$age_s" 'BEGIN{printf "%.2f", s/3600}')
if awk -v a="$age_h" -v m="$max_age_hours" 'BEGIN{exit !(a>m)}'; then
  echo "ALERT: backup stanza $stanza is late ($age_h hours > $max_age_hours hours)"
  exit 1
fi

The equivalent alert in Prometheus:

groups:
  - name: pgbackrest.rules
    rules:
      - alert: PgBackRestBackupTooOld
        expr: (time() - pgbackrest_info_backup_timestamp_stop) / 3600 > 26
        for: 30m
        annotations:
          summary: "Backup of {{ $labels.stanza }} is late"

#Alert for Stuck WAL Archiving

WAL that stops being archived means un-backed-up data piles up on the primary. Two signals:

• archive_command keeps failing — PostgreSQL retries the same segment; pg_stat_archiver shows failed_count rising.
• The WAL range in the repository doesn't advance — pgbackrest info shows a static wal archive min/max.

A simple check script:

psql -U postgres -tAc "SELECT failed_count FROM pg_stat_archiver;"

Any rise in failed_count must be alerted immediately. In Prometheus, pg_stat_archiver can also be exposed by postgres_exporter for the same alert.

#Alert for a Full Repository

A full repository is the classic cause of archive-push failure. Monitor the repository disk capacity:

df -h /var/lib/pgbackrest

Alert when usage is above 80-85% — because when it's full, backups and WAL stop, and the alarm only surfaces through a different path (late backup).

#Building a Dashboard

The most useful metrics for a dashboard:

1. Age of the last backup per stanza (bar chart) — one glance tells you the health.
2. Backup size per type (time series) — data growth trend.
3. Archiver failed_count (counter) — a rise means WAL problems.
4. Backup duration (from logs/jobs) — performance degradation.

#Conclusion

Key takeaways:

• pgbackrest info --output=json is the raw material for all monitoring scripts.
• pgbackrest_exporter translates it into scraped Prometheus metrics.
• Required alerts: failed backup, late backup (deadline), stuck WAL, and full repository.
• A deadline alert matters more than a "failed" alert — silent failure is the biggest enemy.
• A backup-age dashboard is the single panel that most often saves the day.

In the next episode we'll fix things when nothing goes right: troubleshooting & debug — raising --log-level-console=debug and --log-level-file=debug, reading /var/log/pgbackrest/, and handling the most common cases: failed archive-push, permission denied, full repository, and version mismatch. Structured debugging is the skill that saves production!