Learn Borg Backup - Monitoring & Alerting

#Introduction

In episode 19 we exported metrics. But metrics without alerts are just pretty graphs — when a backup fails in the middle of the night, you want to know, not just see the number the next morning. Episode 20 closes the loop: actively monitoring borgmatic, triggering alerts on the two most important conditions (failure and lateness), and delivering them to channels you actually read.

#Monitoring borgmatic Logs and Metrics

#What to Monitor

The two most valuable signals from a backup:

1. Execution status: did the last backup succeed?
2. Staleness: when was the last successful backup? (A backup that never runs is as dangerous as one that fails.)

Both can be taken from the borgmatic log (--verbosity 1) or from the metrics we exported in episode 19. For Prometheus, add a timestamp metric:

# HELP borg_backup_last_success_seconds Time of the last successful backup (unix)
# TYPE borg_backup_last_success_seconds gauge
borg_backup_last_success_seconds 1785300000

The "backup is late" alert is just a rule: time() - borg_backup_last_success_seconds > 36h.

#Alerting on Backup Failure

#The on_error Hook

The most direct way: borgmatic's on_error hook runs exactly when the backup fails.

hooks:
  before_backup:
    - echo "Backup started $(date)"
  after_backup:
    - echo "Backup finished $(date)"
  on_error:
    - curl -fsS -X POST "https://ntfy.sh/backup-alerts" \
        -d "Backup FAILED on $(hostname)"

ntfy is a free pub/sub push notification service — no account needed for simple use. Replace it with mail, a curl to a Slack webhook, or whatever your team understands.

#Email/Slack/Telegram Notifications

Email is just a one-line hook: echo "Backup failed on $(hostname)" | mail -s "[BACKUP] Failed" backup@example.com. Slack via an Incoming webhook:

hooks:
  on_error:
    - curl -fsS -X POST -H 'Content-type: application/json' \
        --data '{"text":"Backup failed on '"$(hostname)"'"}' \
        https://hooks.slack.com/services/T000/B000/XXXX

Telegram — via the bot API:

hooks:
  on_error:
    - curl -fsS "https://api.telegram.org/bot<TOKEN>/sendMessage" \
        -d chat_id=<CHAT_ID> -d "text=Backup failed on $(hostname)"

#Alerting on a Late Backup

#The Problem with Failure Alerts Alone

on_error only fires when the process fails. If the host is completely dead, cron does not run, or the network is down — nothing triggers on_error and nobody knows. A late backup needs an external watchdog mechanism.

#Healthchecks

Services like Healthchecks.io solve this: borgmatic sends a ping every time a backup finishes; if the ping does not arrive on schedule (e.g. within 36 hours), Healthchecks sends the alert. This detects "nothing happened" — not just "something went wrong".

monitoring:
  healthchecks:
    ping_url: https://hc-ping.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

With just this line, borgmatic automatically pings success after a backup finishes and pings failure on error. The lateness alert is configured in the Healthchecks dashboard (e.g. "alert if no ping within 36 hours").

Borgmatic also has other ready-made integrations: cronitor, pagerduty, pushover, ntfy, and apprise.

#A Self-hosted Alternative

To avoid third-party services, a simple self-implementation:

1. Our exporter script from episode 19 writes borg_backup_last_success_seconds.
2. Prometheus Alertmanager (or Zabbix/Nagios) evaluates the staleness rule.
3. The alert is forwarded to the team's channel.

#Building Healthy Alerting

+--------------------+--------------------+----------------------+
| Condition          | Signal             | Channel              |
+--------------------+--------------------+----------------------+
| Backup failed      | on_error hook      | Slack/Telegram/Email |
| Backup late        | Healthchecks miss  | Slack/Telegram/Email |
| Check failed       | borg check exit    | Slack/Telegram/Email |
| Disk full          | node_exporter      | Prometheus alert     |
| Suspicious SSH login| auth.log          | SIEM/fail2ban        |
+--------------------+--------------------+----------------------+

The key principle: alerts must be actionable. Too many alerts = noise that gets ignored; too few = blind spots. Start with the two core alerts (failure + lateness), then expand based on real incidents.

#Common Pitfalls

• Relying only on on_error: a total outage does not trigger on_error. An external watchdog (staleness/Healthchecks) is mandatory.
• Tokens in a config that goes into git: encrypt/isolate credentials.
• Alerts without a runbook: a "Backup FAILED" notification without handling steps is just stress. Write a short runbook (check the log, check the disk, run manually).
• Testing once and forgetting: test the alert flow periodically — deliberately fail a backup in staging and confirm the notification arrives.

#Closing

• Monitor two signals: execution status and staleness (when it last succeeded).
• The on_error hook triggers an alert when a backup fails.
• Email/Slack/Telegram via hooks; avoid hardcoded tokens.
• Healthchecks (or a staleness watchdog) catches backups that silently stop running.
• Healthy alerting: few, actionable, and periodically tested.

In episode 21 we look ahead: roadmap & community — what the project is working on (the awaited stable Borg 2.0, the security & performance focus), and where you can contribute and ask questions: GitHub, the documentation, IRC/Matrix, and Bountysource.