Learning Restic - Monitoring & Alerting

#Introduction

Episode 19 introduced the ecosystem and metrics. Episode 20 answers the scariest question in backup operations: what happens if a backup fails and nobody knows? The short answer: you lose data without realizing it until it's too late.

Monitoring is not a nice-to-have — it is the safety net that turns a "hope it works" backup into a "proven to work" backup.

#What to Monitor

For every backup, record three core things:

• Status: success (0) or failure (non-0).
• Duration: how long the backup took.
• Data added: how much new data was stored.

Example using --json:

restic backup /data --json 2>/tmp/restic.err
status=$?
duration=$(jq '.total_duration' /tmp/restic.err)

The combination of the three detects not just total failure, but also silent degradation: a backup that succeeds but is suddenly very fast could mean the excludes are wrong and data is not being backed up.

#Exporter: Publishing Status

A lightweight approach without new infrastructure: a script writes a metrics file that node_exporter reads (skeleton in episode 19). The full approach: resticprofile with a Prometheus listener (episode 19) that directly exposes metrics on port :9090.

echo "restic_last_success_timestamp $(date +%s)" > /var/lib/node_exporter/restic.prom
echo "restic_backup_duration_seconds $duration" >> /var/lib/node_exporter/restic.prom

#Alert on Failure

In Prometheus, the most important alert is the inverse of what it seems: not "status failed", but "no successful backup in the last X hours". This catches the case of a totally dead server or alerting that was never delivered:

groups:
  - name: restic
    rules:
      - alert: BackupStale
        expr: (time() - restic_last_success_timestamp{job="restic"}) > 86400
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Backup {{ $labels.instance }} has been without success for {{ $value }} seconds"

The second rule — alerting immediately when a backup process fails:

      - alert: BackupFailed
        expr: restic_backup_status{job="restic"} == 0
        for: 5m
        labels:
          severity: critical

#Notifications: Email, Slack, Telegram

Prometheus AlertManager routes alerts to many channels at once. Example Slack webhook:

receivers:
  - name: backup-team
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T000/B000/XXXX'
        channel: '#backup'
        title: '⚠️ {{ .GroupLabels.alertname }}'
        text: '{{ .Annotations.summary }}'
    email_configs:
      - to: 'ops@example.com'
        from: 'alert@example.com'
        smarthost: 'smtp.example.com:587'

The simplest way without AlertManager: the script calls the notification API directly on failure.

if ! restic backup /data; then
  curl -s -X POST "https://api.telegram.org/bot$TOKEN/sendMessage" \
    -d chat_id="$CHAT_ID" -d text="BACKUP FAILED on $(hostname)"
  exit 1
fi

#A Dashboard for the Whole Fleet

With metrics from each host (each labeled instance=hostname), Grafana can show one dashboard for everything:

• Fleet table: each host, last backup status, age of the last snapshot.
• Time series: backup duration per host per day — anomalies become visible.
• Dedup statistics: restic_stats_raw_data vs restore_size per repo.

Search for the community "restic" dashboard on grafana.com and adjust the labels to match your exporter.

#Conclusion

• Monitor status, duration, and data added — not just "success/failure".
• Export metrics via a node_exporter script or resticprofile's Prometheus.
• The best alert: "no success in X hours" — catches silent failures.
• Notify Email/Slack/Telegram via AlertManager or a direct script.
• Test the notification pipeline periodically — don't let channels die.
• One Grafana dashboard for the whole fleet; label per instance.

In the next episode, episode 21, we look ahead: roadmap & community — the development focus on performance/chunking and backend stability, the Go ecosystem contribution, and the community on GitHub, forum.restic.net, IRC, and the docs at restic.readthedocs.io.