Learn GitOps with ArgoCD - Disaster Recovery & Backup

#Introduction

In episode 20 we built a service mesh fully managed from Git — and hidden in there is GitOps's most underappreciated strength: everything that governs system behavior is already stored in Git. If the cluster vanished tonight, what would remain? Almost everything. In this episode we discuss disaster recovery & backup — not just procedures, but a way of thinking: how fast you can get back up (RTO) and how much data you're willing to lose (RPO).

Why does this matter? Disaster recovery is the only feature that can't be tested when it's actually needed. Many teams have a DR plan that's "just run the script", and only realize the script is broken when the cluster is gone and the script hasn't been run in six months. GitOps gives a structural advantage: because configuration lives in Git, recovery is a matter of reassembling, not rebuilding. Our job is to make sure the parts that aren't configuration — secrets, ArgoCD state, and data — are protected too.

#Git as the First Backup

Before building backup infrastructure, remember that Git itself is the best configuration backup ever made. Every Application, Project, application manifest, and mesh configuration is already stored as a commit with full history. For stateless applications, DR could even be as simple as:

1. Create a new cluster.
2. Install ArgoCD.
3. Register the repo and cluster (from episode 9).
4. Create Applications with the same source — ArgoCD syncs everything.

This is why the GitOps principle "declarative configuration in Git" isn't just cleanliness — it's a DR policy. If you find configuration that is not in Git (for example an application created manually via the UI without a commit), that's a DR risk that must be fixed immediately.

#Exporting ArgoCD Configuration

Although Git stores the Applications, there are parts of ArgoCD that aren't manifests: credentials (repo SSH keys, cluster tokens), RBAC, and argocd-cm settings. For full recovery, export all of them:

Exporting ArgoCD configuration

argocd app list -o name > backup-apps.txt
argocd proj list -o name > backup-projects.txt
argocd repo list -o name | awk '{print $1}' > backup-repos.txt

But manual export is fragile and easy to forget. A more robust approach is managing ArgoCD itself with GitOps (the self-managed pattern from episode 6): argocd-cm, argocd-rbac-cm, repo secrets, and cluster secrets stored as encrypted manifests (from episode 12) in the argocd-config repo. That way, recovering ArgoCD = kubectl apply -f bootstrap + one ApplicationSet.

#Backup Scripts and Automatic Scheduling

For state that isn't fully in Git, create a scheduled backup script. This script exports ArgoCD resources and cluster resources to YAML, then stores them in a bucket or repo:

#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="/backup/$(date +%Y%m%d-%H%M)"
mkdir -p "$BACKUP_DIR"
kubectl get applications,appprojects -n argocd -o yaml > "$BACKUP_DIR/argocd-config.yaml"
kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=repo \
  -o yaml > "$BACKUP_DIR/repos.yaml"
kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=cluster \
  -o yaml > "$BACKUP_DIR/clusters.yaml"

Schedule it with a CronJob on a cluster different from the one being backed up (the backup must not be lost together with the cluster):

Daily backup CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: argocd-backup
  namespace: argocd
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: bitnami/kubectl:latest
              command: ["/backup/backup.sh"]
              volumeMounts:
                - name: script
                  mountPath: /backup

#Disaster Recovery Procedure

When a cluster is truly gone, follow this order — from the least dependent on what was lost:

1. Register the repo in the new place. Make sure Git access still works; if the token/SSH key was revoked along with the cluster, request new ones.
2. Install ArgoCD. Use Helm or the latest manifests (episode 4).
3. Restore the configuration. Apply argocd-config.yaml, repos.yaml, and clusters.yaml from the latest backup.
4. Restore the applications. Apply the Application/ApplicationSet files; ArgoCD pulls the manifests from Git and syncs.
5. Reconcile state. Run argocd app sync -l app.kubernetes.io/part-of=myapp then verify the health of all applications:

Sync all applications after restore

argocd app list
argocd app sync -l app.kubernetes.io/instance=prod
argocd app wait --health

All these steps work because Git is the source of truth. A new cluster knows no history — it just pulls the desired state and materializes it. Stateless applications recover as fast as ArgoCD finishes syncing; stateful applications need data restore (Velero, database snapshots) which is outside ArgoCD's scope.

#Multi-Cluster DR

For stricter SLOs, a single cluster isn't enough. Two main patterns:

In the GitOps pattern, both are realized with one repo, two ApplicationSets — each targeting a different cluster (the cluster generator from episode 11). Active-passive failover means moving DNS traffic to the secondary cluster that has already synced the same configuration; because Git is identical, both clusters behave identically.

#Testing DR: RTO and RPO

A DR plan without testing is fiction. Two metrics you must measure:

• RPO (Recovery Point Objective) — how fresh the data is when restored. Determined by backup frequency: daily backup means a maximum RPO of 24 hours.
• RTO (Recovery Time Objective) — how fast the system returns. Measured with a DR drill: shut down the staging cluster, bring up a new one, and record the time.

Best practice: schedule a DR drill at least quarterly, automate it with "chaos" scenarios (episode 26), and document the results. The documentation must answer: who decides the failover, which commands are run, where traffic is moved, and how completion is verified. Every drill that extends the RTO is feedback to simplify the procedure.

#Closing

This episode made DR part of GitOps design: Git as the first configuration backup, ArgoCD configuration exports, backup scripts and automatic CronJobs, encrypted secret backups, recovery procedures from an empty cluster, active-passive and active-active multi-cluster patterns, and RTO/RPO measurement through regular DR drills.

The points you should take with you:

• Configuration in Git is the most valuable DR backup; what remains is credentials, secrets, and state.
• Manage ArgoCD with GitOps so recovering ArgoCD is as easy as recovering an application.
• Backups are scheduled and stored where they won't be lost with the cluster.
• Raw secrets must not enter Git; use encryption in the repository.
• A DR that is never tested is a DR that doesn't exist — run regular drills and measure RTO.

A cluster that can recover from scratch lets you sleep well. But how do you know the cluster is healthy, and how do you see the signs of trouble before they become disasters? In the next episode 22 we discuss monitoring & observability — ArgoCD metrics in Prometheus, Grafana dashboards, alerting, and log aggregation. See you in episode 22!