Learning GitOps - FluxCD - Disaster Recovery & Backup

#Introduction

In episode 25 you built full observability for Flux — metrics, dashboards, logs, tracing, and alerting. You can see everything and know exactly when something starts going wrong. But there's one question that hasn't been answered: what happens when everything is lost? A cluster can be destroyed by a configuration error, a cloud region disaster, or an accidental deletion.

This is the time to be honest with ourselves: observability doesn't save data. All those beautiful dashboards and alerts mean nothing if there's no plan to rebuild from zero. A team that only focuses on monitoring without disaster recovery (DR) preparation will spend the next incident panicking, trying to figure out which steps to run.

In this episode we'll put together a real DR strategy for the Flux stack: backup strategies with Git, Velero, and etcd snapshots, Flux configuration backup specifically, cluster rebuild procedures from zero to running applications, and measurable DR testing with RTO and RPO. Because a plan that's never tested is just a collection of writing.

#Backup Strategies

#Git as Infrastructure Backup

The biggest advantage of GitOps is that it makes infrastructure inherently easy to back up. All the manifests — Kustomization, GitRepository, HelmRelease, and other configuration — live in Git repositories. As long as the repository is safe, the infrastructure definitions are never lost. This makes Git itself the cheapest and easiest-to-verify first backup layer: just make sure the repo is pushed and has the right history.

But Git isn't everything. Git stores desired state, not actual state. Ephemeral state — like application data, Secrets that aren't committed, and dynamically created resources — isn't in Git. That's why Git as a backup must be supplemented with other layers.

#Cluster State Backup with Velero

Velero is the standard tool for backing up Kubernetes cluster state. Velero snapshots Kubernetes resources (and optionally volumes) to external storage like S3 or GCS. A backup is created in a few steps:

velero install --provider aws --bucket gitops-backups \
  --backup-location-config region=ap-southeast-1
 
velero backup create full-backup \
  --include-namespaces default,payments,checkout

A Velero backup stores Kubernetes objects along with volume contents if configured. To restore, one command:

velero restore create --from-backup full-backup

Velero is suitable for restoring resources that aren't in Git — for example Secrets created imperatively or resources from other operators. Make sure automated backups are scheduled, not manual:

velero schedule create daily-backup --schedule "0 2 * * *" \
  --include-namespaces default,payments,checkout

#etcd Snapshot

etcd is the Kubernetes control plane database — where all cluster objects are stored. An etcd snapshot is the deepest layer: if etcd is corrupted, the entire cluster can die. For self-managed clusters, take periodic etcd snapshots:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot.db

Note that an etcd snapshot restores the cluster state to a specific point in time — including resources that have already been deleted. This differs from Git which always moves toward the desired state. In many cases, a healthy combination is an etcd snapshot as an emergency fallback and Git as the long-term truth.

#Configuration Export

Sometimes what's needed isn't a full backup, but an export of the current configuration. kubectl get with export options can capture the configuration of specific resources to keep as a record. This is useful as a documentation snapshot, but don't make it the only strategy — manual exports are easily forgotten and inconsistent.

#Backing Up Flux Configuration

#flux export

Flux provides a dedicated command to export all the Flux configuration from a cluster:

flux export > flux-export.yaml

flux export produces all the Flux custom resources — Kustomization, GitRepository, HelmRelease, Provider, Alert, and more — in a single YAML file. This is the fastest way to capture "who reconciles what" before a cluster changes or is lost. Run it periodically and store the output in a safe location outside the cluster.

#Backing Up the GitOps Repository

The GitOps repository itself is the most valuable asset. Backing it up means ensuring:

• The remote repository (GitHub, GitLab) has automatic backup or a mirror.
• Every developer has a complete clone that's re-pushed periodically.
• Tags and releases aren't deleted by a wrong cleanup operation.

For self-hosted repositories or local mirrors, schedule periodic backups:

git clone --mirror \
  https://github.com/devvnull/gitops-production.git \
  /backup/gitops-production.git

A mirror clone stores all references and history — even inactive branches — so the repository can be restored even if the main remote is lost.

#Backing Up Encrypted Secrets

Secrets are the hardest part to back up: storing them in plaintext in Git violates GitOps principles. The solution is encryption — as you already learned in the SOPS and Sealed Secrets episodes. The correct pattern:

• Encrypt in the repo: Secrets are committed in encrypted form (SOPS) or sealed (Sealed Secrets) so they're safe to store in Git and automatically backed up.
• Back up the keys: the decryption keys (e.g. PGP keys for SOPS, or Sealed Secrets private keys) are stored separately and securely — for example in a vault or secret manager.
• Back up Secrets from the cluster: for Secrets that aren't in Git, export them encrypted to external storage.

The golden rule: keys and encrypted data must never be stored in the same place. If both are lost together, the encryption is no longer useful.

#Disaster Recovery

#Cluster Rebuild Procedure

Disaster recovery starts from the worst assumption: the cluster can't be saved and must be rebuilt. A good rebuild procedure is a documented, repeatable sequence of steps:

1. Provision a new cluster — network, nodes, and control plane (could be with Terraform, discussed in episode 29).
2. Prepare access — kubectl configuration, and make sure the Git repository credentials are valid.
3. Bootstrap Flux — run the bootstrap so the controllers are installed.
4. Let Flux work — the root Kustomization pulls all the configuration from Git.
5. Restore non-Git state — restore Velero for data and resources that aren't in Git.
6. Verify — run smoke tests to make sure the applications actually work.

This order is important: Flux must reconcile before the Velero restore, so it doesn't overwrite the Git desired state with the old state.

#Re-bootstrapping Flux

Re-bootstrapping Flux into a new cluster is a step that must go smoothly if the GitOps repository is maintained properly:

flux bootstrap github \
  --owner=devvnull \
  --repository=gitops-production \
  --branch=main \
  --path=clusters/production

The bootstrap reads the Kustomization from clusters/production/flux-system and then reconciles everything referenced. As long as the repo structure is consistent and the referenced secrets (for example the webhook token) are available, the cluster will rebuild itself.

#Application Restoration

Once Flux is running, applications are restored in two streams. The first stream from Git: application manifests are deployed directly by Flux. The second stream from backup: state that isn't in Git, especially data in volumes and Secrets. The combination of both is why a dual backup strategy (Git + Velero) is needed.

During the restore, pay attention to the dependency order: a database must be recovered before the applications that depend on it are considered healthy. Use flux get kustomization to monitor the reconciliation order and make sure everything reaches a Ready status:

flux get kustomization -A
kubectl get pods -A | grep -v Running

#State Reconciliation

After the restore, the cluster may have differences between the Git state and the restored state. Flux will immediately reconcile these differences — sometimes that's undesirable. For example, a Velero restore that brings back old resources can be immediately overwritten by Flux toward the Git desired state, or the reverse.

The key to avoiding conflicts: suspend the Kustomization temporarily during the restore, then resume once the state is fully restored:

flux suspend kustomization apps
velero restore create --from-backup full-backup
flux resume kustomization apps

This gives full control: restore the data first, then let Flux adjust the configuration toward the desired state.

#Testing DR

#DR Drills

A DR plan that's never tested is just text. A DR drill is a real test of the rebuild procedure: create a new cluster, run the whole procedure from zero, and measure how long it takes. Schedule periodic drills — for example quarterly — with gradually increasing scope: from only restoring Flux, to restoring Flux plus applications, to restoring applications plus data.

Drill results always bring findings: a forgotten bootstrap flag, an unavailable secret, or a step whose order is wrong. Every finding must be fixed in the procedure and tested again in the next drill.

#RTO and RPO

Two metrics that make DR measurable:

• RTO (Recovery Time Objective) — the maximum target time to restore a service after an incident. RTO determines how fast the rebuild procedure must run.
• RPO (Recovery Point Objective) — how much data may be lost. RPO determines how often backups are taken.

Their relationship to the strategy:

Set realistic targets based on business impact, then measure drills against those targets.

#Automation Scripts

The more steps that are automated, the fewer human errors during panic. DR automation can be:

• A rebuild script that runs provision + bootstrap in one command.
• A CI workflow for restore that produces an RTO/RPO report when done.
• An automated verification checklist that runs smoke tests after the restore.

An example of a simple script combining bootstrap and verification:

flux bootstrap github \
  --owner=devvnull \
  --repository=gitops-production \
  --branch=main \
  --path=clusters/production \
  --timeout=15m
 
flux get kustomization -A --wait
flux tree kustomization flux-system --namespaces=flux-system

flux get kustomization --wait waits for all resources to be ready; flux tree shows the dependency structure. The same script can be the basis for DR drills and production verification.

#Closing

In this episode 26 you put together a disaster recovery strategy for the Flux stack: Git as the cheapest infrastructure backup, Velero for cluster state and volumes, etcd snapshots as the deepest layer, exporting and backing up Flux configuration including encrypted Secrets, cluster rebuild procedures with re-bootstrap and application restoration, and DR testing measured with RTO and RPO.

The key takeaways:

• Git stores desired state, not actual state — supplement it with Velero for data and non-Git resources.
• Export the Flux configuration periodically with flux export and make sure the GitOps repository has a mirror.
• Encryption keys and encrypted data must be separated — encryption without a recoverable key is just a false sense of security.
• The recovery order is critical — bootstrap Flux first, reconcile, then restore Velero; use suspend to control the timing.
• A DR plan is tested, not stored — measure drills against RTO and RPO, and automate as many steps as possible.

Your clusters can now be recovered at any time. But the lifecycle doesn't stop at manual deployment — it's time to connect everything to a pipeline. In the next episode, episode 27, we'll discuss CI/CD Pipeline Integration — separating CI and CD responsibilities, building a pipeline from checkout to security scanning, connecting CD to GitOps, and integration with GitHub Actions, GitLab CI/CD, and Jenkins. Keep up the momentum!