Learn Velero - Disaster Recovery Plan

#Introduction

You've mastered eight episodes of technical practice: backup, restore, schedule, hooks, migration. Episode 12 lifts everything up a level: strategy. Because in reality, companies don't go bankrupt from losing data — they go bankrupt from not being able to recover within the time they promised.

Imagine a fire department with the best hoses and pumps, but no evacuation plan. When the fire comes, all the equipment is there but no one knows the sequence. A DR Plan is that "evacuation plan": agreed numbers, documented steps, and regular drills.

#RPO and RTO

#Definitions

Two metrics you must agree on with stakeholders before building anything:

• RPO (Recovery Point Objective): how much data loss is acceptable. If RPO is 24 hours, a daily backup is enough; if RPO is 1 hour, hourly backups are needed.
• RTO (Recovery Time Objective): how fast the service must be back. If RTO is 4 hours, the entire restore process must finish under that.

+----------------+-------------------------------------+
| RPO 24 hours   | daily schedule, long enough TTL     |
| RPO 1 hour     | hourly schedule, TTL 48h            |
| RPO ~0         | cloud snapshot + periodic logical   |
|                | dump                                |
| RTO 4 hours    | automated runbook, Velero image     |
|                | ready                               |
| RTO < 1 hour   | DR standby cluster + deploy         |
|                | pipeline                            |
+----------------+-------------------------------------+

#Translating into Velero Configuration

velero schedule create hourly \
  --schedule="0 * * * *" \
  --include-namespaces prod \
  --ttl 72h \
  --default-volumes-to-fs-backup

RPO 1 hour = hourly schedule. RTO 4 hours demands a fast restore process: Velero installed in the DR region, credentials ready, and a proven runbook.

#Off-site Backup: Different Region

#The 3-2-1 Principle for Kubernetes

The classic backup principle applies to clusters too: 3 copies, 2 different media, 1 in another location. For Kubernetes, the translation:

• 3 copies: data in PVCs, backup in the primary BSL, backup in the secondary BSL.
• 2 media: primary object storage (hot) + archive (e.g. glacier/archive).
• 1 off-site: a bucket in a different region (or a different cloud).

#Configuring a Second BSL

velero backup-location create dr-region \
  --provider aws \
  --bucket velero-backups-dr \
  --config region=ap-southeast-1

Or for cross-cloud — a GCS BSL read during DR on GCP:

velero backup-location create gcs-dr \
  --provider gcp \
  --bucket velero-dr-backups

#Restore Runbook

A runbook is an instruction that anyone can execute — including someone new on the night shift. Compose at least the following steps:

1. Detect the incident and escalate to on-call.
2. Prepare the DR cluster: new cluster in the DR region, correct kubectl context.
3. Install Velero 1.18 with the same plugins:
   velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.14.0 \
     --bucket velero-backups-dr --secret-file ./credentials-velero \
     --backup-location-config region=ap-southeast-1
4. Verify BSL Available and backups visible:
   velero backup-location get && velero backup get
5. Restore with the appropriate mapping:
   velero restore create dr-restore --from-backup <latest> \
     --storage-class-mappings gp2:gp3
6. Verify: velero restore logs, kubectl get pods, check volume data.
7. Switch traffic to the DR cluster, update DNS/Ingress.
8. Record start-finish times for the RTO report.

Keep this runbook in a repository (e.g. your GitOps repo) and make it a commit-able checklist — not in someone's head.

#DR Drill: Regular Practice

#Why It Must Be Practiced

A backup that's never been restored on a new cluster is just a guess. A DR drill turns assumptions into facts: are the credentials still valid? Are the plugin images available? How long does restore actually take? These are only answered by practice.

#Drill Procedure

kubectl config use-context dr-cluster
velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.14.0 \
  --bucket velero-backups-dr --secret-file ./credentials-velero \
  --backup-location-config region=ap-southeast-1
velero restore create drill-restore --from-backup <latest>

Also test the worst-case scenario: delete the source cluster, then restore to a new cluster from zero. This is the only way to prove your backups are truly independent of their origin cluster.

#Completing DR with Monitoring

A backup that silently fails is the most dangerous leak. Schedule automatic checks:

velero backup get | grep -v Completed
kubectl logs -n velero deploy/velero --tail=50

We cover full monitoring in episode 20. For episode 12, just build the habit: a backup that isn't monitored is the same as no backup.

#Closing

Key takeaways:

• RPO = how much data may be lost; RTO = how fast you must recover — both determine your schedule and TTL configuration.
• Apply 3-2-1: off-site backup to a BSL in a different region/cloud.
• Write a restore runbook anyone can execute, and keep it in a repository.
• Regular DR drills turn assumptions into real numbers; delete the cluster → restore to a new one.
• Monitor backups — a silently failing backup helps no one.

In episode 13 next, we enter the security phase: Credential Security & RBAC — storing cloud credentials properly (never commit them!), using IRSA on EKS and Workload Identity on GKE, and restricting user access to the velero namespace.