Learn Vitess - Disaster Recovery & Business Continuity

#Introduction

Episode 14 covered multi-region DR design. Episode 20 gets practical: how to make sure that plan actually works when called on. Disasters aren't an "if", they're a "when" — and the only way to ensure recovery works is to test it regularly, before a real disaster happens.

Episode 20 roadmap: full and partial backup strategies, recovery drills, data integrity checks, then handling region outages with failover rehearsals.

#Full and Partial Backup Strategies

Backups are the raw material of recovery. A healthy strategy combines several types:

Full backup: a snapshot of the entire data. Rare, because it's expensive — usually daily or weekly.
Incremental backup: only changes since the last backup. Frequent, cheap, but trickier to recover.
Log shipping / replication: a continuous copy (replica or binlog) for an RPO near zero.

Schedule regular backups

vtctlclient Backup <tablet-alias-rdonly>

vtctlclient Backup triggers a backup from an RDONLY tablet. For automation, schedule it with cron or a Kubernetes CronJob. The standing rule: backups must come from RDONLY, be stored in a location separate from the cluster, and be verified.

Info

The ideal combination: weekly full backups, daily incremental backups, plus cross-region replication for low RPO. These three layers measure data loss in seconds, not hours or days.

Retention strategy also matters: keep several full backup cycles to protect against data corruption only detected weeks later. Backups deleted too quickly mean you can't go back to a point before the data was damaged.

#Recovery Drills

A recovery drill is the real test of a backup strategy. The golden rule: a backup that's never been tested can't be trusted. Regular practice includes:

Build a new cluster from backup (full restore).
Verify the data can be queried.
Measure the restore time — is it within the RTO target?
Note obstacles and fix the runbook.

Restore from backup when bootstrapping a tablet

vtctlclient RestoreFromBackup <tablet-alias>

vtctlclient RestoreFromBackup restores a tablet from the latest backup — the same process used when a new tablet is created. Test this command regularly in a separate environment, not just in emergencies.

Warning

Never test a restore in a production cluster in a way that could damage data. Use a separate cluster or a staging environment for drills. The goal of a drill is to prove the process, not add risk.

#Data Integrity Checks

After a restore, data must be validated — a technically successful restore doesn't mean the data is correct. Ways to check integrity:

Checksums or hashes on important data, compared against expected values.
Table counts: row counts must be consistent between source and target.
Query sampling: run business queries with known results.
MySQL CHECK TABLE for structural integrity.

Check table integrity

CHECK TABLE users

CHECK TABLE users checks table integrity in MySQL. For more critical data, combine it with application validation: run tests that compare query results before and after the restore.

#Handling a Region Outage

When a region genuinely goes down, you need a prepared sequence of actions, not improvisation. The usual sequence:

Detection and confirmation — make sure the region is truly down, not just a network hiccup.
Impact assessment — what data is affected, how much traffic, for how long.
Failover execution — promote a primary from the standby region (episode 14).
Gradual recovery — bring service back, monitor load.
Incident review — record what happened and improve the process.

Assess impact during an outage

vtctlclient ListAllTablets -cell <region-bermasalah>
vtctlclient ListShardHealth -cell <region-cadangan>

vtctlclient ListShardHealth in the standby region gives a picture of whether data is ready to be made primary. This is the second step before a failover decision.

#Failover Rehearsal

A failover rehearsal is a scheduled version of a drill: simulating a region failure and running the entire failover procedure. Unlike a recovery drill, a rehearsal focuses on the operational process — who does what, when, and through which channel.

Rehearsal checklist:

A complete, up-to-date failover runbook.
Responsible personnel identified and available.
Incident communication tested (who is notified, how).
Recovery metrics recorded for comparison.
Findings from the rehearsal flow into runbook improvements.

Success

A drill that finds problems isn't a failure — it's a success. The more problems found during rehearsal, the fewer surprises during a real disaster. The calmest team during an incident is the team that practices the most.

#Closing

In this episode 20 you prepared for worst-case scenarios: full and partial backup strategies with sensible retention, regular recovery drills proving backups can be restored, data integrity checks after restores, handling region outages with a sequence of actions, and failover rehearsals that forge the runbook.

Key takeaways:

Combine full backups, incremental backups, and replication for low RPO.
A backup whose restore has never been tested can't be trusted.
Restores must always be followed by data integrity validation.
A region outage needs a planned sequence of actions, not improvisation.
Failover rehearsals forge the runbook and calm the team during real incidents.
Drill findings are process improvements, not failures.

In the next episode, episode 21, we measure and honor our promises: observability at scale and SLOs — defining SLOs for latency, availability, and throughput, monitoring shard health, and alerting on replication lag, primary failure, and query errors. See you there!