Learn Vitess - Replication & Failover

#Introduction

A database without availability is just a data warehouse. Episode 6 covers how Vitess keeps data available when hardware fails, processes die, or nodes reboot: through replication (copies of data) and failover (transfer of the primary role). This is a big difference between Vitess and vanilla MySQL — failover in Vitess is automatic and designed to minimize downtime.

Episode 6 roadmap: replica topology and the primary role, failure detection, automatic failover with vtctld and pseudo-GTID, then backup and recovery strategies. By the end, you'll understand what happens behind the scenes when a primary dies.

#Replica Topology and the Primary Role

Within one shard, there's one primary (accepts writes and reads) and several replicas (copy writes from the primary via binlog, serve reads and backups). Vitess calls these roles tablet types: PRIMARY, REPLICA, and RDONLY.

vtctlclient ListAllTablets

The output of vtctlclient ListAllTablets shows each tablet with its role type. A healthy shard 0 usually has one primary tablet and one or more replica tablets.

The difference between REPLICA and RDONLY: both read the same data, but REPLICA is considered promotable to primary, while RDONLY is explicitly never promoted — suitable for heavy read workloads like reporting that must not interfere with the failover path. Vitess directs read traffic to replicas through VTGate, and applications can request "read after write consistency" to make sure reads through a replica don't fall behind.

A tablet's role isn't permanent — it can be changed as needed. For example, when reporting load rises, you can change a REPLICA tablet to RDONLY, or vice versa. Vitess handles this transition without stopping the tablet's service.

vtctlclient ChangeTabletType <tablet-alias> RDONLY

The vtctlclient ChangeTabletType command changes a tablet's role — a useful tool when adjusting read load distribution without adding new instances. Still, remember that turning all REPLICAs into RDONLY removes failover candidates, so always keep at least one REPLICA per shard.

#Failure Detection and Automatic Failover

When the primary dies, writes stop — and every second of downtime is a loss. Vitess detects failures by monitoring tablet health periodically. Multiple parties do the monitoring: vtctld monitors health through the Topology Service, and VTGate monitors the health of every tablet it uses.

Failure detection doesn't mean immediate failover. There's a grace period — a waiting time that gives the primary a chance to come back before failover is triggered. The goal is to avoid unnecessary failover when the network is merely delayed for a moment.

When failover is triggered, the sequence of events looks roughly like this:

1. Health monitoring marks the primary as unhealthy.
2. After the grace period, vtctld selects the best replica as the replacement.
3. The chosen replica runs EmergencyReparentShard and is promoted to primary.
4. The old primary tablet (if it comes back) is demoted to replica and catches up.

#Pseudo-GTID and the Catch-up Mechanism

This is the most technical and interesting part. Traditional MySQL replicates with file position (binlog file + binlog pos). The problem: after failover, a new replica may be at a position that doesn't match other replicas — a file position is relative to the old primary, not the data itself.

Vitess uses pseudo-GTID: a mechanism to find the replication position based on data content rather than file position. Every few thousand transactions, Vitess inserts a unique searchable row. With this, Vitess can pinpoint exactly which transaction a tablet has replicated up to — precisely, and independent of file position.

Pseudo-GTID makes resharding and reparenting far more reliable: Vitess can attach replication from one tablet to another without worrying about skipping or replaying transactions.

vtctlclient PlannedReparentShard -keyspace_shard=commerce/0 \
  -new_primary=<tablet-alias>

The vtctlclient PlannedReparentShard command moves the primary role to a new tablet in a planned manner — applications only feel a brief pause during the transition.

#Backup and Recovery Strategies

Failover handles instant failures. For bigger disasters — for example, all nodes in an availability zone dying — you need backups. Vitess provides a backup engine that runs through VTTablet, storing data snapshots in object storage (e.g., GCS, S3, or Azure Blob).

Vitess backup flow:

• One REPLICA or RDONLY tablet is selected to take the backup, so the primary isn't disturbed.
• VTTablet uses mysqlbackup or another tool to create a consistent snapshot.
• The snapshot is shipped to the storage location configured in vtctld.

To trigger a manual backup:

vtctlclient Backup <tablet-alias>

Restore happens when a tablet is bootstrapped or when creating a new tablet: Vitess detects the latest backup, restores it, then catches up on replication from the primary. The backup storage location must be prepared at install time — helm install of Vitess requires you to specify the backup bucket.

#Durability Policy and Semi-sync

One key decision in failover is the durability policy: how sure you are that a write is safe before VTGate returns success to the application. Vitess supports several policies:

• none — a write is considered successful once it hits the primary, with no replication guarantee. Fastest, but high risk.
• semi_sync — a write waits for at least one replica to acknowledge before succeeding. The most commonly used: trading a little latency for far better failover safety.
• none with RPO — a combination teams need to be aware of: without semi-sync, failover can lose the last transactions that haven't been replicated.

vtctlclient GetKeyspaceDurabilityPolicy commerce

vtctlclient GetKeyspaceDurabilityPolicy displays a keyspace's durability policy. This choice directly affects RPO (how much data may be lost during failover) — we'll revisit it in episodes 14 and 20.

#Closing

In this episode 6 you understood primary and replica topology in Vitess, failure detection and automatic failover via PlannedReparentShard and EmergencyReparentShard, the role of pseudo-GTID in accurately tracking replication position, and object-storage-based backup and recovery strategies.

Key takeaways:

• One shard has one primary and several replicas with the REPLICA and RDONLY types.
• RDONLY can never be promoted; it's for heavy read workloads.
• Failover has a grace period so it doesn't fire on false detections.
• Pseudo-GTID uses data content, not file position, to find the replication point.
• PlannedReparentShard for maintenance, EmergencyReparentShard for emergencies.
• Back up from a replica, store to object storage, and test restores regularly.

In the next episode, episode 7, we put eyes and ears on the cluster: observability and debugging — Prometheus metrics from VTGate and VTTablet, log patterns, slow query tracing, and diagnosing shards with vtctlclient. See you there!