Learn Vitess - Production Hardening & Best Practices

#Introduction

Episode 21 closed with SLOs and alerting — the mark of a measured and maintained service. Episode 22 is the final episode of the Learn Vitess series. There are no heavy new concepts; what remains is filtering and summarizing. Everything you've learned across the previous 22 episodes gets locked into one goal: a Vitess that's secure, managed, and built to last for years in production.

Episode 22 roadmap: a security hardening checklist, scaling and failover readiness, runbooks and ownership, then safe Vitess and MySQL upgrades, closing with a reflection on the journey from episode 0 to here. Treat this episode as an open-book final exam — not to be memorized, but to be used when you actually run Vitess in production.

#Security Hardening Checklist

Hardening isn't one big step, but a collection of small complementary steps. The checklist below can be run as a gate before release or as a periodic evaluation.

Control plane. Limit who can touch Vitess. vtctld and the Topology Service should only be reachable from an internal network or VPN, not publicly exposed. Apply the strictest Kubernetes RBAC you can actually operate, and make sure dashboards and admin APIs can't be reached by applications. Secure admin credentials with scheduled rotation.

Data plane and transport. Make sure TLS is active on all hops: client to VTGate, VTGate to VTTablet, and VTTablet to MySQL (episode 12). Verify NetworkPolicies restrict traffic to only the needed ports and namespaces. Make sure application users have no DDL rights and direct access to tablets is forbidden (episode 13).

Secret management. No secrets in git, no passwords in values.yaml. MySQL, topology, and backup credentials are managed through Kubernetes Secrets and an external secret manager (episode 11). Rotate passwords regularly and audit who can read secrets.

vtctlclient ListShardHealth
vtctlclient GetTopologyInfo
kubectl get networkpolicies -n vitess
kubectl get secrets -n vitess

vtctlclient ListShardHealth verifies shard health, GetTopologyInfo ensures the topology is consistent, kubectl get networkpolicies confirms network isolation is active, and kubectl get secrets lists the secrets present. Run these commands in CI or periodic audits, not just during incidents.

#Scaling and Failover Readiness

Scale and availability can only be trusted if tested. Readiness checklist for production:

Scaling. The sharding key design is final and stable (episode 4). The primary vindex was chosen correctly, and hot-path queries don't scatter (episode 5). Shard capacity is monitored and resharding is planned before a shard fills up (episode 16). RDONLY replicas are available on every shard for reporting and backup (episode 17).

Failover. The PlannedReparentShard and EmergencyReparentShard procedures are documented and have been tested (episode 6). Replication lag is monitored with alerts (episode 21). Backups are restore-tested regularly, and topology backups are recoverable (episodes 18 and 20). A standby region is ready if DR is needed (episode 14).

#Runbooks and Ownership

Operational documentation is the asset most often forgotten until an emergency. A good runbook answers the question: "when X happens, who does what, in what order?"

Suggested runbook structure for every important scenario:

• Symptoms — the metrics and logs that indicate the problem.
• Impact — what's disrupted and how severely.
• Actions — an ordered set of remediation steps, with exact commands.
• Escalation — who to contact if the steps don't work.
• Verification — evidence that the problem is resolved.

Scenarios that must have a runbook: primary failure, region outage, severe replication lag, failed resharding, failed backup, and inconsistent topology.

Ownership complements runbooks: every component and every runbook has a clear owner. Without an owner, maintenance gets ignored. Assign owners for VTGate, VTTablet, Topology Service, backups, and schema. Owners are responsible for the health and documentation updates of their respective areas.

#Upgrading Vitess and MySQL Safely

Upgrades are among the riskiest — and most routine — operations. The principle: plan, test, then execute in stages. Upgrade checklist:

• Read the release notes: know the behavior changes and deprecations in the new version.
• Upgrade staging first: test in an environment that resembles production.
• Full backup: back up data and topology before starting.
• Follow the order: VTGate first (stateless), then VTTablets in stages.
• Monitor: compare metrics before and after (episode 21).
• Rollback plan: know how to back out if the upgrade has problems.

helm list -n vitess
helm upgrade vitess vitess/vitess -f values.yaml --namespace vitess --version 21.0.1
kubectl rollout status deploy/vtgate -n vitess

helm list -n vitess shows the installed version, helm upgrade raises the version, and kubectl rollout status waits until it finishes. MySQL upgrades are similar: verify version compatibility with Vitess, test in staging, then upgrade shard by shard in stages.

#Series Journey Reflection

Let's pause and look back at the path you've traveled.

Phase 1 (episodes 0-2) built the foundation: environment, Vitess history, and the architecture of VTGate, VTTablet, Topology Service, keyspaces, shards, and vindexes. Phase 2 (episodes 3-7) made you comfortable with basic operations: installation, keyspaces and shards, query routing, replication and failover, and observability.

Phase 3 (episodes 8-11) raised the depth: schema and vindexes, transactions and consistency, online migration, and configuration and secrets. Phase 4 (episodes 12-14) opened the security and networking arena: TLS, access control and audit, and multi-region and DR.

Phase 5 (episodes 15-18) took Vitess to an advanced level: performance tuning, resharding, hybrid workloads and OLAP, and control plane and topology. Phase 6 (episodes 19-22) closed the loop: CI/CD and release, disaster recovery, SLOs, and the hardening you've just assembled.

Along the way, there's a recurring pattern: master the vindex before sharding, the fast path before scatter, observe before changing, back up before upgrading, and document before panicking. These patterns aren't specific to Vitess — they apply to any database platform. What you've mastered in this series isn't just feature by feature, but a complete way of operational thinking.

#Closing

This journey is finished, and it's worth remembering as a whole. You started from the most basic questions in episode 0, got to know Vitess and its architecture in episodes 1-2, then gradually mastered operations, security, observability, all the way to large scale and production readiness. Now you no longer ask "what is a shard?", but rather "how is this keyspace sharded, secured, and scaled over the years?".

Key takeaways from the whole series:

• Vitess wins on architecture: stateless VTGate, VTTablet managing MySQL, and Topology Service as the source of truth.
• Good sharding starts with design: choose a stable sharding key, and make sure hot-path queries don't scatter.
• Every transaction wants to be single-shard: co-location is the most powerful design principle.
• Observability, SLOs, and alerting aren't extras — they're the key to changing production with confidence.
• Tested backups, rehearsed failover, and planned upgrades keep Vitess healthy in the long run.
• Documented and executed processes survive; individual habits don't.

Thank you for staying until the final episode. Every concept in this series is now yours — time to go out, build scalable and well-managed databases with Vitess, and keep learning from every incident you handle. The next step is in your hands: run it in the lab, make small mistakes, and let each mistake strengthen the runbook. See you in the next series!