Learn Vitess - Configuration Management & Secrets

#Introduction

Your Vitess cluster is made of dozens of components whose configurations are interconnected: VTGate, VTTablet, Topology Service, and MySQL. Changing one value in one component can affect the entire cluster. Episode 11 covers how to manage this configuration centrally, store credentials securely, and roll back problematic changes.

Episode 11 roadmap: customizing configuration with Helm and ConfigMaps, managing secrets, validating configuration, then rollback strategies for production changes.

#Customizing Configuration with Helm and ConfigMaps

The Vitess Helm chart is customized through the values.yaml file. Important values you'll often change: resource sizes per component, replica counts, backup storage location, and extra flags for VTGate or VTTablet.

values-custom.yaml

vtgate:
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
vtctld:
  resources:
    requests:
      cpu: 250m
      memory: 256Mi
vttablet:
  resources:
    requests:
      cpu: 500m
      memory: 1Gi

Keep this values.yaml file in the repo, then apply it during install or upgrade:

Install with custom values

helm install vitess vitess/vitess -f values.yaml \
  --namespace vitess

Helm translates values.yaml into ConfigMaps and Deployments in the cluster. Configuration scattered across many manifests becomes a single source of truth that can be reviewed and version-controlled.

Info

Important principle: never edit a ConfigMap directly in the cluster. Change it in the values.yaml file in git, then apply it via helm upgrade. Configuration that only exists in the cluster and not in git will be lost and hard to audit.

To inspect the active values a release is using:

View active helm values

helm get values vitess --namespace vitess

#Managing Secrets

Vitess uses many kinds of credentials: the MySQL root password, Topology Service credentials, object storage backup accounts, and credentials for VTGate to VTTablet. The first and most important rule: never put a secret in values.yaml or a ConfigMap.

Use a Kubernetes Secret, and if possible back it with an external secret manager like Vault. Example of creating a secret for the root password:

Create a secret from literals

kubectl create secret generic vitess-mysql \
  --namespace vitess \
  --from-literal=root-password='p4ssw0rd-rahasia'

kubectl create secret generic creates a Secret that Vitess manifests reference. With an external secrets operator, secrets can be pulled automatically from Vault or a cloud secret manager, so passwords never end up in git.

Danger

Make sure secrets never enter git. If a secret was ever committed, treat it as leaked and rotate it immediately. In this repo, .env* and secret files should already be in .gitignore.

When setting up VTGate's access credentials to MySQL, define the user and password in the same Secret:

Create a mysql user secret

kubectl create secret generic vitess-users \
  --namespace vitess \
  --from-literal=vt_appuser=appuser \
  --from-literal=vt_appuser_password='rahasia-app'

#Validating Configuration

Before applying configuration to production, validation is a mandatory gate. A few ways:

YAML validation: make sure values.yaml can be parsed. Helm handles this during helm template.
Helm dry-run: helm template or helm upgrade --dry-run shows the resulting manifests without applying them.
VSchema and DDL validation: Vitess has commands to validate a schema before applying it.

Dry-run before upgrade

helm upgrade vitess vitess/vitess -f values.yaml \
  --namespace vitess --dry-run --debug

helm upgrade --dry-run validates the manifests and shows the diff that would occur — a chance to review before any real change. Combine this with schema validation: vtctlclient ValidateSchema makes sure the schema is consistent across all shards.

Validate schema across shards

vtctlclient ValidateSchemaKeyspace users

vtctlclient ValidateSchemaKeyspace checks that all tablets in a keyspace have the same schema — if not, Vitess flags it as drift that needs fixing.

#Rolling Back Production Changes

Rollback is the safety net. Because configuration is managed by helm, rollback becomes easy:

Rollback helm to a previous version

helm list -n vitess
helm history vitess -n vitess
helm rollback vitess 3 -n vitess

helm history vitess shows the release history, and helm rollback vitess 3 returns the release to revision 3. This process reconstructs the entire configuration from helm's revision snapshot — far safer than manually reverting a single ConfigMap.

For changes made through vtctlclient (for example, a wrong ApplyVschema), Vitess has no automatic undo — this is why keeping configuration in git matters. Always keep VSchema and values files in the repo, and apply changes through a pipeline that can be reviewed and reverted.

Success

The ideal combination: configuration in git, changes through a CI/CD pipeline, dry-run validation before applying, and helm rollback as the safety net. These three layers make configuration changes reversible and auditable.

#Closing

In this episode 11 you understood Vitess configuration management: customizing with Helm and ConfigMaps sourced from git, managing secrets with Kubernetes Secrets and external managers, validating configuration with dry-runs and ValidateSchema, and rolling back production changes with helm rollback.

Key takeaways:

values.yaml is the single source of truth for configuration; keep it in git.
Never edit a ConfigMap manually in the cluster — always through helm upgrade.
MySQL, topology, and backup credentials must be stored as Secrets, not in values.
Use an external secrets manager for production; don't store secrets in git.
Dry-run and ValidateSchemaKeyspace are the validation gates before applying.
helm history and helm rollback are the safe way back.

In the next episode, episode 12, we lock down communication: secure connectivity — TLS between VTGate, VTTablet, and MySQL, client authentication and user management, and network policies for isolating services in Kubernetes. See you there!