Learn Vitess - Observability & Debugging

#Introduction

You won't dare change anything in production without being able to see what's happening. Episode 7 builds your observability capability: collecting metrics, reading logs, tracing queries, and diagnosing problems. In the distributed world of Vitess, debugging without observability is like repairing an airplane blindfolded.

Episode 7 roadmap: metrics and monitoring with Prometheus, log patterns and slow queries, query tracing, then diagnosing shards and replication with vtctlclient. By the end, you'll have a debugging toolkit you can use immediately.

#Metrics and Monitoring

Every Vitess component exposes metrics in Prometheus format at the /metrics endpoint. The most important ones to monitor:

• VTGate: QPS, latency per query, connection count, errors per query type, scatter ratio.
• VTTablet: QPS and latency per shard, replication lag, data size, transaction count.
• Topology Service: latency and availability of topology operations.

The easiest way to see metrics directly is to curl the endpoint (via port-forward):

kubectl port-forward -n vitess svc/vtgate 15001:15001 &
curl -s http://localhost:15001/metrics | grep vtgate

curl -s http://localhost:15001/metrics displays all VTGate metrics. In production, you don't need to curl manually — Prometheus scrapes this endpoint automatically.

For a quick lab setup, install the kube-prometheus-stack:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack

With the bundled Grafana, you can build dashboards from VTGate and VTTablet metrics. There are Vitess community dashboards you can import, but building your own from scratch is also great practice.

#Log Patterns and Slow Queries

Vitess logs are structured and carry context: query identity, source tablet, and duration. Patterns you'll often look for:

• Queries with high duration — slow query candidates.
• VTGate errors like deadline exceeded or primary not serving.
• Replication warnings from VTTablet.

To find slow queries, Vitess provides the query log feature at VTGate and a slow query log at the MySQL level. At the MySQL level, enable the slow query log through tablet configuration. A practical alternative: query the sys database tables for statistics:

SELECT db, query, exec_count, avg_latency
FROM sys.statement_analysis
ORDER BY avg_latency DESC
LIMIT 10

The SELECT ... FROM sys.statement_analysis command lists the slowest queries by average latency — a great starting point for performance investigation.

#Tracing Queries Across Components

One query passes through VTGate, VTTablet, and MySQL — to understand where time is being spent, you need distributed tracing. Vitess supports OpenTracing/Jaeger to follow a single query through every component.

To enable it, configure tracing in VTGate and VTTablet. In the lab, run Jaeger via Docker:

docker run -d --name jaeger -p 16686:16686 -p 6831:6831/udp jaegertracing/all-in-one

Then set the tracing env variables in the VTGate/VTTablet deployments and open the Jaeger UI at http://localhost:16686 to see a flame graph of each query. Tracing is the most powerful tool for finding bottlenecks that aren't visible in aggregate numbers.

#Diagnosing Shards and Replication with vtctlclient

vtctlclient is Vitess's Swiss Army knife. For diagnosis, the most commonly used commands:

vtctlclient ListShardHealth
vtctlclient ListAllTablets
vtctlclient GetTablet <tablet-alias>
vtctlclient GetShardReplication <keyspace/shard>

• ListShardHealth displays the health status of each shard.
• GetTablet gives details for one tablet: type, address, and status.
• GetShardReplication shows the replication map — which tablets are members of a shard.

To see a tablet's replication position, use SQL via the mysql client to that tablet:

SHOW REPLICA STATUS

SHOW REPLICA STATUS shows columns like Seconds_Behind_Source — the replication lag number that's the primary health indicator. If this number keeps growing, the primary is overwhelmed or the network between tablets has issues.

#Closing

In this episode 7 you were equipped to see inside the Vitess cluster: Prometheus metrics from VTGate and VTTablet, log patterns and slow queries, distributed tracing with Jaeger, and diagnosing shards and replication with vtctlclient and SQL.

Key takeaways:

• Every component exposes Prometheus metrics at /metrics.
• Start debugging from VTGate, then drill down to the specific shard and tablet.
• Slow queries can be found via the sys schema or the MySQL slow query log.
• Cross-component tracing reveals bottlenecks hidden in aggregate numbers.
• ListShardHealth, GetTablet, and GetShardReplication are core diagnostic tools.
• Replication lag (Seconds_Behind_Source) is a health indicator you must monitor.

In the next episode, episode 8, we design schemas: schema management and vindexes — configuring vindexes for routing, the difference between global and local vindexes, and designing schemas for sharded and unsharded tables. See you there!