Learn Vitess - Observability at Scale & SLOs

#Introduction

In episode 7 you learned to see the cluster; episode 21 raises that to the level of promises: how fast, how reliable, and how much capacity your service has — measured. SLOs (Service Level Objectives) turn observability from "we monitor" into "we commit and measure". Without SLOs, performance discussions are always subjective.

Episode 21 roadmap: defining SLOs, monitoring shard health, query metrics, then building effective alerting that doesn't make noise.

#Defining SLOs

An SLO is a measurable target for service health. The three most relevant dimensions for Vitess:

Latency: the 99th percentile (P99) of queries in milliseconds. Example: "95% of queries finish under 50ms".
Availability: the percentage of time the service is available. Example: "99.95% monthly availability".
Throughput: the number of queries per second the service can handle. Example: "supporting 50,000 peak QPS".

Example SLO definition

latency_slo:
  target_ms: 50
  threshold_percentile: 95
availability_slo:
  target_percent: 99.95
throughput_slo:
  qps: 50000
replication_lag_slo:
  max_seconds: 30

latency_slo.target_ms: 50 means the target: 95% of queries finish within 50ms. SLOs must be realistic and agreed with business stakeholders — not just numbers engineers hope for.

Info

Start with three simple SLOs: P95 latency, availability, and replication lag. Add new SLOs only when you can genuinely measure them well. An unmeasured SLO is just a piece of writing.

#Monitoring Shard Health

At large scale, operator attention shifts from per-tablet to per-shard. Every shard has health that must be monitored: primary status, replication lag, QPS, and query errors.

Health of all shards

vtctlclient ListShardHealth

vtctlclient ListShardHealth displays the status of each shard. For automated dashboards, parse the JSON output and feed it to Prometheus. An unhealthy shard must be clearly visible — because one problematic shard can drain traffic from an entire keyspace.

Shard metrics monitored routinely:

Seconds_Behind_Source — replication lag (from SHOW REPLICA STATUS).
QPS and latency per shard.
Data size and storage growth.
Number of query errors per type.

#Query Metrics and Bottlenecks

Query metrics reveal patterns: which queries are slow, which shard is busiest, and which query types dominate. At VTGate, query metrics are grouped by keyspace, shard, and table. The keys to finding bottlenecks:

P99 latency per query type: finds the queries that are most often slow.
Scatter ratio: the proportion of queries touching many shards — a signal of poor schema design.
Error rate per shard: finds shards with problems.
Connection wait time: an indicator of a pool running out of connections.

Top vtgate query metrics

curl -s http://localhost:15001/metrics | grep -E 'vtgate_query_(duration|error|rows)' | head

curl -s ... vtgate_query shows VTGate query metrics — the foundation for building latency and error rate dashboards.

#Effective Alerting

Alerting is the bridge from metrics to action. The main rule: alerts must be actionable and quiet. Too many alerts make operators ignore them. Several important alerts for Vitess:

Primary not serving — writes stop, immediate action.
Replication lag beyond the SLO — read data starts going stale, or a sign the primary is overwhelmed.
Query error rate above threshold — the application is failing.
P99 latency beyond the SLO — the service is slower than promised.
Backup failed — the data safety net has a hole.

Example Prometheus alert for replication lag:

Replication lag alert

groups:
  - name: vitess
    rules:
      - alert: HighReplicationLag
        expr: mysql_replication_seconds_behind_source > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Replication lag melebihi 30 detik"

The mysql_replication_seconds_behind_source > 30 rule fires an alert if lag exceeds 30 seconds for 5 consecutive minutes — the for: 5m filter prevents alerts from harmless temporary spikes.

Warning

Every new alert must answer three questions: what should the person receiving it do, how quickly, and has this alert ever led to real action. An alert that's never used is noise that buries important alerts.

#Closing

In this episode 21 you brought observability to the level of promises: defining SLOs for latency, availability, and throughput, monitoring per-shard health, digging into query metrics to find bottlenecks, and building effective, quiet alerting.

Key takeaways:

SLOs must be measurable, realistic, and agreed with the business.
Start with three simple SLOs: P95 latency, availability, and replication lag.
Monitor health per shard, not just per tablet.
Scatter ratio is a schema design signal that needs attention.
Alerts must be actionable and the for filter prevents noise.
Good SLOs and alerting are the foundation for changing production confidently.

In the final episode, episode 22, we summarize everything: production hardening and best practices — a security checklist, scaling and failover readiness, runbook documentation, and safe Vitess and MySQL upgrades. See you there!