Learn Vitess - Managing Keyspaces & Shards

#Introduction

Your Vitess cluster is alive with the single-shard commerce keyspace. Now it's time to level up: create a new keyspace from scratch, then understand how Vitess splits data across many shards. Episode 4 is the heart of Vitess data management — you'll learn to think in terms of keyspaces, shards, and sharding keys.

Episode 4 roadmap: creating a keyspace from scratch, choosing the sharding key and vindex, adding shards, rebalancing, and partitioning best practices. By the end of the episode, you'll have a real two-shard keyspace with vindex-based routing.

#Creating a Keyspace

Creating a keyspace is the first step. Use vtctlclient to create a keyspace with sharding enabled:

vtctlclient CreateKeyspace -sharding_column_name=user_id -sharding_column_type=uint64 users
vtctlclient ListAllKeyspaces

The first command creates a keyspace named users with sharding column user_id of type uint64. vtctlclient ListAllKeyspaces should display both commerce and users.

Now the users keyspace exists, but without shards. To start storing data, a keyspace must have at least one shard. We usually start with one shard named 0 (representing the entire range from the smallest to the largest value):

vtctlclient InitShardMaster -force commerce/0 <tablet-alias>

The vtctlclient InitShardMaster command designates a specific tablet as the shard primary. This is the initialization step that makes the shard ready to accept queries.

#Choosing the Sharding Key and Vindex

#Sharding Key

A sharding key is the column that determines where data lives: rows with a given sharding key value always end up in the same shard. Choosing the sharding key is the most important decision in Vitess. The core requirements: the value must exist in every row and stay consistent for the lifetime of the row — changing a sharding key after the row is stored means moving the row.

Another consideration: the column should be used often in queries. Every query that filters on the sharding key can be routed to a single shard. If your workload rarely filters on that column, all queries will scatter — and sharding becomes pointless.

#Vindex

A vindex connects a sharding key to a shard. The most common vindex is hash (for integer types), which distributes data evenly by hashing the value and mapping it to the shard range. Because it relies on a hash, the distribution is nearly uniform regardless of the underlying value pattern.

Define the VSchema for the users keyspace via a JSON file that gets applied:

{
  "sharded": true,
  "vindexes": {
    "user_hash": { "type": "hash" }
  },
  "tables": {
    "users": {
      "column_vindexes": [
        { "column": "user_id", "name": "user_hash" }
      ]
    }
  }
}

The "sharded": true part marks this keyspace as split, and column_vindexes sets that the user_id column is mapped with the user_hash vindex. Apply it with ApplyVschema:

vtctlclient ApplyVschema -vschema_file=/path/to/vschema.json users

After this, the users table will be sharded by user_id. A query like WHERE user_id = 123 will go straight to a single shard.

#Adding Shards and Rebalancing

#From One Shard to Two Shards

The users keyspace currently has one shard 0. To add shards, we must split shard 0 into two parts: -80 and 80- (in hexadecimal, 80 is the midpoint of the range). This process is called resharding and is covered in depth in episode 16 — for now, understand the result:

Shard 0    : -80    (left half of the range)
Shard 80-  : 80-    (right half of the range)

Every keyspace has a total range from the smallest to the largest value, and shards divide this range. The value mapped by the vindex determines which shard receives it.

#Rebalancing vs Partitioning

Rebalancing means moving data so each shard carries an even load. In Vitess, the main rebalancing is done through planned resharding — not automatically. If one shard fills up, you split it again. This differs from partitioning in MySQL, which splits data within a single server: partitioning doesn't add total capacity, it only organizes data inside the same server.

CREATE TABLE events (
  event_id bigint,
  created_at datetime,
  primary key (event_id, created_at)
) PARTITION BY RANGE (YEAR(created_at))

The PARTITION BY RANGE command above only divides a table within one server. Vitess sharding splits it across servers — the total scale is completely different.

#Partitioning Best Practices in Vitess

A few practical rules worth remembering:

• Choose a balanced sharding key. A column with low cardinality (e.g. status: active/inactive) will pile data into one shard. A high-cardinality column like user_id is far better.
• Consider joins. Tables that are frequently joined together should share the same sharding key, so the join can happen in the same shard (co-located). Cross-shard joins are still supported, but slower.
• Watch out for reference tables. Small tables that rarely change (like a list of countries) don't need sharding — make them unsharded or use a lookup vindex so they're copied to all shards.
• Start from one shard. Don't hesitate to start without sharding. Add shards only when the data is genuinely large.

#Closing

In this episode 4 you created a new keyspace named users, chose the user_id sharding key with the hash vindex, understood shard ranges like -80 and 80-, and distinguished between rebalancing, MySQL partitioning, and Vitess sharding.

Key takeaways:

• The sharding key must always exist, be stable, and be used often in queries.
• A vindex maps column values to shards; hash is the most common choice for integers.
• A keyspace is sharded only if the VSchema declares sharded: true.
• Shards divide the hash range: 0 can be split into -80 and 80-.
• Rebalancing in Vitess means planned resharding, not an automatic process.
• Start with one shard, reshard only when truly needed.

In the next episode, episode 5, we dissect the engine behind all of this: query routing and VTGate — how queries are split, how cross-shard queries work, scatter-gather, and the SQL limitations you need to know. See you there!