Learning MongoDB - Sharding (Horizontal Scalability)

#Introduction

The replica set in episode 15 guarantees high availability and read scaling. But there's a limit that can't be broken: storage and write throughput are still limited by one primary. If a dataset reaches terabytes and writes reach hundreds of thousands per second, no matter how big a server you buy — one physical server will be overwhelmed. The answer is sharding: splitting data across many servers horizontally.

Episode 16 teaches you the horizontal-scale mindset. The roadmap: first we determine when sharding is truly needed, second we dissect the sharded cluster architecture — shards, config servers, and mongos, third we understand the shard key, and fourth we compare ranged versus hashed sharding. Let's get started.

#When Do You Need Sharding?

Sharding adds infrastructure complexity — so don't use it before it's genuinely needed. Two clear signals that the time has come:

1. The dataset is too large for one server — in the terabyte range and above, where one machine's storage doesn't fit or is unreasonably expensive.
2. Write throughput exceeds one replica set's capacity — a single primary can only process a certain number of writes; if this limit is reached while storage is still fine, sharding enables many primaries (one per shard) to write in parallel.

Beyond that, there are supporting indicators: the working set (data accessed often) starts exceeding the server's RAM, and queries keep getting slower even when well indexed. If those signals haven't appeared yet, a scaled-up replica set is simpler and cheaper — sharding is a big architectural decision, not a minor tuning.

#Sharded Cluster Architecture

A sharded cluster has three components with very different roles:

App ---> mongos ---> config servers (metadata)
                |
                +--> shard A (data subset)
                +--> shard B (data subset)
                +--> shard C (data subset)

The key point: applications never talk directly to shards. The application only communicates with mongos — which looks like a regular mongod — and mongos handles the rest. This keeps the application experience simple even though there are dozens of servers behind the scenes.

#Chunks and the Balancer

Data in a sharded collection is split into chunks — continuous data ranges based on the shard key. These chunks are distributed across the shards. When one shard is overloaded and another is idle, the balancer (a background process) moves chunks to even out the load automatically. This is what lets a sharded cluster "heal" its own data distribution.

#Shard Key

#The Shard Key Concept

The shard key is the field (or field combination) that determines how documents are distributed across shards. MongoDB divides the shard key value ranges into chunks, and places each chunk on one shard. Choosing the shard key is the most important and most difficult-to-change decision in a sharded cluster — choose it before the collection grows, because changing the shard key after data is distributed is extremely complex.

#Characteristics of a Good Shard Key

A good shard key has three qualities:

1. High cardinality — very diverse values. If there are only two distinct values (e.g. true/false), data only spreads across two shards — the rest stay idle.
2. Low frequency — no value dominates. A shard key with one very popular value makes one shard receive unbalanced load (a hot shard).
3. Non-monotonically increasing — values don't keep rising. A shard key like createdAt or auto-increment will always route new writes to one last shard — concentrating the load and defeating distribution.

An example of a good shard key: the combination { customerId: 1, orderDate: 1 } in an orders collection — customerId has many unique values, none dominates, and time distribution is spread out.

#Ranged Sharding vs Hashed Sharding

#Ranged Sharding

Ranged sharding splits data based on shard key value ranges. Queries filtering a certain value range can be routed precisely to the shard holding that range — very efficient for range queries and operations grouping nearby values.

sh.enableSharding("app")
sh.shardCollection("app.orders", { customerId: 1 })

Its main weakness: if the shard key rises monotonically, all new writes pile onto one shard (a hot spot). Ranged sharding works best when the shard key is naturally evenly distributed.

#Hashed Sharding

Hashed sharding computes a hash of the shard key value then distributes it evenly across chunks. The hash ensures sequential values spread to different chunks — drastically eliminating the hot spot problem, even for monotonically rising shard keys.

sh.enableSharding("app")
sh.shardCollection("app.events", { deviceId: "hashed" })

The consequence: range queries are not efficient — because sequential values are scattered randomly, a range query has to reach many shards. Hashed sharding is ideal for write-heavy workloads and point-lookup queries (based on a single value), like event logs accessed per device.

#Setting Up a Sharded Cluster in Practice

To understand how a sharded cluster works, there's no better teacher than hands-on practice. Imagine we build a minimal cluster: one mongos, one config server, and two shards. First, start the config server as a single-member replica set:

mongod --configsvr --replSet cfgrs --dbpath /data/configdb --port 27019

The config server must be run with the --configsvr flag and as a replica set. After that, start two shards — each a simple single-member replica set:

mongod --shardsvr --replSet rs1 --dbpath /data/shard1 --port 27017
mongod --shardsvr --replSet rs2 --dbpath /data/shard2 --port 27018

Finally, start mongos as the application entry point:

mongos --configdb cfgrs/localhost:27019 --port 27017

Then register both shards to the cluster via mongos:

mongosh "mongodb://localhost:27017"
 
sh.addShard("rs1/localhost:27017")
sh.addShard("rs2/localhost:27018")
 
sh.status()

sh.status() displays the cluster's complete map: registered shards, sharded databases, and chunk distribution. This is the verification point that your cluster is healthy before traffic comes in.

#Conclusion

In episode 16 you understood sharding as the path to horizontal scale: splitting terabyte datasets and write throughput beyond one replica set's capacity. You got to know the sharded cluster architecture — shards storing data subsets, config servers storing routing metadata, and mongos as the query router — along with the chunk and balancer mechanism that evens out load automatically. Finally, you mastered choosing an ideal shard key (high cardinality, low frequency, non-monotonic) and the difference between ranged and hashed sharding.

Key takeaways:

• Sharding is for terabyte datasets or write throughput beyond one replica set.
• Shards store data, config servers store metadata, mongos routes queries.
• A good shard key: high cardinality, low frequency, and non-monotonically increasing.
• Ranged suits range queries; hashed eliminates hot spots for mass writes.
• The shard key can't be changed — choose it with great care.

In the next episode, episode 17, we secure everything: Security: Authentication, Authorization & Encryption. You'll enable SCRAM authentication, build RBAC with built-in roles and custom roles, then understand encryption at rest, TLS for encryption in transit, and client-side field level encryption for sensitive fields. See you in episode 17!