Learn Elasticsearch - Scaling Elasticsearch Clusters

#Introduction

Your data grows, search traffic rises, and one day a single node can't keep up. The question isn't whether to scale, but how to scale without downtime and without making performance worse. Scaling Elasticsearch isn't just adding machines — there's an art to managing shards, tiers, and allocation.

Episode 18 covers cluster scaling: horizontal scaling by adding nodes, shard allocation strategies, the ideal 10–50 GB shard size, the hot-warm-cold architecture (from episode 10), when to choose scale up vs scale out, and SSD vs HDD storage considerations.

#Horizontal Scaling: Adding Nodes

Elasticsearch is designed for scale out: adding a node adds CPU, RAM, and disk in parallel, without changing the application. When a new node joins, Elasticsearch automatically balances existing shards onto the new node.

GET /_cat/nodes?v

Watch the heap.percent, ram.percent, and cpu columns — after adding a node, make sure the load is spread evenly. If one node remains a hot spot, there's likely a shard distribution problem (or a routing hot spot).

#Shard Allocation Strategies

Shard allocation is managed by the Elasticsearch allocator, with rules you can control. Some important settings:

PUT /_cluster/settings

{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "80%",
    "cluster.routing.allocation.disk.watermark.high": "85%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "90%"
  }
}

Disk watermark is very important: when a node's disk hits the flood_stage threshold, Elasticsearch rejects write operations on indexes in that node to prevent the disk from filling completely. This is a safety you should respect, not fight.

#Ideal Shard Size: 10–50 GB

The rule of thumb you must remember: a single shard should be 10–50 GB, with a comfortable target of around 20–30 GB. Why:

• Shards too small (< 10 GB) → too many shards, overhead and search latency rise.
• Shards too large (> 50 GB) → slow recovery and rebalancing, and a single shard becomes a heavy single point of failure.

A simple calculation: if total data is 200 GB at 50 GB per shard, you need 4 primary shards. For 10 million documents per day over 30 days, size the estimate with an initial snapshot — don't guess.

GET /_cat/shards?v&h=index,prirep,store,docs&s=store:desc

The most common beginner mistake: creating 20 shards for 2 GB of data, or 2 shards for 500 GB. Both hurt later — remember, the number of primary shards can't be changed without a reindex.

#The Hot-Warm-Cold Architecture

From episode 10 we know tiers differentiate storage costs. In this episode we practice dividing nodes by tier:

# node hot (node1)
node.roles: [data_hot, data_content]
node.attr.data_tier: hot
 
# node warm (node2)
node.roles: [data_warm]
node.attr.data_tier: warm
 
# node cold (node3)
node.roles: [data_cold]
node.attr.data_tier: cold

With nodes segregated per tier and an ILM policy (episode 10) moving indexes between tiers, hardware can be bought per need: fast SSD for hot, cheap HDD for cold. This is a cost-efficient form of scaling without increasing total data.

#Vertical Scaling: Scale Up vs Scale Out

Scale up (enlarging one node) and scale out (adding nodes) aren't competing — each has its place:

The practical ceiling of scale up: the 32 GB heap maximum (episode 14), so beyond that point the only path is scale out. The best strategy combines both: scale up for per-node needs, scale out for total capacity.

#Storage: SSD vs HDD

Storage is the decision that affects performance the most. Rules of thumb:

• SSD for hot and warm tiers — indexing and search need high IOPS and low latency.
• HDD for cold/frozen tiers — rare access, large capacity matters more.
• Avoid latent network storage in the hot tier — like NFS, often a source of latency.

One important truth: a full disk is far more dangerous than a slow disk. A full disk makes shards read-only and stops writes. Monitor disk seriously (episode 21) before performance becomes a problem.

#Common Mistakes

1. Too many shards for too little data. Communication overhead rises — calculate from 10–50 GB per shard.

2. Adding nodes just "to be faster". If the bottleneck isn't CPU/memory, a new node doesn't help — measure first (episode 21).

3. All nodes in one tier. No storage savings — apply hot-warm-cold based on data age.

4. Ignoring disk watermark. Writes silently stop at flood — monitor disk and set alerts.

5. Automatic rebalancing during critical operations. Temporarily disable allocation during big maintenance: cluster.routing.allocation.enable: none.

#Conclusion

In episode 18 you mastered cluster scaling: horizontal scaling by adding nodes with automatic rebalancing, shard allocation strategies with disk watermark, the ideal 10–50 GB shard size, the hot-warm-cold architecture with per-tier nodes, scale up vs scale out decisions, and SSD vs HDD choices.

Key takeaways:

• Scale out with new nodes for capacity; watch the real bottleneck.
• One shard should ideally be 10–50 GB — calculate before creating an index.
• Hot-warm-cold separates hardware by data age and saves money.
• Disk watermark is a write safeguard — respect and monitor it.
• Scale up has limits (32 GB heap); scale out is the path to large scale.

The cluster is big now — how do we make it fast and resource-efficient? In episode 19 we'll cover performance optimization and tuning: bulk indexing, refresh interval, translog, index sorting; search optimization with caching, filters, routing; and JVM G1GC tuning, disk I/O, and thread pools. See you there!