Learn Elasticsearch - Cluster Configuration & Node Management

#Introduction

Up until episode 13 we worked on a single node. But Elasticsearch's power is only felt when several nodes unite to form a cluster — and that's where configuration becomes important. Cluster misconfiguration is the most common source of production incidents: split-brain, nodes that can't join, or OOM from a wrong heap.

Episode 14 covers cluster and node configuration: static vs dynamic settings, persistent vs transient, discovery and cluster formation, node role configuration, JVM heap sizing, thread pools, and circuit breakers. This is the foundation that makes all the subsequent scaling episodes make sense.

#Static vs Dynamic Settings

There are two kinds of settings in Elasticsearch:

• Static — read from elasticsearch.yml when the node starts; cannot be changed at runtime without a restart. Examples: node.name, node.roles, path.data, cluster.name.
• Dynamic — can be changed at any time via the Cluster Update Settings API. Examples: cluster.routing.allocation.enable, index.number_of_replicas.

cluster.name: my-cluster
node.name: data-1
node.roles: [master, data]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.seed_hosts: ["node1:9300", "node2:9300", "node3:9300"]

Rule of thumb: use the default settings as much as possible; change only what's truly needed, and write static changes to the config file so they're recorded (infrastructure as code, episode 28).

#Persistent vs Transient

Dynamic settings have two layers:

• Persistent — stored in cluster state, survives restarts. This is the right way to change dynamic settings in production.
• Transient — applies until the cluster restarts or it's removed. Only for quick experiments.

PUT /_cluster/settings

{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  },
  "transient": {
    "cluster.routing.allocation.disk.threshold_enabled": false
  }
}

#Discovery and Cluster Formation

When a node starts, it must find other nodes and form a cluster. The discovery mechanism uses several seed hosts as an initial list:

discovery.seed_hosts: ["node1:9300", "node2:9300", "node3:9300"]
cluster.initial_master_nodes: ["master-1", "master-2", "master-3"]

cluster.initial_master_nodes is only used the first time a cluster is formed (bootstrap) to elect the initial master. After that, the master is chosen through voting. The number of master-eligible nodes must be odd to avoid tied votes — we'll return to the quorum topic in episode 29.

#Node Roles

Node roles are configured in node.roles (static):

In a small cluster, one node runs all roles. In large-scale production, separate them: dedicated master nodes (lightweight), dedicated data nodes (heavy), and dedicated coordinating nodes to manage search traffic without storing data. Roles determine the load and hardware profile — episode 18 goes deeper on this.

#JVM Heap Sizing

The golden rules of heap configuration:

• Half of RAM, max 32 GB. Heap above 32 GB makes the JVM stop using compressed oops — a memory waste that actually decreases performance.
• The remaining memory is used by Lucene for file cache (OS page cache) — don't give so much heap that the OS cache starves.
• -Xms must equal -Xmx — avoiding heap resizing at runtime.

-Xms8g
-Xmx8g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=500

Monitor heap usage: safe values are below 75–85%. If it keeps approaching 100%, it doesn't necessarily mean you should raise the heap — it could be a heavy aggregation query or bloated field data. Episodes 19 and 21 cover performance diagnosis.

#Thread Pools

Every node has thread pools for different kinds of work:

Thread pool sizes are computed automatically from the CPU count. In almost all cases you don't need to change them — what you need to understand is how to read overload signals: a RejectedExecutionException in the logs means a thread pool is full. When that happens, the answer isn't to raise the thread pool, but to add nodes or fix queries.

#Circuit Breakers

The circuit breaker is the last line of defense: it monitors the estimated JVM memory used by certain operations, and rejects requests as the limit approaches — preventing the out-of-memory that would take down a node. The three main breakers:

GET /_nodes/stats/breaker

If you often see Data too large, data for [<breaker>]... errors in search responses, that's a signal of memory-hungry aggregations or sorting — not a reason to raise the breaker. A too-loose breaker just delays the node's death.

#Common Mistakes

1. -Xms different from -Xmx. The JVM performs unnecessary resizing — make them equal.

2. An even number of master-eligible nodes. Voting can deadlock (tie) — keep the count odd.

3. Changing dynamic settings via transient in production. Use persistent and record the change.

4. Raising thread pools on overload. That masks the symptom, not the cause — find the root problem (heavy queries, too few nodes).

5. Raising circuit breakers. Almost always wrong; fix memory usage, not its limit.

#Conclusion

In episode 14 you mastered cluster and node configuration: static vs dynamic settings, persistent vs transient, discovery with seed hosts and bootstrap, node roles, JVM heap sizing with the half-RAM-max-32GB rule, thread pools, and circuit breakers.

Key takeaways:

• Static settings need a restart; dynamic can be changed at runtime (prefer persistent).
• Discovery uses discovery.seed_hosts; bootstrap once with initial_master_nodes.
• Heap = half of RAM, max 32 GB, -Xms = -Xmx.
• RejectedExecutionException means overload, not a reason to raise thread pools.
• Circuit breakers reject requests to prevent OOM — respect their signals.

A correctly configured cluster can still be breached — what separates a production cluster from a learning cluster is security. In episode 15 we'll cover security fundamentals: enabling xpack.security, built-in users and realms, user and role management, authentication methods (native, LDAP/AD, SAML, API keys, service tokens), and authorization with RBAC, DLS/FLS, and role mapping. See you there!