Learn Elasticsearch - Core Concepts & Main Architecture

#Introduction

In episode 1 we learned why Elasticsearch was born: turning Lucene — a search library — into a distributed, easily accessible engine. Now we enter the most important part of understanding Elasticsearch: how it works under the hood. Without this understanding, you'll type commands like a memorizing robot — it runs, but you don't understand why.

Episode 2 covers the architectural foundation: the document-index hierarchy, the inverted index mechanism, sharding and replication, node types, the indexing and search processes from start to finish, as well as consistency trade-offs. This is the most theory-dense episode in the series — you'll keep referring back to it constantly.

#Document, Index, and Hierarchy

#Document as the Data Unit

The basic unit of data in Elasticsearch is the document: a JSON object storing fields with values. One document represents one entity — one product, one log, one article. Unlike SQL tables, documents don't have to have exactly the same fields; one document may have fields that another document doesn't.

{
  "name": "Kaos Polos Premium",
  "price": 99000,
  "category": "fashion",
  "in_stock": true,
  "tags": ["casual", "basic"]
}

#Hierarchy: Index and Document

Elasticsearch has three hierarchy levels: cluster (a collection of nodes), index (a collection of documents with the same mapping, analogous to a table), and document (analogous to a row). The type concept — the level between index and document — has been removed since Elasticsearch 8.0; now one index contains only one type of document.

An index has settings (how many shards, how many replicas, refresh configuration) and mapping (the data type definition of each field). Both are covered in depth in episodes 4 and 5.

#Inverted Index

This is the heart of Elasticsearch. The inverted index is a data structure that maps each term to the list of documents containing it. The easiest analogy: the index at the back of a textbook. You don't read the whole book to find the word "sharding" — you open the index, look up "sharding", and are directed straight to the relevant pages.

{
  "elasticsearch": ["doc-1", "doc-2"],
  "search": ["doc-1"],
  "analytics": ["doc-2"]
}

When you search for the word "search", Elasticsearch just looks up that term in the inverted index and immediately knows document doc-1 matches — without scanning all documents. Search speed is therefore nearly constant, independent of the number of documents. This is why LIKE in SQL loses badly: LIKE must read every row, while Elasticsearch points directly to the right document.

#Sharding and Replication

#Sharding: Splitting Data

An index doesn't have to reside on a single machine. Elasticsearch divides an index into shards — independent pieces of data spread across several nodes. The number of primary shards is determined when the index is created and cannot be changed afterwards (except via reindex, episode 13). This is the scaling-out mechanism: 100 million documents can be spread across 10 shards on 5 nodes, and each node only handles part of the work.

#Replication: Copying for Resilience and Performance

Each primary shard has replica shards — exact copies residing on different nodes. Replicas have two functions: resilience (if a node dies, a replica replaces the primary) and read performance (searches can be distributed across primary and replicas). All write operations go to the primary, then spread to replicas asynchronously.

curl -s localhost:9200/_cat/shards?v

Note the pri and rep columns: the number of primaries and replicas. You'll often see rep 0 in early setups — for production, the minimum is 1 replica.

#Node and Cluster Architecture

A node is one Elasticsearch instance; a collection of nodes forms a cluster. One node can play several roles at once, determined by node.roles in elasticsearch.yml:

In a small cluster, one node runs all roles. In a large production cluster, roles are separated for load isolation — we'll cover this in episodes 14 and 18. The master node records the cluster state: mappings, settings, the location of every shard, and existing indexes.

#Behind the Scenes

#Indexing Flow

When a document is sent for indexing: the receiving node (coordinating) computes routing to determine which primary shard handles the document, sends the document to the node owning that primary shard, writes to the translog (a write log for recovery) and an in-memory buffer, then periodically forms a new segment — an immutable Lucene file storing the inverted index. This new data is only visible to searches after a refresh (default 1 second).

#Search Flow

When a query comes in: the coordinating node sends the query to every shard of the searched index, each shard searches locally, returns results plus a relevance score, and the coordinating node merges the results by sorting on score. This is why the number of shards affects search performance — the more shards, the more nodes you have to "ask".

#Relevance and Scoring

The relevance score is calculated with the BM25 model: the more often a word appears in a document (term frequency) and the rarer that word is across the whole index (inverse document frequency), the higher the score. Document length is also taken into account — a short document containing the word is considered more relevant.

#Segment and Merge Process

Lucene writes data as segments that are immutable — they can't be changed, only newly created. Over time, many small segments accumulate, and Lucene runs merge: combining small segments into one large segment. During a merge, deleted documents are truly discarded. This is a background process that runs continuously, and understanding segments is the basis for configuring refresh_interval and the translog in episode 19.

#CAP Theorem and Eventual Consistency

Elasticsearch emphasizes availability and partition tolerance, and for writes its consistency is eventual: newly indexed documents may not be immediately visible on all replicas (replication takes time). Reads with _primary and _replica preferences allow you to choose whether to read from the primary or let replicas serve, trading off consistency vs performance.

There is one important exception: real-time search for newly written documents — Elasticsearch guarantees a document is visible one second after it's written (thanks to the refresh interval). "Real-time" in Elasticsearch means near-real-time, not millisecond-consistent like a transactional database.

#Conclusion

In episode 2 you understood Elasticsearch's core architecture: the document as the JSON data unit, the index as a collection of documents with the same mapping, the inverted index as the secret to search speed, shards and replicas as the distribution and resilience mechanism, node roles within a cluster, as well as the indexing and search flows involving the translog, segments, refresh, and merge. You also understood Elasticsearch's position in the CAP theorem: available and partition tolerant, with eventual consistency.

Key takeaways:

• Data hierarchy: cluster → index → document; the type concept has been removed in 8.x.
• The inverted index turns search into a lookup, not a scan — speed is nearly constant on big data.
• The number of primary shards cannot be changed after an index is created.
• Replicas provide both resilience and read performance.
• Searches run across all shards, then results are merged at the coordinating node.
• Relevance uses BM25; data is written to immutable segments that are merged periodically.

Now the theory is enough to start practicing. In episode 3 we'll install Elasticsearch and run the first instance: installation methods (APT, archive, Docker), starting and stopping the service, checking cluster health, understanding the JSON response structure, and getting to know Dev Tools in Kibana. See you there!