Learn Elasticsearch - Performance Optimization & Tuning

#Introduction

Elasticsearch is fast not because of magic — speed is the result of a series of trade-offs configured correctly. Unending incoming data, searches that must answer in milliseconds, and limited resources demand that you understand what needs tuning and what should stay at the default.

Episode 19 covers three tuning areas: indexing performance (bulk, refresh interval, translog, replicas during indexing, index sorting), search performance (query optimization, caching, filters, shard count, routing), and resource optimization (JVM G1GC, disk I/O, memory pressure, thread pools).

#Indexing Performance

#Bulk Indexing

The first rule of fast indexing: always use the bulk API (episode 4). Never send documents one by one. Configure the right batch size:

POST /produk/_bulk
{ "index": { "_id": "1" } }
{ "name": "produk 1", "price": 50000 }
{ "index": { "_id": "2" } }
{ "name": "produk 2", "price": 75000 }

An ideal bulk contains 1–10 thousand documents or 5–15 MB total, then try increasing gradually while watching took. The optimal size depends on the hardware — don't copy numbers from a tutorial without measuring yourself.

#Refresh Interval

Newly indexed data only becomes visible after a refresh (default 1 second). For bulk indexing loads, refreshing every second means a lot of segment synchronization work. For now, raise the interval so indexing becomes much faster:

PUT /produk/_settings

{
  "index.refresh_interval": "30s"
}

After the loading finishes, restore it to 1s (or automatically). Exception: applications that need instant search results shouldn't raise this interval carelessly — remember the near-real-time trade-off from episode 2.

#Translog

The translog (episode 2) guarantees write safety before data is segmented. The durability setting has two modes:

{
  "index.translog.durability": "async",
  "index.translog.sync_interval": "5s"
}

The request mode (default) flushes the translog to disk per request — the safest, but slow for mass loads. The async mode speeds up indexing with the risk of losing the last few seconds of data if a node crashes. Use async only for mass indexing where data can be re-indexed; for important data, keep request.

#Replicas During Indexing

Replicas slow down indexing because every write is copied. For mass loading, set replicas to 0 during the load, then raise them back afterward:

{
  "index.number_of_replicas": 0
}

After finishing, restore to 1: PUT /produk/_settings with {"index.number_of_replicas": 1}. This is a classic trick that significantly cuts large load times.

#Index Sorting

Index sorting configures the storage order of documents within a segment. If your searches are often sorted or filtered by a specific field — for example @timestamp — sort the index by that field so time-range queries become far more efficient:

{
  "settings": {
    "index.sort.field": "@timestamp",
    "index.sort.order": "desc"
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" }
    }
  }
}

Index sorting can only be set when the index is created — watch for it from the start. The trade-off: indexing costs slightly more, but range queries and top-N are much faster.

#Search Performance

#Query Optimization

Query optimization principles:

• Use filter, not query, for yes/no criteria — results are cached (episode 8).
• Limit the fields scanned with _source filtering and stored_fields.
• Use search_after, not large from (episode 6).
• Avoid leading wildcards (*foo) which force term scanning.

#Caching: Three Layers

The request cache works per shard and is invalidated when new documents arrive — so it's effective for data that rarely changes. The field data cache holds structures in memory; make sure the fields you aggregate are keyword/numeric types and rarely change.

#Shard Count and Routing

The right shard count (10–50 GB per shard, episode 18) affects search latency — every shard adds communication cost. Pre-filtering with routing limits searches to a subset of shards:

GET /produk/_search?routing=fashion

When the application already knows the target category, routing saves a drastic amount of search time.

#Resource Optimization

#JVM and G1GC

G1GC is the default GC in Elasticsearch 8.x — designed for large heaps with controlled pauses:

-Xms8g
-Xmx8g
-XX:+UseG1GC
-XX:G1HeapRegionSize=4m
-XX:MaxGCPauseMillis=500

GC monitoring is the most important signal: long pauses and promotion failed indicate a problem. Episode 21 covers JVM metrics in detail.

#Disk I/O

Indexing and merging are I/O-heavy work. Make sure: segment merging runs on a scheduled background (index.merge.scheduler.max_thread_count), and don't force merges during an indexing load. SSD for the hot tier (episode 18) is an investment that pays off.

#Memory Pressure

Memory pressure measures the load on the heap relative to capacity. If pressure is high and RejectedExecutionException appears often, the solution isn't changing configuration — it's adding nodes or fixing wasteful queries/aggregations.

#Common Mistakes

1. Bulk sizes memorized without measuring. The optimal batch size depends on hardware — measure with took and iterate.

2. 1s refresh interval during mass loads. Slow it down first, restore it afterward.

3. 1 replica during big loads. Set 0 during the load, restore after finishing.

4. Large from for pagination. Use search_after.

5. Index sorting applied too late. It must be set from the start — design it into the first mapping.

#Conclusion

In episode 19 you mastered performance tuning: bulk indexing with measured batches, refresh interval and async translog for mass loads, zero replicas during loading, index sorting; search optimization with filters/caching/routing and search_after; and resource tuning for G1GC, disk I/O, and memory pressure monitoring.

Key takeaways:

• Always bulk — measure the batch size, don't guess.
• Refresh and translog are a speed vs data safety trade-off.
• Replica 0 during load, 1 after — the most effective mass-indexing trick.
• Filters are cached, the request cache is prone to invalidation — use each per data type.
• Good tuning is measured and iterative, not copying configuration.

Performance is optimal — but one corrupted-disk incident can wipe out everything. In episode 20 we'll cover snapshot and restore as a backup strategy: repository types (filesystem, S3, GCS, Azure), snapshot lifecycle management (SLM), full and incremental snapshots, index recovery, partial restore, cross-cluster restore, and disaster recovery planning. See you there!