Learn Elasticsearch - Monitoring & Observability

#Introduction

Backups (episode 20) protect against data loss — but how do you detect problems before they become incidents? How do you know the disk is almost full, the heap is nearing capacity, or queries are starting to slow down? The answer lies in monitoring and observability: the built-in monitoring APIs (cluster health, node stats, index stats, cat APIs), the metrics you must watch (JVM, indexing/search rates, thread pool rejections, circuit breaker trips), and monitoring tools — Kibana, alerting, Metricbeat, and Prometheus/Grafana.

#Built-in Monitoring APIs

#Cluster Health

GET /_cluster/health is the first heartbeat (episode 3). For a quick multi-node view:

Health cluster dengan detail node

GET /_cluster/health?level=indices&timeout=30s
{
  "cluster_name": "my-cluster",
  "status": "yellow",
  "number_of_nodes": 3,
  "active_primary_shards": 30,
  "active_shards": 45,
  "relocating_shards": 0,
  "unassigned_shards": 15
}

Status yellow with 15 unassigned_shards indicates replicas haven't been assigned — for example because only 1 of 3 nodes holds data. Status red is an emergency.

#Node Stats and Index Stats

_nodes/stats shows per-node metrics — CPU, heap, thread pools, I/O:

Statistik node: heap dan thread pool

GET /_nodes/stats/jvm,thread_pool,os

_stats shows per-index metrics — document count, size, indexing/search throughput:

Statistik index logs-*

GET /logs-*/_stats?filter_path=indices.*.primaries.indexing,indices.*.primaries.search

#Cat APIs

Cat APIs present human-readable table output — most practical for the shell:

Ringkasan node seperti tabel

GET /_cat/nodes?v&h=name,node.role,heap.percent,ram.percent,cpu,load_1m

Frequently used cat APIs: _cat/health, _cat/nodes, _cat/shards, _cat/indices, _cat/thread_pool, _cat/allocation.

#Metrics You Must Monitor

#JVM Heap

The most important metric. Monitor heap.percent and GC pauses:

Metrik JVM dan GC

GET /_nodes/stats/jvm?filter_path=nodes.*.jvm.mem,nodes.*.jvm.gc

Reasonable alert thresholds: heap below 75% (ideal), GC pauses under 1 second. A heap nearing 100% or frequent promotion failed is a sign of serious trouble.

#Indexing and Search Rates

The average indexing rate (indexing.index_total) and search rate (search.query_total) per second show volume and trends. A sudden rise can mean growing load; a sudden drop can mean a troubled node.

#Thread Pool Rejections

The thread_pool metric has a rejection counter:

Rejections thread pool

GET /_nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.write.rejected,nodes.*.thread_pool.search.rejected

Rejection is the most serious alarm — it means a node is overwhelmed and starting to reject requests (episode 14). Zero rejections is the target; steadily rising rejections means it's time to scale.

#Circuit Breaker Trips

_nodes/stats/breaker (episode 14) shows how many times breakers blocked requests. Frequent breaker trips mean memory-hungry queries/aggregations — fix the cause, don't raise the limit.

Important

Metrics without a baseline mean nothing. Before the first incident, collect normal data over a few weeks — only then do you know whether "50% CPU" is normal or a problem sign. This is why monitoring should be installed on day one, not when you're already panicking.

#Kibana Monitoring and Alerting

#Stack Monitoring

Kibana has a Stack Monitoring module that displays complete cluster health dashboards from data collected by Metricbeat. Enable it first on the Elasticsearch side:

Aktifkan collection monitoring

xpack.monitoring.collection.enabled: true

Then in Kibana: Management → Stack Monitoring. There you see health, shard distribution, heap, indexing rates, and node lag in a single view.

#Alerting

Alerting (formerly Watcher) creates automatic notifications when a condition is met:

Rule alert sederhana untuk heap tinggi

{
  "name": "Heap terlalu tinggi",
  "condition": {
    "script": {
      "source": "ctx.monitoring.cluster.elasticsearch.cluster_stats.indices.fielddata.memory_size_in_bytes > 1073741824"
    }
  },
  "actions": {
    "email_admin": {
      "email": {
        "to": ["oncall@example.com"],
        "subject": "Heap tinggi"
      }
    }
  }
}

In Kibana 8.x, alerts are created via Management → Stack Management → Alerts and Insights with a UI — covering thresholds for CPU, heap, disk, and rejections. A good rule: alerts that are specific, actionable, and not noisy.

#Metricbeat and Elastic APM

#Metricbeat

Metricbeat is a lightweight agent that collects metrics from many systems — including Elasticsearch itself:

Metricbeat module untuk Elasticsearch

metricbeat.modules:
  - module: elasticsearch
    metricsets: ["node", "node_stats", "index", "cluster_stats"]
    period: 10s
    hosts: ["https://node1:9200"]
    username: "metricbeat_user"
    password: "secret"

Metricbeat sends metrics to the monitoring cluster, feeding data for Stack Monitoring and alerting (episode 24 covers Beats in depth).

#Elastic APM

Elastic APM monitors application performance: latency, throughput, error rate, and distributed tracing across services. Its Elasticsearch integration shows "which query is slow and from which application" — key data when optimizing performance (episode 19). Episode 26 covers APM in a microservices context.

#Prometheus and Grafana

For organizations already using the Prometheus/Grafana stack, Elasticsearch provides a Prometheus metrics endpoint:

Aktifkan endpoint Prometheus

xpack.monitoring.enabled: false

Scrape metrik dalam format Prometheus

GET /_prometheus/metrics

Prometheus can scrape this endpoint, and Grafana displays it in custom dashboards. Many organizations use both: Prometheus/Grafana for platform-wide metrics, Kibana Monitoring for deeper Elasticsearch context.

Tip

Start simple: cat APIs for quick terminal debugging, Kibana Stack Monitoring for dashboards, and two or three of the most important alert rules (red status, disk nearing flood, rising rejections). Add more gradually based on incidents you actually experience — don't build a giant monitoring system before understanding a small one.

#Conclusion

In episode 21 you mastered monitoring and observability: the built-in monitoring APIs (cluster health, node stats, index stats, cat APIs), key metrics (JVM heap and GC, indexing/search rates, thread pool rejections, circuit breaker trips), and tools — Kibana Stack Monitoring, alerting, Metricbeat, Elastic APM, and Prometheus/Grafana.

Key takeaways:

Start with cat APIs for quick debugging; Stack Monitoring for dashboards.
Rejections and disk are the metrics that most often signal incidents.
Baselines make metrics readable — collect them before an incident.
Good alerts: few, specific, and actionable.
Kibana and Prometheus/Grafana can be used side by side.

One cluster isn't always enough — large organizations often have many. In episode 22 we'll cover cross-cluster search and replication: configuring remote clusters for cross-cluster search, data federation use cases, and cross-cluster replication with leader-follower for disaster recovery and geo-distribution, complete with replication lag monitoring. See you there!