Learn Elasticsearch - Data Streams & Time-Series Data

#Introduction

Logs, metrics, and IoT events share one thing: they are append-only (almost never updated) and arrive continuously. Managing such data with a regular index — one big index that keeps swelling — is a recipe for disaster: searches get slower, ILM gets harder, and storage goes uncontrolled. Data streams are Elasticsearch's modern answer for append-only data. Episode 11 covers the data stream concept (including the differences from regular indexes), backing indices and automatic rollover, data stream templates, TSDB mode for metrics with dimensions and metrics, downsampling, and runtime fields.

#Data Streams vs Regular Indexes

At a glance a data stream looks like an index — you can write to it and search it. But behind the scenes it's an abstraction layer over many indexes:

Because data streams are designed to be append-only, they're a perfect fit for logs, metrics, and event streams. Rule of thumb: if the data will never be updated, use a data stream.

#Backing Indices and Automatic Rollover

When you write a document to the logs data stream, Elasticsearch stores it in the currently active backing index — for example logs-2026.08.03-000001. When the active index reaches the rollover condition (configured via an ILM policy), Elasticsearch automatically creates a new backing index logs-2026.08.03-000002 and makes it active. You don't need to touch anything — see the list with GET /_data_stream/logs:

{
  "data_streams": [
    {
      "name": "logs",
      "timestamp_field": { "name": "@timestamp" },
      "indices": [ { "index_name": "logs-2026.08.03-000001", "status": "active" } ]
    }
  ]
}

A data stream must have a timestamp field — @timestamp by default — and every incoming document must include it.

#Creating a Data Stream with a Template

A data stream isn't created with PUT; it appears when the first index is created with a name pattern matching a data stream template. The template is a regular index template marked with "data_stream": {}:

{
  "index_patterns": ["logs-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "number_of_shards": 2, "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "message": { "type": "text" }
      }
    }
  }
}

When the first document is sent to logs-test, the logs data stream is automatically created complete with its first backing index, settings from the template, and the logs-policy ILM policy attached — so rollover and tier transitions (episode 10) run automatically.

#Writing to and Searching a Data Stream

Writing is no different from a regular index — just write to the data stream name via POST /logs/_doc:

{ "@timestamp": "2026-08-03T10:00:00Z", "level": "error", "message": "koneksi ke database timeout" }

Searching is the same too — the data stream automatically covers all its backing indexes:

{
  "query": {
    "bool": {
      "filter": [
        { "term": { "level": "error" } },
        { "range": { "@timestamp": { "gte": "now-7d" } } }
      ]
    }
  }
}

#TSDB Mode for Metrics

For metric data (continuously measured numbers — CPU, latency, temperature), Elasticsearch 8.x has a Time Series Database (TSDB) mode. It's not separate storage, but a special way of indexing time-series data so it's far more efficient.

With TSDB, mappings are split into two kinds of fields:

• Dimensions — labels that group data, e.g. host.name, service.name. They act as the series identity.
• Metrics — measured values, e.g. cpu.usage, latency.p50.

{
  "index_patterns": ["metrics-*"],
  "data_stream": {},
  "template": {
    "settings": { "index.mode": "time_series", "index.routing_path": ["host.name", "service.name"] },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "host.name": { "type": "keyword" },
        "service.name": { "type": "keyword" },
        "cpu.usage": { "type": "float" }
      }
    }
  }
}

index.routing_path declares the dimensions. With TSDB, all data with the same combination of dimensions is stored sequentially on disk — so time-range queries for a single host become much faster, and storage is denser because metric values are delta-encoded. Remember: TSDB can only be used by data streams, and the measurement interval is configured via index.time_series.start_time and end_time.

#Downsampling for Long-Term Storage

Metrics don't need to be stored at full precision forever. Per-second metrics over a year consume enormous storage, yet a year later you only need the summary. Downsampling answers this: summarizing metrics into a coarser resolution — for example from every 10 seconds to every 1 hour — by storing aggregations (sum, avg, min, max, etc.) per interval. Elasticsearch 8.x supports automatic downsampling in the ILM phase — the warm/cold phase can include a downsample action:

{
  "policy": {
    "phases": {
      "hot": { "actions": { "rollover": { "max_age": "1d" } } },
      "cold": { "min_age": "30d", "actions": { "downsample": { "fixed_interval": "1h" } } }
    }
  }
}

After 30 days, per-10-second metrics are automatically summarized to per-1-hour — storage drops dramatically while the answer to "average CPU of host X last month" stays accurate. Combine this with the cold/frozen ILM tiers for minimal long-term storage costs.

#Runtime Fields

Sometimes we want to project a new field without changing the original data. Runtime fields are fields computed at query time, not at indexing time — register them via PUT /logs/_mapping:

{
  "runtime": {
    "service_name": {
      "type": "keyword",
      "script": { "source": "emit(doc['service.full.name'].value.substring(0, 6))" }
    }
  }
}

This service_name field can be used immediately for querying, aggregation, and sorting — without touching the original documents or mappings. When to use it? When the transformation logic is unstable or rarely used — avoiding the cost of a reindex. If queries keep using a runtime field and performance becomes an issue, that's when you move it to a real field via an ingest pipeline (episode 12) or a reindex.

#Common Mistakes

1. Sending documents without @timestamp. A data stream rejects documents that lack the timestamp field.
2. TSDB without index.routing_path. Time series mode needs dimension declarations — without them the index fails to be created.
3. Updating documents in a data stream. Update operations are rejected; if forced, use reindex to a regular index.
4. Skipping the template. Without a data stream template, regular indexes with the old name pattern get created and aren't managed by ILM.
5. Downsampling without understanding intervals. A wrong fixed interval ruins data granularity.

#Conclusion

In episode 11 you mastered data streams and time-series: the append-only data stream concept with backing indices, automatic rollover, templates that unify settings/mappings/ILM, TSDB mode with dimensions and metrics, downsampling, and runtime fields.

Key takeaways:

• A data stream is an abstraction over many backing indexes; writes always go to the active index.
• Data streams are append-only — perfect for logs, metrics, and events.
• TSDB compacts metrics with dimensions and metrics; downsampling summarizes old metrics.
• Runtime fields give flexibility without altering the original data.

Data from various sources isn't always tidy — formats are messy, fields differ. In episode 12 we'll cover ingest pipelines and data preprocessing: the set/remove/rename/grok processors, conditional processing, pipeline chaining, log parsing with grok, enrich, GeoIP, user agent, and the script processor. See you there!