Learn Elasticsearch - Index Lifecycle Management (ILM)

#Introduction

Time-series data like logs and metrics grows continuously without stopping. If every index is treated the same — stored on fast storage, with many replicas — storage costs will explode. Yet data value isn't constant: yesterday's logs are accessed very often, logs from 3 months ago are almost never opened, and logs from 2 years ago just need to be kept for compliance. Index Lifecycle Management (ILM) answers this problem: automating an index's journey from "hot and fast" to "cold and cheap" based on age and size. Episode 10 covers the hot-warm-cold-frozen architecture, ILM phases, rollover, creating and attaching policies, index priority, allocation filtering, and searchable snapshots for the frozen tier.

#ILM Concepts and the Tier Architecture

ILM treats an index like a commodity whose value declines over time. Each tier has different storage characteristics:

These tiers are implemented via data nodes with data tier attributes (data_hot, data_warm, data_cold, data_frozen). Allocation filtering in the ILM phase directs each index to the right node.

#ILM Phases and Rollover

#Rollover

With a date-based index pattern, you don't want to create new indexes manually. Rollover automatically creates a new index when the old one reaches a certain condition — for example, aged 7 days or sized 50 GB:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_age": "7d", "max_size": "50gb", "max_primary_shard_size": "20gb" }
        }
      }
    }
  }
}

Why is rollover important? An index left to grow unbounded becomes hard to manage and slows searches. With rollover, each "generation" index stays at a controlled size, and ILM can treat each generation differently.

#The Hot, Warm, Cold, Frozen, Delete Phases

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": { "rollover": { "max_age": "7d", "max_size": "50gb" }, "set_priority": { "priority": 100 } }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": { "number_of_replicas": 1, "include": { "data_tier_preference": "data_warm" } },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": { "allocate": { "include": { "data_tier_preference": "data_cold" } }, "set_priority": { "priority": 0 } }
      },
      "frozen": {
        "min_age": "90d",
        "actions": { "searchable_snapshot": { "snapshot_repository": "backup-s3" } }
      },
      "delete": { "min_age": "365d", "actions": { "delete": {} } }
    }
  }
}

Let's read the flow: an index lives 7 days or up to 50 GB in the hot tier (write priority 100), then enters warm (shrunk to 1 shard, force-merged, priority lowered to 50), after 30 days enters cold with priority 0, after 90 days becomes a searchable snapshot in the frozen tier, and is finally deleted after 1 year.

#Creating and Attaching Policies

#Creating a Policy

PUT /_ilm/policy/logs-policy

With the ILM policy JSON body above. The same policy can be used by many indexes.

#Attaching a Policy to an Index

PUT /logs-2026.08.03

{
  "settings": {
    "index.lifecycle.name": "logs-policy",
    "index.lifecycle.rollover_alias": "logs",
    "index.routing.allocation.include.data_tier": "data_hot",
    "index.number_of_shards": 2,
    "index.number_of_replicas": 1
  }
}

There are two ways to attach: via settings when creating the index, or — even better — automatically via an index template (episodes 4 and 11), so that no index "escapes" lifecycle management.

#Index Priority and Allocation Filtering

Index priority (0–100) determines the recovery order when the cluster performs recovery — for example after a mass restart. Indexes with higher priority are recovered first. The hot phase has the highest priority because its data is the most active.

Allocation filtering controls which nodes an index may be placed on. In a policy, the allocate action with include/exclude moves the index to the correct tier. This is also useful for maintenance: POST /_cluster/allocation/explain helps find out why a shard hasn't been assigned.

#Searchable Snapshots for the Frozen Tier

The frozen phase uses searchable snapshots: an index whose data lives in a snapshot repository (S3, for example), with a small amount of metadata and local cache on the node. The result: very cheap storage, while searching remains possible — only with higher latency because data must be fetched from cold storage on first access (then cached).

That's an interesting trade-off: 2-year-old data can be stored almost for free, and if it's ever needed for a compliance investigation, it can still be searched. Episode 20 covers snapshots thoroughly.

#Monitoring ILM

ILM runs in the background; make sure you can see its progress:

GET /_ilm/explain/logs-*

{
  "indices": {
    "logs-2026.08.03": {
      "index": "logs-2026.08.03", "managed": true, "policy": "logs-policy",
      "phase": "hot", "action": "rollover", "step": "check-rollover-ready"
    }
  }
}

_ilm/explain shows which policy is attached, which phase is running, and which step is being executed. If there's an error, the step_info field carries the details — this is the primary debugging tool when a policy isn't behaving as expected.

#Common Mistakes

1. Rollover without an alias. Rollover needs index.lifecycle.rollover_alias pointing to an alias containing the index.

2. Force merge in the hot phase. It chokes write performance. Only do it in warm.

3. A tier that doesn't exist in the cluster. If there's no data_cold node, cold indexes can't be allocated — check node availability per tier.

4. All priorities the same by default. If every index has the same priority, recovery order is uncontrolled. Set different priorities per phase.

5. Changing a policy doesn't trigger instant re-evaluation. ILM evaluates periodically — it takes a few minutes before new actions run.

#Conclusion

In episode 10 you mastered ILM: the hot-warm-cold-frozen architecture, phases with their actions (rollover, shrink, force merge, allocate, searchable_snapshot, delete), creating and attaching policies via index templates, index priority, allocation filtering, searchable snapshots for the frozen tier, and monitoring with _ilm/explain.

Key takeaways:

• ILM automates the index lifecycle by age and size.
• Rollover keeps index size under control; it happens in the hot phase.
• Shrink and force merge are warm-phase actions.
• The frozen tier uses searchable snapshots for the cheapest storage.
• Attach policies via templates so every index is managed.
• _ilm/explain is ILM's primary debugging tool.

ILM and the date-based index pattern lead us to the next concept. In episode 11 we'll cover data streams and time-series data: data streams vs regular indexes, backing indices, automatic rollover, data stream templates, TSDB mode with dimensions and metrics, downsampling, and runtime fields. See you there!