Learn Elasticsearch - Reindex & Update By Query Operations

#Introduction

Changing a mapping (episode 5) or an analyzer (episode 7) can't be applied to documents that are already indexed. Wrongly-typed fields, new analyzers, or changed formats — all of them require rebuilding the data. Fortunately Elasticsearch provides the Reindex API and Update By Query.

Episode 13 covers when a reindex is needed, the Reindex API with filters and scripts, reindexing from a remote cluster, Update By Query for mass updates, Delete By Query, and handling version conflicts and throttling.

#When a Reindex Is Needed

Reindex is the process of copying documents from a source index to a target index with new mappings/settings. It's needed when:

• The number of primary shards needs to change (it can't be changed in place).
• The mapping needs fixing — for example a field that's wrongly text/keyword.
• The analyzer changes and old documents need re-analysis.
• Fields need to be merged, split, or removed via script.
• The index needs to move between clusters or tiers.

The best pattern is always the same: create a new index with the correct mapping, reindex, then move the alias (episode 4) from the old index to the new one — the application never stops.

#Basic Reindex API

POST /_reindex
{
  "source": { "index": "produk" },
  "dest": { "index": "produk-v2" }
}

dest must already exist (or be created automatically with dynamic mapping — better to create it explicitly first with the target mapping). To keep the source alive during the process, set "dest": { "index": "produk-v2", "version_type": "external" } so source documents that change during the reindex don't conflict.

#Reindex with Query Filter and Script

Not all data must be moved — sometimes a subset is enough, and often transformation is needed. Both can be combined in a single request:

{
  "source": {
    "index": "produk",
    "query": {
      "term": { "category.keyword": "fashion" }
    }
  },
  "dest": { "index": "produk-v2" },
  "script": {
    "source": "ctx._source.price_diskon = ctx._source.price * 0.9"
  }
}

The Painless script above adds a price_diskon field to every document moved. The query filters which documents come along — only fashion products get reindexed. The filter + script combination makes reindex very flexible.

#Working Asynchronously

Reindexing large data isn't an instant process. By default, the _reindex request runs synchronously and can time out for large datasets — for production, run it asynchronously with ?wait_for_completion=false:

POST /_reindex?wait_for_completion=false
GET /_tasks/<task_id>?pretty

The first response returns a task_id; _tasks then shows how many documents succeeded and how many failed. This is the standard pattern for reindexing millions of documents.

#Reindex from a Remote Cluster

Reindex can also pull data from another cluster — for example migrating data from an old Elasticsearch (version 7.x) to a new 8.x cluster. The source is configured as a remote cluster:

POST /_reindex
{
  "source": {
    "remote": {
      "host": "https://old-cluster:9200",
      "username": "migration",
      "password": "secret"
    },
    "index": "produk"
  },
  "dest": { "index": "produk-v2" }
}

Note: query clauses on a remote source are not supported (limited), so data filtering should be done after the data arrives. Remote reindex also requires an HTTPS connection (episode 16) and the remote_cluster_client role on the node.

#Update By Query

Update By Query modifies documents matching a query in place, without creating a new index. That's the difference from reindex: no copying to another index. For example, adding a field to all old log documents:

POST /produk/_update_by_query
{
  "query": {
    "term": { "category.keyword": "fashion" }
  },
  "script": {
    "source": "ctx._source.price = ctx._source.price * 0.85"
  }
}

Update By Query runs the search + update in a single operation and can also run asynchronously with ?wait_for_completion=false. It's the fastest solution for transformations that don't require mapping changes.

#Delete By Query

The opposite: Delete By Query removes documents matching a query without deleting the index:

POST /logs-2026.07/_delete_by_query
{
  "query": {
    "range": {
      "@timestamp": { "lte": "2026-07-01T00:00:00Z" }
    }
  }
}

Deleted documents don't disappear from disk immediately — the deletion is marked in the segment, and the space is only freed during merge (episode 2). For large storage cleanups, consider a force merge after delete by query.

#Conflicts and Throttling

#Version Conflicts

During a reindex/update, documents can be changed by other operations — leaving version conflicts (409). Two handling strategies: version_type: external on the reindex (the source version is preserved) and "conflicts": "proceed" (skip conflicting documents):

POST /_reindex
{
  "source": { "index": "produk" },
  "dest": { "index": "produk-v2", "version_type": "external" },
  "conflicts": "proceed"
}

#Throttling

Mass operations use cluster resources; letting them run free can starve other services. Limit the speed with requests_per_second — for example 1000 requests per second:

POST /_reindex
{
  "source": { "index": "produk" },
  "dest": { "index": "produk-v2" },
  "conflicts": "proceed",
  "requests_per_second": 1000
}

Throttling is very useful when a reindex runs alongside production traffic — slower during the day, faster at night.

#Common Mistakes

1. Forgetting ?wait_for_completion=false for large data. Synchronous requests can time out — make them asynchronous and monitor the task.
2. Reindex without an explicit target mapping. Dynamic mapping in the target index can produce a mapping that's just as wrong.
3. Forgetting the alias. After a reindex, applications pointing at the old index don't follow automatically — moving the alias is the final step.
4. Update by query without conflict configuration. Parallel operations produce 409 — set conflicts: proceed when it's safe.
5. No throttling on a busy cluster. A full-speed reindex can trip circuit breakers — limit the rate.

#Conclusion

In episode 13 you mastered mass operations: when a reindex is needed, the Reindex API with query filters and scripts, asynchronous reindex with task tracking, reindexing from a remote cluster, Update By Query, Delete By Query, and handling version conflicts and throttling.

Key takeaways:

• Reindex to change mapping/shard/analyzer; create a new index, copy, move the alias.
• Reindex can run asynchronously with wait_for_completion=false and be monitored via the Tasks API.
• Update By Query modifies documents in place; Delete By Query deletes the matching ones.
• Handle version conflicts with version_type: external and conflicts: proceed; use throttling (requests_per_second) to keep the cluster safe.

Everything so far has happened on a single instance. But real Elasticsearch lives as a cluster. In episode 14 we'll cover cluster configuration and node management: static vs dynamic settings, discovery and cluster formation, node roles, JVM heap sizing, thread pools, and circuit breakers. See you there!