Learn Elasticsearch - Cross-Cluster Search & Replication

#Introduction

A single cluster is indeed powerful, but large organizations rarely live with just one cluster: a cluster per region, per team, per environment, or per function (for example a separate observability cluster from the application cluster). Scattered data creates two needs: searching data across all clusters at once, and copying data between clusters for resilience.

Episode 22 covers the two features that answer those needs: cross-cluster search (CCS) — searching across clusters without copying data — and cross-cluster replication (CCR) — continuously copying indexes from a leader cluster to a follower for disaster recovery and geo-distribution.

#Cross-Cluster Search: Concepts

CCS lets one cluster transparently execute searches on other clusters. You don't need to copy data — just point at the remote cluster as a remote cluster, then search it like a local index:

GET /remote-logs:logs-*/_search

The remote-cluster:index notation tells Elasticsearch where to look. Results are merged by the coordinating node in the local cluster. This is the basis of data federation: many data sources, one search point.

#Setting Up a Remote Cluster

Remote clusters are registered via cluster settings:

PUT /_cluster/settings

{
  "persistent": {
    "cluster.remote.remote-logs.seeds": [
      "logs-node1:9300",
      "logs-node2:9300"
    ]
  }
}

Every local node must have the remote_cluster_client role to connect. Check the connection status:

GET /_remote/info

#Running a Cross-Cluster Search

Cross-cluster searches use the remote name as the index prefix:

{
  "query": {
    "match": { "message": "timeout" }
  }
}

In the request body, you specify the indexes in the index parameter of the request: POST /remote-logs:logs-*,remote-prod:app-logs-*/_search. For cross-cluster aggregations, CCS supports full aggregations — data is merged at the coordinating node.

CCS security and performance need attention:

• CCS requires TLS (episode 16) and cross-cluster credentials — use remote_cluster_client with API keys.
• Every CCS search adds latency (network hops) and burdens the source cluster.
• Use ccs_minimize_roundtrips to reduce round-trips; results can be slower for large aggregations.

#CCS Use Cases and Considerations

An important consideration: if a remote cluster is unreachable, a CCS search fails entirely (or partially, per the skip_unavailable setting). Set "skip_unavailable": true so one down cluster doesn't kill the whole search.

#Cross-Cluster Replication: Concepts

CCR copies indexes continuously between clusters with a leader-follower model: one index on the leader cluster, its copy on the follower cluster kept up to date. Its characteristics:

• One-way (leader → follower). For two-way, create a reverse replication too — carefully, with an eye on conflicts.
• Follower indexes are read-only — all writes must happen on the leader.
• Copies are near-real-time with measurable lag (milliseconds to seconds).

#Setting Up CCR

The requirements are the same as CCS: a registered remote cluster and active TLS. Then create a follower index on the target cluster using the remote name as the source:

PUT /remote-logs:logs-prod/_ccr/follow

{
  "remote_cluster": "remote-logs",
  "leader_index": "logs-prod"
}

Once running, changes on the leader (index, update, delete) are copied to the follower. To match the initial mapping, use _ccr/auto_follow with an index pattern:

PUT /_ccr/auto_follow/logs-follow-pattern

{
  "remote_cluster": "remote-logs",
  "leader_index_patterns": ["logs-*"]
}

With auto follow, every new logs-* index on the leader automatically gets a follower — very useful for time-series data that rolls over continuously.

#CCR Use Cases and Lag Monitoring

#Monitoring Replication Lag

Replication lag — the time difference between a write on the leader and its appearance on the follower — is CCR's most important metric:

GET /remote-logs:logs-prod/_ccr/info

{
  "follower_indices": [
    {
      "follower_index": "logs-prod",
      "leader_index": "logs-prod",
      "remote_cluster": "remote-logs",
      "time_since_last_read_millis": 850,
      "follower_checkpoint": 4210
    }
  ]
}

A small time_since_last_read_millis means low lag. Growing lag indicates network problems, an overwhelmed follower, or a slow leader — monitor and alert on it (episode 21). Don't forget to add ccr metrics to monitoring.

#Common Mistakes

1. Using CCS when you need a copy. CCS without network reduction doesn't replace CCR for DR.

2. Forgetting skip_unavailable. One down cluster kills the entire CCS search.

3. Writing to a follower index. Followers are read-only — write to the leader, then wait for sync.

4. Ignoring replication lag. Growing lag is a waiting disaster — monitor and alert.

5. TLS not enabled between clusters. CCS/CCR connections must be encrypted.

#Conclusion

In episode 22 you mastered cross-cluster search and replication: the CCS concept with remote clusters and data federation, setting up remote clusters with seed hosts, running cross-cluster searches along with their performance and security considerations, then CCR with the leader-follower model, follow and auto follow setup, and replication lag monitoring.

Key takeaways:

• CCS searches across clusters without copying data; CCR copies data continuously.
• CCS needs skip_unavailable so one down cluster doesn't kill searches.
• CCR is one-way, follower read-only, with measurable lag.
• Auto follow keeps time-series data synced automatically.
• Monitor time_since_last_read_millis as a replication health indicator.

Connected clusters provide big data — and big data can be processed further. In episode 23 we'll cover Elasticsearch's machine learning features: anomaly detection with ML jobs, single vs multi-metric, population analysis; data frame analytics for outliers, regression, and classification; and the 8.x NLP features — ELSER, semantic search, and embeddings integration. See you there!