Learn Cilium - Multi-Cluster & ClusterMesh

#Introduction

Modern organizations rarely stop at a single cluster. There are clusters per region, per environment, or per team — and suddenly a need emerges: an application in cluster A wants to call a service in cluster B. Episode 17 covers ClusterMesh, Cilium's feature for connecting those clusters transparently.

With ClusterMesh, pods in different clusters can communicate with each other as if they were in one big cluster: identity still applies, policies still apply, and services can be discovered across clusters. And with the MCS API (Multi-Cluster Services) being stable in Cilium 1.20, the way to define cross-cluster services becomes standard and clean.

One aspect worth realizing from the start: ClusterMesh is not a state replicator. It connects control planes and dataplanes between clusters, not application data. Understanding this matters so you do not set the wrong expectation when a cross-cluster service does not carry over the local pod state.

#ClusterMesh Concepts

ClusterMesh works by connecting the kube-apiserver and the Cilium dataplane between clusters. The clusters that join form a "mesh". Each cluster uses the same identity range — meaning the identity of a pod in cluster A is recognized and trusted in cluster B. Without this, cross-cluster policies could not be consistent.

Connection security between clusters is guaranteed by shared TLS certificates. Each cluster has a key pair and a CA that form the mesh trust domain. Consequently, ClusterMesh is not just a VPN between clusters — it is an extension of the identity model we have built since episode 5.

Because identity is recognized across clusters, the policies we wrote in episodes 6 and 13 apply across clusters without changes. A pod in cluster A labeled app=backend has the same identity in cluster B, so a fromEndpoints rule targeting that label works in both directions. This consistency is why ClusterMesh feels like one big cluster.

#Connecting Clusters

Connecting two clusters is done via the Cilium CLI. From the first cluster, enable ClusterMesh:

cilium clustermesh enable --service-type LoadBalancer

cilium clustermesh enable --service-type LoadBalancer installs the ClusterMesh components and exposes its service so other clusters can reach it. Save the generated values because the second cluster will use them.

From the second cluster, connect to the mesh:

cilium clustermesh connect --context cluster-b

cilium clustermesh connect --context cluster-b connects the active cluster to the first one using the prepared configuration. Verify the connection:

cilium clustermesh status

cilium clustermesh status shows the connection status to all clusters in the mesh. All connections must show an OK status before cross-cluster services are tested.

Before connecting many clusters, consider the geographic distance and latency between them. ClusterMesh does not optimize cross-cluster routes; each connection takes whatever path is available. For clusters across regions with high latency, think about whether your need really requires cross-region service discovery or whether ingress-level failover is enough.

#Cross-Cluster Service Discovery and the MCS API

To expose a service to other clusters, Kubernetes defines a standard called the MCS API: a ServiceExport resource to mark a service that may be visible across clusters, and a ServiceImport to represent it in the consumer cluster. In Cilium 1.20, the MCS API became stable — this is one of the most important releases for multi-cluster.

Export the service from the source cluster:

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: svc-akunting
  namespace: akunting

A ServiceExport named svc-akunting marks that this service may be discovered from other clusters. In the consumer cluster, a ServiceImport represents the same service, and pods can call it using the exact same name.

Verify that the cross-cluster service is visible:

kubectl get serviceimports.multicluster.x-k8s.io

kubectl get serviceimports.multicluster.x-k8s.io shows the services that have been imported from other clusters. From here, cross-cluster DNS works automatically — applications do not need to know which cluster provides the service.

It is worth understanding that the MCS API and ClusterMesh complement each other: ClusterMesh provides the data path between clusters, while the MCS API provides the standard way to export and import services. Both are stable in 1.20, and their combination is the foundation of the currently recommended multi-cluster architecture.

#Cross-Cluster Failover

The combination of ClusterMesh and ServiceImport opens up failover scenarios: one service exported from two different clusters. When one cluster has an outage, endpoints from the other cluster remain available and the Cilium dataplane forwards traffic to the healthy endpoints.

Note that this failover is not magic: applications that use per-cluster local state still need a data replication strategy. ClusterMesh handles connectivity and service discovery, not application state synchronization. For stateless services, cross-cluster portability works almost perfectly — this is the base pattern for multi-region architectures.

To test failover safely, do routine drills: temporarily shut down a service in one cluster, observe whether traffic moves to the other cluster, then bring it back up. These drills should be scheduled regularly, because a failover that has never been tested tends to fail when it is actually needed.

#Checking Mesh Health

Inter-cluster connections can be observed from several layers. Start from the ClusterMesh status, then connection details, down to proof in the dataplane:

cilium clustermesh status
kubectl exec -n kube-system -it ds/cilium -- cilium-dbg clustermesh status --verbose

cilium clustermesh status shows a summary of connections to all clusters. cilium-dbg clustermesh status --verbose gives per-connection details — including TLS status and the number of synchronized endpoints. When a cluster in the mesh shows an unhealthy status, the details here are the first source of information.

Also check that identity is consistent between clusters:

kubectl get ciliumidentity -n default
kubectl get ciliumnodes -o wide

kubectl get ciliumidentity -n default shows identities; kubectl get ciliumnodes -o wide shows nodes from all clusters in the mesh. If the number of nodes matches the total across all clusters, dataplane synchronization is working normally.

An important operational note: ClusterMesh adds cross-cluster control plane traffic. Every policy or endpoint change is synchronized to all clusters, so watch the kube-apiserver load on very large clusters. For large-scale multi-cluster architectures, plan this control plane capacity from the start — a topic we will touch on again in episode 21 when discussing production architecture.

#Closing

Key takeaways:

• ClusterMesh connects Cilium clusters with a shared trust domain.
• Identity is consistent across clusters, so policies and mTLS still apply.
• cilium clustermesh enable and cilium clustermesh connect link clusters.
• The MCS API, stable in 1.20, uses ServiceExport and ServiceImport.
• Cross-cluster services are reached by the same name from any cluster.
• Cross-cluster failover works for stateless services with data replication.

In the next episode 18, we will cover Cilium in GitOps and Policy as Code — managing CiliumConfig, helm values, and CNP/CCNP policies as code with Argo CD or Flux, plus best practices for versioning, review flow, and gradual rollout. This changes how teams operate Cilium from manual to reproducible.