Learn Cilium - Cilium Service Mesh & Gateway API

#Introduction

Service meshes have long been synonymous with sidecars: a proxy injected into every pod, consuming extra CPU and memory. Cilium Service Mesh offers a different direction: because Cilium already has an eBPF dataplane on every node, it can provide service mesh features — mTLS, L7 routing, and observability — without injecting a sidecar into every pod.

Episode 15 dissects the Cilium Service Mesh architecture, the identity-based mTLS mechanism, the Gateway API implementation with HTTPRoute and TLSRoute, and the steps to integrate it with a cluster that already uses Cilium for networking. After this episode, you will be able to compare it fairly with Istio — which we will do in episode 22.

Before diving into details, let's clarify a difference that often confuses: Cilium Service Mesh is not a literal replacement for or imitation of Istio. It takes a different approach — leveraging the dataplane that already exists — so the trade-offs between features and efficiency differ too. We will discuss that comparison at greater length in episode 22.

#Service Mesh Without Sidecars

The basic concept is simple: instead of a proxy per pod, Cilium uses an L7 proxy (Envoy) that runs as a per-node daemonset. All pods on a node share the same proxy, and the eBPF dataplane directs traffic that needs L7 processing to that proxy. Pods are not modified, there are no extra init containers, and there is no per-pod resource bloat.

Enable the service mesh components at install time:

cilium install --set l7Proxy=true --set envoy.enabled=true

--set l7Proxy=true enables the L7 proxy and envoy.enabled=true ensures the per-node Envoy component is installed. With this, L7 routing and mTLS features are already usable without injecting a sidecar.

In the latest versions, Cilium is also testing ztunnel — a lighter layer data path for processing mesh traffic without the full Envoy overhead. We will touch on ztunnel implementation details again when we discuss the 1.19 and 1.20 releases in episode 20.

This daemonset pattern has an important consequence: per-node proxy capacity must be planned for, because all pods on that node share the same Envoy. A pod with very high L7 traffic can affect its neighbors on the same node. For workloads with extreme L7 needs, consider placing them on a separate node or giving Envoy more resources.

#Identity-Based mTLS

mTLS (mutual TLS) guarantees two directions: the client verifies the server, and the server verifies the client — all with cryptography. In the Cilium Service Mesh, the identity used for certificates is not a CN or a service account, but the Cilium identity we learned about in episode 5. This means identity-based access policies automatically align with the mTLS mechanism.

The consequence is interesting: even if a pod steals another pod's IP, mTLS still rejects it because its identity is different. This closes one of the biggest weaknesses of purely network-based policies. Service-to-service encryption is handled transparently — the application does not need to know, similar to the transparent encryption in episode 14, but this time at the service level.

The certificates used for mTLS are issued and rotated automatically by Cilium. You do not need to inject secrets or manage a CA manually for the common case. Integration with the ClusterMesh trust domain (episode 17) makes these certificates valid across clusters, so mTLS and policies remain consistent across the entire mesh.

#Gateway API: HTTPRoute and TLSRoute

The Gateway API we touched on in episode 12 becomes the foundation of Cilium's service mesh routing. HTTPRoute handles path- and header-based routing, while TLSRoute handles TLS passthrough termination. Here is an example route that splits traffic by path:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: route-api
spec:
  parentRefs:
    - name: gw-utama
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: api-v1
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /v2
      backendRefs:
        - name: api-v2
          port: 8080

path: {type: PathPrefix, value: /v1} matches requests with the /v1 prefix and forwards them to the api-v1 backend. This is an example of L7 routing executed by Cilium's proxy at the cluster entry path — no separate ingress rewrite is needed.

Besides HTTPRoute and TLSRoute, the Gateway API also supports GRPCRoute for gRPC-based routing. You can pick the route type that fits your application's protocol: HTTPRoute for REST, GRPCRoute for gRPC, and TCPRoute for L4. The route type you choose determines how Cilium processes and routes traffic at the gateway.

#Integration with Existing Clusters

Cilium Service Mesh does not force an all-or-nothing choice. Because the service mesh stands on top of the dataplane that already exists, you can adopt it incrementally:

1. Make sure Cilium networking is already stable (episodes 3-4).
2. Enable the L7 proxy and Envoy, then run cilium connectivity test.
3. Apply an HTTPRoute or L7 policy for one pilot service.
4. Observe the performance impact with Hubble before expanding scope.

The advantage of this incremental adoption is important: there is no "big bang" like the one that usually happens with a sidecar-based service mesh, where all pods have to be restarted at once.

#Verifying Mesh Components and mTLS

After enabling the service mesh components, make sure everything is really running before writing your first route:

kubectl get ds -n kube-system cilium-envoy
cilium status | grep -i envoy
kubectl get pods -n kube-system -l k8s-app=cilium-envoy -o wide

kubectl get ds -n kube-system cilium-envoy shows the per-node Envoy DaemonSet — the number of pods must match the number of nodes. cilium status | grep -i envoy shows the L7 proxy status from the agent's perspective. If these components are not ready, HTTPRoute and mTLS will not work even if they have been applied.

To prove mTLS is working, observe encrypted L7 traffic:

hubble observe --type l7 --since 10m

hubble observe --type l7 --since 10m shows L7 flows processed by the proxy. These flows show the HTTP method, path, and status code — a direct window into what applications are actually doing at layer seven, not just IPs and ports.

Before enabling mTLS for the whole mesh, run a small experiment: apply an mTLS policy for one pair of services, observe the connections from Hubble, then expand. This pilot-service pattern is the same as the canary policy pattern in episode 18 — it reduces the risk of a global change that is hard to roll back if some workload turns out to be incompatible.

#Closing

Key takeaways:

• Cilium Service Mesh is sidecarless: the Envoy proxy runs per node, not per pod.
• mTLS uses Cilium identity, so it stays in sync with label-based policies.
• l7Proxy and envoy.enabled activate the mesh components.
• HTTPRoute and TLSRoute are the primary routing tools from the Gateway API.
• Adoption can be incremental because the mesh stands on the existing dataplane.
• Measure the overhead with a connectivity test before expanding scope.

In the next episode 16, we will cover runtime security with Tetragon — eBPF-based security for monitoring process execution, file access, and detecting suspicious behavior. You will install Tetragon, write a TracingPolicy, and connect it with Cilium for a comprehensive security posture.