Learn Cilium - Kube-Proxy Replacement & Basic Service Mesh

#Introduction

In episode 4 we saw a glimpse of how Cilium handles Services. Episode 8 covers that topic in full: kube-proxy replacement. Since the start of the series we mentioned that iptables-based kube-proxy does not scale — now it is time to prove it and see its replacement in action.

We will discuss how Cilium replicates all Service types via eBPF, what socket load balancing, session affinity, and DSR (Direct Server Return) are, as well as the kube-proxy-replacement: strict configuration and its interaction with cloud load balancers.

An initial note: kube-proxy replacement is transparent to workloads. Pods and Services keep working the same way; what changes is the path inside the kernel. This is one of those Cilium features that most often makes people hesitant before trying it, even though the risk of failure can be measured with cilium connectivity test.

#Why Replace kube-proxy

kube-proxy translates Services into iptables or ipvs rules on every node. The main problem with iptables: every time a Service changes, the entire chain must be updated — an expensive operation in large clusters. On top of that, every packet has to traverse many rules before finding a match.

Cilium replaces this with a hash table in eBPF: one service entry, one lookup, straight to the selected backend. No long chains and no massive updates. When a new Service is created, Cilium simply adds one row to the kernel table — an operation that is nearly instantaneous regardless of cluster size.

There is another, subtler reason: predictability. In iptables-based kube-proxy, rule updates happen in batches, and during that short period some packets can see an inconsistent state — some of the old Service, some of the new one. In Cilium, every Service entry is updated atomically in the kernel table, so there is no transition period where behavior is uncertain.

It is also worth noting that kube-proxy replacement is not only about performance. It simplifies the networking stack: one fewer component to operate, monitor, and debug. Reducing the number of moving parts is an operational win that is not always visible in benchmarks.

#Socket Load Balancing

One of eBPF's advantages that iptables cannot replicate: socket load balancing. For connections within the same node, Cilium can perform load balancing directly at the socket level, even before the packet leaves the application process. This means pod-to-pod communication on the same node does not need to go through the full kernel networking path — the result is very low latency.

This mechanism is automatically active when kube-proxy replacement is active. You do not need to change any application code; only the path the packet takes inside the kernel changes. Implementation details can be seen via cilium-dbg bpf lb list inside the agent pod:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg bpf lb list

cilium-dbg bpf lb list shows the load balancing entries in the data plane: service IP, port, backends, and backend health. Compare it with kubectl get svc — the two must be in sync.

One important note: socket load balancing is only active for connections that can be optimized at the socket level — usually pod-to-pod communication on the same node. For traffic that crosses nodes or enters from outside, the path still goes through eBPF programs at the network hooks. So this optimization complements, rather than replaces, load balancing on the general path.

#Session Affinity and DSR

Session affinity ensures connections from one client are always directed to the same backend for a certain period. This matters for applications with in-memory state. Enable it as usual on the Kubernetes Service, and Cilium applies it in the eBPF table:

apiVersion: v1
kind: Service
metadata:
  name: svc-affinity
spec:
  selector:
    app: frontend
  ports:
    - port: 80
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 600

sessionAffinity: ClientIP tells Cilium to keep the same backend for the same source IP. The implementation runs in the kernel, so it does not add significant latency.

One thing that is often asked: can session affinity be combined with balanced load balancing? The answer is yes, with a trade-off. Session affinity makes distribution depend on the source IP pattern; if one client dominates traffic, its selected backend will receive more load. For even load, leave session affinity off; enable it only when the application truly needs it.

DSR (Direct Server Return) is a technique where the backend's response returns to the client directly without passing through the node that received the initial request. This avoids a bottleneck at the ingress node and cuts one hop per connection. Cilium supports DSR for traffic from outside the cluster when the dsr mode is enabled.

#kube-proxy-replacement: strict

strict mode is a configuration where Cilium fully takes over kube-proxy's job. There is no more kube-proxy iptables working; all Services are handled by the eBPF data plane. Enable it at install time:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=strict

helm upgrade cilium cilium/cilium applies configuration changes without reinstalling. After the upgrade, verify the status:

cilium status | grep KubeProxyReplacement

cilium status | grep KubeProxyReplacement must show Strict mode. In this mode, the kube-proxy daemonset can be deleted to save resources, although leaving it is also safe as long as it does not interfere.

It is important to know: in strict mode, Cilium still works with cloud load balancers (AWS ELB, GCP LB, Azure LB). The cloud load balancer points to the NodePort on the nodes, and the eBPF data plane forwards the traffic to the backends. You do not need to change how external ingress works at all.

#Verifying the kube-proxy Replacement

After upgrading to strict mode, do not immediately trust the configuration — verify it from several angles:

cilium status | grep KubeProxyReplacement
kubectl exec -n kube-system -it ds/cilium -- cilium-dbg status --brief

cilium status | grep KubeProxyReplacement shows the active mode. cilium-dbg status --brief shows a summary directly from inside the node, including the KubeProxyReplacement and BPF LoadBalancing status — two lines that must show active status.

The most convincing verification step is to temporarily turn off kube-proxy and observe whether Services still work:

kubectl scale ds kube-proxy -n kube-system --replicas=0
cilium connectivity test
kubectl scale ds kube-proxy -n kube-system --replicas=1

kubectl scale ds kube-proxy -n kube-system --replicas=0 temporarily stops the kube-proxy daemon. If cilium connectivity test still passes, the eBPF data plane has truly taken over. Remember to scale the replicas back up after the test in a lab environment.

An important note for clusters using ipvs: iptables- and ipvs-based kube-proxy behave differently when stopped. Cilium manages this transition explicitly, so the scenario above is safe to do in a lab — but in production, do this test in a maintenance window and prepare a rollback before shutting down the old component.

#Closing

Key takeaways:

• Kube-proxy replacement swaps iptables chains for eBPF hash tables.
• Socket load balancing shortens the packet path between pods on the same node.
• Session affinity is implemented directly in the data plane via sessionAffinity: ClientIP.
• DSR cuts one hop for responses to traffic from outside the cluster.
• Strict mode makes Cilium fully replace kube-proxy.
• Cloud load balancers keep working normally on top of the kube-proxy replacement.

In the next episode 9, we will discuss IPAM (IP Address Management) — the cluster-pool mode, the Multi-Pool that is stable in version 1.19, cloud-native modes such as ENI and Azure, and dual-stack IPv4 and IPv6 support. The IPAM mode choice determines the pod capacity per node and the addressing strategy of your entire cluster.