Learn Cilium - Basic Networking (Pod-to-Pod & Service)

#Introduction

With Cilium installed, let us look at what it actually does every day: flowing traffic between pods and to services. Episode 4 dissects Cilium's basic networking — address allocation, the pod-to-pod path, masquerading, and how Services are handled entirely in eBPF.

Why is this episode important? Because almost all subsequent topics — policy, observability, even service mesh — stand on top of this basic networking. If you understand how a packet finds its way from pod A to pod B, understanding policy and Hubble will be much easier.

Throughout this episode we will use commands that run inside the Cilium agent pod. Make sure you are comfortable with the kubectl exec -n kube-system -it ds/cilium -- <command> pattern because this pattern will be used again and again in later troubleshooting episodes. These tools are the window into what is actually happening in the data plane.

#Address Allocation and Endpoints

When a pod is created, the runtime calls the Cilium CNI plugin. Cilium allocates an IP address from the IPAM pool (full details in episode 9), attaches a veth, and registers the pod as an endpoint. Every endpoint has an identity, an IP address, and metadata from the pod in question.

To see all endpoints on a node, run inside the agent pod:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg endpoint list

cilium-dbg endpoint list shows the endpoint table: IP address, pod labels, identity, and health status. The Identity column here is the number that will later become the key to Cilium's security model in episode 5.

#Pod CIDR and Subnets

Every cluster has a pod address space (pod CIDR) determined when the cluster is created. Cilium divides this space into per-node subnets, and each node allocates addresses for local pods from its own subnet. The subnet size determines the maximum pod capacity per node — a topic we will dig deeper into in episode 9 on IPAM.

Allocated addresses are recorded in the data plane and can be seen with the same command as the endpoint list. If you find an unknown address in the endpoint list, it is most likely a pod from a neighboring node currently being communicated with, or a leftover endpoint that has not been cleaned up.

#Pod-to-Pod: Intra- and Cross-Node

When two pods are on the same node, packets move through the bridge and veth. The eBPF programs at the egress and ingress hooks ensure packets reach the destination endpoint quickly, without leaving the physical network. This is the fastest path in the system.

When pods are on different nodes, the path depends on the datapath mode:

• Tunnel mode: packets are wrapped in VXLAN or Geneve and sent to the destination node.
• Direct routing: packets are sent directly to the destination node's IP based on the routing table.

In both modes, Cilium still applies identity and policy at both ends of the path. To test pod-to-pod connectivity, run a simple client and server:

kubectl run test-a --image=nginx --port=80
kubectl run test-b --image=curlimages/curl --command -- sh -c "sleep 3600"
kubectl exec test-b -- curl http://$(kubectl get pod test-a -o jsonpath='{.status.podIP}')

kubectl get pod test-a -o jsonpath='{.status.podIP}' fetches the destination pod's IP address dynamically. If the curl succeeds, the pod-to-pod path is working.

#Masquerading

When traffic leaves the cluster toward the internet, the pod IP address must be translated so the external network can route it. This process is called masquerading — equivalent to NAT. In Cilium, masquerading is done by eBPF programs, not iptables. This reduces node load and makes the process faster.

Masquerading behavior can be seen later through Hubble in episode 7: traffic leaving the cluster will show the source NAT process. To simply confirm that a pod can access the internet, run:

kubectl exec test-b -- curl -s -o /dev/null -w "%{http_code}" https://www.google.com

If the command above prints the code 200, masquerading and egress routing are working correctly.

One detail to understand: masquerading only applies to traffic that leaves the cluster's routing scope. For pod-to-pod communication within the cluster, the pod's original address is preserved so identity-based policy can work without interference. This means you can clearly distinguish between internal traffic that keeps its original identity and external traffic that has been NATed.

#Service Handling in eBPF

Kubernetes Services (ClusterIP, NodePort, LoadBalancer) are replicated in Cilium as lookup tables in eBPF, not as iptables chains. Every packet heading to a Service IP is looked up in this table and directly directed to one of the backend pods based on the load balancing algorithm.

See how Services look from the data plane perspective:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg service list

cilium-dbg service list shows the ClusterIP and NodePort Services along with their backends. Compare it with kubectl get svc — the two must be in sync. Because the handles live in the kernel, adding a Service does not create long rule chains like in kube-proxy; only a single entry in the hash table changes.

The load balancing algorithm used by Cilium is selectable: the most common are random and maglev. Maglev is a consistent hashing algorithm that ensures connections from the same client always land on the same backend as long as the backend list does not change — behavior similar to session affinity (episode 8) but computed purely deterministically in the kernel. The algorithm choice is set through the data plane configuration at install time.

To test Service access, create a Service for the test-a pod then call it from the test-b pod using the DNS name:

kubectl expose pod test-a --port=80 --name=svc-test
kubectl exec test-b -- curl -s http://svc-test:80

kubectl expose pod test-a --port=80 --name=svc-test creates a ClusterIP Service that is replicated directly into eBPF. The http://svc-test:80 call from another pod will be load balanced to the test-a pod by the data plane, not by kube-proxy.

#Inspecting the Packet Path Between Nodes

To understand how packets move between nodes, the data plane stores all the information needed. From inside the agent pod, run:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg node list
kubectl exec -n kube-system -it ds/cilium -- cilium-dbg bpf tunnel list

cilium-dbg node list shows the list of nodes known to the data plane along with the addresses used for inter-node communication. cilium-dbg bpf tunnel list shows the eBPF tunnel table — if this table contains entries, the data plane is running in tunnel mode and inter-node packets are wrapped accordingly.

These commands are useful when you suspect inter-node problems: whether the destination node is known to the data plane, and via which path packets are sent. Compare the results on a healthy node and a problematic node; small differences such as a wrong address or a missing tunnel entry are often the main clue.

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg bpf lb list

cilium-dbg bpf lb list completes the picture: if the load balancer table already contains the correct backends, the connectivity problem is most likely not in the data plane but in policy or the application. Reading the data plane in the order node, tunnel, then load balancer is a consistent pattern for all subsequent episodes.

#Closing

Key takeaways:

• Every pod becomes an endpoint with an IP, identity, and labels registered in the agent.
• Intra-node pod-to-pod goes through the bridge; cross-node goes through tunnel or direct routing.
• Masquerading is handled by eBPF so egress traffic can be externally routed.
• Services are replicated as eBPF hash tables, not iptables chains.
• cilium-dbg service list and cilium-dbg endpoint list are the primary data plane reading tools.

In the next episode 5, we will discuss the identity-based security model — how Kubernetes labels are mapped to numeric identity, identity-based security instead of IP-based, the difference between default allow versus deny, and how policy is evaluated directly in the data plane. This is the foundation of all Cilium policies we will build in episode 6 and beyond.