Learn Cilium - Core Concepts & Main Architecture

#Introduction

After understanding the history and motivation behind Cilium, it is now time to dissect how Cilium works behind the scenes. Episode 2 is the architecture map: what the Cilium Agent is, what the Cilium Operator is, how the eBPF data plane works, and which components make up the Cilium ecosystem.

Why is it important to understand the architecture before practicing? Because almost all of the troubleshooting in episode 19 later is rooted in understanding these components. When you see cilium status showing an agent that is not ready, you need to know which component is having the problem and why. Let us build that understanding now.

#The Two-Process Architecture: Agent and Operator

#The Cilium Agent on Every Node

The Cilium Agent is the main process that runs on every node as a DaemonSet. Its job is heavy: compiling eBPF programs according to the configuration, loading them into the kernel, keeping in sync with the Kubernetes API, managing endpoints (pods), and executing all networking decisions. When a new pod is created, it is the agent that attaches the veth, allocates the IP, and attaches the eBPF programs for that pod.

To see the agent running in your cluster:

kubectl get pods -n kube-system -l k8s-app=cilium

kubectl get pods -n kube-system -l k8s-app=cilium shows one agent pod per node. Because it runs as a DaemonSet, the number of pods will always equal the number of healthy nodes.

#The Cilium Operator in the Cluster

Unlike the agent that runs per node, the Cilium Operator is a process that runs per cluster (usually two replicas for redundancy). The operator does not handle the data path; it manages cluster-wide state: maintaining IP allocation quotas (IPAM), managing the identity list, handling garbage collection, and ensuring configuration is consistent across nodes.

This separation is architecturally important: everything that needs speed and local knowledge lives in the agent, while everything that needs the full picture lives in the operator. If the operator dies, network traffic keeps flowing, but operations like IP allocation for new pods will be disrupted.

#The eBPF Data Plane

The data plane is the data path where packets traverse the kernel. In Cilium, the data plane is built almost entirely from eBPF programs attached to several hooks at once: when a packet arrives from the NIC (tc ingress), when a packet goes out (tc egress), and for socket-level load balancing. No iptables is involved in the main path.

With eBPF, Cilium can handle L3 (routing between pods and between nodes), L4 (Service load balancing, port-based policy), and L7 (HTTP, gRPC, and Kafka inspection) in a single pipeline. Decisions are made inside the kernel at kernel speed, not in user-space. This is what makes Cilium excel in latency and throughput.

What needs to be emphasized: the term "without iptables" does not mean iptables is removed from the system. The kernel still has iptables, but the main pod-to-pod and pod-to-service data path no longer depends on it. Some additional functions, such as masquerading for certain traffic, still use other kernel mechanisms as needed.

To see the data plane status on a node, run it inside the agent pod:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg status

cilium-dbg status shows a summary of endpoints, controllers, ipam, health, and kernel. The Kernel Version output here is useful for confirming the node kernel meets the eBPF requirements.

#Main Components of the Cilium Ecosystem

Here are the components you will encounter frequently throughout the series:

• Cilium Agent: the per-node process that manages the data plane and endpoints.
• Cilium Operator: the per-cluster process that manages IPAM, identity, and garbage collection.
• Cilium CLI: the installation and verification tool from outside the cluster.
• Hubble: the observability component with relay, CLI, and UI.
• CiliumNetworkPolicy (CNP): networking policy scoped to a single namespace.
• ClusterwideNetworkPolicy (CCNP): policy that applies across namespaces.
• IPAM: management of pod IP address allocation.

Most of these components are installed automatically when Cilium is installed. Hubble needs to be explicitly enabled (episode 7), while CNP and CCNP are CustomResources we define via YAML.

#Who Does What: Division of Responsibilities

To make it easier to remember, here is the division of main responsibilities in the Cilium architecture:

• Cilium Agent: compiles eBPF, manages endpoints, enforces policy, and reports node state.
• Cilium Operator: allocates identity, manages IPAM, and performs garbage collection.
• Hubble Relay: collects flows from all agents and provides an observability API.
• Hubble UI: displays flows visually from the relay data.
• Cilium CLI: a helper tool from outside the cluster for install and verification.

When one component has a problem, the first question that always arises: which component is responsible for the failing function? This mindset will be very helpful in episode 19 when we discuss troubleshooting. Keep this table in mind, because it will be used throughout the series.

#The Journey of a Single Packet

Let us follow one simple packet from pod A to pod B on the same node:

1. The packet leaves the container of pod A through the veth toward the bridge.
2. The eBPF program at the egress hook reads pod A's identity from the socket metadata.
3. The data plane determines the destination endpoint based on pod B's IP address.
4. The egress and ingress policies are evaluated against both identities.
5. If allowed, the packet is forwarded to pod B's veth on the fast path.

When pod A and pod B are on different nodes, the path is extended with tunnel encapsulation (VXLAN/Geneve) or direct routing between nodes — depending on the datapath mode chosen at installation. We will compare these two modes in episode 3.

Note that in both cases, identity and policy are considered at every hop. This is what distinguishes Cilium from other CNIs: the network is not just a path, but also a security decision that is evaluated continuously. We will look deeper into this in episode 5.

#Seeing the Architecture Directly

Architecture concepts feel abstract until we see them with kubectl. Let us look at the components installed from episode 3:

kubectl get ds -n kube-system cilium
kubectl get deploy -n kube-system cilium-operator
kubectl get pods -n kube-system -l k8s-app=cilium -o wide

kubectl get ds -n kube-system cilium shows the agent DaemonSet. The DESIRED and READY columns must be equal — if not, there is a node whose agent failed to run. kubectl get deploy -n kube-system cilium-operator shows the operator deployment; note the AVAILABLE column which indicates how many replicas are ready.

To see the agent and operator processes more deeply, inspect their logs:

kubectl logs -n kube-system -l k8s-app=cilium --tail=20
kubectl logs -n kube-system deploy/cilium-operator --tail=20

kubectl logs -n kube-system -l k8s-app=cilium --tail=20 shows the last 20 log lines from all agents. When troubleshooting (episode 19), this is the pattern we use first: check whether there is an error on the agent of the problematic node, then compare it with a healthy agent node.

The mindset to remember: the agent is the "factory floor worker" that handles packets every day, while the operator is the "head-office management" that takes care of administration. When a packet fails, the problem is most likely on the relevant node's agent; when identity or IP allocation is having problems, the operator deserves suspicion first.

#Closing

Key takeaways:

• The Cilium Agent runs per node and manages the eBPF data plane and endpoints.
• The Cilium Operator runs per cluster and manages IPAM, identity, and global state.
• The eBPF data plane handles L3, L4, and L7 without iptables in the main path.
• The Cilium ecosystem includes the CLI, Hubble, CNP, CCNP, and IPAM.
• A single packet is processed through endpoints, identity, and policy, then forwarded to its destination.
• The tunnel vs direct routing datapath mode is determined at installation.

In the next episode 3, we will install Cilium — both via cilium install and the Helm chart, verify it with cilium status and cilium connectivity test, choose between tunnel and direct routing, and understand the integration with the containerd and CRI-O runtimes. This is the first episode where your cluster really changes.