Learn Cilium - Bandwidth Management & QoS

#Introduction

In a dense cluster, one bandwidth-hungry workload can starve other workloads on the same node. Episode 11 discusses Cilium's tool for handling this: the eBPF-based bandwidth manager. With the bandwidth manager, you can set throughput limits per pod, use BBR pacing for efficiency, and observe the results directly from the data plane.

This topic falls into the QoS (Quality of Service) category. Unlike CPU and memory, which are governed by the Kubernetes scheduler, bandwidth has long been difficult to control at the CNI level. Cilium changes that by moving enforcement into the kernel via eBPF.

This topic is often considered niche, yet its impact is felt every day. In a cluster shared by many teams, one workload saturating the network can break the SLOs of all tenants on the same node. The bandwidth manager gives you the tool to set limits — and just as importantly, a way to measure whether those limits are being enforced.

#The eBPF-Based Bandwidth Manager

The bandwidth manager is a Cilium feature that enables eBPF qdisc (queue discipline) on every pod. Instead of using classic heavyweight QoS mechanisms, Cilium attaches eBPF programs that limit transmission speed directly. As a result, bandwidth limits are enforced with minimal overhead and high accuracy.

Enable the bandwidth manager at install time:

cilium install --set bandwidthManager.enabled=true

cilium install --set bandwidthManager.enabled=true enables this feature. For kernel 5.18 and above, Cilium automatically uses the more modern FQ-BBR; on older kernels, it uses a fallback algorithm that still enforces bandwidth well.

#Pacing with BBR

BBR (Bottleneck Bandwidth and Round-trip propagation time) is a Google congestion control algorithm focused on throughput, not just avoiding packet loss. In the context of the bandwidth manager, BBR does pacing: packets are released gradually at the specified rate, instead of being sent in bursts.

The result of BBR pacing: smoother bandwidth usage, small buffers, and consistent throughput — especially for high-latency traffic. Because pacing runs in the kernel via eBPF, applications do not need to change at all. You only set the limit, and the kernel enforces it.

One practical advantage of BBR worth understanding: it reduces packet loss. Classic congestion control algorithms (Reno or CUBIC) react to packet loss, so on networks with small buffers they can cause unnecessary throughput reductions. BBR measures the bottleneck proactively, achieving high throughput with fewer wasted packets — a good result even on limited bandwidth.

#The kubernetes.io/egress-bandwidth Annotation

The easiest way to set a bandwidth limit per pod is with an annotation on the pod or deployment. This annotation is already recognized by Kubernetes and picked up by the Cilium bandwidth manager:

apiVersion: v1
kind: Pod
metadata:
  name: pod-limit
  annotations:
    kubernetes.io/egress-bandwidth: "10M"
spec:
  containers:
    - name: nginx
      image: nginx

kubernetes.io/egress-bandwidth: "10M" limits the pod's outgoing traffic to 10 megabits per second. The value is written as a string with a clear unit, for example 1M, 10M, or 1G. Cilium reads this annotation when the pod is created and applies the appropriate qdisc on the pod's veth.

Note the value format: the value must be written as a string because it is consumed by the Kubernetes annotation admission, not a numeric value. A common mistake is writing 10 without a unit or 10MB with capital letters — make sure to use the recognized format, i.e., a number followed by a binary unit such as M or G.

#Understanding egress-bandwidth vs Kubernetes QoS

#Relationship to Kubernetes QoS

Kubernetes has a QoS concept for CPU and memory (Guaranteed, Burstable, BestEffort), but not for networking. The kubernetes.io/egress-bandwidth annotation is the bridge: it uses the same format as other QoS annotations, and Cilium executes it in the data plane.

Keep in mind that this annotation only governs egress, not ingress. To govern ingress bandwidth, you must use other mechanisms outside the pod — for example limits at the load balancer or traffic shaping at the gateway. This is an asymmetry that often surprises teams using this feature for the first time.

#Verifying the Rate Limit

To prove that the limit is really enforced, we can observe from the data plane and measure throughput directly. Check the bandwidth table inside the agent pod:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg bpf bandwidth list

cilium-dbg bpf bandwidth list shows the bandwidth manager entries: endpoint identity, address, and the applied limit. If the pod-limit pod appears with a 10M value, the qdisc is active.

A more tangible measurement: run a download from inside the pod and observe the speed:

kubectl exec pod-limit -- sh -c "wget -qO /dev/null http://speedtest-berat:80/file.bin"

The command above will download a large file and its throughput will be capped at roughly the annotation value. Compare it with a pod without the same annotation — that pod should be able to reach far higher speeds. The impact on latency can also be observed: a limit that is too low will add queuing at the qdisc, visible as increased RTT in ping.

For more precise measurement, you can use a tool like iperf3 if available in the pod image. Run an iperf server in one pod and a client in the limited pod, then compare the results with a pod without the annotation. This difference in numbers is the proof that the bandwidth manager is truly enforcing the limit on the data path.

#Kernel Prerequisites and Qdisc

The bandwidth manager depends on kernel capabilities, so first check whether the node supports the required features:

uname -r
cat /sys/module/sch_fq/parameters/nopacing 2>/dev/null || echo "modul belum dimuat"
tc qdisc show dev eth0

uname -r shows the kernel version; for FQ-BBR, Cilium requires kernel 5.18 or newer. tc qdisc show dev eth0 shows the queue discipline on the interface — when the bandwidth manager is active, you will see an eBPF qdisc attached to the pod's veth interface.

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg status | grep -i bandwidth

cilium-dbg status | grep -i bandwidth shows the bandwidth manager status on that node. If this line does not appear or shows a disabled state, check the kernel version and the bandwidthManager.enabled value in Helm — the two most common causes before the feature works.

One practice worth adopting: document the bandwidth limits you apply along with their reasons, for example in manifest comments or a platform documentation page. When later there is a question like "why is this pod limited to 10M?", the answer must be findable in the repository, not in memory — the same principle as episode 18 on GitOps.

#Closing

Key takeaways:

• The bandwidth manager enforces bandwidth limits per pod via eBPF qdisc.
• BBR does pacing so transmission is smooth and throughput is consistent.
• The kubernetes.io/egress-bandwidth annotation sets the per-pod egress limit.
• cilium-dbg bpf bandwidth list shows the active bandwidth entries.
• Verify the limit by measuring throughput from inside the pod.
• Limits that are too tight raise latency because the queue fills up.

In the next episode 12, we will discuss egress and ingress gateways — how CiliumEgressGatewayPolicy funnels outgoing traffic through a specific IP for NAT, and how CiliumIngressGateway and the Gateway API integration open the door into the cluster. This connects our internal networking with the outside world.