Learn Cilium - Performance & Troubleshooting

#Introduction

At this point you already know how to install and configure Cilium. Episode 19 flips the perspective: what happens when things do not go as expected? Networking at the kernel level is hard to see with the naked eye, so Cilium provides specialized diagnostic tools to dissect problems.

We will use cilium-dbg to read endpoints, identity, and policies; cilium-bugtool to collect diagnostic data; cilium monitor to watch traffic directly in the dataplane; and we will learn the troubleshooting patterns that appear most often in production.

One discipline we will use throughout this episode: diagnose before acting. Most networking incidents can be solved just by reading data correctly — not by blindly changing configuration. All the tools we use today exist to produce that data.

#Diagnostic Tool: cilium-dbg

cilium-dbg (formerly named cilium-agent) is a command that runs inside the agent pod. Each subcommand reads the dataplane state on that node. The three most frequently used commands:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg endpoint list
kubectl exec -n kube-system -it ds/cilium -- cilium-dbg identity list
kubectl exec -n kube-system -it ds/cilium -- cilium-dbg policy get

cilium-dbg policy get shows the active policies along with the allowed identities. When a pod cannot communicate, the diagnostic order starts here: is the endpoint there, is the identity populated, and does the policy allow the target identity pair.

Besides the three main commands above, cilium-dbg has many other subcommands: cilium-dbg status for node health, cilium-dbg bpf policy get to see policies at the bpf map level, and cilium-dbg troubleshoot to check connectivity from the kernel's perspective. Explore the available subcommands; every command is a window into a different dataplane layer.

#Recommended Diagnostic Flow

#Recommended Step Sequence

When an application cannot connect, follow this order so you do not miss the cause:

1. Observe flows with hubble observe --verdict DROPPED to see whether packets are being dropped.
2. If dropped, read the drop reason and the identities involved.
3. Check the policy with cilium-dbg policy get to compare the active rules.
4. Check the endpoint with cilium-dbg endpoint list to make sure identity is populated.
5. If the endpoint is problematic, check the agent and operator logs for initial clues.

This order handles the most common cases from top to bottom: from the symptom (drop) to the root (policy, identity, or configuration). Do not start from the bottom — checking configuration without looking at the symptoms usually wastes time with no results.

#cilium-bugtool and cilium monitor

When a problem is hard to guess, collect as much data as possible with cilium-bugtool — this command produces a tarball containing status, logs, and a dataplane snapshot:

kubectl exec -n kube-system -it ds/cilium -- cilium-bugtool

cilium-bugtool creates an archive you can share with teammates or send to Cilium maintainers. This bundle contains agent logs, endpoint state, policies, and relevant kernel configuration — far more complete than just copying the output of a single command.

To observe traffic directly on the data path, use cilium monitor:

kubectl exec -n kube-system -it ds/cilium -- cilium monitor

cilium monitor shows dataplane events in real time: packets received, forwarded, and dropped along with their reasons. This is the right tool when you suspect the problem is in the kernel, not in the application.

If the output of cilium monitor is too much (because it really does monitor all node traffic), combine it with filters: cilium monitor --type drop only shows drop events, and cilium monitor --type policy-verdict shows only policy decisions. These filters turn a noisy event stream into something readable.

#hubble observe --verdict DROPPED

The most effective combination for policy problems: observe drops from Hubble, then compare them with the active policies. We already met this pattern in episode 7:

hubble observe --verdict DROPPED --since 30m

hubble observe --verdict DROPPED --since 30m shows every rejected flow complete with a reason, usually Policy denied. From here, you know exactly which identity pair is being blocked, and you can decide: fix the policy, or the application really is trying to access something it should not.

It is also important to understand the difference in data sources: cilium monitor reads events directly from the dataplane of the node where the command runs, while hubble observe reads from the relay that aggregates all nodes. For a problem limited to one node, cilium monitor is more direct; for a problem spread out, hubble observe gives the full picture.

#Common Troubleshooting

Some of the most frequently encountered problems and their resolution patterns:

• Policy rejects traffic: check cilium-dbg policy get, match the source and destination identities, then make sure the rule selector really covers the labels in use. Widen the selector or add a rule as needed.
• Missing or wrongly labeled identity: new pods sometimes do not get an identity yet. Check cilium-dbg endpoint list and the operator logs; make sure the pod labels are correct because identity is derived from labels.
• IPAM runs out of pool: cilium-dbg ipam list shows the full allocations. The solution is to add pool capacity, shrink the per-node mask, or use Multi-Pool (episode 9).
• Upgrade between minors: follow the official Upgrade Guide; skipping one minor is usually safe, and every upgrade must be tested with cilium connectivity test.
• Kernel or eBPF problems: check the kernel version and ls /sys/kernel/btf. A kernel that is too old makes eBPF features fail to load — the most common symptom is Cilium pods crash-looping.

One more problem that often appears: a symptom that looks like a Cilium problem turns out not to be a Cilium problem at all. Before blaming the dataplane, make sure the basic flow is correct — the pod is running, the service selector matches, and the DNS name used is actually valid. Many "networking incidents" end up being a misspelled service name or a selector that does not match.

#Practice Scenario: A Pod Cannot Connect

Let's practice the flow above in one complete scenario. A frontend pod cannot access the backend pod. Run the diagnosis in order:

kubectl get pods -o wide
kubectl exec -n kube-system -it ds/cilium -- cilium-dbg endpoint list
hubble observe --verdict DROPPED --since 10m

kubectl get pods -o wide confirms both pods are really running and have IPs. cilium-dbg endpoint list checks whether both pods are registered as endpoints with a populated identity. hubble observe --verdict DROPPED --since 10m shows whether packets are being dropped and by which policy.

If the dropped flow shows an identity pair that should be allowed, the next step is to check the policy:

kubectl exec -n kube-system -it ds/cilium -- cilium-dbg policy get
kubectl get cnp -A
kubectl get ccnp

cilium-dbg policy get shows the active policies from the dataplane side. kubectl get cnp -A and kubectl get ccnp show the policy resources from the control plane side. Compare the two: if the policy exists in the control plane but does not appear in the dataplane, there is a synchronization problem — and if it is in neither, the problem is elsewhere, such as the wrong namespace or a label selector that does not match.

Practice patterns like this one — with dummy workloads in a separate namespace — are the best way to build troubleshooting reflexes before a real incident. Repeat with variations: a pod deliberately given the wrong label, a policy deliberately restricted by port, and identity deliberately left empty. The more scenarios you have seen, the faster you read the symptoms in production.

#Closing

Key takeaways:

• cilium-dbg endpoint list, identity list, and policy get are the first diagnostic commands.
• cilium-bugtool produces a complete diagnostic bundle to share.
• cilium monitor observes dataplane events in real time.
• hubble observe --verdict DROPPED reveals the reason traffic is rejected.
• Policy, identity, and IPAM are the three most common problem areas.
• Always test upgrades with cilium connectivity test.

In the next episode 20, we will cover the latest stable Cilium features in 1.19 and 1.20 — Network Policy refinements, the stable Multi-Pool IPAM, IPv6 progress, ztunnel transparent encryption, the stable MCS API for ClusterMesh, plus dataplane and Hubble updates. This brings you to the edge of current Cilium development.