Learn Cilium - Hubble Observability

#Introduction

The policies we created in episode 6 need to be proven, and that is the main reason Hubble exists. Hubble is Cilium's observability layer that reads flow logs directly from the eBPF data plane. Every connection that passes through Cilium produces a flow: who the source is, where it is heading, what protocol, and what the data plane's decision was — allowed or dropped.

Episode 7 covers how to enable Hubble, read flows with hubble observe, use the Hubble UI, and most importantly: turn flow logs into evidence for verifying policies and finding dropped traffic.

#Enabling Hubble

Hubble is not active by default during Cilium installation. Enable it with the Cilium CLI including the UI:

cilium hubble enable --ui

cilium hubble enable --ui installs the Hubble relay and Hubble UI components. The relay collects flows from all agents and provides an API for both the CLI and the UI. This process takes a few minutes until the hubble-relay pod is ready.

If using Helm, the equivalent configuration is hubble.relay.enabled=true and hubble.ui.enabled=true. Whatever the method, make sure everything is ready before continuing:

kubectl get pods -n kube-system -l k8s-app=hubble-relay
kubectl get pods -n kube-system -l k8s-app=hubble-ui

kubectl get pods -n kube-system -l k8s-app=hubble-relay must show the relay with Running status before we use the hubble CLI.

#hubble status and hubble observe

After Hubble is active, check the CLI connection status to the relay:

hubble status

hubble status shows the relay version and the number of flows currently being handled. If it shows an error, check the port-forward or connection configuration.

To see flows directly, use hubble observe:

hubble observe --since 5m

hubble observe --since 5m shows the flows of the last 5 minutes with the format: time, source, destination, protocol, and verdict. If there is no traffic yet, re-run the pod-to-pod curl from episode 4 or episode 6 to generate new flows.

#Understanding Flow Logs

One flow log line contains important information you need to get used to reading:

• Source and destination: the endpoint or IP communicating.
• Protocol and port: for example TCP 8080 or HTTP GET.
• Identity: the identity number involved at both ends.
• Verdict: FORWARDED means allowed, DROPPED means rejected by policy.
• Drop reason: the reason for rejection, usually Policy denied.

For more detailed flows, the JSON format provides all fields including labels:

hubble observe --since 10m --output json

hubble observe --output json produces complete output suitable for processing with jq or automated pipelines — a pattern that will be used again when doing production observability in episode 21.

#Hubble UI

Hubble UI provides a browser-based visual dashboard. After the port-forward, open it in your browser:

cilium hubble ui --open

cilium hubble ui --open port-forwards to the Hubble UI and opens it in your browser. On the dashboard you can see the cluster map, filter flows by namespace, service, or identity, and observe traffic patterns visually. The UI uses data from the same relay, so the information shown is consistent with hubble observe.

#Finding Dropped Traffic

Hubble's most valuable function for policy managers: finding rejected traffic. The verdict filter is very helpful:

hubble observe --verdict DROPPED --since 30m

hubble observe --verdict DROPPED shows only the flows dropped by the data plane, complete with the reason for rejection. This is the fastest way to answer the classic question: "why can't my applications reach each other?" — usually the answer is here, not in the application logs.

Combine filters to speed up diagnosis:

hubble observe --verdict DROPPED --namespace default

hubble observe --verdict DROPPED --namespace default narrows the search to one namespace. This pattern will become your first reflex when troubleshooting policies in episode 19.

#Closing

Key takeaways:

• Hubble reads flow logs directly from the eBPF data plane without extra agents.
• cilium hubble enable --ui enables the relay and dashboard.
• hubble observe shows flows; hubble status checks the relay connection.
• One flow contains source, destination, protocol, identity, verdict, and drop reason.
• The Hubble UI provides a visual view via cilium hubble ui --open.
• hubble observe --verdict DROPPED is the main door for policy troubleshooting.

In the next episode 8, we will discuss kube-proxy replacement and basic service mesh — how Cilium replicates ClusterIP, NodePort, and LoadBalancer via eBPF with socket load balancing, session affinity, and DSR, as well as the kube-proxy-replacement: strict configuration and cloud load balancer compatibility. This fundamentally changes how Services work in your cluster.