Learn Calico - Basic Observability

#Introduction

A "healthy" CNI isn't something you just feel — you have to be able to prove it. Episode 7 equips you with ways to read Calico's state: from one-line debugging commands to Prometheus metrics you can put on a dashboard.

Calico observability works on three layers: status (are the components alive and is BGP up), policy state (what is applied to which workload), and dataplane (are the rules actually installed). You'll be able to answer the question "why can't pods access each other?" with data, not guesses.

#Debugging with calicoctl

#Node and BGP Status

The first command to memorize is calicoctl node status:

calicoctl node status

Expected output:

Calico process is running.
 
IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 172.18.0.2   | node-to-node mesh | up    | 10:20:05 | Established |
+--------------+-------------------+-------+----------+-------------+

The STATE column contains up or down, and the INFO column is usually Established. If Established doesn't appear, BGP isn't finished yet — episodes 8 and 19 will cover it.

#Inspecting Dataplane Resources

The following commands are your Calico observability starter kit:

calicoctl get nodes
calicoctl get ippool -o wide
calicoctl get workloadendpoints -A
calicoctl get networkpolicy -A
calicoctl get felixconfiguration default -o yaml

calicoctl get workloadendpoints -A shows pod endpoints with their node, IP, and interface. If a pod doesn't show up here, Calico hasn't seen that pod yet — usually a kubelet or CNI problem.

#Flow Logs

#What Are Flow Logs

Flow logs are the connection records Calico generates — who talked to whom, on which port, and whether it was allowed or denied. In Calico Open Source, flow logs are available via a Fluentd integration, while the fuller edition uses a managed log pipeline. The pattern is the same: collect dataplane logs, send them to a pipeline, then visualize.

Here's an example of following the Felix logs on each node to see the actions taken:

kubectl logs -n calico-system ds/calico-node | grep -iE "action|deny|allow"

Grepping "action|deny|allow" shows the lines recording dataplane policy decisions. This is the best shortcut before building a full logging pipeline.

#Prometheus Metrics

#Felix and Typha Metrics

Felix and Typha expose Prometheus metrics on port 9091/metrics. Enable scraping first by labeling the pods so they can be scraped, then access them directly from inside a pod:

kubectl exec -n calico-system ds/calico-node -- sh -c \
  "curl -s localhost:9091/metrics | grep calico_felix"

Important metrics to monitor:

• calico_felix_iptables_total_*: the number of rules installed per chain.
• calico_felix_active_local_endpoints: active endpoints on this node.
• calico_felix_num_...: global counters that indicate Felix health.
• Typha metrics with the typha_ prefix: client counts and connection errors.

#Monitoring Service

For automatic scraping, point a service monitor at the labeled pods (a common pattern with kube-prometheus-stack). The services that expose metrics are usually named felix-metrics-svc and typha-metrics-svc in the calico-system namespace:

kubectl get svc -n calico-system | grep metrics
kubectl get svc -n calico-system felix-metrics-svc -o yaml

With a ServiceMonitor, these metrics automatically flow into Prometheus and can be shown in Grafana alongside other Kubernetes metrics.

#Dataplane Troubleshooting

#Inspecting Rules on a Node

Sometimes the API status looks right but traffic is still blocked. That's when you drop down to the dataplane and inspect iptables/nftables on the node:

kubectl exec -n calico-system ds/calico-node -- iptables -L -n | head -40
kubectl exec -n calico-system ds/calico-node -- iptables -S cali-PREROUTING

iptables -L -n shows the chains programmed by Felix. If the cali-* chains are missing, Felix failed to program the dataplane — check the Felix logs and daemon status.

#Investigating Endpoints and Policy

For "pod can't access" cases, follow this sequence:

kubectl get pods -o wide
calicoctl get workloadendpoints -n <namespace> -o wide
calicoctl get networkpolicy -n <namespace>
kubectl exec <client> -- nc -zv <target> <port>

The calicoctl get workloadendpoints step confirms Calico sees both pods, then kubectl exec ... nc -zv tests the real connection. If the connection fails but the policy looks correct, continue checking on the source and destination nodes.

#Conclusion

Episode 7 builds your observability habits: reading status with calicoctl node status, inspecting dataplane resources, following flow logs, pulling Prometheus metrics, and investigating the rules inside a node.

Key takeaways:

• calicoctl node status is the first gateway: check BGP state and processes.
• A missing workload endpoint means Calico hasn't seen the pod.
• Flow logs can be grepped directly from Felix logs before building a pipeline.
• Felix and Typha expose metrics on port 9091/metrics.
• Dataplane rules can be verified with iptables inside calico-node.
• Investigation order: endpoints, policy, then a real connection test.

In the next episode, episode 8, we cover BGP peering and routing — node-to-node peer configuration, full-mesh, route reflectors, BGPConfiguration, BGPPeer, peering with external routers, and the MetalLB integration.