Learn Calico - Performance & Troubleshooting

#Introduction

All the previous episodes built up your capabilities, but in the real world what you'll need most often is the ability to fix things. Episode 19 is a toolbox of Calico troubleshooting techniques: from reading logs to solving the most common cases seen in the field.

Our approach is sequential and methodical: gather information, narrow down the problem layer (API, BGP, dataplane, IPAM), then apply the fix. Most Calico incidents are actually simple — the problem is we don't know how to look at the symptoms properly.

#Core Diagnostic Tools

#calicoctl for Inspection

The first set of commands always used during an incident:

calicoctl node status
calicoctl get nodes
calicoctl get ippool -o wide
calicoctl get workloadendpoints -A -o wide
calicoctl get networkpolicy -A -o wide

calicoctl get workloadendpoints -A -o wide shows pod endpoints. If a pod doesn't appear here, Calico hasn't seen that pod yet — the problem is in kubelet/CNI, not policy.

#Felix and Typha Logs

When status looks healthy but traffic is broken, check the Felix logs:

kubectl logs -n calico-system ds/calico-node | grep -iE "error|warn|felix" | tail -50
kubectl logs -n calico-system deploy/typha --tail=50

Grepping "error|warn|felix" in the calico-node logs is the fastest way to find dataplane messages. A typha with many errors points to API server communication problems.

#Investigating Applied Policy

For "policy rejected traffic" cases, check what's attached to the pod:

kubectl get pods -o wide
kubectl describe networkpolicy -n <namespace>
calicoctl get workloadendpoints -n <namespace> -o wide

kubectl describe networkpolicy shows the rules matching a particular pod selector.

#Common Cases and Their Solutions

#Pods Can't Communicate

Checklist order: make sure both pods appear in workloadendpoints, make sure there's a route on the destination node, and make sure no policy is rejecting. Test connectivity directly, then compare with policy:

kubectl exec <client> -- ping -c 3 <ip-target>
kubectl exec -n calico-system ds/calico-node -- ip route | grep <cidr-target>

If the ping fails but the route exists, the next suspect is policy. If the route is missing, the problem is in BGP or IPAM.

#Policy Rejecting Traffic That Should Pass

Likely causes: a wrong selector, a deny policy in a higher tier, or an FQDN/DNS rule that hasn't been allowed yet. Check the tier order and the action taken:

calicoctl get tiers
calicoctl get globalnetworkpolicy -o wide
kubectl logs -n calico-system ds/calico-node | grep -iE "deny|drop" | tail -20

The "deny|drop" lines in the Felix logs show the dataplane action taken on rejected packets.

#BGP Not Coming Up

BGP that isn't Established is usually due to a wrong peer IP, a mismatched AS number, or port 179 being blocked. Verify:

calicoctl node status | grep -A3 BGP
calicoctl get bgppeer -o wide
kubectl exec -n calico-system ds/calico-node -- \
  sh -c "ss -tn | grep 179"

ss -tn | grep 179 confirms the TCP connection to the BGP port is up. If it's not, check the firewall between the nodes.

#IPAM Problems

Exhausted or mispooled IPs usually show up as pods stuck in ContainerCreating with kubelet events. Check:

kubectl describe pod <pod> | grep -i -E "failed|ipam"
calicoctl ipam show --show-blocks
calicoctl ipam release --show-ip 192.168.0.10

calicoctl ipam show --show-blocks shows the remaining addresses per block; if a block is full, Calico will request a new block from the IPPool.

#Upgrading Versions Safely

#Upgrade Principles

Calico upgrades are done through the operator: update the operator version, then the operator upgrades the components. Follow the official steps and check the release notes for skipped versions:

helm repo update
helm upgrade calico projectcalico/tigera-operator --version v3.32.1
kubectl get tigerastatus

helm upgrade ... --version v3.32.1 replaces the operator image, and kubectl get tigerastatus monitors the component upgrade progress.

#Rollback

If an upgrade goes wrong, return to the previous version the same way, then confirm the datastore is still compatible. Always back up Calico resources before upgrading:

calicoctl get --export -o yaml > calico-backup.yaml
kubectl get crd | grep projectcalico.org | wc -l

#Wrap-Up

Episode 19 equips you with troubleshooting instincts: reading status with calicoctl, investigating Felix and typha logs, and solving common cases from pod connectivity to version upgrades.

Key takeaways:

• Diagnose in order: status, endpoints, routes, then policy.
• A pod missing from workloadendpoints means a kubelet/CNI problem.
• The Felix logs carry deny/drop actions for policy cases.
• BGP not up: check the peer IP, AS number, and port 179.
• IPAM: ipam show --show-blocks to see allocations and remaining addresses.
• Upgrade via the operator; back up resources before every upgrade.

Next, in episode 20, we cover the latest stable features — Calico v3.31 and v3.32 refinements, including ClusterNetworkPolicy, improved observability, migration from Flannel, and new VM support.