Learn Cilium - Production-Ready Architecture

#Introduction

Twenty episodes built the skills from scratch. Episode 21 puts everything into one framework: how to design a production-ready Cilium architecture. Not just installing Cilium, but planning IPAM, kube-proxy replacement, encryption, observability, upgrades, and daily operations coherently.

This blueprint is the result of experience handling real clusters. Every part — from the IPAM choice to the incident runbook — has its own rationale and is interconnected. You do not need to apply everything at once, but you do need to know which direction the architecture should grow.

One principle applies to the entire sequence above: document every decision along with its rationale. Six months later, the team will ask why direct routing was chosen instead of tunnel — and the answer should be found in the documentation, not in memory. An undocumented architectural decision is a decision that is hard to undo.

#Architecture Planning

Architecture decisions are best made before workloads arrive. The recommended order:

1. IPAM: calculate the maximum number of pods per node and choose cluster-pool or Multi-Pool (episode 9).
2. Kube-proxy replacement: enable strict mode from the start (episode 8) for performance and simplicity.
3. Datapath: choose direct routing or tunnel based on your network topology (episode 3).
4. Encryption: enable WireGuard (episode 14), especially if the cluster spans public networks.

An example production installation combining these decisions:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set encryption.enabled=true \
  --set encryption.type=wireguard \
  --set ipam.mode=cluster-pool \
  --set routing-mode=direct

--set kubeProxyReplacement=strict and the other options above are one package of decisions replicated across all environments. Store these values as a helm values file in the GitOps repository (episode 18) so they stay consistent between clusters.

#Observability and Monitoring

Production without observability means flying blind. Two layers must exist:

• Hubble: for flow-level observability and policy debugging (episode 7).
• Prometheus: for performance metrics, endpoint health, and long-term alerting.

Enable Prometheus metric collection at install time:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --set prometheus.enabled=true \
  --set operator.prometheus.enabled=true

prometheus.enabled=true makes the Cilium agent expose metrics for Prometheus to scrape. Key metrics to monitor: endpoint count, agent health, flow rate, and dropped packets. Build dashboards and alerts from these metrics — for example, an alert when the drop percentage suddenly rises.

#CI/CD and Operations

Changes to Cilium must pass the same quality gates as application changes. cilium connectivity test is the primary gate:

steps:
  - name: connectivity-test
    run: |
      cilium install --context staging --version 1.20.0
      cilium connectivity test --context staging

cilium connectivity test --context staging runs the full set of connection scenarios against the staging cluster before a change is carried to production. This pipeline prevents broken configuration from spreading.

Node sizing also falls under operations: reserve resources for the Cilium agent and operator (usually a few percent of the node), and make sure the kernels of all nodes meet the eBPF requirements before they join the cluster.

In addition to the connectivity test, also include policy validation in the pipeline: kubectl apply --dry-run=server to make sure policy manifests are valid before applying, and preview with kubectl diff to see what will change. This prevents syntax errors from reaching the cluster through the automated path.

#Upgrade Strategy and Runbook

Routine upgrades are part of life. The recommended pattern:

1. Upgrade one minor at a time (episode 20).
2. Test in staging with cilium connectivity test before production.
3. Use rolling node updates so traffic never stops completely.
4. Have an incident runbook for the worst-case scenarios.

A minimal incident runbook for Cilium:

• Application cannot connect: start from hubble observe --verdict DROPPED (episode 19).
• Cilium agent restarts repeatedly: check the agent logs and kernel eBPF support.
• IPAM exhausted: cilium-dbg ipam list, add pool capacity or widen the mask.
• After every step, re-verify with a connectivity test.

#Node Sizing and Resources

One of the most often deferred decisions is calculating the resources Cilium needs. Start from a check of the actual state:

kubectl top pod -n kube-system -l k8s-app=cilium
kubectl top pod -n kube-system -l app.kubernetes.io/name=cilium-operator
kubectl describe nodes | grep -A3 -B3 "eBPF\|cilium"

kubectl top pod -n kube-system -l k8s-app=cilium shows the agent's CPU and memory usage per node. kubectl top pod -n kube-system -l app.kubernetes.io/name=cilium-operator does the same for the operator. The numbers from kubectl top become the basis for realistic requests and limits, not estimates.

A sizing rule of thumb: allocate about 20 percent headroom above peak usage for the agent, and make sure the operator has enough resources for large clusters. The kernels of all nodes must meet the eBPF requirements before joining the cluster — check with ls /sys/kernel/btf from episode 1, because a node with an old kernel will become an operational burden later.

One thing that is often forgotten: observability resources also count. Hubble Relay and UI need their own CPU and memory, and if you store flows long-term, the disk for the flow database grows over time. Include all of this in the cluster cost calculation from the start, not after the bill arrives.

#Closing

Key takeaways:

• IPAM, kube-proxy replacement, datapath, and encryption decisions are made before workloads arrive.
• Production helm values are stored as code for consistency across clusters.
• Hubble for flow observability; Prometheus for metrics and alerting.
• cilium connectivity test becomes a quality gate in the CI/CD pipeline.
• Upgrades happen one minor at a time, tested in staging first.
• Minimal incident runbook: drop diagnosis, agent restarts, and IPAM exhaustion.

In episode 22, the final episode, we will cover the alternative ecosystem and final reflection — a comparison of Cilium vs Calico vs Flannel vs Weave, kube-proxy replacement vs traditional, Istio vs Cilium Service Mesh, a recap of the journey 0-21, a production-grade Cilium checklist, and the future direction of Cilium with eBPF, Tetragon, and multi-cluster.