Learn Flannel - Subnet Management

#Introduction

Every Flannel network is rooted in one mechanism: the subnet lease. Without an understanding of leases, you will struggle to explain why one node gets the IPs 10.244.1.0/24 while another gets 10.244.2.0/24, let alone what happens during a conflict.

Episode 6 dissects this mechanism from concept to troubleshooting. You will learn how flanneld claims a subnet, where leases are stored, and how to make sure all nodes stay in sync.

#The Subnet Lease Concept

#One Large Pool, Small Per-node Subnets

Flannel takes a large pool, by default 10.244.0.0/16, and divides it into small per-node subnets, by default /24. This way a cluster can host up to 256 nodes and every node has 254 Pod addresses.

This process is automatic: when flanneld starts, it requests a lease from the datastore. The datastore guarantees two nodes never get the same subnet, because every allocation is recorded and locked.

kubectl get lease -n kube-system
kubectl get lease -n kube-system worker-1 -o yaml

The kubectl get lease -n kube-system command shows one Lease object per node. The holderIdentity field on the worker-1 Lease will show the subnet claimed by that node.

#Lease Validity

Every lease has a validity period and is renewed periodically by flanneld. If a node dies for a long time and its lease expires, its subnet can be reallocated to a new node. This is important to understand: a lease is not permanent ownership, but a loan that must be renewed.

#Datastore: Kube vs etcd

#Kubernetes API: The Default Mode

Kube mode uses the API server as the datastore, enabled via the --kube-subnet-mgr flag. flanneld uses its ServiceAccount to create and update Lease objects in the kube-system namespace. The advantage: no extra components, and consistency is guaranteed by the etcd behind the API server.

#etcd: The Historical Mode

etcd mode uses etcd directly as the datastore, without going through the Kubernetes API. This mode is useful for non-Kubernetes setups or upgrades from legacy installations, but it is more complex because you must manage the etcd endpoints, prefix, and certificates manually.

kubectl -n kube-flannel get ds kube-flannel-ds -o yaml | grep -E "kube-subnet-mgr|kube-subnet-mgr="

If the --kube-subnet-mgr argument is present in the DaemonSet, Flannel uses the Kubernetes API datastore.

#Seeing Leases in Action

#Reading subnet.env on a Node

The runtime result of the subnet manager is stored in the file /run/flannel/subnet.env. This file is read by the CNI plugin when creating a new Pod:

cat /run/flannel/subnet.env

The output of cat /run/flannel/subnet.env contains FLANNEL_NETWORK, FLANNEL_SUBNET, FLANNEL_MTU, and FLANNEL_IPMASQ. The FLANNEL_SUBNET value shows the subnet claimed by this node.

#Checking Remote Leases

To make sure all nodes are in sync, compare the leases in the API server with the routes on the node:

ip route | grep flannel
kubectl get lease -n kube-system -o jsonpath='{.items[*].spec.holderIdentity}'

Every subnet in the leases must appear as a route on all other nodes. If there is a lease without a matching route, inter-node synchronization has a problem.

#Troubleshooting Subnets

#Lease Conflict

A lease conflict happens when two nodes claim the same subnet. Common causes: a cluster restored from backup without cleaning up leases, or two clusters using the same 10.244.0.0/16 network that get merged together. The fix: delete the conflicting Lease object after confirming the node is indeed dead.

kubectl delete lease -n kube-system <node-name>
kubectl rollout restart ds/kube-flannel-ds -n kube-flannel

#Subnet Not Formed

If a lease never appears for a node, check the flanneld logs. Common causes: the ServiceAccount lacks permissions, or the net-conf.json ConfigMap is invalid so flanneld fails to start.

kubectl logs -n kube-flannel -l k8s-app=flannel

#Inter-node Synchronization

If some nodes don't see another node's subnet, check routes and FDB as we already learned. On the vxlan backend, make sure UDP port 4789 is open; on host-gw, make sure nodes are mutually reachable. Episode 19 will take you deeper into the complete troubleshooting workflow.

#Best Practices for Managing Leases

Make lease checks part of your operational routine. A healthy allocation appears as one subnet per active node. If the number of leases is unreasonable, for example far more than the number of nodes, there are old nodes that haven't been cleaned up. Clean them up only after confirming the node will truly not come back.

#Conclusion

Episode 6 completed your understanding of subnet management: the lease concept, dividing the pool into per-node subnets, the difference between the kube and etcd datastores, and how to resolve conflicts and synchronization issues.

Key takeaways:

• Every node claims one /24 subnet from the 10.244.0.0/16 pool.
• Leases are stored as Lease objects in the kube-system namespace in kube mode.
• Leases have a validity period and are renewed periodically by flanneld.
• subnet.env in /run/flannel holds the runtime results the CNI plugin reads.
• Lease conflicts usually come from backup restores or colliding CIDRs.
• Subnets that fail to form are usually due to permissions or an invalid net-conf.json.

In the next episode, episode 7, we will discuss iptables/nftables traffic management — masquerade rules for outbound traffic, forwarding between namespaces, the importance of br_netfilter since kubeadm 1.30, and the iptables versus nftables traffic manager modes. This determines how Pods talk to the outside world.