Learn Flannel - Performance Optimization & MTU

#Introduction

Flannel works well by default, but networks are not always ideal: host MTUs differ, some hops have smaller packet limits, and network-heavy applications demand maximum throughput. This is where optimization begins.

Episode 16 covers calculating the correct MTU, how to avoid fragmentation, and enabling direct routing for intra-subnet traffic — all wrapped up with how to measure the results.

#Calculating the Correct MTU

#The 50-Byte VXLAN Overhead

Every packet that passes through a VXLAN tunnel adds 50 bytes of overhead. The effective MTU inside a Pod is the host MTU minus that overhead. If the host uses 1500, the Pod MTU must be 1450.

ip link show eth0
ip link show flannel.1

The ip link show eth0 command shows the host MTU, and ip link show flannel.1 shows the MTU calculated by flanneld. The difference must be 50 for the VXLAN backend.

#Avoiding Fragmentation

The most common mistake is a Pod MTU larger than what the tunnel path can carry. As a result, packets get fragmented along the way, performance plummets, or TCP connections fail. The typical symptom: small pings succeed, but large transfers stall midway.

The rule of thumb: lower the MTU to match the lowest path across the entire cluster. When in doubt, choose a smaller, stable MTU over a larger but fragile one.

#Setting the MTU in Flannel

#The MTU Field in net-conf

By default flanneld calculates the MTU automatically. You can force a specific value through the MTU field in net-conf.json:

net-conf.json: |
  {
    "Network": "10.244.0.0/16",
    "MTU": 1450,
    "Backend": {
      "Type": "vxlan"
    }
  }

The value 1450 is safe for hosts with an MTU of 1500. For hosts with 9000-byte jumbo frames, you can use 8950, as long as every hop on the network allows it.

#Testing the MTU After Changing It

After changing the MTU and restarting the DaemonSet, test with a packet the size of the MTU:

kubectl exec -it <pod> -- ping -c 2 -s 1450 -M do <ip-pod-node-lain>

The -M do option on ping requests don't fragment. If a 1450-byte packet succeeds, the MTU is configured correctly. If it fails, the MTU is too large for the path.

#Direct Routing for Intra-subnet

#When Direct Routing Works

Traffic between nodes on the same host subnet does not need encapsulation: the nodes can reach each other directly. The DirectRouting option in the VXLAN backend takes advantage of this by using a direct route for intra-subnet traffic, and falling back to VXLAN for cross-subnet traffic.

net-conf.json: |
  {
    "Network": "10.244.0.0/16",
    "Backend": {
      "Type": "vxlan",
      "DirectRouting": true
    }
  }

#Verifying the Routes from Direct Routing

Once enabled, routes to nodes on the same subnet become direct routes without the flannel.1 interface:

ip route | grep flannel

The output of ip route | grep flannel shows routes via dev flannel.1 for nodes on different subnets, and via dev eth0 for nodes on the same subnet. That is proof that direct routing is working.

#A Simple Benchmark

#Measuring Throughput

To prove the impact of the optimization, measure throughput before and after the change. A simple tool you can use: iperf inside a Pod.

kubectl run iperf-srv --image=networkstatic/iperf --restart=Never --command -- iperf -s
kubectl run iperf-cli --image=networkstatic/iperf --restart=Never --rm -it --command -- iperf -c <ip-pod-srv>

The iperf output shows the bandwidth between the two Pods. Compare the results across a regular vxlan backend, DirectRouting, and host-gw. The pattern will be consistent: the less overhead, the higher the throughput.

#Interpreting the Results

Don't just chase the highest number. Consider the trade-offs: host-gw is indeed the fastest but demands that nodes can reach each other directly. VXLAN with DirectRouting offers a good compromise for mixed networks. Use the benchmark data to support your decision in episode 21.

#Conclusion

Episode 16 optimized Flannel's performance: calculating the MTU with the 50-byte overhead, avoiding fragmentation, enabling direct routing, and measuring the results with a benchmark.

Key takeaways:

• The 50-byte VXLAN overhead determines the correct Pod MTU.
• An MTU that is too large causes fragmentation and dropped connections.
• The MTU field in net-conf.json can force the desired value.
• DirectRouting speeds up intra-subnet traffic without encapsulation.
• Verify the MTU with a ping -M do sized to the MTU.
• The iperf benchmark provides data for choosing the backend and configuration.

In the next episode, episode 17, we will step outside a single cluster: multi-cluster and advanced topology — building a cross-cluster overlay, understanding Flannel's limitations, mixing nodes with different backends, and advanced topologies such as IPv6 and restricting subnet allocation.