Learn Calico - Encapsulation & Overlay Options

#Introduction

In episode 4 you met ipipMode and vxlanMode. Now let's dissect both: what's the difference between IPIP and VXLAN, when to use each, and when you shouldn't use an overlay at all. This is an architectural decision that affects performance and network compatibility.

A simple rule guides us: use an overlay when the underlying network can't route pod IPs; use direct routing when the underlying network is ready to accept pod routes. Both are configured per IPPool.

#IPIP vs VXLAN

#How IPIP Works

IPIP wraps a pod packet inside another IP packet: the original IP header becomes the payload, and the outer header uses the node IP. The tunnel header is very thin — about 20 bytes — because there's only one IP header. The encapsulated protocol must be IP, and external ports and connections don't need to know about the tunnel.

#How VXLAN Works

VXLAN wraps packets in UDP (port 4789) with a VXLAN header. The header is larger, but because it's UDP-based, VXLAN can traverse networks that filter exotic protocols, and it's friendlier for clouds that restrict certain protocols. VXLAN is also the primary choice when working with VMs or infrastructure that doesn't support IPIP.

#Quick Comparison

                     IPIP                  VXLAN
Tunnel header        ~20 byte              ~50 byte
Transport            IP-in-IP              UDP 4789
Cloud support        limited               broad
MSS/MTU overhead     small                 larger

The header difference is small, but in a high-traffic cluster, 20-30 bytes per packet can add up. VXLAN more often "just works" on public clouds because it's UDP-based.

#Choosing the Encapsulation Mode

#Mode per IPPool

The mode is set via ipipMode and vxlanMode with the values Always, CrossSubnet, or Never. Here's an example of a VXLAN IPPool for cross-subnet only:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 192.168.0.0/16
  blockSize: 26
  ipipMode: Never
  vxlanMode: CrossSubnet
  natOutgoing: true

vxlanMode: CrossSubnet means the tunnel is only used when the destination node is on a different subnet — traffic between nodes in the same subnet stays direct. The Always value encapsulates all inter-node traffic.

#Changing the Mode Without Downtime

Change the IPPool, then restart calico-node so the new routes are programmed:

kubectl patch ippool default-ipv4-ippool --type merge \
  -p '{"spec":{"ipipMode":"Never","vxlanMode":"Always"}}'
kubectl rollout restart ds/calico-node -n calico-system
calicoctl get ippool default-ipv4-ippool -o yaml

kubectl patch ippool changes the mode online, and calicoctl get ippool -o yaml verifies the result.

#Direct Routing: No Overlay

#When to Use Direct Routing

If the underlying network can already route pod IPs — for example, on-prem with routers peered via BGP, or a cloud with private networking that supports pod IPs — set both modes to Never:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 192.168.0.0/16
  blockSize: 26
  ipipMode: Never
  vxlanMode: Never
  natOutgoing: true

With direct routing, pod packets go out as-is (NAT only for non-pod destinations), full MTU, and no tunnel overhead. This is the most preferred mode when possible because it's the fastest and most transparent.

#MTU: The Often-Forgotten Detail

Every tunnel layer reduces the MTU. If packets are too large, fragmentation or drops occur. Adjust the MTU with FelixConfiguration:

kubectl patch felixconfiguration default --type merge \
  -p '{"spec":{"vxlanMTU":1450,"vxlanVNI":4096}}'

vxlanMTU: 1450 compensates for the VXLAN header on networks that use an MTU of 1500. If your underlying network uses a jumbo MTU, adjust this value carefully.

#Combinations: Cloud and On-Prem

#Cloud Patterns

On public clouds that don't route pod IPs (for example, a VPC without a mode like AWS "IPs directly", or GCP where not all overlays are supported), Calico's default uses IPIP Always. You can keep this pattern, or switch to VXLAN CrossSubnet to reduce inter-subnet overhead.

#On-Prem and Bare-Metal Patterns

On bare-metal with BGP to routers, direct routing (Never/Never) is the production standard. The most common combinations you'll see in the field:

calicoctl get ippool -o yaml | grep -E "ipipMode|vxlanMode"
kubectl exec -n calico-system ds/calico-node -- ip -s link show tunl0

Check tunl0 (the IPIP tunnel) and the VXLAN interface (vxlan.calico) to confirm the tunnel is actually being used. An interface with zero statistics means its mode isn't active for traffic.

#Conclusion

Episode 9 closes the encapsulation topic: IPIP for minimal overhead, VXLAN for broad compatibility, direct routing for maximum performance, and MTU as the hidden variable that often causes trouble.

Key takeaways:

• IPIP uses an IP-in-IP header (~20 bytes); VXLAN uses UDP (~50 bytes).
• Mode values: Always, CrossSubnet, and Never.
• CrossSubnet balances: direct within a subnet, tunnel across subnets.
• Direct routing is fastest when the underlying network can route pods.
• MTU must be adjusted in FelixConfiguration according to the tunnel mode.
• Verify tunnels via the tunl0 and vxlan.calico interfaces.

In the next episode, episode 10, we cover egress access control — EgressGatewayPolicy and egress IPs for outbound traffic with a fixed IP, NAT to external networks, and the practice of securing access to external databases and source IP whitelists.