MTU Optimization

MTU Optimization

MTU (Maximum Transmission Unit) is the largest packet size a network path can carry without fragmentation. The standard Ethernet MTU is 1500 bytes. When MTU is misconfigured — particularly in VPN tunnels, VXLAN overlays, or cloud networks with encapsulation overhead — it causes mysterious connectivity problems: large transfers hang, TCP connections established but no data flows, or HTTPS works but large downloads time out. Understanding MTU troubleshooting is essential for cloud and data center networking.

What is MTU?

Standard Ethernet frame:
  [Ethernet header: 14B][IP header: 20B][TCP header: 20B][Payload][FCS: 4B]
                         |←────────────── MTU = 1500 bytes ──────────────→|

With VPN/tunnel overhead:
  [Outer header][Inner packet ← must be smaller than 1500B to fit]
  VPN adds 50-100 bytes overhead → effective MTU for inner traffic = 1400-1450B

If inner packet is 1500B and effective MTU is 1400B:
  → fragmentation (if DF bit not set) or drop (if DF bit set)
  → connection appears to establish but hangs on data transfer

Path MTU discovery

Path MTU Discovery (PMTUD) is the TCP mechanism that finds the maximum packet size a path can handle. It sets the Don't Fragment (DF) bit on packets. If a router needs to fragment but DF is set, it sends back an ICMP "Fragmentation Needed" message. PMTUD breaks when these ICMP messages are blocked by firewalls — a common cause of "black hole" routing.

# Check current MTU of all interfaces
ip link show | grep mtu

# Check MTU of specific interface
ip link show ens3 | grep mtu

Output

2: ens3:  mtu 1500 qdisc fq_codel state UP

Finding the correct MTU

# Method: ping with specific packet sizes to find the maximum that works
# -M do = don't fragment
# -s = packet size (payload, add 28 for IP+ICMP header = total size on wire)

# Start high and work down:
ping -c 3 -M do -s 1472 8.8.8.8    # 1472+28=1500 (standard MTU)
ping -c 3 -M do -s 1400 8.8.8.8    # 1400+28=1428
ping -c 3 -M do -s 1450 8.8.8.8    # Binary search between 1400-1472

When MTU is too large

ping: local error: message too long, mtu=1400
# → The path cannot handle 1500B. The path MTU is 1400B.
# Set interface MTU to 1400 (or 1380 for some headroom)

# Find exact path MTU using tracepath
tracepath -n 8.8.8.8 | grep -i mtu

Jumbo frames

# Jumbo frames: MTU = 9000 bytes (or 9216 on some hardware)
# Use case: storage networks (iSCSI, NFS), high-throughput LAN transfers
# Requirement: ALL devices in the path must support jumbo frames

# Temporarily set jumbo frame MTU (test first!)
sudo ip link set ens3 mtu 9000

# Test that jumbo frames work end-to-end
ping -c 3 -M do -s 8972 STORAGE_SERVER_IP    # 8972+28=9000

⚠️ WARNING: Do not enable jumbo frames on an interface that connects to a switch that does not support them. The switch will drop jumbo frames silently, making the server appear unreachable on that interface. Enable jumbo frames only on storage/private networks where you control all devices in the path and can verify switch support.

Setting MTU in Netplan

# /etc/netplan/00-installer-config.yaml
network:
  version: 2
  ethernets:
    ens3:
      mtu: 1500       # Standard Ethernet — default, usually no need to set
      dhcp4: false
      addresses: [192.168.1.10/24]

    # Storage network with jumbo frames:
    ens4:
      mtu: 9000
      dhcp4: false
      addresses: [10.100.0.5/24]

sudo netplan apply
ip link show | grep mtu    # Verify MTU was applied

Conclusion

MTU mismatches cause symptoms like "connection works but no data transfers" or "small requests work, large downloads hang" — these are classic PMTUD black hole symptoms. Diagnose with ping -M do -s PAYLOAD_SIZE to find the path MTU. Set MTU persistently in Netplan using the mtu: key. For VPN/tunnel interfaces, lower the MTU to account for encapsulation overhead (typically 1380-1450 for most VPNs). For storage networks, jumbo frames (9000) significantly improve throughput but require all-or-nothing deployment across the entire path.