Network Troubleshooting

Network Troubleshooting

Network problems have five common causes: interface is down, IP/route is misconfigured, DNS is broken, a firewall is blocking, or the service is not listening. The systematic approach works from layer 1 (physical) up to layer 7 (application). Checking each layer in order prevents the mistake of debugging the wrong layer — the most common networking debugging error is jumping to DNS or firewall before confirming the interface is even up.

Systematic layer-by-layer approach

Diagnostic ladder (always check in this order):
  L1: Is the interface up?
       ↓
  L2: Does it have a valid IP address?
       ↓
  L3: Can you reach the default gateway?
       ↓
  L3: Can you reach an external IP (e.g., 8.8.8.8)?
       ↓
  L7: Can you resolve DNS names?
       ↓
  L4: Is the target port open and reachable?
       ↓
  L7: Is the service returning the expected response?

Interface and IP checks

# Step 1: Is the interface up and does it have an IP?
ip addr show

Healthy interface vs problem states

# Healthy:
2: ens3:  mtu 1500 ...
    inet 192.168.1.10/24 ...

# Problem: interface down (no LOWER_UP flag, no inet line)
2: ens3:  mtu 1500 ...

# Problem: APIPA — DHCP failed
2: ens3:  ...
    inet 169.254.x.x ...

# If interface is down, bring it up
sudo ip link set ens3 up

# If DHCP is expected, force DHCP renewal
sudo dhclient -v ens3

# Check for errors at the driver level
sudo dmesg | grep ens3 | tail -10

Gateway and routing checks

# Step 2: Verify routing table and default gateway
ip route show | grep default
# Should show: default via GATEWAY_IP dev INTERFACE

# Step 3: Ping the gateway (layer 3 local reachability)
ping -c 3 $(ip route show default | awk '{print $3}')

# If gateway ping fails:
# - Check ARP: ip neigh show | grep GATEWAY_IP
# - Check VLAN tagging (if gateway is on a VLAN)
# - Check physical cable/switch port

# Step 4: Ping an external IP (bypasses DNS, tests internet routing)
ping -c 3 8.8.8.8
ping -c 3 1.1.1.1

# If external IP ping fails but gateway ping works:
# - Problem is upstream of your gateway (ISP, datacenter)
# - Check if your IP is NAT'd correctly

DNS and firewall checks

# Step 5: Test DNS resolution
dig google.com              # Full DNS test
dig @8.8.8.8 google.com    # Test bypassing local resolver

# If external IP works but DNS fails:
resolvectl status            # Check configured DNS servers
sudo systemctl status systemd-resolved

# Quick fix for DNS issue: test with Google DNS directly
# If "dig @8.8.8.8 works" but "dig google.com fails":
#   → Problem is with systemd-resolved or your configured DNS server

# Step 6: Check if a firewall is blocking
sudo iptables -L -n -v | grep -v "0     0"    # Show active rules with hits
sudo ufw status verbose

Service and port checks

# Step 7: Is the service listening on the expected port?
ss -tlnp | grep :80      # Is nginx listening on 80?
ss -tlnp | grep :3306    # Is MySQL listening?
ss -tlnp | grep :22      # Is SSH listening?

# Step 8: Can you reach the port from a remote host?
# From another machine:
nc -zv 192.168.1.10 80        # TCP port test
nc -zv 192.168.1.10 22
telnet 192.168.1.10 80        # Alternative

# From the same machine:
curl -v http://localhost:80
curl -v http://192.168.1.10

# Step 9: Test full application response
curl -I https://example.com   # Check HTTP response headers
wget -q -O /dev/null --server-response http://192.168.1.10 2>&1 | head -10

Conclusion

The diagnostic ladder prevents wasted time: always start at layer 1 (interface up?), confirm IP and route, then ping gateway, then ping external IP, then test DNS, then test the port. Each step isolates the layer where the problem exists. The three most common server network problems in practice: (1) interface has no IP because DHCP failed, (2) wrong or missing default route, (3) firewall rule blocking traffic. Test each layer explicitly rather than assuming — a problem at layer 3 may look like a DNS problem from the application's perspective.