Network Troubleshooting Methodology

Layered troubleshooting—eliminate one layer at a time from L1 to L7. MTU black holes (ICMP filtered, causing TCP handshake to complete but data to stall), DNS resolution anomalies, TCP handshake timeouts—each classic failure has a specific diagnostic path. mtr is better suited than ping/traceroute for locating intermittent packet loss.

Overview

The core methodology for network troubleshooting is layered diagnosis: from L1 (Physical) to L7 (Application), eliminating one layer at a time. The most common failure modes are: MTU black holes (ICMP filtered, causing TCP to stall), DNS resolution anomalies, and data transmission stalling after TCP connection establishment (buffer/window issues). mtr is better suited for long-term packet loss monitoring than ping/traceroute—it displays packet loss rates and latency jitter for every hop end-to-end.

Layered Diagnosis

L1 (Physical): ip link, ethtool eth0 | grep Link
L2 (Data Link): ip neigh, tcpdump arp
L3 (Network): ping -M do, traceroute, ip route
L4 (Transport): ss -tlnp, tcpdump 'tcp[tcpflags] & tcp-syn != 0'
L5-6 (TLS): openssl s_client -connect host:443, tshark 'tls.alert_message.level==2'
L7 (Application): curl -v, dig, ss -tianp (Recv-Q / Send-Q)

Classic Failure Modes

"Network is up but inaccessible"

ping succeeds → L3 OK, L4 may have issues

Check:
  ss -tlnp | grep <port>            Is the service listening?
  iptables -L -n -v                  Is the firewall blocking?
  tcpdump 'tcp[tcpflags] & tcp-rst != 0' Who is sending RST?
  cat /proc/sys/net/netfilter/nf_conntrack_count Is the conntrack table full?

MTU Black Hole

Symptoms: ping succeeds, TCP SYN+ACK completes, data transmission stalls.

Root Cause: MTU on the path is smaller than the packet size + DF=1 + ICMP Frag Needed is filtered
  → Packets are dropped along the path, sender is unaware

Detection: ping -M do -s 1472 <target> → fails (but small pings succeed) → MTU black hole
Mitigation: iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

DNS Issues

dig <domain> +short                   What does it return?
dig +trace <domain>                   Where does the recursive path break?
tcpdump -nn 'port 53'                 Was the request sent? Is there a response?

Intermittent Timeouts

mtr -r -c 100 <target>               Which hop starts dropping packets + jitter?
ethtool -S eth0 | grep -E 'err|drop'  NIC packet drops?
cat /proc/net/dev                      Soft drops?
tc -s qdisc show dev eth0              Qdisc drops (bufferbloat?)

References

mtr: bitwizard.nl/mtr
iptables: netfilter.org/documentation

Keywords: network troubleshooting, L1-L7, MTU black hole, DNS, TCP RST, conntrack, bufferbloat