Network Fault Diagnosis Methodology

The goal of diagnosis is to narrow the fault boundary using evidence and to verify whether the fix altered the original symptoms. Protocol layering provides a map for inspection, but it is not necessary to check every layer from the physical cable to the application, nor should you permanently exclude the entire network layer just because a single ping succeeded.

"Cannot access" must be expanded into: which client, which network namespace, which time window, which domain and port, which protocol, and at which step the error occurred. You should also distinguish between conditions such as all requests failing, only some backends failing, only large responses failing, slow first access, and failures after long connections have been idle.

Selecting the Most Discriminative Controls from Application Symptoms

Successfully changing DNS, proxy, HTTP version, and target IP simultaneously makes it difficult to know which change was effective. Each control should record what was changed and which conditions remain uncontrollable.

DNS: Finding a record does not mean the application used that answer

# Specify resolver, retain status code, TTL, and answer content.
dig @192.0.2.53 app.example A
dig @192.0.2.53 app.example AAAA

# Let dig perform iterative queries to observe public delegation behaviors.
dig +trace app.example

+short is suitable for quickly extracting results but hides much failure context. +trace does not show every step inside enterprise recursive resolvers and may bypass split DNS; it requires direct access to the corresponding authoritative servers from the current environment. Failure does not necessarily mean the recursive resolver used by the application also failed. BIND dig

The application might use system resolution, hosts, browser DoH, its own cache, or proxy-side resolution. First record the remote IP the application actually connects to, then compare it with the expected answer. NXDOMAIN, SERVFAIL, timeouts, and returning incorrect addresses have different diagnostic paths; do not uniformly attribute them to "DNS instability."

To test a service for an HTTPS domain using a specific IP, use --resolve to preserve the SNI, certificate verification, and HTTP host identity corresponding to the URL hostname; directly changing the URL to an IP and using -k changes multiple conditions.

# Explicitly test direct connection; replace target address, name, and path with actual environment values.
curl --noproxy '*' --resolve app.example:443:192.0.2.10 \
  --connect-timeout 5 --max-time 15 -sS -o /dev/null \
  -w 'ip=%{remote_ip} code=%{http_code} total=%{time_total}\n' \
  https://app.example/health

Here, --noproxy '*' explicitly excludes the influence of proxy variables; if the original problem lies in the proxy path, you should separately retain proxy settings for comparison. A fixed IP might also select a different CDN node, so "fixed IP works" only supports continuing to check resolution and node differences; it does not immediately prove a DNS protocol fault. curl --resolve

Connections: Separate the outbound SYN and the return response

# Route query is performed within the current namespace, without initiating remote probes.
ip route get 192.0.2.10
ip -6 route get 2001:db8::10

# View listening and connection states, filtering by actual ports.
ss -ltnp 'sport = :443'
ss -tinp 'dport = :443'

Route results are affected by source address, mark, policy rules, VRF, and namespace; queries in the current shell may not reproduce container or business flows with special marks. Services listening on 127.0.0.1, a specific address, IPv6 address, or another namespace will also change external reachability. ip route

Ping uses ICMP, while business traffic may use TCP, UDP, different address families, or even different paths. A successful ping only proves that this specific ICMP probe received a response; ping being disabled does not mean the business is unavailable. See tcpdump and Wireshark for packet capture details.

TLS: Correctly Distinguish Transport Handshake from Identity Verification

openssl s_client -connect 192.0.2.10:443 -servername app.example \
  -verify_hostname app.example -verify_return_error -brief </dev/null

-servername provides SNI, and -verify_hostname checks the expected hostname; the two cannot replace each other. The default diagnostic behavior of s_client may continue after certificate verification errors; using -verify_return_error explicitly treats verification failure as a failure condition; private CAs require configuring the actual trusted CA file. OpenSSL s_client

Successful TLS establishment does not verify HTTP routing, user authentication, or backend dependencies. If only the application fails, compare its SNI, ALPN, client certificates, CA bundle, and actual connection target; do not exclude all TLS configuration differences just because "the command line can handshake."

Breaking Down a Slow Request into Explainable Time Segments

curl --noproxy '*' --connect-timeout 5 --max-time 15 -sS -o /dev/null \
  -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} first=%{time_starttransfer} total=%{time_total}\n' \
  https://app.example/health

These times are typically cumulative from the start of the operation to the respective stages, not five distinct durations that can be directly added. The following assumes a new direct HTTPS connection, no redirects, no connection reuse, and that request body sending impact is included in the wait before the first byte.

from decimal import Decimal as D
cumulative = list(map(D, ['0', '0.020', '0.060', '0.140', '0.340', '0.400']))
stages = [b - a for a, b in zip(cumulative, cumulative[1:])]
assert stages == list(map(D, ['0.020', '0.040', '0.080', '0.200', '0.060']))
assert sum(stages) == D('0.400')
print('DNS 20, TCP 40, TLS 80, Pre-first-byte 200, Receive 60 ms')

time_starttransfer - time_appconnect includes request sending, network round-trips, service queuing, and processing; it cannot all be recorded as server-side computation time. Proxies, redirects, connection reuse, multiplexing, and failed requests change the interpretation of fields; do not forcibly calculate missing or zero stages as normal measurements. curl write-out

If TLS is fast but the first byte is slow, continue correlating request IDs, proxy upstream times, service queues, and dependency calls. If the first byte is fast but downloading is slow, compare response size, application output rhythm, bandwidth, retransmissions, and receive backpressure. An HTTP 503 indicates an HTTP response has already been generated; prioritize checking the responder and upstream status, rather than just repeating pings.

Why can five timing values not be added?

Increasing TLS alone raises appconnect, starttransfer and total; these are not three independent costs. Only adjacent differences form additive durations.

MTU Blackhole: Confirm with a Set of Mutually Supporting Evidence

The typical hypothesis is: somewhere in the path, larger packets cannot fit, packets are dropped, and the sender does not correctly learn or adapt to a smaller PMTU. Successful handshakes and small requests, but stalled large data, are clues rather than exclusive symptoms; application buffering, policy filtering by content, and receive window pauses can also look similar.

# IPv4, without IP options: 1472 B payload + 8 B ICMP + 20 B IP = 1500 B.
ping -4 -c 3 -M do -s 1472 192.0.2.10
ping -4 -c 3 -M do -s 1200 192.0.2.10

Large ping failures may come from local PMTU checks, peer ICMP policies, the path, or the return path; a single test cannot confirm a blackhole. First distinguish between local message too long, receiving ICMP errors, and complete timeouts, then observe whether actual business large packets, retransmissions, and ICMP Packet Too Big/Fragmentation Needed are related. ping manual

ipv4_probe = 1472 + 8 + 20
ipv6_probe_without_extensions = 1452 + 8 + 40
assert ipv4_probe == ipv6_probe_without_extensions == 1500
path_mtu = 1400
assert path_mtu - 20 - 20 == 1360  # Upper bound of data under IPv4 TCP base header.
assert path_mtu - 40 - 20 == 1340  # IPv6 without extension headers.
print('Calculate probe payload, IP packet length, and TCP data budget separately')

These budgets do not yet account for TCP options or additional encapsulation. Fixes should target the real path MTU, necessary ICMP feedback, or applicable PLPMTUD mechanisms; MSS clamping only affects the relevant direction of TCP negotiation and cannot fix arbitrary UDP/QUIC, nor does it automatically apply to already established connections. Therefore, a global firewall modification should not be treated as a diagnostic step. DPLPMTUD

Intermittent Issues: Look at Time Correlation, Not Just Cumulative Values

mtr summarizes responses through repeated probes, helping to identify continuous or intermittent patterns, but rate-limiting probe responses by intermediate routers does not mean business packets are also dropped. If hop 4 shows 60% no response, but hops 5 and the target are normal, it usually does not support the conclusion that "hop 4 dropped 60% of business traffic."

# Use TCP target port probes closer to business when applicable; still not real business requests.
mtr --report --report-cycles 100 --tcp --port 443 app.example

Each hop's response has its own return path; ECMP, probe fields, and protocol differences also affect observations. Only when subsequent hops and the target show similar degradation consistently is it worth checking this segment together with end-to-end business evidence, and the return path still needs attention. mtr manual

Linux TCP's ss queue fields also need to distinguish states: For LISTEN, Recv-Q/Send-Q represent the completed queue waiting for accept and its limit; for established connections, the receive queue represents data not yet read by the application, and the send queue involves the sequence space already written but not yet acknowledged, including data not yet sent; they cannot all be called "unacknowledged data on the line." These definitions can be cross-referenced with Linux tcp_diag_get_info(). ss manual

Writing Conclusions as Recheckable Causal Chains

A useful record could be: "10:02–10:08, new IPv6 connections on Node A timed out before TLS, IPv4 control was normal; server side did not see corresponding IPv6 SYN, client interface showed retransmissions; current evidence narrows the boundary to between the two collection points, but has not yet excluded missing captures by intermediate mirrors." This is more conducive to next-step collaboration than "network jitter, restarted."

After fixing, repeat the original failure conditions and compare error rates, time distributions, and resource side effects. One successful request only proves success at that time; intermittent faults require observation windows sufficient to cover the original fault conditions. If temporary measures restore service but the root cause is undetermined, record the restoration facts and root cause hypotheses separately.