ICMP and PMTUD

A webpage may complete the TCP handshake but stall when transmitting larger responses. This could indicate a path size feedback issue: small handshake packets pass through, but data packets exceed a certain egress MTU, and the returned ICMP messages fail to reach the sender. However, this phenomenon could also stem from filtering, congestion, or server-side pauses. The value of ICMP lies in providing feedback that can be correlated with the original packet, moving diagnosis from "guessing causes" to "verifying evidence."

Two Protocol Families: Do Not Mix Type Numbers

ICMPv4 uses Protocol number 1 in IPv4; ICMPv6 uses Next Header number 58. They serve for IP error reporting and information exchange, providing no general reliable byte stream. Echo can carry data, but this does not make ICMP an application transport service like TCP.

This is a common subset table; do not use it to categorize all unreachable errors as the same. ICMPv6 also carries messages for Neighbor Discovery and Router Discovery; bluntly blocking the entire protocol will affect basic network operation. RFC 792: ICMPv4, RFC 4443: ICMPv6

Why Error Messages Can Find the Original Connection

The generic header of ICMP consists of Type, Code, and Checksum, followed by structures that vary by type. Error messages reference the original IP packet that triggered the error. Endpoints correlate this feedback to the corresponding transport or probe by combining addresses, protocols, and available ports/sequence numbers.

Take IPv4 "Fragmentation Needed" as an example. The first 8 bytes of the ICMP header contain a 16-bit Next-Hop MTU, followed by the original IP header and the beginning of the original data. The classic minimum reference range includes the original IP header and the first 8 bytes of data, but receiving more referenced content is not abnormal. When parsing the referenced packet, you must still check its actual IHL, length, and available bytes; do not trust the claimed original Total Length to blindly read out of bounds. RFC 1191: MTU Field and Feedback

ICMPv4 checksums cover the ICMP message; ICMPv6 checksums also use the IPv6 pseudo-header. A successful checksum only indicates that the corresponding error detection passed; it does not authenticate the sender. Endpoints must verify that the referenced content matches their existing send state to prevent unrelated or forged feedback from arbitrarily changing connection states.

Error feedback is also not reliably delivered: typically, no further errors are generated for another ICMP error to avoid recursion. Multicast, broadcast, and non-first fragments have protocol-specific restrictions and exceptions; IPv6 "Packet Too Big" has significant exceptions. Add rate limiting, packet loss, and filtering, and no feedback does not mean no error. RFC 4443: Error Generation Rules

ping: A Single Matching Round-Trip Sample

Echo Identifier and Sequence help correlate requests and replies. The Identifier should not be fixedly understood as the process PID. RTT includes the outbound trip, peer processing, and return trip; it cannot be simply divided by two to assume one-way latency, nor can it represent application request latency under high load.

If the reply TTL is 57, it only supports the inference that the return trip passed through approximately seven decrements, provided the initial TTL was indeed 64. The initial value is configurable, and there may be tunnels or address translations in between, so you cannot determine the operating system based on this, nor can you directly infer the forward hop count.

In Linux iputils, ping -s refers to the number of Echo data bytes. For IPv4 without options, -s 1472 corresponds to a 1500 B IP packet (20+8+1472); for IPv6 without extension headers, the corresponding value is 1452. Ethernet headers and FCS are not included in this IP MTU. iputils ping manual

# Replace placeholders with the target to be diagnosed; this is for command illustration only and no probes were executed in this round.
ping -4 -c 3 -M do -s 1472 <TargetIPv4>
ping -4 -c 3 -M do -s 1464 <TargetIPv4>
ping -6 -c 3 -M do -s 1452 <TargetIPv6>

-M do is constrained by the kernel's PMTU check; sending too large a packet may result in a local error rather than reaching an intermediate router. It is necessary to distinguish between local send failures, errors returned by the network, and wait timeouts. Three probe samples are only for quick observation and are insufficient to estimate a stable packet loss rate.

traceroute: Gradually Extending Probe Lifespan

Starting with TTL=1, the first forwarding router exhausts it and returns Time Exceeded under certain conditions; increasing the TTL allows the probe to reach further nodes. How arrival is ultimately determined depends on the probe protocol: UDP often utilizes Destination Port Unreachable, ICMP Echo utilizes replies, and TCP can utilize SYN-ACK or RST results.

The time displayed for each hop is the round-trip time from the probe source to that feedback node and back. You cannot simply subtract adjacent RTTs to claim you obtained the link latency between two routers. The return path may differ, and device priorities for handling ICMP may also vary.

Traditional probes may change the ECMP hash when altering the UDP destination port, so consecutively printed addresses do not necessarily belong to a single true fixed path. To locate business issues, try to align the target, protocol, port, time, and direction, and treat the path map as an observation result.

PMTUD: Fully Tracking a 1500 → 1492 Adjustment

Assuming IPv4/TCP without options, the receiver previously advertised MSS=1460 in the SYN. The sender transmits a 1500 B IP packet containing 1460 B of TCP data with DF=1; the router's egress MTU is 1492, and the feedback carries 1492. After the sender verifies the feedback and updates the path estimate, the effective data upper limit for subsequent data packets can drop to 1492−20−20=1452 B.

What changes here is the sender's current segmentation size, not re-handshaking mid-connection to change the receiver-advertised MSS to 1452. If there are 12 B of TCP options, the data must be further reduced by these 12 B under the same PMTU; IPv6 or tunnels also require accounting for actual header budgets.

TCP will retransmit unacknowledged data intervals as needed; IP itself does not guarantee retransmission of the original packet. The path may change again, or there may be smaller subsequent bottlenecks, so receiving a 1492 once does not mean the unique correct value has been permanently found. RFC 1191

IPv6 corresponds to Packet Too Big; the basic header has no DF bit, and routers do not fragment; extension headers also take up space, so they cannot be ignored just because the basic header is fixed. Classic PMTUD requires receiving trusted size feedback; the specific cached objects, lifetime, and re-probing behavior are determined by implementation and configuration. RFC 8201

Does Too Big change the advertised MSS?

Retain the handshake advertisement to identify the variable actually adjusted. Drop feedback and resend, then reset, deliver feedback and resend. Receiving feedback never retroactively makes the original packet successful.

Black Holes: Collecting Evidence via Causal Chains

A credible diagnosis can sequentially confirm: the sender indeed transmitted a packet of a certain size; the corresponding egress could not accommodate it; where the error feedback was generated, arrived, or was lost; and whether subsequent sends still used packets that were too large. If you can only look at endpoints, combine controllable packet size changes, retransmissions, and application progress to narrow the scope, while keeping other causes in mind.

Packet capture also needs to pay attention to TSO/GSO/GRO: "jumbo frames" on the host may be views before segmentation or after aggregation, and may not have been transmitted online at that size. MSS clamping only applies to TCP and cannot fix all UDP/QUIC path issues.

PLPMTUD and Linux Implementation Switches

The probe layer can use probes and explicit acknowledgments to verify whether a certain packet size is available, reducing reliance on ICMP as the sole feedback channel. DPLPMTUD requires states for base size, upward search, probe failure retry, and black hole recovery; a single large packet loss may just be congestion and cannot simply be dichotomized to announce the maximum value. If a UDP application has no acknowledgment mechanism, you cannot assert that the remote end received it based on local send success alone. RFC 8899: DPLPMTUD

Linux net.ipv4.tcp_mtu_probing documentation distinguishes 0 (disabled), 1 (enabled when corresponding black hole is detected), and 2 (always enabled). It is a TCP implementation behavior, not a global switch to fix all transport protocols; applicability must be combined with kernel, connection, and network policies. This article retains mechanism descriptions and does not use an unconditional sysctl write to replace diagnosis. Linux IP sysctl documentation

Local Validation and Length Budgets

import struct

def checksum(data):
    if len(data)%2:
        data += bytes(1)
    total = sum(struct.unpack('!%dH' % (len(data)//2), data))
    while total>>16:
        total = (total&65535)+(total>>16)
    return (~total)&65535

message = bytearray(struct.pack('!BBHHH',8,0,0,0x1234,7)+b'hello')
message[2:4] = struct.pack('!H',checksum(message))
assert checksum(message) == 0
message[-1] ^= 1
assert checksum(message) != 0
assert 1500-20-8 == 1472
assert 1500-40-8 == 1452
assert 1492-20-20 == 1452
assert 1492-20-32 == 1440
assert 1464+8+20 == 1492
print('ICMP ones’ complement checksum, odd-byte padding, and MTU data budget passed')

Here, only a partial message of ICMPv4 Echo is constructed and length calculated; no network probes were sent, nor were PMTU, MSS, or sysctl changed.

Continue Reading

• IPv4 Protocol: DF, header checksum, and fragment reassembly.
• IPv6 Protocol: Extension headers, source-side fragmentation, and autoconfiguration.
• Routing and BGP: Forwarding table selection and control plane convergence.