PCIe and High-Speed Interconnects

A Gen 4×4 SSD connected to a Gen 4 motherboard does not necessarily achieve a dedicated ~8 GB/s bandwidth: the slot may only be wired for two lanes, or it may share the uplink with other devices via a switch chip or chipset. To explain performance, first map the actual topology, then distinguish between link speed, encoded bandwidth, transaction overhead, and device processing speed.

Lanes are Physical Parallelism, Generations are Link Speed Levels

A single lane contains one pair of transmit differential lines and one pair of receive differential lines, allowing both directions to operate simultaneously. An x4 link uses four lanes for cooperative transmission, not four independently addressable devices; an x16 link involves 16×4=64 high-speed signal wires, not including power, ground, reference clocks, and sideband control.

The mechanical length, electrical routing, and currently negotiated width of a slot can differ. An x16-long slot might only be wired for x4; the motherboard may also reallocate port resources after inserting another device. Root ports supporting bifurcation can configure a group of lanes into multiple independent links, but passive breakout cables do not magically implement PCIe Switch functionality; both firmware and hardware must support the corresponding split.

Bandwidth is Stated as Unidirectional First, Then Generation Overhead is Deducted

The values for Gen 1–5 below have already deducted line coding overhead, but transaction packets, link control, and other overheads have not been deducted. GB uses decimal 10⁹ bytes, not GiB.

For example, Gen 4×4: 16×10⁹×128/130×4/8 ≈ 7.877×10⁹ byte/s. Full-duplex means the other direction has corresponding capability; you cannot treat bidirectional throughput and 15.754 GB/s as the limit for a single unidirectional SSD read.

Gen 6's 64 GT/s uses PAM4, where each symbol carries two bits, so the symbol rate no longer corresponds one-to-one with the nominal GT/s as in NRZ. It uses fixed 256-byte FLITs and adds mechanisms like error correction; you can no longer simply multiply by 128/130 to call it the final effective bandwidth. A TLP area within one FLIT can hold multiple transaction packets, with transaction headers and possible padding inside the area; application payload cannot directly equal the entire FLIT size. PCI-SIG: FLIT, PAM4, and FEC

The PCIe 7.0 128 GT/s specification was released in June 2025. Specification release, controller support, mass-produced motherboards, and the actual link capability of a specific machine are different states; do not directly infer current device available speeds based solely on specification generations. PCI-SIG: PCIe 7.0 Release

Topology: Each Segment is Point-to-Point, but the System May Still Share Bottlenecks

The Root Complex connects the processor/memory system to the PCIe hierarchy, with Root Ports being the respective ports; Endpoints provide device functionality. Switches expand the hierarchy and route transactions via uplink and downlink ports; they do not allow all downlink ports to obtain a complete, dedicated uplink independently.

Assume two SSDs each have Gen 4×4 downlinks, sharing a Gen 4×4 uplink, simultaneously transmitting 5 GB/s to host memory. The total demand is 10 GB/s, which already exceeds the uplink's encoded limit of ~7.877 GB/s. Even ignoring other overheads, it is impossible for each SSD to maintain 5 GB/s long-term; actual allocation depends on arbitration and traffic, and may not be strictly half-half.

Storage, networking, and USB devices hanging off the chipset may also share the host interconnect. Intel DMI, chipset connections on different AMD platforms, have their own widths and implementations; you cannot fixate on "DMI is essentially PCIe x8" or directly equate AMD chipset uplinks with Infinity Fabric. Checking specific processor and motherboard block diagrams is more reliable than memorizing a unified lane count for a brand.

P2P transactions may be forwarded at local hierarchy levels or restricted by platform, ACS, IOMMU, or driver support. Seeing the same Switch does not guarantee that any DMA between two devices can proceed directly. PCIe Driver Model in the Kernel

Three-Layer Protocol: Receiving Correctly at the Link Layer Does Not Mean the Device Has Completed the Business

In common non-FLIT modes, the link layer uses sequence numbers, LCRC, and ACK/NAK replay mechanisms; Gen 6 FLIT mode protection and replay units differ, so old packet formats cannot be generalized. Error correction/replay also consumes bandwidth; repeatedly correctable errors should not be simply dismissed as having "no performance impact."

Memory writes are typically posted requests, not requiring a corresponding transaction Completion to be returned; memory reads are non-posted requests, requiring a Completion to be returned, usually carrying data. The absence of a Completion for writes does not mean writes can arbitrarily bypass all operations, nor does it mean the SSD has committed data to disk. Register readbacks, DMA barriers, device command completions, and storage persistence belong to different guarantee levels.

The Requester uses Tags to manage incomplete read requests; one request may be split into multiple Completions. Read throughput is limited not only by link speed but also by the amount of data that can be in-flight and the round-trip latency. Assuming 32 requests allowed, each 256 B, with an average round-trip of 2 μs, the ideal window upper limit is approximately 32×256/2μs = 4.096 GB/s; even if the link has higher available bandwidth, it may not be fully utilized. Increasing concurrency can help, but you will also encounter Tag, buffer, credit, device, and memory limits. Intel: Factors Affecting PCIe Throughput

MPS limits the maximum data payload of a transaction packet, and MRRS limits the maximum size of an initiated read request; these are not the same parameter. Larger requests may also be split into multiple Completions; adjusting these values must conform to the capabilities of the entire path and device, and cannot guarantee that "setting everything to maximum is fastest."

Why wait when the link is not full?

The default 32 requests and 2 μs give a 4.096 GB/s window budget. More concurrency eventually reaches the link ceiling; more RTT moves that knee right. This is a read-Completion window, not the transaction direction of an NVMe application read.

$$R\le \min \left(C_{link},\frac{N{B}_{request}}{RTT}\right)$$

From Power-On to Driver Usage: Understand Training and Enumeration in Two Steps

The physical link goes through receive detection, training sequences, width/speed establishment, and corresponding equalization to enter an available state, commonly described by LTSSM state transitions. The highest capabilities of both ends, wiring, firmware policies, and signal quality determine the final result, which may drop to a narrower or lower speed; recovery states may also be used for re-training.

Subsequently, firmware/OS discovers devices and bridges via the configuration space, assigning bus numbers and resource windows. BARs describe the address space regions required by the device; after the host assigns addresses, the driver can map and access MMIO according to device rules. BARs are not ordinary RAM sticks, nor do they "copy" the physical device into RAM.

When devices perform DMA, they also need bus mastering capability, correct mapping, IOMMU permissions, and driver synchronization. If the link is in L0 and lspci can list the device, but no data transfer occurs, it may be a driver, resource, or DMA configuration issue, not necessarily requiring a slot change. IOMMU and DMA API

Physical Design Cannot Rely Solely on Memorizing a Set of Equal-Length Numbers

PCIe serial receivers recover timing from data via CDR, but many platforms still provide a reference clock; "embedded clock" does not mean the circuit board lacks a REFCLK. Common Clock, SRNS, and SRIS are different reference schemes and must match the capabilities of both ends and SSC configuration; you cannot assert that SRIS is universally adopted now. Intel: Applicable Background for SRIS

AC coupling is typically arranged according to the transmit path, with corresponding transmitters in each direction; do not mechanically add a set of capacitors at both ends of the link. Capacitance values, placement, differential impedance, loss, via stubs, and length skew requirements should be determined according to target specifications, packaging, and vendor routing guidelines. Intra-pair skew and inter-lane skew are different constraints; uniformly writing "all P/N must be less than 5 mil" lacks conditions and obscures more important loss budgets.

Redrivers primarily compensate for analog signal loss, while Retimers also perform clock/data recovery and re-transmission; their power consumption, latency, compatibility, and training support differ, so you cannot assume that adding one amplifier can infinitely extend the link. Continuous reference planes, return paths, and connector transitions remain fundamental.

What NVMe, ReBAR, and CXL Add on PCIe

NVMe is a storage command and queue model, while PCIe is one of the common local transport foundations; NVMe over Fabrics can also use other network transports. Multi-queues reduce contention among different submitters and maintain concurrency, but the number/depth of queues representable in the specification does not mean every controller implements the upper limit. Disk reads are "reads" in business semantics, but when the controller sends data to host RAM, it typically issues a PCIe Memory Write; do not confuse business direction with transaction type. Intel: I/O Data Flow Example

Resizable BAR allows selecting the BAR window size from sizes supported by the device; a larger window may reduce overhead such as VRAM window remapping. 256 MiB is a common window size for some older GPUs, not a unified upper limit for all PCIe BARs; ReBAR does not automatically increase lane bandwidth or guarantee reduced copying. Firmware address space, GPU, driver, and application jointly determine availability and benefits. PCI-SIG: Resizable BAR

CXL leverages the PCIe physical foundation, adding protocol semantics such as CXL.io, CXL.cache, and CXL.mem, serving consistency and memory access scenarios. Ordinary PCIe devices will not become CXL memory devices simply by being inserted into a similar slot; host, device, firmware, and software all require corresponding support. CXL Alliance: Protocol and Conformance Positioning

GPU workload sensitivity to PCIe depends on whether data resides in VRAM long-term, the frequency of exchanges, VRAM capacity, and synchronization methods. Measure the real working set to assess a change from x16 to x8; GPU series names do not imply a fixed performance penalty.

In Linux, Check Current Link First, Then Maximum Capability

lspci -t
lspci -nn
# Replace the address with the target device BDF seen in the previous step.
lspci -s 0000:01:00.0 -vv

Pay attention to LnkCap (Capabilities) and LnkSta (Current Speed/Width), and check uplinks for bridges and root ports. When reading configuration space is restricted by permissions, the output may be incomplete. Linux PCI sysfs also provides read-only attributes such as current_link_speed, current_link_width, max_link_speed, and max_link_width; not all fields may exist under certain devices or configurations. See the Linux PCI sysfs ABI for fields and units.

Do not assert load status based on a single idle sample. Devices may dynamically downclock for energy saving, and link anomalies may truly cause degradation; analysis should combine load, AER logs, and path topology. This only provides reading methods, without modifying link control, re-training, or resource registers.

Recalculate Encoding and In-Flight Window

from fractions import Fraction as F

per_lane = F(16_000_000_000)*F(128,130)/8
x4 = per_lane*4
assert x4 == F(102_400_000_000,13)
assert float(x4)/1e9 < 8
assert 2*5e9 > x4
window_bytes = 32*256
window_rate = F(window_bytes)/F(2,1_000_000)
assert window_rate == 4_096_000_000
assert window_rate < x4
needed = x4*F(2,1_000_000)
assert needed > window_bytes
print('Gen 4×4 Encoded Unidirectional Limit:', round(float(x4)/1e9, 6), 'GB/s')
print('Window Limit for 32 requests of 256 B, 2 μs RTT:', float(window_rate)/1e9, 'GB/s')

The above only verifies budgets under explicit assumptions, without testing actual GPU, SSD, or PCIe throughput.

Continue Reading

• USB and Type-C: Capability negotiation for external high-speed connections.
• Storage Technologies: Devices, interfaces, and access models.
• PCB Design Fundamentals: High-speed paths, references, and return paths.