PCIe and High-Speed Interconnects

What is PCIe

PCIe (PCI Express) — The high-speed serial bus in modern computers, connecting the CPU to peripherals such as GPUs, SSDs, and network cards.

Lanes and Generations

Lane

The minimum unit of PCIe is 1 Lane

x1:  1 lane
x4:  4 lanes (Common: NVMe SSD)
x8:  8 lanes
x16: 16 lanes (Common: GPU)

Each Lane is an independent full-duplex serial link
  TX+/TX- (Transmit) + RX+/RX- (Receive) = 4 signal lines per Lane

Physically, x16 = 64 signal lines (16×4)

Speeds by Generation

8b/10b (Gen1/2): 2.5 GT/s × 8/10 = 2 Gbps = 250 MB/s
128b/130b (Gen3+): 8 GT/s × 128/130 ≈ 7.88 Gbps ≈ 985 MB/s
→ Encoding overhead is significantly reduced for Gen3+

"GT/s" = Gigatransfers per second (includes encoding overhead)
"GB/s" = Gigabytes per second of effective data

Common Configurations

Typical PCIe Lanes for Consumer CPUs:
  AMD Ryzen: 24~28 lanes (4 connect to PCH, 16+4 for GPU+NVMe)
  Intel Core: 16~20 lanes (4 connect to PCH)

Graphics Cards: x16 Gen4/5 (but x8 is usually sufficient, with only 1~3% performance loss)
NVMe SSDs: x4 Gen4/5 (Gen4 x4 = ~8 GB/s)
10GbE Network Cards: x4 Gen3

Backward Compatibility

Gen4 devices can be inserted into Gen3 slots → Speed drops to Gen3
Gen3 devices can be inserted into Gen4 slots → Operate at Gen3

Link Training:
  Upon power-on, both ends automatically negotiate → Determine the highest speed and maximum number of lanes supported

Physical Layer

Signals

Per Lane:
  TX+ TX- : Differential Transmit (AC coupled, capacitor blocks DC)
  RX+ RX- : Differential Receive (AC coupled)

Reference Clock: 100MHz (HCSL or LVDS differential)
  Common Clock (Source Synchronous) or Separate Clock (Independent)
  Currently, SRIS (Separate Refclk Independent SSC) is mostly used

Gen3+ uses 128b/130b encoding + scrambling
No separate clock line; the receiver recovers the clock from the data (CDR)

PCB Design Key Points

PCIe Traces:
  - Differential pairs must be length-matched (P/N skew <5mil)
  - Characteristic Impedance: 85Ω differential (Gen3), sometimes 100Ω
  - Avoid reference plane discontinuities (crossing splits)
  - AC coupling capacitors (100nF~220nF) should be placed close to the transmitter
  - Minimize vias (each via disrupts impedance)

PCIe Protocol Layers

Three-Layer Model:

Transaction Layer:
  Generates/Consumes TLPs (Transaction Layer Packets)
  Address routing / ID routing / Implicit routing

Data Link Layer:
  Ensures reliable transmission
  ACK/NAK, Retransmission, CRC Checksum
  Generates/Consumes DLLPs (Data Link Layer Packets)

Physical Layer:
  8b/10b or 128b/130b encoding
  Serial-to-Parallel conversion, Link Training

Common Applications

NVMe over PCIe

The NVMe protocol runs over PCIe
AHCI (the traditional protocol for SATA) runs over PCIe → Replaced by NVMe

NVMe Advantages:
  - 64K Queue Depth (AHCI is only 32)
  - 64K Queues (AHCI is only 1)
  - Lower Latency (no need to access the SATA controller)

GPU Connection

Consumer GPUs: x16 PCIe Direct Connect to CPU
  Gen3 x16 = 16 GB/s (RTX 20/30 Series)
  Gen4 x16 = 32 GB/s (RTX 40 Series)
  Gen5 x16 = 64 GB/s (RTX 50 Series)

Reality: In gaming scenarios, Gen3 x8 has almost no performance loss.
         Gen4 for consumer GPUs is mainly for future headroom.
         High bandwidth is more critical for compute/ML scenarios (Multi-GPU interconnects).

Resizable BAR (Smart Access Memory)

Traditional: CPU can only access GPU VRAM through a 256MB BAR window
ReBAR: CPU can map all GPU VRAM at once → Reduces copying, improves performance

Requirements: BIOS Support + GPU Driver Support + PCIe Support

Connection to CPU

Keywords: PCIe, Lane, GT/s, NVMe, Root Complex, Endpoint, Differential Signal, ReBAR, Gen