--- title: PCIe and High-Speed Interconnects url: https://doc.liz6.com/en/hardware/06-interfaces-and-communication/04-pcie-and-high-speed-interconnects locale: en area: hardware tags: - hardware - interfaces-and-communication date: 2026-06-30 modified: 2026-07-16 description: PCIe and High-Speed Interconnects What is PCIe PCIe (PCI Express) — The high-speed serial bus in modern computers, connecting the CPU to peripherals such as GPU… --- # PCIe and High-Speed Interconnects ## What is PCIe **PCIe (PCI Express)** — The high-speed serial bus in modern computers, connecting the CPU to peripherals such as GPUs, SSDs, and network cards. PCIe Tree Topology: Device Hierarchy under a Single Root Complex CPU PCIe RC Memory Controller Switch Endpoint (GPU) EP EP EP (NVMe SSD, Network Card...) RC = Root Complex (typically within the CPU or PCH); Switch = Switching chip, expands more PCIe ports downwards EP = Endpoint (end device), i.e., the terminal devices such as NVMe SSDs and network cards connected under the Switch --- ## Lanes and Generations ### Lane ``` The minimum unit of PCIe is 1 Lane x1: 1 lane x4: 4 lanes (Common: NVMe SSD) x8: 8 lanes x16: 16 lanes (Common: GPU) Each Lane is an independent full-duplex serial link TX+/TX- (Transmit) + RX+/RX- (Receive) = 4 signal lines per Lane Physically, x16 = 64 signal lines (16×4) ``` ### Speeds by Generation | Gen | Rate/Lane (Unidirectional) | Bandwidth/Lane (Bidirectional) | x16 Bandwidth | Year | |-----|-------------------|-------------------|----------|------| | 1.0 | 2.5 GT/s | 250 MB/s | 4 GB/s | 2003 | | 2.0 | 5.0 GT/s | 500 MB/s | 8 GB/s | 2007 | | 3.0 | 8.0 GT/s | ~1 GB/s | ~16 GB/s | 2010 | | 4.0 | 16.0 GT/s | ~2 GB/s | ~32 GB/s | 2017 | | 5.0 | 32.0 GT/s | ~4 GB/s | ~64 GB/s | 2019 | | 6.0 | 64.0 GT/s | ~8 GB/s | ~128 GB/s | 2022 (Just Commercialized) | ``` 8b/10b (Gen1/2): 2.5 GT/s × 8/10 = 2 Gbps = 250 MB/s 128b/130b (Gen3+): 8 GT/s × 128/130 ≈ 7.88 Gbps ≈ 985 MB/s → Encoding overhead is significantly reduced for Gen3+ "GT/s" = Gigatransfers per second (includes encoding overhead) "GB/s" = Gigabytes per second of effective data ``` ### Common Configurations ``` Typical PCIe Lanes for Consumer CPUs: AMD Ryzen: 24~28 lanes (4 connect to PCH, 16+4 for GPU+NVMe) Intel Core: 16~20 lanes (4 connect to PCH) Graphics Cards: x16 Gen4/5 (but x8 is usually sufficient, with only 1~3% performance loss) NVMe SSDs: x4 Gen4/5 (Gen4 x4 = ~8 GB/s) 10GbE Network Cards: x4 Gen3 ``` ### Backward Compatibility ``` Gen4 devices can be inserted into Gen3 slots → Speed drops to Gen3 Gen3 devices can be inserted into Gen4 slots → Operate at Gen3 Link Training: Upon power-on, both ends automatically negotiate → Determine the highest speed and maximum number of lanes supported ``` --- ## Physical Layer ### Signals ``` Per Lane: TX+ TX- : Differential Transmit (AC coupled, capacitor blocks DC) RX+ RX- : Differential Receive (AC coupled) Reference Clock: 100MHz (HCSL or LVDS differential) Common Clock (Source Synchronous) or Separate Clock (Independent) Currently, SRIS (Separate Refclk Independent SSC) is mostly used Gen3+ uses 128b/130b encoding + scrambling No separate clock line; the receiver recovers the clock from the data (CDR) ``` ### PCB Design Key Points ``` PCIe Traces: - Differential pairs must be length-matched (P/N skew <5mil) - Characteristic Impedance: 85Ω differential (Gen3), sometimes 100Ω - Avoid reference plane discontinuities (crossing splits) - AC coupling capacitors (100nF~220nF) should be placed close to the transmitter - Minimize vias (each via disrupts impedance) ``` --- ## PCIe Protocol Layers ``` Three-Layer Model: Transaction Layer: Generates/Consumes TLPs (Transaction Layer Packets) Address routing / ID routing / Implicit routing Data Link Layer: Ensures reliable transmission ACK/NAK, Retransmission, CRC Checksum Generates/Consumes DLLPs (Data Link Layer Packets) Physical Layer: 8b/10b or 128b/130b encoding Serial-to-Parallel conversion, Link Training ``` --- ## Common Applications ### NVMe over PCIe ``` The NVMe protocol runs over PCIe AHCI (the traditional protocol for SATA) runs over PCIe → Replaced by NVMe NVMe Advantages: - 64K Queue Depth (AHCI is only 32) - 64K Queues (AHCI is only 1) - Lower Latency (no need to access the SATA controller) ``` ### GPU Connection ``` Consumer GPUs: x16 PCIe Direct Connect to CPU Gen3 x16 = 16 GB/s (RTX 20/30 Series) Gen4 x16 = 32 GB/s (RTX 40 Series) Gen5 x16 = 64 GB/s (RTX 50 Series) Reality: In gaming scenarios, Gen3 x8 has almost no performance loss. Gen4 for consumer GPUs is mainly for future headroom. High bandwidth is more critical for compute/ML scenarios (Multi-GPU interconnects). ``` ### Resizable BAR (Smart Access Memory) ``` Traditional: CPU can only access GPU VRAM through a 256MB BAR window ReBAR: CPU can map all GPU VRAM at once → Reduces copying, improves performance Requirements: BIOS Support + GPU Driver Support + PCIe Support ``` --- ## Connection to CPU PCIe Connection to CPU: Direct Connect vs. via PCH (Intel / AMD Comparison) Intel CPU PCH (Chipset) DMI (Essentially PCIe x8) PCIe x16 (Direct Connect) PCIe x4 (NVMe Direct Connect) SATA/USB/Ethernet/WiFi More PCIe Lanes AMD CPU PCH PCIe x4 (IF Bus) PCIe x16/x8 PCIe x4 (NVMe) Similar to Intel Topology (SATA/USB/More Lanes) PCIe Direct Connect to CPU (Independent x16 GPU, CPU Direct NVMe) = Lower Latency, Best for GPU/NVMe; PCIe via PCH (SATA/USB/Network Cards, etc.) = Shares DMI/IF Bus Bandwidth, then uplinks to CPU. --- *Keywords: PCIe, Lane, GT/s, NVMe, Root Complex, Endpoint, Differential Signal, ReBAR, Gen*