---
title: RDMA (Remote Direct Memory Access)
url: https://doc.liz6.com/en/linux-kernel/06-network-subsystem/05-RDMA
locale: en
area: linux-kernel
tags:
- linux-kernel
- network-subsystem
date: 2026-06-30
modified: 2026-07-16
description: 'Coverage: RDMA Architecture → InfiniBand/RoCE/iWARP → verbs API → Queue Pair → Memory Region → Kernel Bypass → Integration with Linux Network Stack Kernel Version: 3.x ~ 6.x'
---

# RDMA (Remote Direct Memory Access)

> Coverage: RDMA Architecture → InfiniBand/RoCE/iWARP → verbs API → Queue Pair → Memory Region → Kernel Bypass → Integration with Linux Network Stack
> Kernel Version: 3.x ~ 6.x

## Overview

RDMA allows one machine to directly read and write the memory of another machine, **bypassing the remote CPU and kernel**. Latency drops from ~50μs in traditional TCP/IP to ~1μs, with zero CPU involvement in data transfer. Linux provides a unified RDMA interface via `ib_core` plus vendor drivers (mlx5, qedr, hns).

## Three Transport Types

| | InfiniBand | RoCEv2 | iWARP |
|---|---|---|---|
| Network Layer | IB L2/L3 | UDP/IP (dst port 4791) | TCP/IP |
| Hardware | IB HCA | Converged Ethernet NIC | Converged Ethernet NIC |
| Dependencies | IB Switches | DCB/PFC (Lossless Ethernet) | Standard TCP Offload |
| Kernel Support | `ib_core` | `ib_core` + `rdma_rxe`(SW) | `iw_cm` |

## Core Abstraction: Queue Pair (QP)

```
Each QP = Send Queue (SQ) + Receive Queue (RQ) + Completion Queue (CQ)

SQ: RDMA operations initiated locally (SEND, RDMA_WRITE, RDMA_READ)
RQ: Buffers waiting for remote SENDs (post buffers first, filled after receiving SEND)
CQ: Completion notifications (hardware writes CQE upon operation completion)

QP State Machine:
  RESET → INIT → RTR (Ready To Receive) → RTS (Ready To Send)
```

## Memory Region (MR)

```c
// RDMA hardware writes directly to remote memory → requires memory "registration" (pinning)
// Registration: ibv_reg_mr() → hardware locks physical pages + generates access key (rkey/lkey)
// 
// Why is registration needed?
//   RDMA HCA bypasses the kernel → cannot rely on CPU page tables for address translation
//   → requires direct mapping of physical addresses + locked pages (no swap/page fault)

// On-Demand Paging (ODP) / Implicit MR (IMR):
//   Reduces registration overhead → pin/unpin on demand
```

## Verbs API

```c
// User space: libibverbs

// Find devices
ibv_get_device_list()

// Create QP
ibv_create_qp(pd, &qp_init_attr)

// Post work request (send)
ibv_post_send(qp, &wr, &bad_wr)
// RDMA_WRITE: Direct write to remote memory (most common)
// RDMA_READ:  Direct read from remote memory
// SEND:       Send message (remote side needs to pre-post RECV)

// Completion
ibv_poll_cq(cq, num_entries, &wc)  // Read CQE
```

## Kernel Bypass and Linux Integration

<svg viewBox="0 0 720 380" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,'Source Han Sans CN','Microsoft YaHei',sans-serif" role="img" aria-label="Kernel Bypass Comparison: Traditional TCP vs RDMA Path, and RDMA Connection Establishment Still Requires Kernel Involvement">
  <defs>
    <marker id="rdmaArrow" markerWidth="10" markerHeight="8" refX="8" refY="3" orient="auto"><path d="M0,0 L8,3 L0,6 Z" fill="#475569"/></marker>
  </defs>
  <rect width="720" height="380" fill="#ffffff"/>
  <text x="360" y="28" text-anchor="middle" font-size="17" font-weight="700" fill="#1f2933">Kernel Bypass Comparison: Data Plane Bypasses Kernel, Connection Establishment Still Relies on Kernel</text>

  <text x="48" y="84" font-size="13" font-weight="600" fill="#64748b">Traditional TCP</text>
  <text x="48" y="100" font-size="11" fill="#94a3b8">Via Kernel</text>
  <rect x="130" y="66" width="70" height="40" rx="5" fill="#e2e8f0"/><text x="165" y="90" text-anchor="middle" font-size="11" fill="#334155">app</text>
  <line x1="200" y1="86" x2="222" y2="86" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="222" y="66" width="80" height="40" rx="5" fill="#e2e8f0"/><text x="262" y="90" text-anchor="middle" font-size="11" fill="#334155">socket</text>
  <line x1="302" y1="86" x2="324" y2="86" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="324" y="66" width="80" height="40" rx="5" fill="#e2e8f0"/><text x="364" y="90" text-anchor="middle" font-size="11" fill="#334155">TCP/IP</text>
  <line x1="404" y1="86" x2="426" y2="86" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="426" y="66" width="100" height="40" rx="5" fill="#e2e8f0"/><text x="476" y="90" text-anchor="middle" font-size="11" fill="#334155">netdevice</text>
  <line x1="526" y1="86" x2="548" y2="86" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="548" y="66" width="60" height="40" rx="5" fill="#e2e8f0"/><text x="578" y="90" text-anchor="middle" font-size="11" fill="#334155">NIC</text>

  <text x="48" y="178" font-size="13" font-weight="700" fill="#0f766e">RDMA</text>
  <text x="48" y="194" font-size="11" fill="#14b8a6">Bypasses Kernel</text>
  <rect x="130" y="160" width="70" height="40" rx="5" fill="#f0fdfa" stroke="#99f6e4"/><text x="165" y="184" text-anchor="middle" font-size="11" fill="#115e59">app</text>
  <line x1="200" y1="180" x2="222" y2="180" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="222" y="160" width="70" height="40" rx="5" fill="#f0fdfa" stroke="#99f6e4"/><text x="257" y="184" text-anchor="middle" font-size="11" fill="#115e59">verbs</text>
  <line x1="292" y1="180" x2="314" y2="180" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="314" y="160" width="110" height="40" rx="5" fill="#f0fdfa" stroke="#99f6e4"/><text x="369" y="184" text-anchor="middle" font-size="11" fill="#115e59">User-space Driver</text>
  <line x1="424" y1="180" x2="446" y2="180" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="446" y="160" width="60" height="40" rx="5" fill="#f0fdfa" stroke="#99f6e4"/><text x="476" y="184" text-anchor="middle" font-size="11" fill="#115e59">NIC</text>
  <text x="516" y="184" font-size="11" fill="#0f766e">(Fully Bypasses Kernel!)</text>

  <line x1="60" y1="218" x2="660" y2="218" stroke="#e2e8f0" stroke-width="1"/>

  <text x="60" y="246" font-size="13" font-weight="700" fill="#9a3412">But Connection Establishment Still Requires Kernel: RDMA CM Follows Full Process</text>
  <rect x="90" y="256" width="170" height="36" rx="5" fill="#ffedd5"/><text x="175" y="278" text-anchor="middle" font-size="10.5" fill="#9a3412">RDMA CM (Connection Management)</text>
  <line x1="260" y1="274" x2="280" y2="274" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="280" y="256" width="200" height="36" rx="5" fill="#ffedd5"/><text x="380" y="278" text-anchor="middle" font-size="10.5" fill="#9a3412">IP Address Resolution + Route Lookup</text>
  <line x1="480" y1="274" x2="500" y2="274" stroke="#475569" stroke-width="1.6" marker-end="url(#rdmaArrow)"/>
  <rect x="500" y="256" width="140" height="36" rx="5" fill="#ffedd5"/><text x="570" y="278" text-anchor="middle" font-size="10.5" fill="#9a3412">Establish RC QP</text>

  <rect x="60" y="304" width="600" height="54" rx="8" fill="#eef2ff" stroke="#c7d2fe"/>
  <text x="76" y="326" font-size="12.5" fill="#3730a3">RDMA connects directly to the NIC via verbs, completely bypassing the kernel protocol stack in the data plane, reducing latency from ~50μs to ~1μs;</text>
  <text x="76" y="346" font-size="12.5" fill="#3730a3">However, during the connection establishment phase (RDMA CM), the kernel is still required for address resolution and route lookup; it is not "zero kernel involvement".</text>
</svg>

## References

- **Source Code**: `drivers/infiniband/core/`, `drivers/infiniband/hw/mlx5/`, `include/rdma/`
- **Documentation**: `Documentation/infiniband/`
- **LWN**: "RDMA and the Linux kernel"

*Keywords: RDMA, InfiniBand, RoCE, QP, Memory Region, verbs, kernel bypass, ib_core*