RDMA (Remote Direct Memory Access)

Coverage: RDMA Architecture → InfiniBand/RoCE/iWARP → verbs API → Queue Pair → Memory Region → Kernel Bypass → Integration with Linux Network Stack Kernel Version: 3.x ~ 6.x

Overview

RDMA allows one machine to directly read and write the memory of another machine, bypassing the remote CPU and kernel. Latency drops from ~50μs in traditional TCP/IP to ~1μs, with zero CPU involvement in data transfer. Linux provides a unified RDMA interface via ib_core plus vendor drivers (mlx5, qedr, hns).

Three Transport Types

Core Abstraction: Queue Pair (QP)

Each QP = Send Queue (SQ) + Receive Queue (RQ) + Completion Queue (CQ)

SQ: RDMA operations initiated locally (SEND, RDMA_WRITE, RDMA_READ)
RQ: Buffers waiting for remote SENDs (post buffers first, filled after receiving SEND)
CQ: Completion notifications (hardware writes CQE upon operation completion)

QP State Machine:
  RESET → INIT → RTR (Ready To Receive) → RTS (Ready To Send)

Memory Region (MR)

// RDMA hardware writes directly to remote memory → requires memory "registration" (pinning)
// Registration: ibv_reg_mr() → hardware locks physical pages + generates access key (rkey/lkey)
// 
// Why is registration needed?
//   RDMA HCA bypasses the kernel → cannot rely on CPU page tables for address translation
//   → requires direct mapping of physical addresses + locked pages (no swap/page fault)

// On-Demand Paging (ODP) / Implicit MR (IMR):
//   Reduces registration overhead → pin/unpin on demand

Verbs API

// User space: libibverbs

// Find devices
ibv_get_device_list()

// Create QP
ibv_create_qp(pd, &qp_init_attr)

// Post work request (send)
ibv_post_send(qp, &wr, &bad_wr)
// RDMA_WRITE: Direct write to remote memory (most common)
// RDMA_READ:  Direct read from remote memory
// SEND:       Send message (remote side needs to pre-post RECV)

// Completion
ibv_poll_cq(cq, num_entries, &wc)  // Read CQE

Kernel Bypass and Linux Integration

Traditional TCP Via Kernel app socket TCP/IP netdevice NIC

RDMA Bypasses Kernel app verbs User-space Driver NIC (Fully Bypasses Kernel!)