RDMA: Queues, Memory Registration, and Completion Semantics

After a work request is successfully posted, the device may still be reading from the source buffer; after the remote memory is updated, the application may not have consumed it yet. RDMA reduces processing and copying in specific data paths, but resource authorization, completion, and reuse must still be clearly defined.

1. InfiniBand, RoCE, and iWARP

RoCE deployments often pair ECN, congestion control, and appropriate PFC policies, but PFC is not a logical prerequisite that "must be enabled network-wide unconditionally whenever RoCEv2 is present." The actual effects of packet loss and pause policies depend on hardware and network design and require explicit verification; PFC itself can also cause head-of-line blocking and congestion propagation.

Linux software implementations such as RXE and SIW are helpful for learning/testing corresponding RDMA interfaces, but the data plane still consumes CPU and goes through software processing. Do not directly equate behaviors measured with software RDMA to the performance or bypass capabilities of hardware RNICs.

2. QP, CQ, and PD are Managed Objects

A QP typically contains an SQ and an RQ: the SQ receives actively initiated SEND, READ, WRITE, and other work requests; the RQ provides buffers needed to receive SEND operations. A CQ is a separately created and associated object, not necessarily a private third queue defined within the QP structure.

Multiple QPs can share a CQ; send_cq and recv_cq can be the same or different. When using SRQ, multiple QPs can share receive resources. The capabilities requested at creation must be verified against the actual capabilities returned by the provider, especially regarding maximum WR, SGE, and inline sizes. See ibv_create_qp.

PD (protection domain) participates in access control for associated resources like QPs and MRs. Placing objects into the correct PD and satisfying requirements for addresses, keys, and access flags is necessary to legally use registered memory; knowing a pointer value is not sufficient to issue DMA.

RC QPs often go through RESET, INIT, RTR, and RTS states, requiring configuration of addresses, paths, PSN, access permissions, and retry parameters. RDMA CM can assist with connection management, or parameters can be exchanged and configured by the program separately; not all applications must use the same CM flow. Connection, reliability, and supported operations for UD and other QP types differ.

3. Memory Region: What Registration Promises

Standard MR registration establishes memory address ranges and permissions with the provider/RNIC, typically involving page pinning, DMA mapping, and hardware translation tables, consuming limited resources. lkey is used for local SGE access validation, and rkey is used for corresponding remote access validation.

Mechanisms like ODP can handle page mapping on-demand in supported environments, eliminating the need to pin all pages from creation to destruction. Therefore, statements like "registration always permanently locks all pages" and "RDMA can never page fault" are too absolute.

The meaning of addresses received by the remote side depends on MR settings such as IOVA/zero-based, and cannot always be treated as unconverted local CPU pointers. Access flags also determine whether remote read/write/atomic operations are allowed; when remote writes are needed, associated local write permission requirements must also be set according to the interface. See ibv_reg_mr.

Keys Are Not Encryption Keys

rkey is an access authorization identifier and does not provide transport encryption, authentication, or confidentiality encapsulation for application data. It should be exchanged through trusted protocols along with address and length information to limit exposure scope, and in-flight operations on the remote side must be handled when resources are revoked.

Registering large address ranges for the entire process may reduce management overhead but also expands the impact of erroneous remote writes. MR/MW range, permissions, key lifetime, and connection identity must be designed together, rather than solely pursuing reduced registration overhead.

4. Differences in Responsibilities for SEND, WRITE, and READ

RC SEND may trigger RNR and retries when remote receive resources are insufficient, and may ultimately fail. WRITE_WITH_IMM writes data to the specified remote address; do not mistake it for a standard SEND writing data into receive SGEs; however, receive notifications still consume corresponding RQ/SRQ work resources.

RDMA atomic operations only provide atomicity for supported sizes, alignments, operations, and scopes, and do not automatically make arbitrary multi-field data structures transactionally consistent.

5. Post Success vs. Completion Success

ibv_post_send() success indicates that the work has been submitted and accepted, not that the device has finished using the data. When a batch WR chain fails midway, bad_wr points to the first failed item, and previous items may have been successfully submitted; do not release the entire buffer batch upon seeing an error.

Non-inline send buffers must typically remain alive until the corresponding completion condition is met; inline data is copied into the submission structure, allowing earlier reuse of the original data according to the inline contract, but queue resources and completion management still apply. See ibv_post_send.

CQ Return Count and WC Status Must Be Checked Separately

The number of completion records and the success/failure of each work item are two separate levels of judgment.

Failed completions only guarantee that certain fields are valid, such as wr_id, status, qp_num, and vendor_err. CQ event notifications are not WCs themselves: events prompt the application to poll, and polling retrieves the completion records. Request events and acknowledge them according to the notification interface contract to avoid races between sleep and new completions.

Choosing unsignaled sends can reduce CQE count, but this does not mean all buffers can be immediately reused arbitrarily. Recovery watermarks must be established based on QP order, operation/fence semantics, and subsequent signaled completions, and error paths must also be considered. Long-term non-consumption of CQs may lead to overrun; do not treat it as a minor issue of just losing a statistical log entry. See ibv_poll_cq.

6. Remote Memory Updates Are Not Application Transaction Commits

Treat the remote record slot as an object requiring full management:

RNIC data movement does not automatically fill in these state transitions for the application.

"WRITE payload first, then WRITE ready=1" cannot be treated as a universal publish protocol without considering conditions such as QP, operation order, memory visibility, and relaxed ordering. Cross-QP operations also do not have automatic global ordering. Notifications must be designed according to the transport/provider's completion and fence contracts, combined with CPU synchronization methods.

Remote Updates, Consumption, and Persistence Are Confirmed Separately

Local success completion of a standard WRITE does not equal server application consumption, nor does it mean the server has persisted the data to disk. If business requires persistence confirmation, the server should reply after completing corresponding storage and metadata steps, or use a dedicated solution that explicitly supports persistence semantics.

RDMA READ also does not automatically obtain a consistent snapshot: the remote side may modify multi-field objects during the read. Version checking, locking, or copy protocols are still necessary.

What is visible below are the request, transport, CQE, remote application consumption, and persistence, but each action only fulfills its own conditions. The source can be immediately reused after local completion, and then check if the remote content has been persisted.

7. Actual Scope of "Kernel Bypass"

Hardware user-space verbs fast paths can be accessed directly by the provider via mapped queues/doorbells, reducing system calls per operation. The kernel still handles resource creation, mapping authorization, events, exceptions, ODP, and device lifecycle, and the application still executes submission and poll logic.

Fix Measurement Boundaries Before Comparing Performance

Busy polling often trades CPU for low latency; event-driven approaches reduce idle spinning but introduce notification costs. Buffer pre-registration, message size, NUMA, queue depth, congestion, and provider all affect performance. When comparing TCP with RDMA, specify whether it is single-operation latency, throughput, CPU/byte, or end-to-end business completion time.

Some hardware RoCE data paths do not pass through the standard host IP socket stack, so do not assume nftables socket/forwarding rules or tcpdump automatically cover all RDMA traffic. Resource permissions, NIC/switch counters, and RDMA events must be observed together. See User-space verbs.

8. Read-Only Viewing of RDMA Devices and Ports

from pathlib import Path

root = Path('/sys/class/infiniband')
devices = sorted(root.glob('*'))
if not devices:
    print('no RDMA devices visible in this environment')
for dev in devices:
    print(dev.name)
    for port in sorted((dev / 'ports').glob('*')):
        for field in ('state', 'phys_state', 'link_layer'):
            try:
                print(' ', port.name, field, (port / field).read_text().strip())
            except OSError:
                pass

Tools like rdma link show can supplement kernel RDMA object information. Device visibility does not prove that remote QP, MR, routing, or application protocols are configured successfully; lack of visible devices only describes the current environment and cannot be used to verify actual transmission.

For deeper implementation details, refer to libibverbs, librdmacm, and providers in rdma-core, as well as the kernel drivers/infiniband/. Prioritize tracking resource creation, a single completion, and error cleanup, rather than just looking at post functions.