6 min read #linux-kernel #block-device-layer
On this page

NVMe and Storage

Coverage: NVMe Protocol (SQ/CQ/PRP/SGL) → Linux NVMe Driver → NVMe Multipath → NVMe-oF → SCSI Subsystem (sd/sr) → Device Mapper (dm-crypt/LVM/dm-thin) → Bcache/dm-cache Kernel Version: 3.3 ~ 6.x

NVMe Protocol Basics

Queue Model

NVMe uses a PCIe + ring queue model, replacing AHCI's fixed-depth command slots:

Each pair of SQ (Submission Queue) + CQ (Completion Queue) forms an I/O channel.
  SQ: Host → Controller (Host writes, Controller reads)
  CQ: Controller → Host (Controller writes, Host reads)

Advantages of multiple queues:
  - Each CPU core has its own SQ/CQ → Zero lock contention
  - Supports up to 64K queues, with 64K commands per queue → Extremely high parallelism
  - Interrupts: MSI-X per CQ → Independent interrupts per queue → Natural IRQ affinity binding to cores

Command Format

// include/linux/nvme.h
struct nvme_command {
    u8  opcode;        // NVME_CMD_READ, NVME_CMD_WRITE, NVME_CMD_FLUSH, ...
    u8  flags;
    u16 command_id;    // Unique ID within the associated SQ

    u32 nsid;          // Namespace ID

    // PRP (Physical Region Page) or SGL (Scatter-Gather List)
    __le64 prp1;       // First physical page or PRP list pointer
    __le64 prp2;

    // Logical Block Address (LBA) + Transfer Length
    __le64 slba;       // Starting LBA
    __le16 nlb;        // Number of Logical Blocks (0-based)
};

PRP vs SGL:

  • PRP: Simple linked list, each entry points to a physical page. Supports up to 2 inline entries + a PRP list. Suitable for small I/O.
  • SGL: General-purpose scatter-gather list, supporting segments of arbitrary length. Required for NVMe 1.2+ / NVMe-oF.

Doorbell Mechanism

Submitting a command:
  1. Host writes the NVMe command to an SQ entry (ring buffer)
  2. Host writes the SQ Tail Doorbell (MMIO write, notifying the controller of new commands)
  3. Controller fetches the command → Executes → DMA data → Writes to CQ entry
  4. Controller sends an MSI-X interrupt
  5. Host writes the CQ Head Doorbell (notifying the controller that CQ entries have been consumed)

Entire process: 1 MMIO write for submission, 0 MMIO reads (CQ resides in Host DDR)
Comparison with AHCI: Each submission requires multiple rounds of MMIO reads (reading PxCI, etc.)
  → NVMe latency is 3~5x lower

Linux NVMe Driver

Driver Layering

// drivers/nvme/host/
// nvme-core.c:  Protocol implementation (admin queue, IO queue, identify, abort, ...)
// nvme.h:       Data structures (nvme_command, nvme_completion)
// pci.c:         PCIe transport layer (MMIO doorbell, MSI-X, PRP mapping)
// fabrics.c:     NVMe-oF transport layer (TCP/RDMA/FC)

Queue Initialization Flow

nvme_probe(): From PCIe Enablement to IO Queue Readiness nvme_probe() Enable PCIe Device + Map BAR nvme_dev_map() Create Admin SQ/CQ (Single Pair, for Management Commands) nvme_alloc_admin_tags() → blk_mq_init_queue() Identify Controller + Namespace (via Admin Queue) nvme_identify_ctrl() → Get Queue Count, Capabilities, Firmware Version Create IO SQ/CQ · Allocate based on CPU topology (num_possible_cpus() or nr_io_queues) · Each IO queue → One blk-mq hardware context (hctx) · MSI-X Interrupts: request_irq() per CQ blk_mq_alloc_tag_set() → Associate with request_queue The Admin queue comes first; only after obtaining queue count and capabilities are IO queues laid out according to CPU topology; each IO queue has an independent hctx and independent MSI-X interrupt, laying the foundation for subsequent IRQ affinity binding.

queue_rq: Submission Path (Hot Path)

// drivers/nvme/host/pci.c
static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
                                    const struct blk_mq_queue_data *bd) {
    struct nvme_queue *nvmeq = hctx->driver_data;
    struct request *rq = bd->rq;

    // 1. Construct NVMe command
    struct nvme_command *cmd = &nvmeq->sq_cmds[nvmeq->sq_tail];
    nvme_map_rq(rq, cmd);  // Construct PRP/SGL + LBA + Length

    // 2. Write NVMe command to SQ
    //    (Already in the SQ ring buffer, no copy needed)

    // 3. Update SQ tail pointer
    nvmeq->sq_tail = (nvmeq->sq_tail + 1) & (nvmeq->q_depth - 1);

    // 4. Write doorbell (MMIO write)
    writel(nvmeq->sq_tail, nvmeq->q_db);

    return BLK_STS_OK;
}

// Interrupt: nvme_irq()
//   → nvme_process_cq() → Iterate through CQ entries
//     → nvme_handle_completion() → blk_mq_complete_request()
//       → mq_ops->complete() → end_that_request_last()

NVMe Multipath

// drivers/nvme/host/multipath.c
// The same namespace can be accessed via multiple I/O paths:
//   - NVMe-oF: Multiple controllers under the same subsystem
//   - PCIe: Dual-port NVMe drives

// nvme-core automatically aggregates multiple paths for the same namespace into one block device
// I/O Scheduling:
//   Default: round-robin (ANA: Asymmetric Namespace Access)
//   Optimized Path: I/O policy knob

NVMe-oF (NVMe over Fabrics)

Extends the NVMe protocol to networks:
  NVMe/TCP:  Standard TCP transport (Most universal)
  NVMe/RDMA: InfiniBand/RoCE (Lowest latency, ~10μs)
  NVMe/FC:   Fibre Channel (Traditional data center stack)

Linux Implementation:
  Target (Server): drivers/nvme/target/ + nvmet-tcp/nvmet-rdma
  Host:             drivers/nvme/host/fabrics.c + nvme_tcp/nvme_rdma

SCSI Subsystem

// drivers/scsi/
// Three-layer architecture:
//   Upper Level: sd (Disk), sr (CD-ROM), st (Tape)
//   Mid Level:   Generic command construction, Error recovery (eh_*)
//   Lower Level: Vendor HBA drivers
//
// Key Structures:
//   scsi_cmnd: A single SCSI command
//   scsi_host_template: Operations registered by the HBA driver
//   queuecommand(): Submit SCSI command → HBA

// SCSI 5.x+ uses blk-mq by default (scsi_mod.use_blk_mq=1)
// Maps SCSI tags to blk-mq tags → Unified queue management

Device Mapper: Block Device Virtualization

// drivers/md/
// The Device Mapper (DM) framework allows the creation of virtual block devices:

// Common DM Targets:
//   dm-linear:    Linear mapping (Foundation of LVM)
//   dm-crypt:    Block device encryption (LUKS)
//   dm-thin:     Thin provisioning
//   dm-cache:    SSD caching for HDD (dm-cache / dm-writecache)
//   dm-raid:     MD RAID is no longer developed; DM RAID takes over
//   dm-multipath: Multipath I/O (SAN)

// CRYPT (dm-crypt):
//   Key: LUKS header on disk → Contains encryption algorithm, key slots, PBKDF
//   Data Path: bio → encrypt/decrypt per sector → Submit to underlying device

cryptsetup luksFormat /dev/sda2
cryptsetup open /dev/sda2 cryptroot
dmsetup table  # View all DM devices

Bcache: SSD Caching for HDD

// drivers/md/bcache/
// Bcache uses SSD as a read/write cache for HDDs
// 
// Cache Modes:
//   writeback:    Writes go to SSD first → Async writeback to HDD (High performance, risk on power loss)
//   writethrough: Writes go to both SSD and HDD simultaneously (Safe, but write performance is limited)
//   writearound:  Writes bypass SSD directly to HDD (SSD used only for reads)

// Kernel 6.x natively supports bcache (Merged into mainline)

Debugging and Observation

# NVMe Driver Information
nvme list
nvme id-ctrl /dev/nvme0  # Controller capabilities (Queue count, Supported features)
nvme id-ns /dev/nvme0n1  # Namespace details

# NVMe Command-level Tracing
echo 1 > /sys/kernel/debug/tracing/events/nvme/enable
cat /sys/kernel/debug/tracing/trace_pipe

# SMART Data
nvme smart-log /dev/nvme0
# Key Metrics: temperature, percentage_used (Lifespan), media_errors

# DM Devices
dmsetup ls --tree

# Bcache
cat /sys/block/bcache0/bcache/state
echo 1 > /sys/block/sda/bcache/writeback_rate_debug

# SCSI Layer Tracing
echo 1 > /sys/kernel/debug/tracing/events/scsi/enable

References and Further Reading

  • Source Code: drivers/nvme/host/pci.c (PCIe NVMe Driver), drivers/nvme/target/ (NVMe-oF), drivers/scsi/, drivers/md/, drivers/md/bcache/
  • NVMe Specification: nvmexpress.org (base spec 1.4c/2.0)
  • Kernel Documentation: Documentation/nvme/, Documentation/device-mapper/
  • LWN: "The Linux NVMe driver", "Bcache design"

Keywords: NVMe, SQ/CQ, doorbell, PRP, SGL, blk-mq, nvme_queue_rq, NVMe-oF, SCSI, device mapper, dm-crypt, Bcache

Related pages

© 2026 liz's doc