NVMe and Storage

A single namespace can have multiple paths, and even after a command completes, it may still rely on volatile cache. To understand NVMe, you must separately locate storage objects, queue protocols, data addresses, and recovery states.

1. Controller, Namespace, and Subsystem

The controller provides the host access interface and management capabilities; the namespace provides the logical block address space; the subsystem can contain multiple controllers and provide shared namespaces. A namespace does not necessarily correspond to an independent NAND chip, nor can it be simply equated with a partition.

A namespace can be accessed via different paths. Whether it is recognized as the same storage object depends on identifiers such as the subsystem, NSID, and UUID/NGUID/EUI, as well as sharing attributes, rather than just comparing capacity or /dev names. Device numbers may change after a path disappears; persistent location should use the correct stable identifiers.

The Identify command returns capabilities and format information, but actual I/O queue resources still require negotiation and creation. The controller's supported maximum, the number actually used by the driver, and the current concurrency of the workload are three different values.

2. SQ/CQ: Two Producer-Consumer Protocols

In the PCIe path, the host writes commands to the submission queue and notifies the controller of the new tail according to the protocol. After execution, the controller writes back the completion queue entry, which the host processes and advances the CQ head.

One SQ is associated with one CQ; the protocol allows multiple SQs to share one CQ. Linux drivers commonly use a paired organization, but this is not a protocol-mandated one-to-one correspondence. MSI-X vectors also do not need to be bound one-to-one with each queue; polled queues may not use interrupts. The number of queues is determined by hardware, driver, CPU topology, and resources, and does not guarantee exclusive, lock-free access per CPU.

The CQ's phase tag helps distinguish whether a ring slot has already been written by the controller as a valid completion in the current cycle. After the host detects a valid state, it must read the remaining fields according to DMA/memory ordering rules. Commands can complete out of order; requests are located using SQ identifiers and command identifiers, so you cannot assume the earliest submitted command finishes first.

Fill and consume one ring turn, then revisit slot zero. Old CID bytes deliberately remain visible: head and expected phase establish validity, not whether a slot looks nonempty.

Why the doorbell cannot be a fixed "one MMIO per operation"

Before publishing a command, you must ensure that related writes can be seen by the device according to the protocol. The host can batch-publish multiple SQ entries before notifying, and it also updates the CQ head. Doorbell buffers, polling, and interrupt coalescing mechanisms further alter access frequency.

Therefore, while NVMe can reduce control overhead, there is no cross-platform universal inference that "every I/O requires exactly one MMIO, which is five times faster than AHCI." Completion latency is also affected by flash, cache, queue waiting, CPU, NUMA, and power states.

3. Specific Examples of LBA, NLB, and PRP

Taking a 4096-byte logical block as an example, reading 8192 bytes from LBA 256:

You must first determine the namespace format to correctly interpret command units.

This differs from the Linux block layer's 512-byte sector unit; the driver must perform the conversion. The actual struct nvme_command contains unions for different command formats; fields are not a single flat set of members common to all commands; multi-byte fields are encoded in little-endian format according to the protocol.

PRP describes data addresses accessible to the controller

With a controller page size of 4096 bytes, a starting offset of 1024, and a total length of 8192, three pages are required:

Therefore, two more pages are needed after the first page before determining the meaning of PRP2.

• PRP1 points to the start of the first segment, allowing for page-internal offsets.
• If the remaining data fits in one page, PRP2 can directly point to that data page.
• In this example, the remainder requires two pages, so PRP2 points to a PRP list, which provides the addresses of subsequent pages.

A PRP list does not mean "PRP2 is always the second data pointer"; there are linking rules when the list crosses pages. Addresses are established via the DMA API; do not interpret "Physical" in the name as a CPU virtual address or an unconverted physical address that can be filled in directly.

SGL uses descriptors with type and length information; supported capabilities vary by command set, controller, and transport. Data transfer for NVMe/RDMA and NVMe/TCP cannot simply apply the PCIe host memory PRP mechanism. Entry points: NVM Express Specification and kernel include/linux/nvme.h.

4. Linux Submission and Recovery Paths

drivers/nvme/host/core.c handles shared protocols and controller/namespace lifecycles; pci.c handles PCIe transport; tcp.c, rdma.c, fc.c, etc., handle their respective transports; fabrics.c provides common fabrics functionality.

PCIe initialization typically makes the device and management queues available first, then retrieves capabilities, negotiates resources, creates I/O queues, and scans namespaces. During probing and recovery, there are asynchronous work items; you cannot treat a list of function names from a specific version as a fixed order of synchronous execution.

When submitting a request, the driver sets up the command, maps the data, handles resource failures, and then publishes the command to the controller. Queue wrapping cannot blindly use index & (depth - 1) without proof: this bitwise operation only holds for powers-of-two depths.

A timeout indicates that the host has not yet confirmed completion; it does not prove that the device did not execute the write. The driver will implement abort, reset, reconnect, or failure handling based on status, and will handle old requests and late completions. Upper layers cannot infer that old side effects did not occur at all based on "retry success." Completion still requires distinguishing the persistence semantics of normal write cache, Flush, and FUA.

5. Native Multipath and ANA

Linux native NVMe multipath aggregates paths from multiple controllers into a single logical access object when identifier and configuration conditions are met. ANA describes the access state of paths, such as optimized and non-optimized; the host uses this along with I/O policies to select paths.

Strategies such as NUMA, round-robin, and queue-depth have their uses in versions that support them; you cannot hardcode the default strategy as round-robin. Actual status should be read from the running system. Path selection is also not another name for the I/O elevator: the former selects the access path, while the latter arranges requests within the queue.

Multipath improves path redundancy but does not automatically provide data replicas, backups, or replication across independent disks. During fault recovery, requests may queue or fail; their latency is affected by controller and connection recovery policies. See NVMe multipath.

NVMe/TCP, RDMA, and FC extend NVMe access to respective transports. Network congestion, authentication/encryption configuration, reconnection, and target backends all alter performance and reliability; do not assign RDMA an unconditional fixed microsecond latency.

6. Where SCSI and device-mapper Fit

SCSI upper-layer drivers include disk sd, optical drive sr, and tape st; the middle layer handles general command management and error recovery; lower-layer drivers connect to HBAs or specific transports. SATA may also appear as SCSI disks in Linux via libata and other paths, so /dev/sdX cannot be judged as a physical bus solely by name.

device-mapper maps logical block addresses to lower-layer devices according to targets, and can be composed in multiple layers:

LVM is a user-space volume management system that can configure various DM targets; it is not merely a "dm-linear command wrapper." The interfaces and management relationships between MD, DM RAID, and LVM need to be distinguished.

Why Thin Pool Monitors Both Data and Metadata

Thin volumes declare logical capacity first, allocating data blocks only upon actual write; snapshots share blocks, and subsequent writes may trigger allocation and mapping updates. Data pool full and metadata pool full are different faults, potentially entering queuing, read-only, or error paths; specific behavior depends on configuration and status.

Snapshots still depend on the original pool's data and metadata; they are not independent backups. File deletion does not guarantee immediate pool space reclamation; it also depends on discard, snapshot references, and target support. See thin provisioning.

7. Where Data Is When Cache Mode Changes

Recovery basis when mode changes cannot rely solely on comparing speeds during normal operation.

It is not "SSD is safe, HDD is dangerous," but rather determining the persistence of dirty data, mapping metadata, and flush agreements. Cache devices with power-loss protection can mitigate some power-loss risks, but device failure itself may still result in loss of unwritten data.

dm-cache, dm-writecache, and bcache are different implementations; their management interfaces and migration/removal processes are not universal. bcache is not an abbreviation for bcachefs. See dm-cache, bcache.

8. Read-Only Observation and Location

lsblk -o NAME,TYPE,SIZE,LOG-SEC,PHY-SEC,PKNAME,MOUNTPOINTS

After installing nvme-cli, you can use nvme list and nvme list-subsys to view objects and paths, and nvme id-ctrl / id-ns to view capabilities of specific objects. SMART's percentage_used is an estimate of endurance consumption, not a "remaining life countdown"; error counts need to be combined with trend changes and specific error interpretations.

When an application sees I/O errors, sequentially confirm the logical volume, DM/RAID, namespace/path, controller, and lower-layer device. Do not accumulate statistics of the same data across multiple stacked layers into physical throughput. Block layer measurements and completion semantics are covered in Block Device Layer; key hierarchies are covered in Crypto API and Storage Encryption.