Block Device Layer

A single application write may be split into multiple device commands, or it may remain in the cache. Along the block layer, you must track data ranges, queueing positions, and completion promises separately; you cannot directly equate application IOPS with the number of device commands.

1. One write does not equal one device command

Buffered writes typically modify the page cache first; subsequent writeback generates block I/O. Cache-hit reads may not access the device at all. Direct I/O bypasses some cache paths, but alignment, filesystem, and device constraints may still cause a single call to be broken down into multiple requests.

A bio describes a block data operation and its memory range; it is not an atomic unit of a disk transaction. It can be cloned, split, and remapped; multiple compatible bios can be merged into the same request. Stacking layers such as device-mapper and RAID may also generate I/O directed at different lower-layer devices.

In modern kernels, the current position and remaining length are mainly described by bi_sector and bi_size in bio.bi_iter, and bi_status is of type blk_status_t. Do not treat the pseudo-structure of old flat members as the currently compilable API.

Three types of "segments"

• bio_vec describes the memory range, including the starting page, offset, and length; modern multipage bvecs can span pages and do not necessarily correspond one-to-one with a single page.
• The bio iterator describes the portion currently being processed; you cannot deduce the remaining I/O solely based on bi_vcnt.
• The DMA scatter-gather segments ultimately used by the drivers undergo mapping and merging, and their count need not match the original bvec count.

Refer to IOMMU and DMA API for mapping rules.

2. Sector units and splitting/merging

The same device byte position uses different units at different layers:

The block layer sector is fixed at 512-byte counts; this does not imply that the device's logical block must also be 512 bytes.

Two consecutive 4 KiB bios, with device intervals [2048,2056) and [2056,2064) sectors respectively; if the operation, flags, queue limits, etc., are compatible, they may be merged into a single 8 KiB request. Whether the physical memory addresses are adjacent is not the only condition, because requests can use SG (scatter-gather) to represent non-contiguous memory.

Conversely, a 256 KiB bio may be split into multiple requests on paths that allow only 128 KiB transfers, or have stricter segment counts/boundaries. The constraints are not just total bytes, but also include maximum segment count, single-segment length, boundaries, alignment, and special constraints for different operation types.

Therefore, application IOPS, bio count, request count, and device command count are not the same metric. An increased merge rate may reduce command overhead, but this does not mean the number of business requests has decreased.

3. blk-mq: Multi-entry parallelism, not completely lock-free

blk_mq_ctx provides a software submission context per CPU; blk_mq_hw_ctx manages the state for driver dispatch queues. CPUs select hctx based on queue mapping; read, write, poll, and other queue types can also have different mappings.

hctx is a software structure, not the SQ memory directly read by the controller. The driver converts requests into device commands and places them into its own queues. The number of software queues, hardware queues, CPUs, and interrupt vectors are not guaranteed to be equal, and you cannot describe ctx as a generic per-cgroup queue.

Multi-queues disperse shared hotspots, but the queue list, scheduler, tag allocation, and driver may still require locks or atomic operations. A scheduler may also maintain shared state across the request_queue, such as the scheduling state of mq-deadline; you cannot uniformly state that "all schedulers run independently on each hctx." See the blk-mq documentation.

Tags and Backpressure

Tags are used to identify and manage in-flight requests, allowing the completion path to locate the request. There are different configurations such as scheduler tags, driver tags, and shared tag sets; you cannot assume that each request occupies the same fixed hardware tag from creation to completion.

When resources are insufficient, requests may wait, be delayed in dispatch, re-queued, or return with NOWAIT conventions indicating they cannot be executed immediately; this is different from returning a media error after the device has received the command. After accepting a request, the driver must still complete it according to the protocol; do not confuse "temporarily busy" with "owns the request but will complete it later."

4. Plug: Short-term batching, not transactions

Plug associates with the current task, temporarily storing requests suitable for batching, increasing merge opportunities, and reducing per-request dispatch overhead. The basic call form is as follows; this is a local structure within the existing kernel submission path, not for user-space programs:

struct blk_plug plug;
blk_start_plug(&plug);
/* Submit bios here that are correctly established and managed for their lifetime. */
blk_finish_plug(&plug);

The parameter is struct blk_plug *, not current. Plug does not guarantee that all I/O will be held until finish: conditions such as blocking scheduling and capacity can prompt early dispatch. It does not guarantee atomic completion for the entire batch, nor does it guarantee completion in submission order.

While batching increases throughput, it may also increase wait times, so the batch size cannot be expanded indefinitely. Dependencies between reads and writes, flushes, and filesystem transaction ordering are still guaranteed by their respective mechanisms.

5. Schedulers address policy trade-offs

Do not replace measurement with "NVMe must use none, desktop must use BFQ." Storage media, device internal concurrency, mixed read/write patterns, cgroups, background writeback, and CPU costs will change the conclusions.

read_expire / write_expire do not guarantee the maximum latency from application submission to completion; writes_starved controls the read/write selection strategy and is not an SLA for each business. Scheduler parameter units and available options should be read from the actual sysfs. Refer to mq-deadline, Kyber, and BFQ.

6. Quantitative relationships between queue depth, throughput, and latency

Under a stable system and the same measurement boundary, use Little's Law L = λ × W:

Align statistical boundaries first, then interpret this mean.

This 20 is an observed mean, not a requirement to set a specific sysfs parameter to 20. Queues fluctuate, and both software and hardware have their own quotas. Increasing concurrency may improve throughput when the device is not saturated; after saturation, increasing depth further often mainly increases queueing and tail latency.

Similarly, 100000 data operations of 4096 bytes each amount to approximately 409.6 MB/s, or 390.625 MiB/s. IOPS must be interpreted in conjunction with I/O size, read/write ratio, synchronous semantics, and latency.

In the simplified closed queue below, increase service parallelism and then request depth. After the throughput plateau, locate the extra requests.

$$\lambda =\frac{\min (Q,K)}{d},W=\frac{Q}{\lambda }$$

This ideal closed-loop model keeps Q requests in flight, with K service slots and service time d. λ is completions per second; W is average time within the same boundary.

7. Completion, ordering, and persistence

Block layer completion notification indicates that the operation has ended under the current device stack contract; you cannot assert that data has entered non-volatile media solely based on a normal write completion. Volatile write caches change this boundary.

Native device support and block layer simulation produce different command sequences. Neither directly guarantees atomicity for arbitrary write ranges. See Write Cache Control.

The application's fsync() is coordinated by the filesystem for data and necessary metadata, then pushed down as appropriate block layer operations. Directory persistence involved in file creation/renaming also requires consideration of directory synchronization; you cannot interpret the entire file transaction based on a single bio flag.

8. Read-only viewing of actual queue parameters

from pathlib import Path

fields = ('logical_block_size', 'physical_block_size', 'max_sectors_kb',
          'max_hw_sectors_kb', 'nr_requests', 'scheduler', 'write_cache')
for dev in sorted(Path('/sys/block').glob('*')):
    if dev.name.startswith(('loop', 'ram')):
        continue
    print(dev.name)
    for field in fields:
        try:
            print(' ', field, (dev / 'queue' / field).read_text().strip())
        except OSError:
            pass
    print(' ', 'visible hctx directories:', len(list((dev / 'mq').glob('[0-9]*'))))

In the scheduler output, square brackets indicate the currently selected item; the absence of an mq directory may be due to stacked devices or other situations. nr_requests cannot be universally interpreted as the maximum number of commands per queue for physical devices, and max_sectors_kb does not have a fixed default value applicable to all devices.

9. Returning from statistics to evidence

The await in iostat -x includes wait and service times within the corresponding device statistical boundaries; it is not the end-to-end latency of application requests. %util approaching 100% does not equal "throughput has reached the absolute upper limit" for parallel devices, nor can it be treated as CPU utilization.

blktrace/block layer tracepoints can observe queueing, dispatch, and completion, but splitting, merging, retries, partial completions, and event loss can cause simple address-based pairing to fail. Determine identity and hierarchy first, then calculate time differences. Statistical fields are in Block Device Statistics, and tracing methods are in Tracing and Debugging.