IOMMU and DMA API

Addresses in device descriptors cannot be directly cast from CPU pointers; unmap is not a command to stop hardware. DMA requires managing addresses, direction, visibility, and handover of ownership.

1. Three Types of Addresses Cannot Be Cast to Each Other

An IOMMU can translate IOVAs issued by devices into physical page addresses; without an IOMMU, DMA addresses may also be affected by host bridge offsets, etc. dma_addr_t cannot be dereferenced directly, nor obtained by arbitrarily casting CPU pointers. Even if the values happen to be identical on a specific machine, this is not a portable guarantee.

Drivers set the DMA mask according to hardware capabilities. The address width of descriptors and data payloads can differ; set the streaming mask and coherent mask separately if necessary. A narrower mask may not be supported on the platform, so handle API return values; do not claim device support for 64-bit DMA just because the CPU is 64-bit.

2. Coherent and Streaming Solve Different Access Patterns

dma_alloc_coherent() returns a CPU address and outputs a DMA address for device use. When freeing, use the matching device, size, CPU address, and DMA handle. Coherent memory exempts this region from explicit cache maintenance for normal access, but it does not mean all writes are immediately visible to the device in source code order, nor does it automatically composite operations into atomic ones.

For example, when publishing a descriptor, write the address and length first, execute the appropriate dma_wmb(), and finally set the valid bit according to the device protocol; after the device completes, the CPU may also need dma_rmb() to safely read data published with the completion flag. MMIO doorbell access ordering must also satisfy I/O accessor and hardware requirements. Barriers do not equal waiting for DMA completion.

"Streaming" in streaming mappings does not mean mapping only once, nor does it promise to be faster than coherent; performance depends on the platform, cache, mapping overhead, and access patterns. Concepts and address rules are in the DMA HOWTO, and specific interfaces are in the DMA API.

3. Direction is Based on Data Flow Between Device and Memory

• DMA_TO_DEVICE: Device reads from memory, e.g., sending network packets.
• DMA_FROM_DEVICE: Device writes to memory, e.g., receiving network packets.
• DMA_BIDIRECTIONAL: Both sides write, and ownership and synchronization protocols must still be followed.

Do not just mark "read" or "write," as the CPU and device perspectives are exactly opposite. Use accurate directions when determined; the device, range, and direction used for map, sync, and unmap must match the API's pairing requirements.

A single receive can be understood as: the CPU allocates a suitable buffer and maps it as FROM_DEVICE, submitting the returned DMA address to the device; unmap after confirming the device has truly completed, then the CPU reads. Unmapping is not a command to stop hardware; you must first confirm the device is no longer accessing the memory.

When Mapping is Not Revoked, CPU and Device Must Still Hand Over Access Rights

When retaining a mapping long-term, call dma_sync_single_for_cpu() after confirming device completion before the CPU enters the access phase; call dma_sync_single_for_device() before handing back to the device. These calls may involve cache maintenance or bounce buffer copying, but they do not prevent the other party from accessing simultaneously on behalf of the driver.

In the simplified model of non-coherent platforms, CPU cache and RAM contents are shown separately. Ownership is transferred via synchronization actions; observe which step is missing between device completion and CPU readiness to read.

What Memory Can Be Mapped Directly

dma_map_single() is suitable for contiguous memory meeting constraints, such as properly allocated kmalloc buffers. Stack memory, kernel image addresses, and virtually contiguous addresses returned by vmalloc cannot be directly used as generic single-segment DMA buffers. vmalloc underlying pages need to be handled via appropriate per-page/SG paths.

You should also avoid sharing cache lines between CPU fields and device write areas; on non-coherent platforms, such sharing can cause data overwrites. dma_mapping_error(dev, addr) checks if mapping failed; a DMA address of zero does not automatically represent failure.

4. Two Counts for Scatter-gather

SG allows expressing a single data transfer using multiple memory fragments. The DMA layer can merge adjacent fragments, so the number of segments seen by the device may be less than the number of input entries.

Below is partial API usage, assuming sgl is correctly established and orig_nents > 0:

int mapped_nents = dma_map_sg(dev, sgl, orig_nents, direction);
if (mapped_nents == 0)
    return -EIO;

/* Program hardware DMA segments: use the mapped count and sg_dma_* fields. */
struct scatterlist *sg;
int i;
for_each_sg(sgl, sg, mapped_nents, i) {
    dma_addr_t addr = sg_dma_address(sg);
    unsigned int len = sg_dma_len(sg);
    /* Fill addr/len into actual hardware descriptors; hardware not started here. */
    dev_dbg(dev, "DMA segment %d: %pad / %u\n", i, &addr, len);
}

/* The actual driver must confirm hardware has stopped using the buffer before executing the next line. */
dma_unmap_sg(dev, sgl, orig_nents, direction);

The mapping layer generates segments from the device's perspective and must also retain information needed to restore the original mapping. Do not use the merged count for all interfaces. See the SG API documentation.

5. IOMMU and SWIOTLB Can Coexist

IOMMU provides address translation, permissions, and isolation via page tables, and may also map discrete physical pages into contiguous IOVAs. It does not guarantee that any SG becomes a single segment; it is still constrained by segment length, boundaries, and mapping resources. Interrupt remapping is a related but independent capability provided by the platform; you cannot assume all interrupt safety conditions are met just because DMA translation exists.

SWIOTLB provides bounce buffers: intermediate buffers accessible by device operations, where the DMA layer moves data at corresponding synchronization points. Besides limited device addressing, shared memory constraints in confidential computing may also use bounces; therefore, it is not simply "if IOMMU exists, no copy; if no IOMMU, then SWIOTLB."

The direction of copying relates to the DMA direction: TO_DEVICE requires sending data prepared by the CPU to a region readable by the device, while FROM_DEVICE retrieves device results when the CPU receives data. Whether copying is needed and at which synchronization point it occurs is determined by the mapping and synchronization paths; do not assume bidirectional copying on every map/unmap.

The SWIOTLB pool size parameter has its own units; do not treat slot counts directly as KiB. For example, with common 2 KiB slots, 65536 slots is 128 MiB, not 64 MiB. Specific parameters, dynamic pools, and limits are subject to the target kernel's SWIOTLB documentation.

6. Domain, Group, and VFIO

Groups are influenced by topology, ACS, and device behavior, and are not fixed as one group per PCI function.

When handing over a function from a group to untrusted user space, you must also control other related devices in the group to prevent driver access that could bypass isolation boundaries. This does not mean every function must actually be presented to the same VM; specific binding and ownership constraints are managed by the VFIO/IOMMU interface.

VFIO allows controlled exposure of device resources, DMA mappings, and interrupts. The underlying guest memory for device access remains host physical pages; the protection goal is to ensure it can only access authorized mappings, not to write "cannot access host memory at all." Special modes without IOMMU cannot be considered to have the same isolation guarantees. See the VFIO documentation.

Reading Order

For address exceptions, first check DMA mask, return values, and addresses written to descriptors; for data corruption, check direction, completion timing, cache line sharing, and barriers; for post-release corruption, check if unmap/free occurred before hardware and callbacks finished. IOMMU faults indicate that a device access violated mapping conditions, but you cannot conclude that the page table implementation is wrong solely based on a fault.

Related Reading: PCIe and USB Driver Model, DRM and Graphics.