IOMMU and DMA API

Coverage: IOMMU hardware → DMA API (coherent/streaming) → swiotlb → IOMMU domain → VFIO → IOMMU in virtualization Kernel version: 2.6 ~ 6.x

IOMMU Principles

IOMMU provides IO address translation for devices (analogous to CPU MMU):
  Device-view address (IOVA) → IOMMU page table → Physical address

Functions:
  1. Address translation: Devices can only "see" IOVA → unaware of real physical addresses
  2. Isolation: Each device/domain has independent page tables → A cannot access B's memory
  3. Interrupt remapping: MSI addresses translated by IOMMU → prevents MSI injection attacks
  4. Large contiguous blocks: Dispersed physical pages → mapped as contiguous IOVA from device perspective → simplifies DMA buffer allocation

Implementation:
  Intel: VT-d (DMAR table)
  AMD:   AMD-Vi (IVRS table)
  ARM:   SMMU (System MMU)

DMA API

// include/linux/dma-mapping.h
// Device drivers should not manually allocate/release physical memory for DMA → must use DMA API

// === Coherent DMA ===
// Usage: descriptor rings, device status pages (CPU and device frequently read/write the same region)
// Characteristics: Always consistent between CPU and device → cache automatically synchronized
void *dma_alloc_coherent(struct device *dev, size_t size,
                          dma_addr_t *dma_handle, gfp_t gfp);
void dma_free_coherent(struct device *dev, size_t size,
                        void *cpu_addr, dma_addr_t dma_handle);

// === Streaming DMA ===
// Usage: data buffers (one-time, fixed direction)
// Characteristics: Requires explicit sync → better performance than coherent
dma_addr_t dma_map_single(struct device *dev, void *cpu_addr,
                           size_t size, enum dma_data_direction dir);
void dma_unmap_single(struct device *dev, dma_addr_t dma_handle,
                       size_t size, enum dma_data_direction dir);

// direction: DMA_TO_DEVICE (write), DMA_FROM_DEVICE (read), DMA_BIDIRECTIONAL

// === Scatter-Gather ===
int dma_map_sg(struct device *dev, struct scatterlist *sg,
                int nents, enum dma_data_direction dir);

Internal Implementation: Platform Selection

With IOMMU:
  dma_map_single → iommu_dma_map_page() → creates mapping in IOMMU domain
  → Zero-copy (mapping from IOVA to physical address, no data copying required)

Without IOMMU:
  dma_map_single → if DMA addressable range < physical address range → SWIOTLB bounce buffer
  → Allocates temporary buffer in low address space → copies data (performance loss)

SWIOTLB: Software Bounce Buffer

When device DMA address range is limited (e.g., 32-bit PCI devices, can only access 0~4GB)...:
  SWIOTLB reserves bounce buffers in the low address space (DMA zone)
    dma_map:  Data from high address → copy → bounce buffer (within DMA reachable range)
    dma_unmap: bounce buffer → copy → user buffer (if DMA_FROM_DEVICE)

  Boot parameter: swiotlb=65536 (allocates 64MB bounce buffer)

IOMMU Domain and Group

// drivers/iommu/
// domain: Abstraction of IOMMU page tables (a set of IOVA → physical mappings)
// group: Hardware isolation unit (devices in the same group can access each other)

// IOMMU group is the isolation boundary for VFIO:
//   Devices within a group cannot be isolated from each other → must be assigned together to the same VM
//   /sys/kernel/iommu_groups/*/devices/*

VFIO: Userspace Device Drivers

// drivers/vfio/
// Allows userspace programs to directly (but safely) access hardware:
//   1. IOMMU isolation: Device mapped to its own IOMMU domain → cannot access host memory
//   2. MMIO passthrough: Maps device BARs → userspace can directly mmap
//   3. Interrupt forwarding: Device interrupts → IOMMU interrupt remapping → eventfd → QEMU/userspace

// QEMU: vfio-pci → PCI passthrough → guest directly accesses hardware
// SPDK:  Userspace NVMe driver → zero-copy data path

References

• Source Code: kernel/dma/, drivers/iommu/intel/, drivers/iommu/arm/, drivers/vfio/
• Kernel Documentation: Documentation/core-api/dma-api.rst
• LWN: "The DMA API", "IOMMU groups and VFIO"

Keywords: IOMMU, DMA API, dma_map_single, coherent DMA, swiotlb, IOMMU domain, VFIO