KVM Memory and Device Virtualization

Guest page faults, second-stage translation failures, and device DMA out-of-bounds accesses occur along different control paths. First, distinguish between CPU address translation and device access authorization, then examine shared queues and migration.

Two-Stage Address Translation

The guest OS manages the page table from GVA to GPA; x86 EPT/NPT then maps GPA to HPA and enforces host-controlled permissions. The two-stage permissions jointly determine whether access is allowed. The VMM's HVA is the perspective for host software to manage backing; it is not an address directly usable by the guest.

The classic User Memory Interface registers a correspondence between a GPA range and VMM user-space memory using memslots. Registration does not mean physically locking all pages in the range at once; host page faults, reclamation, and mapping invalidation still require KVM coordination. New mechanisms like private guest memory have different interfaces, so not all VM memory can be assumed to be ordinary readable/writable HVA. Memory Interface

Memslot offset calculation does not yield a physical page number

Assume a standard user memory slot covers GPA [0x10000000, 0x10003000), with the corresponding HVA start at 0x70000000. GPA 0x10001234 falls within the slot, with an offset of 0x1234, so the corresponding HVA is 0x70001234. This addition only yields a position in the VMM address space and cannot derive the HPA; the host page table and backing state determine which physical page currently holds it.

Even if the address start is within the slot, it does not guarantee the entire access stays inside. For example, reading 8 bytes from 0x10002ffc causes the last 4 bytes to exceed the slot's exclusive upper bound. The following model requires the entire non-empty access to reside within a single slot and does not handle splitting cross-slot requests.

GPA_BASE, HVA_BASE, SIZE = 0x10000000, 0x70000000, 0x3000

def slot_hva(gpa, length):
    if not 0 <= gpa < 2**64 or not 0 < length <= 2**64:
        raise ValueError("Invalid address or length")
    offset = gpa - GPA_BASE
    if not 0 <= offset < SIZE or length > SIZE - offset:
        raise ValueError("Entire access must be within this slot")
    return HVA_BASE + offset

assert slot_hva(0x10001234, 8) == 0x70001234
assert slot_hva(0x10002ffc, 4) == 0x70002ffc
for gpa, length in [(0x10002ffc, 8), (0x10003000, 1),
                    (GPA_BASE - 1, 1), (GPA_BASE, 0), (2**64 - 1, 8)]:
    try:
        slot_hva(gpa, length)
    except ValueError:
        pass
    else:
        raise AssertionError("Should reject out-of-bounds or illegal access")

In fixed-width languages, checking "whether the length exceeds remaining space" avoids integer wraparound when calculating gpa + length. For actual device backends, if no RAM slot is found, one should determine whether it belongs to an MMIO device or an invalid access according to the machine address map, rather than assuming any hole is a readable/writable device.

EPT violation does not mean the guest modified the page table

When the guest updates its own GVA page table, it usually does not require re-establishing all GPA-to-HPA mappings. An EPT violation reflects that the second-stage translation does not exist or permissions are not met, which may correspond to on-demand mapping establishment, write-protection tracking, or access to regions treated as MMIO.

Page faults generated by the guest's own page tables are also different from second-stage issues: the former may need to be handled by the guest kernel. When diagnosing, one must distinguish which layer the fault occurred in, rather than attributing all faults to the guest modifying page tables.

Hardware nested paging reduces the maintenance burden of software shadow page tables, but the KVM MMU still has complex mapping and invalidation management, and nested virtualization introduces additional combinations. One should not view the shadow MMU merely as legacy from old CPUs that no longer exist. KVM MMU

Change guest PTE and EPT permissions separately. Both stages retain the same page offset, but addresses beyond a failed stage are not shown as completed translations.

Why page table traversal cost cannot be fixed at 24 steps

Four-level guest page tables superimposed on four-level nested page tables, under idealized conditions of no cache and no huge pages, can derive multiple memory accesses: fetching each guest page table entry itself also requires second-stage translation. Actual hardware has TLBs, page table walk caches, huge pages, and different levels, so not every load re-traverses the full path.

Huge pages may reduce translation pressure, but huge pages, alignment, and continuity at both guest and host levels must be considered. Dirty page tracking, sharing, or invalidation may split mappings; one cannot simply look at HugePages in the guest and assert that huge pages are also used in the second stage.

Expanding a fully unmitigated four-level walk

Assume the guest page table walks g levels, and accessing each guest page table entry's GPA requires a full h-level second-stage page table walk; all relevant caches are unmitigated, no huge pages terminate early, and additional transactions like accessed/dirty bit updates are ignored.

To obtain one guest page table entry, h second-stage page table entry reads are needed, plus the guest page table entry read itself, totaling h+1 reads. There are g such entries. After obtaining the final GPA, h more second-stage page table entry reads are needed, and finally 1 read for the target data:

• Translation-related reads: g × (h+1) + h.
• Including the final data read: (g+1) × (h+1).

def uncached_walk(g, h):
    assert g >= 1 and h >= 1
    translation = sum(h + 1 for _ in range(g)) + h
    return translation, translation + 1

assert uncached_walk(4, 4) == (24, 25)
assert uncached_walk(4, 3) == uncached_walk(3, 4) == (19, 20)
assert uncached_walk(3, 3) == (15, 16)

This explains what "24" usually refers to, and why it cannot be written as a fixed cost per access: when the TLB hits, the full traversal above does not occur; the page containing the page table entry and the final data page may not have the same second-stage leaf level. The table sets them uniformly to h only to compare hypothetical changes, not implying that giving the guest a huge page necessarily reduces all h by 1.

virtio defines device protocols, not backend location

The virtio driver and device exchange buffer descriptors via negotiated capabilities and virtqueues. PCI, MMIO, etc., are transport methods; virtio-net, virtio-blk, etc., are device types; the backend can be QEMU, kernel vhost, user-space services, or compatible hardware. Virtio

Split virtqueues separate the descriptor table, available, and used areas; packed virtqueues use another form of publish and recycle. One cannot directly describe all packed implementations using split field flows.

Barriers and publish ordering maintain this protocol, not just for performance optimization.

Kick and interrupts are notifications, not a strict one-request-to-one-notification mapping: they can be suppressed or merged, and the driver can also poll. After buffers are handed to the device, they cannot be arbitrarily modified or freed while the device is still using them; shared memory does not mean there is no ownership protocol, nor does it mean zero-copy. Virtio Specification

Two requests published sequentially can be recycled in reverse order

Taking a split queue without negotiated VIRTIO_F_IN_ORDER as an example, the driver first publishes descriptor chain head 0, then publishes chain head 1; the device can complete 1 first and 0 later. The driver must find the request based on the chain head ID in used, and cannot hard-match "the first completed item" to "the first submitted request." When IN_ORDER is negotiated, corresponding rules should be followed, and this reverse-order example should not be applied.

Below models only two device read-only request buffers and their lifecycles. The device state includes periods that have been published but the device has not started or is still processing; used indicates completion of published items that the driver has not yet reclaimed. Queue index wraparound, descriptor chains, barriers, and notification mechanisms are handled separately by full implementations.

from collections import deque

class Buffers:
    def __init__(self):
        self.state = {0: "driver", 1: "driver"}
        self.data = {}
        self.used = deque()

    def write(self, ident, payload):
        if self.state[ident] != "driver":
            raise RuntimeError("Buffer has not returned to driver, cannot rewrite")
        self.data[ident] = bytes(payload)

    def publish(self, ident):
        assert self.state[ident] == "driver" and ident in self.data
        self.state[ident] = "device"

    def complete(self, ident):
        assert self.state[ident] == "device"
        self.state[ident] = "used"
        self.used.append(ident)

    def reclaim(self):
        ident = self.used.popleft()
        assert self.state[ident] == "used"
        self.state[ident] = "driver"
        return ident, self.data[ident]

queue = Buffers()
queue.write(0, b"request A"); queue.publish(0)
queue.write(1, b"request B"); queue.publish(1)
try:
    queue.write(0, b"overwritten")
except RuntimeError:
    pass
else:
    raise AssertionError("Cannot reuse prematurely")
queue.complete(1)
assert queue.reclaim() == (1, b"request B")
assert queue.state[0] == "device"
queue.complete(0)
assert queue.reclaim() == (0, b"request A")
queue.write(0, b"request C")
queue.publish(0)
assert queue.state == {0: "device", 1: "driver"}

"Did not receive an interrupt" cannot be used to assert that completion has not occurred, and "received an interrupt" cannot be used to assert that only one request was completed. Reclamation is based on used information visible per protocol; interrupts and kicks only prompt the other end to check the queue. Memory ordering requirements ensure that when the receiver sees index advancement, descriptors, data, or completion information are already visible; the above sequential Python calls do not verify true weak memory ordering. Virtio split queue and IN_ORDER

Boundaries of vhost and vhost-user

Kernel vhost-net and other backends move part of the queue data processing to the kernel, reducing round-trips for corresponding requests entering the QEMU device model. QEMU still handles configuration and control; it is not that the process completely exits VM execution.

vhost-user places the backend in another user-space process, passing queue, memory, and notification information via protocol. The name containing vhost does not mean all backends are in the kernel. vhost-vsock handles AF_VSOCK semantics and should not be written as "AF_UNIX crossing VM boundaries."

Backend choices change copy, scheduling, memory sharing, and fault isolation paths. Performance requires measuring actual queue counts, packet or block sizes, CPU placement, and load; one cannot promise a uniform 50% latency reduction.

VFIO: Allocating Device Access Rights and DMA Isolation

VFIO exposes controlled device access capabilities, usable for passthrough or user-space drivers. The CPU's EPT/NPT does not automatically limit DMA for devices; IOMMU mapping, device isolation topology, and interrupt remapping require separate configuration. VFIO

IOMMU groups in traditional interfaces express sets of isolatable devices, often not equal to a single PCI function. Before unbinding a device, one must understand group devices, reset scope, and host dependencies; theoretical tutorials should not pretend to complete the entire lifecycle configuration with a single unbind line.

New interfaces may use IOMMUFD or other methods to manage device address spaces, but this does not mean ignoring underlying hardware isolation limits. DMA can only access authorized ranges, which is the goal after correct configuration, not an automatic guarantee completed by a single "bind vfio-pci" operation.

Passthrough migration is not absolutely impossible

What recovery materials does the device need to provide before migration

Migration requires saving and restoring device state, handling in-progress DMA, and correctly recording device modifications to memory. Devices without these capabilities cannot migrate transparently; devices with VFIO migration support and meeting QEMU and target compatibility conditions may support corresponding migration flows. VFIO Migration

Therefore, it should be written as a capability and compatibility issue, rather than "all passthrough devices cannot migrate online." The consistency boundaries for the entire VM's memory, CPU, and devices are seen in QEMU and KVM Collaboration.