KVM Architecture

When a guest exits to KVM, it does not necessarily cause KVM_RUN to return to user space. Within the execution loop, it is crucial to distinguish between hardware exits, kernel processing, and VMM processing. KVM provides the execution interface; QEMU is a common consumer, not the only prescribed one.

Hardware examples focus on x86 VMX/SVM, while other architectures use different execution levels and mechanisms.

The Kernel Handles Execution Boundaries; The VMM Organizes the Entire Machine

KVM utilizes Linux infrastructure such as scheduling and memory management, and implements hardware virtualization control through architecture-specific implementations. User-space VMMs typically handle machine configuration, firmware, device models, and migration coordination; some device data paths can enter the kernel or operate as independent backends.

The kvm module on x86 also contains corresponding architecture-specific code. kvm_intel/kvm_amd then interface with vendor-specific mechanisms; one cannot simplify the entire kvm.ko into a purely architecture-agnostic core. The directories virt/kvm and arch/x86/kvm can help distinguish source code responsibilities.

Two Types of "Exits" Need to Be Separated

VMX root/non-root and guest internal CPL0/CPL3 are different dimensions: the guest kernel can still execute at its own privilege level. Ordinary instructions are usually executed directly by the CPU; hardware control specifies which events or operations require an exit.

Hardware VM exits first return to KVM. If KVM can handle it, such as certain page mappings or interrupt status updates, it can directly re-enter the guest; only in cases requiring user-space processing does KVM_RUN return. Multiple hardware entries and exits can occur during a single KVM_RUN.

Therefore, counting ioctl calls cannot replace counting hardware VM exits. Exits are not all triggered by "sensitive instructions"; external events and nested paging issues can also trigger them.

Independently select one guest operation, CPUID, MMIO, or host signal, and count hardware exits and KVM_RUN returns separately. The two boundaries can be crossed individually; do not mix counts using a single "exit count."

File Descriptor Hierarchy

Different ioctls must be used with the correct category of fd. Interface capabilities are queried via KVM_CHECK_EXTENSION; do not rely solely on the kernel version number; some extensions also require explicit enabling. VM lifecycle is tied to its references; closing a duplicate fd does not necessarily destroy the entire VM immediately. KVM API

Below only probes interfaces without creating a VM:

#include <fcntl.h>
#include <linux/kvm.h>
#include <stdio.h>
#include <sys/ioctl.h>
#include <unistd.h>

int main(void) {
    int fd = open("/dev/kvm", O_RDWR | O_CLOEXEC);
    if (fd == -1) { perror("open /dev/kvm"); return 1; }
    int version = ioctl(fd, KVM_GET_API_VERSION, 0);
    if (version < 0) { perror("KVM_GET_API_VERSION"); close(fd); return 1; }
    if (version != KVM_API_VERSION) {
        fprintf(stderr, "unsupported KVM API: %d\n", version);
        close(fd); return 1;
    }
    int memory = ioctl(fd, KVM_CHECK_EXTENSION, KVM_CAP_USER_MEMORY);
    if (memory < 0) { perror("KVM_CHECK_EXTENSION"); close(fd); return 1; }
    int bytes = ioctl(fd, KVM_GET_VCPU_MMAP_SIZE, 0);
    if (bytes < 0) { perror("KVM_GET_VCPU_MMAP_SIZE"); close(fd); return 1; }
    printf("API=%d user_memory=%d vcpu_mmap=%d\n", version, memory, bytes);
    close(fd);
    return 0;
}

Save as kvm_probe.c and compile with cc -D_GNU_SOURCE -Wall -Wextra kvm_probe.c -o kvm_probe. The absence of /dev/kvm or lack of permissions is a different issue from interface capability support; retain specific errors during diagnosis.

vCPU is an Execution Context Scheduled by the Host

QEMU's common KVM mode assigns one host thread per vCPU. The number of vCPUs is not the number of physical cores; threads may be preempted, migrated to different CPUs, or waiting for I/O/interrupts; CPU pinning and NUMA placement affect results.

KVM uses scheduling notifications and other mechanisms to maintain architecture execution state during host switches. Steal time, virtual clocks, and TSC offset/scaling have specific interfaces; do not describe every thread switch as "recalculating all time offsets."

HLT indicates the guest has no work to do and is waiting for a wake-up; whether it blocks in the kernel or returns to user space depends on configuration; it is not equivalent to idle injection used to limit host power consumption.

The Run Loop Must Handle Errors and Completion Protocols

On error returns, do not blindly read the shared exit_reason which may still belong to the previous exit.

After returning 0, the user space handles exit reasons such as KVM_EXIT_IO, KVM_EXIT_MMIO, etc. Some I/O operations require re-entering KVM_RUN to complete on the kernel side; when preparing to save state or migrate, you must consider these incomplete operations.

Which Completion Boundaries Must Be Crossed for One Input Instruction

Assume the guest executes "read one byte from a port, then add 1 to the result." The device model is in user space. After the first KVM_RUN returns due to KVM_EXIT_IO, the VMM finds the virtual device for that port and fills the value 7 into the shared run area. However, "buffer filled with 7" and "guest input instruction has submitted the result" are still two distinct states.

The KVM API explicitly requires such operations to re-enter the kernel for completion. When you need to stop after completion but before the next guest instruction, you can use immediate_exit or pending unmasked signals depending on capability support; completing pending operations occurs before subsequent signal checks. immediate_exit support should be queried via KVM_CAP_IMMEDIATE_EXIT. Migration also requires coordinating other vCPUs, devices, and memory; here we only handle one easily overlooked local boundary. KVM Run Area and I/O Completion Conventions

Below is the state machine for three abstract instructions IN → ADD → HALT. pc is the instruction index, not an x86 byte address; it is used to demonstrate completion order and does not simulate true instruction decoding, hardware exits, or KVM ioctls.

import errno

class GuestModel:
    def __init__(self):
        self.pc = 0
        self.acc = 0
        self.waiting_input = False
        self.input_byte = None
        self.exit_reason = None

    def run(self, immediate_exit=False):
        # Complete the input operation from the previous exit first, then handle the early return for this round.
        if self.waiting_input:
            if self.input_byte is None or not 0 <= self.input_byte <= 255:
                raise ValueError("Must provide an input byte first")
            self.acc = self.input_byte
            self.pc += 1
            self.waiting_input = False
            self.input_byte = None
        if immediate_exit:
            return -1, errno.EINTR  # exit_reason does not represent a new exit this time
        while True:
            if self.pc == 0:
                self.waiting_input = True
                self.exit_reason = "IO_IN"
                return 0, 0
            if self.pc == 1:
                self.acc = (self.acc + 1) & 255
                self.pc += 1
            else:
                self.exit_reason = "HALT"
                return 0, 0

for value in (0, 7, 255):
    guest = GuestModel()
    assert guest.run() == (0, 0) and guest.exit_reason == "IO_IN"
    guest.input_byte = value
    assert (guest.pc, guest.acc, guest.waiting_input) == (0, 0, True)
    assert guest.run(immediate_exit=True) == (-1, errno.EINTR)
    assert (guest.pc, guest.acc, guest.waiting_input) == (1, value, False)
    assert guest.exit_reason == "IO_IN"  # Old field remains; do not process it again as a new I/O
    assert guest.run() == (0, 0) and guest.exit_reason == "HALT"
    assert (guest.pc, guest.acc) == (2, (value + 1) & 255)

The trace for value 7 stops at pc=1, acc=7 after completing the input; running further yields 8. If registers are saved after writing to the buffer but before re-entering the kernel, the hidden state of the pending operation is not fully expressed by ordinary visible states, and the restore side may be unable to continue the same operation.

The second call in the model also illustrates a detail: EINTR does not mean that all previous work did not happen. This round may have already completed the previous pending operation; however, it does not provide a new successful exit, so you cannot call the device model again based on the leftover IO_IN field. A normal run loop should first check the ioctl return value before deciding which exit fields to read.

I/O Data Interpreted by Offset and Count

The data_offset of KVM_EXIT_IO is relative to the start of the entire struct kvm_run mapping; the data is a compact array of count elements, each size bytes. For example, size=2, count=3 means 6 bytes, not just processing the first 16-bit value. For OUT, the VMM reads this data; for IN, the VMM fills the device return value back into the same region.

The offset is not a guest virtual address, nor a guest physical address; using it directly to query guest page tables will access the wrong address space. MMIO exits use a different set of fields, including physical address, length, and data, and cannot be mixed with the layout of port I/O. During processing of the shared region, ensure that the execution of the same vCPU and device processing are serialized according to the predetermined thread protocol, avoiding rewriting data from the previous round while re-entering KVM_RUN.

The Shared Run Area is Not a Complete Register Mirror

KVM_GET_VCPU_MMAP_SIZE gives the vCPU shared mapping size; do not assume it is one page. struct kvm_run is used for execution control and exit data, not as a general-purpose complete register mirror; registers are usually handled via corresponding ioctls, or under explicitly supported and enabled synchronization mechanisms.

Interrupt Injection Depends on Whether the Guest Can Receive Them

Device completion, virtual timers, and vCPU IPIs can all generate virtual interrupts. Setting the pending state does not guarantee immediate delivery before the next guest instruction; it must also comply with guest interrupt masking, priority, and hardware injection conditions. kick is used to prompt the vCPU to process new work and does not mean unconditionally running its handler immediately.

Kernel irqchip, irqfd, and hardware interrupt virtualization can reduce some user-space round trips, but the path depends on architecture capabilities. Address translation is covered in Memory and Device Virtualization; thread and migration collaboration is covered in QEMU and KVM.