QEMU and KVM Collaboration

Coverage: QEMU-KVM process model → KVM_CREATE_VM/KVM_RUN → memory layout → vCPU thread loop → device emulation → live migration Kernel version: 2.6.20 ~ 6.x

Process Model

flowchart TD
    QEMU["One VM = One QEMU Process"]

    QEMU --> MAIN["🧵 Main Thread<br/>event loop, monitor"]
    QEMU --> IO["🧵 IO Thread (iothread)<br/>handles virtio device IO"]
    QEMU --> VCPU["🧵 vCPU Threads × N<br/>each executes KVM_RUN"]
    QEMU --> MIGRATE["🧵 Migration Thread<br/>live migration"]

    classDef root fill:#e3f2fd,stroke:#1565c0
    classDef thread fill:#fff3e0,stroke:#ef6c00
    class QEMU root
    class MAIN,IO,VCPU,MIGRATE thread

VM Creation Flow

// QEMU → KVM:
kvm_fd = open("/dev/kvm", O_RDWR);

// 1. Create VM
vm_fd = ioctl(kvm_fd, KVM_CREATE_VM, 0);

// 2. Allocate guest RAM (managed by QEMU):
guest_ram = mmap(NULL, ram_size, PROT_READ|PROT_WRITE,
                 MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0);

// 3. Register with KVM:
struct kvm_userspace_memory_region region = {
    .slot = 0,              // memory slot ID
    .guest_phys_addr = 0,   // start of guest physical address
    .memory_size = ram_size,
    .userspace_addr = (uintptr_t)guest_ram,
};
ioctl(vm_fd, KVM_SET_USER_MEMORY_REGION, &region);

// 4. Create vCPU:
for (int i = 0; i < num_vcpus; i++) {
    vcpu_fd[i] = ioctl(vm_fd, KVM_CREATE_VCPU, i);
    mmap_size = ioctl(kvm_fd, KVM_GET_VCPU_MMAP_SIZE, 0);
    vcpu->kvm_run = mmap(NULL, mmap_size, ..., vcpu_fd[i], 0);
    // kvm_run is shared memory: KVM writes exit_reason and register state here
}

vCPU Execution Loop

// QEMU: kvm_cpu_exec() (accel/kvm/kvm-all.c)
int kvm_cpu_exec(CPUState *cpu) {
    struct kvm_run *run = cpu->kvm_run;
    int ret;

    do {
        // 1. Inject pending interrupts (if any)
        if (cpu->interrupt_request)
            kvm_arch_put_registers(cpu);

        // 2. Enter guest!
        ret = ioctl(cpu->kvm_fd, KVM_RUN, 0);

        // 3. VM exit → handle:
        switch (run->exit_reason) {
        case KVM_EXIT_IO:
            kvm_handle_io(run->io.port, run->io.direction, run->io.size, run->io.count,
                          run->io.data_offset);
            break;
        case KVM_EXIT_MMIO:
            address_space_rw(run->mmio.phys_addr, ...);
            break;
        case KVM_EXIT_IRQ_WINDOW_OPEN:
            // Interrupts can be injected now
            break;
        case KVM_EXIT_SHUTDOWN:
            // Guest triple fault → stop
            return -1;
        }
    } while (ret == 0);

    return ret;
}

IO Thread and virtio Handling

// QEMU iothread: independent event loop
//   → epoll on eventfd (virtio kick)
//   → QEMU handles virtio requests:
//       virtio-blk: read/write → map to host file (qcow2/raw)
//       virtio-net: tap fd → read/write Linux tun/tap
//
// vhost (bypassing QEMU):
//   ioeventfd + irqfd → kernel handles directly
//   → QEMU only configures, does not participate in the data path

Live Migration

QEMU + KVM Collaboration:
  1. Source: Mark all guest RAM as read-only
     → Subsequent writes generate write faults → record dirty pages
  2. Iterative copy: Pre-copy RAM to destination
  3. Dirty page convergence: Dirty rate ↓ each round → until threshold
  4. Stop-and-copy: Pause source VM → copy last dirty pages + register state → start destination
  5. Downtime ~100ms (depends on workload)

References

• Source Code: QEMU accel/kvm/kvm-all.c (KVM acceleration core), Linux virt/kvm/kvm_main.c
• Kernel Documentation: Documentation/virt/kvm/api.rst (ioctl API manual)

Keywords: KVM_RUN, kvm_run, ioctl, KVM_CREATE_VM, vCPU thread, iothread, live migration