QEMU and KVM Collaboration

Adding IOThreads does not guarantee that all device operations run in parallel, nor does running pre-copy migration for a few more rounds guarantee a reduction in dirty pages. QEMU and KVM collaboration requires simultaneous management of execution context, device state, and migration consistency. The complete backend may also reside in other processes.

Threads Are Not a Fixed One-to-One Mapping for Devices

When a vCPU thread enters KVM_RUN, the main event loop handles management and corresponding device events; devices configured with IOThreads can use independent event loops. Not every virtio device automatically gets its own thread; thread pools, backends, and device implementations also participate in I/O. IOThreads

Multithreading allows for parallelism but introduces state synchronization and execution context constraints. Mechanisms such as QEMU's global lock, device locks, and AioContext are responsible for specific states; one cannot infer that all device operations are parallel or lock-free simply by "adding IOThreads."

From Creation to an Executable Machine

Simply calling CREATE_VM, registering RAM, and CREATE_VCPU does not mean the operating system can start: configurations for CPU mode, boot code or firmware, interrupt controllers, and devices are still missing. The simplified flow in tutorials must be distinguished from a "complete launcher."

After creating backing via mmap, physical pages may be allocated on demand; MAP_NORESERVE does not guarantee that memory will always be sufficient later. The memslot expresses the guest address mapping, but the VMM must still maintain the backing lifecycle; changes or unmap operations need coordination with KVM. KVM API

Two Levels of Return from the Execution Loop

Hardware exits can be handled by KVM and re-enter the guest directly; the VMM's exit handling only enters when KVM_RUN returns. When an error is returned, handle errno first; upon successful return, read run->exit_reason; do not interpret every ioctl return as a normal device request.

For MMIO/PIO reads/writes that need to be completed, the VMM fills in the data and re-enters KVM to complete the corresponding guest operation. If only registers are saved before migration while ignoring unfinished exits, state boundaries may be lost. QEMU already has architecture and device collaboration logic; it is inappropriate to use a few lines of pseudo-C functions lacking error handling as its real source code.

eventfd Reduces Notification Detours, Does Not Carry All Data

ioeventfd can convert matching guest I/O writes into eventfd notifications; irqfd can associate corresponding eventfd signals with guest interrupt injection. Request data remains in buffers described by the virtqueue or device protocol; eventfd is not the content of every complete request.

The kernel vhost backend can reduce the round-trip of QEMU user space processing data requests based on this, but initialization, feature negotiation, device state, and migration control still require the VMM. User-space backends have another set of shared memory and inter-process protocols.

Pre-copy Migration and Convergence

While the source VM is running, memory continues to be modified, and the migrator needs to track pages that remain dirty after this round. Write-protection triggering tracking is one way; hardware dirty bits, dirty bitmap/ring, etc., may also participate; not all mechanisms can be described as "setting all RAM to read-only at the beginning."

If the rate of continuously dirtying pages approaches or exceeds the rate the transfer can handle, repeated pre-copy may not converge. The amount of dirty pages in the new round is not guaranteed to be less than the previous round; bandwidth, compression, workload throttling, and switching strategies jointly affect the result. Migration Framework

At this point, you cannot wait for "copying a few more rounds to naturally clear to zero." Real convergence is also affected by repeated writes to the same page, page granularity, and windows; you cannot treat the two rates directly as a precise prediction formula.

From 8 GiB to Final Downtime: Calculate Round by Round

Assume the memory to be copied is M=8 GiB, the effective send rate is B=2 GiB/s, and the new different dirty pages generated each round and needing to be sent in the next round grow at D=0.5 GiB/s. Assume fixed rates, no compression, and temporarily ignore page granularity rounding and additional transmission overhead. This is a round-aggregated model; D represents the data volume of different dirty pages, not the original bytes of application execution stores.

When the send amount for this round is R, the time is R/B; the new workload generated during this time is min(M, D×R/B). When not hitting the M limit, the shrinkage factor is D/B=1/4.

If the fixed CPU/device switching overhead is 40 ms, stopping the source immediately after the third round requires 0.125/2 s + 40 ms = 102.5 ms. Running a fourth round leaves 0.03125 GiB, i.e., 32 MiB, with an estimated downtime of 15.625 + 40 = 55.625 ms, which finally falls within the 60 ms budget for this example.

from fractions import Fraction as F

def precopy(memory, bandwidth, dirty_rate, rounds):
    assert memory > 0 and bandwidth > 0 and dirty_rate >= 0 and rounds >= 0
    remaining = F(memory)
    rows = []
    for _ in range(rounds):
        seconds = remaining / bandwidth
        new_dirty = min(F(memory), dirty_rate * seconds)
        rows.append((remaining, seconds, new_dirty))
        remaining = new_dirty
    return rows, remaining

rows, remaining = precopy(F(8), F(2), F(1, 2), 4)
assert [r[2] for r in rows] == [F(2), F(1, 2), F(1, 8), F(1, 32)]
fixed_stop = F(40, 1000)
assert rows[2][2] / 2 + fixed_stop == F(1025, 10000)
assert remaining / 2 + fixed_stop == F(55625, 1000000) <= F(60, 1000)
for dirty in (F(2), F(3)):
    unstable, left = precopy(F(8), F(2), dirty, 4)
    assert left == 8 and all(row[2] == 8 for row in unstable)
assert precopy(F(8), F(2), F(0), 1)[1] == 0

When D≥B, this model regenerates all 8 GiB every round, and continuing to increase rounds does not shorten the final downtime. The actual workload's hot page set may be small: the same page written ten thousand times might still only need to be sent once in a certain statistical window. Therefore, judgments should be made based on different dirty page amounts, windows, and actual bandwidth; do not directly substitute memory write bandwidth counts into D. QEMU's dirty-limit mechanisms help convergence by limiting dirty workloads, but the cost includes impacting the source workload. QEMU Dirty limit

Clearing Dirty Bits After Copying May Erase Newly Occurring Writes

Consider a page with a current value of 0, and the dirty bit is already set. After the migrator copies 0 to the target, the guest changes the source page to 1, and the dirty bit remains 1. If the migrator unconditionally clears the dirty bit at this moment to indicate "this page copy is complete," it erases the evidence of the new write; the next round will neither see the dirty page nor supplement the value 1.

A correct protocol needs to establish clear tracking boundaries, separating the old dirty state processed in this round from subsequent new modifications. Below, use atomic exchange of dirty sets to express this boundary: take out the current round's set and start tracking the next round, then copy; modifications occurring during the copy enter the new set. It does not implement real KVM dirty bits, write-protection restoration, memory barriers, or DMA tracking.

class Memory:
    def __init__(self):
        self.pages = {0: 0}
        self.dirty = {0}

    def write(self, page, value):
        self.pages[page] = value
        self.dirty.add(page)

    def take_epoch(self):
        old, self.dirty = self.dirty, set()
        return old  # In the model, this operation cannot be interrupted by writes

# Incorrect order: clearing after copy swallows new modifications.
bad = Memory()
target = {0: bad.pages[0]}
bad.write(0, 1)
bad.dirty.clear()
assert not bad.dirty and target != bad.pages

# Correct boundary: new modifications after this round's copy are left for the next round.
source = Memory()
target = {}
for page in source.take_epoch():
    target[page] = source.pages[page]
source.write(0, 1)
assert source.dirty == {0} and target[0] == 0
for page in source.take_epoch():
    target[page] = source.pages[page]
assert target == source.pages and not source.dirty
for value in range(2, 102):
    source.write(0, value)
assert len(source.dirty) == 1  # One hundred writes, still only one different dirty page

Real systems have multiple layers of tracking including KVM, QEMU RAM accounting, and migration sending; you cannot directly replace take_epoch() in the model with any clear ioctl. Kernel re-protection of pages, hardware dirty bit collection, and sending order need to jointly ensure that "modifications after reading are not lost." Page content may also change during copying; finally, after stopping the source, re-synchronizing the remaining dirty state forms the final consistent switch boundary.

Change effective bandwidth B and dirtying rate D, then choose when to finish pre-copy. Watch how the remaining set and stop-and-copy time behave as D/B approaches one.

$$M_{n+1}=\min \left(M,\frac{D}{B}{M}_n\right),T_{\mathrm{stop}}\approx \frac{M_{n+1}}{B}+T_{\mathrm{fixed}}$$

M is total memory and M_n the set copied in round n. The model assumes constant bandwidth and an effective distinct-page dirtying rate, capped by total memory; repeated writes to one page do not add unlimited dirty bytes.

Consistency at the Last Moment Involves More Than Just RAM

Therefore, migration consistency cannot be judged solely by "RAM has been copied."

Post-copy can fetch missing pages on demand after the target resumes, changing the trade-off between downtime and page fault waiting, and making source/target fault recovery after switching more complex. It is not equivalent to "running the last round of pre-copy faster" and should not be compared using a single downtime number.

Before and after successful migration, it must be clear which instance has external write rights to avoid split-brain operation. Whether passthrough devices can be migrated depends on device state and DMA tracking capabilities; see Memory and Device Virtualization.

Find the Layer First When Observing Problems

When CPU usage is high, distinguish between guest effective execution, frequent exits, and host scheduling waits; when I/O latency occurs, distinguish between guest queuing, notifications, QEMU/backend execution, and actual storage network; when migration stalls, look at dirty page rates, effective bandwidth, and device blocking.

/dev/kvm availability detection can be found in KVM Architecture. Further source code can be entered from QEMU accel/kvm/kvm-all.c, Linux virt/kvm, and corresponding arch directories; version-related paths should be based on the actual checkout.