Linux Context Switching

The scheduler answers "who runs next," and context switching is responsible for handing the CPU from prev to next. This is not just about saving a set of general-purpose registers; it may also involve:

• User address space and CR3/PCID;
• Kernel stack and callee-saved registers;
• FPU/SIMD/extended state;
• Architecture state such as perf, ptrace, I/O bitmap, and TLS;
• Cleanup of RCU, membarrier, rseq, and scheduler locks.

"Context switch time," "latency from task wakeup to running," and "subsequent performance loss due to cache/TLB coldness" are three different metrics and cannot be summarized by a fixed microsecond value.

1. Call Path

Typical path:

schedule()
  └─ __schedule()
       ├─ pick_next_task()
       └─ context_switch(rq, prev, next, rf)
            ├─ prepare_task_switch()
            ├─ Switch or borrow mm
            ├─ prepare_lock_switch()
            ├─ switch_to(prev, next, prev)
            └─ finish_task_switch(prev)

context_switch() is defined in kernel/sched/core.c. The code below retains only the control flow; it is a commented simplified version, not source code that corresponds line-by-line with any specific kernel version:

prepare_task_switch(rq, prev, next);
arch_start_context_switch(prev);

if (!next->mm) {                 // Switching to a kernel thread
    enter_lazy_tlb(prev->active_mm, next);
    next->active_mm = prev->active_mm;
    // Maintain active_mm reference based on whether prev was a user task
} else {                         // Switching to a user task
    membarrier_switch_mm(rq, prev->active_mm, next->mm);
    switch_mm_irqs_off(prev->active_mm, next->mm, next);
    // Delayed release of borrowed active_mm when switching back from kernel thread to user task
}

prepare_lock_switch(rq, next, rf);
switch_to(prev, next, prev);
return finish_task_switch(prev);

Always refer to the current context_switch() source code for your target version, as mm reference management and synchronization steps may change.

2. mm and active_mm

User tasks own an mm; pure kernel threads have no user address space, so mm == NULL. However, the CPU still must load a set of page tables, so kernel threads temporarily borrow the previous user task's active_mm.

"Kernel thread borrowing user page tables" does not mean it can access arbitrary user addresses. Execution in kernel mode is still constrained by page tables, KPTI, SMEP/SMAP, and explicit user-access APIs.

3. switch_mm_irqs_off(), CR3, and PCID

On x86-64, address space switching ultimately revolves around CR3 and TLB state. PCID adds an address space tag to TLB entries, allowing CR3 switches without unconditionally discarding all non-global entries.

However, "having PCID means never flushing" is also incorrect. The following situations may still require invalidation:

• ASID/PCID is recycled and reallocated;
• Page table contents change;
• KPTI requires maintaining user/kernel CR3 combinations;
• The mm's TLB generation indicates the current CPU is lagging;
• Security fixes or architectural constraints require stronger invalidation.

Linux tracks the current mm, ASID, and TLB generation in the per-CPU cpu_tlbstate. The specific implementation is located in arch/x86/mm/tlb.c.

4. What switch_to() Actually Saves

The switch_to macro on x86-64 calls __switch_to_asm(prev, next). The assembly entry mainly does three things:

1. Push callee-saved registers such as rbp, rbx, r12–r15 onto prev's kernel stack.
2. Write the current rsp to prev->thread.sp, then load next->thread.sp.
3. Restore registers from next's kernel stack, then jump to the C function __switch_to() to complete the architecture state switch.

# arch/x86/entry/entry_64.S (simplified)
pushq %rbp
pushq %rbx
pushq %r12
pushq %r13
pushq %r14
pushq %r15

movq %rsp, TASK_threadsp(%rdi)
movq TASK_threadsp(%rsi), %rsp

popq %r15
popq %r14
popq %r13
popq %r12
popq %rbx
popq %rbp
jmp __switch_to

General-purpose registers are not all copied into a thread_struct array. Callee-saved state is mainly saved in the inactive_task_frame on the kernel stack, and thread.sp allows the kernel to locate this frame.

"Who does switch_to return to?"

After executing switch_to(prev, next, prev), the CPU is already using next's kernel stack and task context. Code execution continues from the call chain left behind when next was last switched out.

Therefore, a more accurate statement is: switch_to does return, but the return happens in the context of the incoming task; the third parameter receives the task that was actually switched out, for finish_task_switch() to perform cleanup. Simply saying "it doesn't return from a C perspective" obscures this dual-return semantics.

5. Modern x86 FPU State Switching

Modern Linux no longer describes regular FPU context as the classic lazy-FPU model where "TS is set on every switch, and the new task's first floating-point instruction triggers #NM to save the old state."

The switch_fpu() in the current scheduling path will:

1. Save the old task's FPU registers to fpstate if needed;
2. Set TIF_NEED_FPU_LOAD;
3. Restore registers before returning to user mode or before the kernel needs to use that state.

If the task returns to the same CPU that still holds its register state, the kernel can avoid unnecessary restores. XFD/#NM can still be used to enable dynamic extended states like AMX on demand, but this cannot be generalized as the scheduling strategy for all FPU/SIMD states.

Refer to the current switch_fpu() and x86 FPU documentation.

6. TLB Shootdown is Not a Fixed Step of Ordinary Context Switching

After operations like munmap(), permission changes, or page table reclamation modify the page tables of a certain mm, other CPUs that are currently or previously running that mm may still cache old TLB entries. The kernel needs to make relevant CPUs perform invalidation operations, coordinating via IPI if necessary.

Modify page tables
  ├─ Invalidate corresponding entry/range on this CPU
  └─ Initiate shootdown to remote CPUs holding old TLB state for this mm
       └─ Wait for necessary completion confirmation

This is related to but not equivalent to "the scheduler switching the CPU from A to B": an ordinary context switch might only load an existing ASID, while a page table update might trigger a shootdown without any task switch.

7. How to Measure

First, clarify which metric you want to measure.

7.1 Count Switches

perf stat -e context-switches,cpu-migrations,page-faults -- ./workload

grep -E 'voluntary_ctxt_switches|nonvoluntary_ctxt_switches' \
  /proc/<pid>/status

A high count does not automatically mean poor performance. I/O-bound programs generate many voluntary switches; the issue depends on latency targets, CPU utilization, and cache impact.

7.2 Observe Scheduling Events

perf sched record -- ./workload
perf sched timehist
perf sched latency

perf sched latency is closer to the latency of a task waiting for scheduling, rather than the cycles spent on a single switch_to assembly instruction.

7.3 Measure Wakeup-to-Run Latency

bpftrace -e '
tracepoint:sched:sched_wakeup {
    @wake[args->pid] = nsecs;
}
tracepoint:sched:sched_switch /@wake[args->next_pid]/ {
    @wake_to_run_us = hist((nsecs - @wake[args->next_pid]) / 1000);
    delete(@wake[args->next_pid]);
}'

This histogram measures the time from "task wakeup" to "actually getting the CPU," including run queue wait time, and should not be labeled as pure context-switch cost.

7.4 Microbenchmark

perf bench sched pipe -l 100000

It measures the comprehensive cost of two tasks going back and forth through a pipe, including syscall, wakeup, scheduling, and switching. When reporting results, at least note: CPU, kernel version, mitigations, CPU affinity, frequency policy, whether it crosses NUMA nodes, and whether it involves threads or processes.

8. Common Misconceptions

• "A context switch takes a fixed 1–5 μs": Different measurement definitions and hardware make direct comparison impossible.
• "Thread switching doesn't touch the address space": Switching within the same mm usually avoids CR3 switching, but there are still scheduling, stack, register, and potential FPU costs.
• "PCID eliminates TLB flushes": It reduces unconditional flushes, but does not eliminate page table invalidations and ASID recycling.
• "All FPU states are restored on first use": This is an outdated generalization; modern implementations use eager save and return-to-user restore, handling dynamic extensions separately.
• "TLB shootdown is context switching": The former is a page table consistency mechanism, while the latter is a transfer of task execution rights.
• "The interval of finish_task_switch tracepoints is the switch cost": It usually mixes in task running time or queuing time; measurement boundaries must be defined first.

9. Upstream Resources

• Linux context_switch()
• x86 __switch_to_asm()
• x86 TLB implementation
• x86 FPU scheduler hooks
• Scheduler tracing documentation
• perf sched documentation

When reading kernel articles, the most reliable habit is to separately label "stable concepts," "implementation in a specific version," and "measurement results." Context switching has existed for a long time, but the security hardening, FPU policies, TLB management, and scheduler cleanup surrounding it are not static.