Context Switch

Coverage: context_switch() full path → switch_mm (page tables/TLB/PCID) → switch_to (registers/FPU) → lazy TLB → Switching overhead analysis Kernel versions: 2.6 ~ 6.x, with emphasis on PCID/ASID and FPU optimizations in 5.x+

Overview

Context switching is the "execution side" of the scheduler: the scheduler decides "who runs," while context switching handles "how to switch." Each switch requires the kernel to save the CPU register state of the current process, restore the registers of the next process, and handle address space switching (page tables, TLB) and FPU/SIMD context.

The typical overhead of a context switch is between 1 and 5 μs, depending on:

• Whether the address space changes (process switch vs. thread switch)
• Whether the TLB needs flushing (PCID/ASID can reduce this)
• Whether the FPU is dirty (requiring save/restore of the XSAVE area)
• Cache locality (a freshly woken process may still have data in the cache)

Entry Point: context_switch()

// kernel/sched/core.c
// Called by schedule() → __schedule()
static __always_inline struct rq *
context_switch(struct rq *rq, struct task_struct *prev,
               struct task_struct *next, struct rq_flags *rf) {

    prepare_task_switch(rq, prev, next);

    // === Step 1: Switch Address Space ===
    // If the two processes have different mm → page tables need switching
    if (!next->mm)
        // Kernel thread: no mm → borrow prev's active_mm
        enter_lazy_tlb(prev->active_mm, next);
    else
        // User process: normal switch
        switch_mm_irqs_off(prev->active_mm, next->mm, next);

    // === Step 2: Switch Register Context ===
    // Save prev's registers → Restore next's registers → Return to where next last stopped
    switch_to(prev, next, prev);

    // Execution is now in next's context
    // However, prev's kernel stack and rq are still in use—finish_task_switch() will clean up

    return finish_task_switch(prev);
}

Key Insight: After context_switch() calls switch_to(), execution is already in next. switch_to does not return (from the C perspective); it passes control to the return point of switch_to where the next process was previously switched out. When prev is scheduled again, it resumes execution from the return point of switch_to to execute finish_task_switch().

switch_mm: Address Space Switching

This is the most expensive part of context switching, especially when crossing address spaces.

x86-64 Implementation

// arch/x86/mm/tlb.c
void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
                         struct task_struct *tsk) {
    unsigned cpu = smp_processor_id();

    if (likely(prev != next)) {
        // Address space really needs switching

        // 1. Switch CR3 (Page Table Root Pointer)
        u64 new_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid);

        if (IS_ENABLED(CONFIG_PTI)) {
            // KPTI: Two page tables → Write kernel CR3 during switch
            // User CR3 is switched in entry_SYSCALL_64
            load_new_mm_cr3(next->pgd, new_asid, true);
        }

        // 2. Flush TLB for this CPU (if necessary)
        //    PCID allows non-global TLB entries to be tagged without flushing
        //    Without PCID → Writing CR3 flushes all non-global entries

        // 3. If other CPUs need to be notified to flush TLB → Send IPI (TLB shootdown)
        //    See arch/x86/mm/tlb.c: flush_tlb_mm_range()

        this_cpu_write(cpu_tlbstate.loaded_mm, next);
    } else {
        // prev == next: Thread switch or idle process reuse → No need to change CR3
        // But still need to confirm lazy TLB state
    }
}

PCID: Avoiding TLB Flushes

// x86 PCID (Process-Context Identifier):
//   One 12-bit ASID (Address Space Identifier) per address space
//   Attaching ASID when writing CR3 → Entries for another ASID in the TLB won't hit
//   And don't need to be flushed
//
// Without PCID: Every write to CR3 → Flushes all non-global TLB entries → High cache miss rate
// With PCID:    Switching address spaces → Only writes CR3 + new ASID → TLB retains old entries
//               (Old entries have old ASID, won't cause false hits)
//
// The kernel allocates and recycles ASIDs via per-CPU ASID bitmaps

Lazy TLB

// When switching kernel threads (mm == NULL):
//   enter_lazy_tlb(prev->active_mm, next)
//      → next->active_mm = prev->active_mm
//      → Does not write CR3 (continues using prev's page tables)
//      → Kernel address space is shared, no need to switch
//
// This is why kernel thread switching is particularly fast—no TLB flush overhead

ARM64 Equivalent Implementation

// ARM64 uses ASID (8/16 bit) instead of PCID
// TTBR0_EL1: User space page tables (different for each process)
// TTBR1_EL1: Kernel space page tables (shared by all processes)
//
// switch_mm(): Writes TTBR0_EL1 + ASID
//   → No extra instructions needed to flush TLB (ASID distinguishes hardware-side)
//
// VMID (Virtual Machine ID):
//   Virtualization extension, managed by hypervisor
//   TLB tag for stage-2 page tables

switch_to: Register Context

// arch/x86/entry/entry_64.S
// switch_to() is a macro that expands to:
//
// 1. Save prev's general-purpose registers to the kernel stack
// 2. Save prev's RIP/RSP to task_struct.thread
// 3. Restore next's RIP/RSP
// 4. Restore next's general-purpose registers
// 5. Jump to next's RIP

thread_struct — Saved Hardware Context

// arch/x86/include/asm/processor.h
struct thread_struct {
    // C compiler protected (callee-saved) registers — implicitly saved by switch_to's compiler behavior
    unsigned long           sp;         // Kernel stack pointer (RSP)
    unsigned long           sp0;        // Stack pointer used at privilege level 0

    // FPU/SIMD state
    struct fpu              fpu;        // Embedded fpu structure

    // IO bitmap (ioperm)
    struct io_bitmap        *io_bitmap;

    // Debugging
    unsigned long           debugreg[8];
    unsigned long           cr2, trap_nr, error_code;

    // Virtualization
    unsigned long           gs_base;
};

FPU Lazy Switching

// arch/x86/kernel/fpu/core.c
// FPU switching strategy under modern x86 (XSAVE/XSAVES):

// 1. Each process has an independent fpstate (XSAVE area, up to ~10KB for AMX)
// 2. "Lazy" handling during switch:
//    - FPU state is not automatically saved/restored in switch_to()
//    - Sets TS (Task Switched) or XFD bit
//    - New process uses FPU for the first time → #NM exception → Kernel performs XRSTOR
//
// 3. Why? The XSAVE area is large (AMX can be several KB), and many processes never use FPU
//    Lazy saving avoids meaningless data movement

// Exceptions to lazy saving:
//   - If the next process explicitly needs FPU (checked via __fpu_state_active)
//   - And the FPU is dirty → Must perform XSAVE/XSAVES in switch_to

Architectural Differences

TLB Shootdown

When modifying global page table entries (e.g., munmap), all other CPUs need to be notified to flush their TLBs:

// kernel/smp.c → arch/x86/mm/tlb.c
void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
                         unsigned long end, unsigned int stride_shift,
                         bool freed_tables) {
    // 1. Local: Flush own TLB
    // 2. Remote: Send IPI (Inter-Processor Interrupt) to all CPUs running this mm
    //    → Target CPU receives IPI → Flushes TLB in interrupt handler → Replies completion
    // 3. Wait for all remote CPUs to complete (tracked via per-CPU flush state)
}

The overhead of multi-core TLB shootdowns is significant (IPI + waiting), so the kernel implements various optimizations:

• Lazy TLB mode: If the target CPU is in a kernel thread (not running user space), flushing can be deferred
• Batching: Multiple TLB operations are merged into a single IPI
• Self-snoop: Some hardware can automatically detect page table changes, eliminating the need for software IPIs

Switching Overhead Analysis

Typical numbers (x86-64, ~3GHz):

Measurement Methods

# 1. perf: Latency histogram
perf sched record -- sleep 1
perf sched latency -s max

# 2. bpftrace: Directly measure schedule() latency
bpftrace -e '
kprobe:finish_task_switch {
    $ns = nsecs - @last;
    if (@last > 0) { @lat_us = hist($ns / 1000); }
    @last = nsecs;
}'

# 3. /proc/<pid>/sched
cat /proc/self/sched | grep nr_switches
# nr_voluntary_switches: Process actively yields (block/sleep/yield)
# nr_involuntary_switches: Preempted due to time slice expiration
# High involuntary → CPU-intensive; High voluntary → IO-intensive

References and Further Reading

• Kernel Documentation: Documentation/x86/tlb.rst, Documentation/arch/x86/topology.rst
• LWN:
  • "PCID and the TLB" (lwn.net/Articles/729754/)
  • "Lazy FPU state restoration" (lwn.net/Articles/675957/)
• Source Files:
  • kernel/sched/core.c — context_switch()
  • arch/x86/mm/tlb.c — switch_mm_irqs_off()
  • arch/x86/kernel/process_64.c — __switch_to()
  • arch/x86/entry/entry_64.S — switch_to at the assembly level
  • arch/x86/kernel/fpu/core.c — FPU lazy switching

Keywords: context_switch, switch_mm, switch_to, PCID, ASID, TLB shootdown, lazy TLB, FPU lazy restore, XSAVE