Linux context switching

A read() on an empty pipe provides a concrete example: A waits for data, the CPU runs B, B writes and wakes A, and only later does A resume its read.

Distinguish kernel entry, waiting for an event, waiting for CPU time, and resuming execution. They all affect call latency but suggest different remedies.

Examples use the generic scheduler and x86-64, checked against Linux 7.2. Internal function ordering and architecture details are not a stable ABI.

1. Start with a blocking read

A calls read() on a pipe with no data and at least one writer. System-call entry enters A's kernel execution path, saving the applicable user return state. The current task is still A. If data were already available, the call could return without switching tasks at all.

With no data, the kernel arranges the wait, checks the condition, and lets A sleep when necessary. Only after the scheduler selects B does the CPU save A's execution context and resume B.

B's write wakes A, restoring its opportunity to run. When A is selected later, it resumes its own waiting path, rechecks the condition, reads the data, and returns to user space.

Higher priority cannot make unproduced data appear. Find the interval before choosing a scheduling remedy.

A hardware interrupt does not automatically switch tasks either. It can interrupt A and return to A. A task handoff occurs only when an applicable exit path or another scheduling point selects a different task. Kernel entries, syscalls, and context switches count different things.

2. context_switch: address space and execution flow

After choosing next, follow three logical stages in kernel/sched/core.c:

1. Prepare accounting and the old/new mm relationship.
2. Use architecture switch_to to change stacks and the necessary register state.
3. Finish the handoff in the new task's context, accounting for the task just switched out.

The next section shows why finishing on the new stack matters.

Sharing mm can save address-space work; it does not eliminate context preservation. Different mm objects do not imply unconditional flushing of the entire TLB. On x86, CR3 identifies the page-table root and, when enabled, a PCID. Tagged translations can coexist, but cached context and invalidation generations still determine whether reuse is valid. A tag cannot legitimize an obsolete mapping.

A kernel thread's active_mm maintains a suitable address-space context without an owned user mm. References and deferred release cooperate with architecture lazy-TLB protocols. This cannot be reduced to assigning mm = &init_mm or freeing the old mm before the handoff.

3. Why switch_to seems to return on another call chain

Every task has its own kernel stack. When A calls the scheduler, its unfinished kernel call chain remains on A's stack. The low-level switch saves the stack position and architecture state needed to resume A, then installs B's saved stack position. If B previously paused through the scheduling path, execution continues on B's suspended call chain and local state.

Choose A or B directly, then change a local value on the selected task. Compare shared and separate mm scenarios: sharing an address space never merges these kernel stacks.

Two saved contexts serve different returns

x86-64 switch assembly saves callee-saved registers and the stack position. Calling conventions, compiler frames, and entry paths preserve other required values. Counting switch_to's pushes and pops does not count the complete user context.

B resumes, but A is the task to finish switching out

Suppose B previously switched out in favor of D. When A now switches to B, the resumed scheduler cleanup must identify A, not the old D, as the task just switched out. The low-level interface and common cleanup coordinate this cross-stack handoff. A newly created task has no ordinary previous scheduler return to resume; it uses a prepared initial context and startup trampoline.

An exiting task cannot immediately free the stack the CPU is still using. Switch to another task first, then perform the applicable deferred release and accounting. Scheduler locks, preemption counts, and RCU also cross this boundary through dedicated protocols. switch_to is not an isolated “save one array, restore another” utility.

4. TLS, FPU, and TLB make costs workload-dependent

Threads sharing mm still have separate thread-local storage. x86-64 FS/GS-related state must match the current task, or a thread-local access could reach someone else's data. Performance monitoring, debug registers, and I/O bitmaps can add work depending on enabled features.

Saving and loading FPU state need not coincide

FPU/SIMD extended state belongs to individual tasks. Modern x86 Linux should not be explained with the old “set TS, save the previous task on the next #NM” lazy-FPU diagram.

Local arch/x86/include/asm/fpu/sched.h saves old state when required and coordinates loading through protocols including TIF_NEED_FPU_LOAD. Some restoration may be deferred to appropriate return/use boundaries. Dynamic extensions such as AMX add state-size, permission, and on-demand handling considerations. x86 extended state

Mapping invalidation is separate from task switching

TLB shootdown prevents CPUs from using obsolete translations after a page-table change. It can happen without a task switch and need not happen on every different-mm switch.

The next figure keeps CPU 1 in M1 while CPU 0 can select M1 or M2. First test reuse of a valid tagged translation, then change M1's page table without changing either CPU's task. Both old M1 translations must become unusable.

After the low-level handoff, the resumed task may still pay for lost cache, TLB, or branch-predictor locality. These later costs can explain throughput loss better than switch assembly alone. Separate direct handoff work, indirect locality recovery, and run-queue waiting before comparing optimizations.

5. Waiting protocols: a scheduler cannot recover a lost notification

An unsafe consumer checks ready == false, intending to sleep. A producer publishes data and wakes the queue before the consumer registers. The producer sees no waiter; the consumer then registers and sleeps even though the condition is true. Without another event, it may wait indefinitely.

Kernel waiting interfaces coordinate registration, task state, condition checks, and scheduling. Publication also needs matching locks or memory ordering. READ_ONCE/WRITE_ONCE constrain particular accesses but do not alone establish a complete cross-CPU publication/wait protocol. wake_up() does not deposit a persistent ticket in a notification mailbox: the woken task rechecks the condition, and multiple consumers may compete for one item.

Coordinate checking with entering the wait

A user-space condition variable expresses a similar constraint. The consumer checks under a mutex; pthread_cond_wait() atomically releases it and enters the wait, then reacquires it before returning. A loop checks the condition again. The producer updates the condition under the same mutex. Even a notification that precedes waiting need not cause a hang: a later protected check can observe the persistent condition.

Try check → publish → wait in the next figure, then switch to the protected protocol without resetting. The operation order remains the same; the mutex changes which action can complete in the vulnerable window.

A 1000-round handoff using the condition

Save as handoff.c and compile with cc -std=c11 -O2 -Wall -Wextra -Werror -pthread handoff.c -o handoff. The two threads alter only their own synchronization objects, not system scheduling settings.

#define _POSIX_C_SOURCE 200809L
#include <assert.h>
#include <pthread.h>
#include <stdio.h>
#include <time.h>

static pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t cv = PTHREAD_COND_INITIALIZER;
static int turn; /* 0: main may publish; 1: worker may consume */
static int consumed;
enum { ROUNDS = 1000 };

static void *worker(void *unused) {
    (void)unused;
    for (int i = 0; i < ROUNDS; ++i) {
        assert(pthread_mutex_lock(&mu) == 0);
        while (turn != 1)
            assert(pthread_cond_wait(&cv, &mu) == 0);
        ++consumed;
        turn = 0;
        assert(pthread_cond_signal(&cv) == 0);
        assert(pthread_mutex_unlock(&mu) == 0);
    }
    return NULL;
}

int main(void) {
    pthread_t thread;
    struct timespec start, end;
    assert(clock_gettime(CLOCK_MONOTONIC, &start) == 0);
    assert(pthread_create(&thread, NULL, worker, NULL) == 0);
    assert(pthread_mutex_lock(&mu) == 0);
    for (int i = 0; i < ROUNDS; ++i) {
        while (turn != 0)
            assert(pthread_cond_wait(&cv, &mu) == 0);
        turn = 1;
        assert(pthread_cond_signal(&cv) == 0);
    }
    while (turn != 0)
        assert(pthread_cond_wait(&cv, &mu) == 0);
    assert(consumed == ROUNDS);
    assert(pthread_mutex_unlock(&mu) == 0);
    assert(pthread_join(thread, NULL) == 0);
    assert(clock_gettime(CLOCK_MONOTONIC, &end) == 0);
    double seconds = (end.tv_sec - start.tv_sec)
                   + (end.tv_nsec - start.tv_nsec) / 1e9;
    printf("%d round trips; %.0f ns/round including startup and join\n",
           ROUNDS, seconds * 1e9 / ROUNDS);
    assert(pthread_cond_destroy(&cv) == 0);
    assert(pthread_mutex_destroy(&mu) == 0);
}

Do not define NDEBUG for this assertion-driven example. while cannot be replaced by if: returning from a wait does not establish that the condition holds. With one peer waiter, signal suffices; multiple consumers require a fresh design for conditions, wakeup counts, and shutdown.

The output measures round-trip latency for this synchronization workload, including mutexes, condition variables, possible futex calls, scheduling waits, startup, and join. It is not pure context-switch time. Do not assume exactly two switches per round without event counts; the threads may also run on different CPUs.

6. From counts to causes

Voluntary/involuntary counts in /proc/PID/status or getrusage() can show trends. Voluntary includes blocking and does not mean the application explicitly called yield; involuntary does not imply a system fault. Compare increments with completed requests, thread counts, and CPU use over the same interval.

For wakeup latency, associate a target TID's sched_wakeup with its later selection as next in sched_switch. Handle repeated wakeups, exit and TID reuse, cross-CPU timestamps, and missing capture boundaries. Arbitrarily subtracting two same-PID records is insufficient. The result includes competition and scheduling paths, not isolated switch_to cost.

These are investigation paths, not conclusions from a single counter. Change thread counts, batching, or lock granularity to address measured waiting or execution costs.

Read __schedule() / context_switch() in kernel/sched/core.c, then architecture switch_to, mm/TLB, and FPU paths. For why this next task was selected, revisit the scheduler. For signal-interrupted waits and EINTR, continue to signals and ptrace.