Synchronization Debugging

Coverage: lockdep (lock dependency graph) → lockstat (contention statistics) → KCSAN (data race detection) → hung task detector → NMI watchdog → CONFIG_DEBUG_ATOMIC_SLEEP Kernel versions: 2.6 ~ 6.x

Overview

Concurrency bugs are among the most difficult issues to debug in the kernel: deadlocks, data races, and sleeping in atomic context. These bugs are often non-deterministic—triggering only under specific timing conditions. Fortunately, the kernel includes a powerful synchronization debugging infrastructure that can catch them when they occur (or even before they happen).

Lockdep: Lock Dependency Validator

// kernel/locking/lockdep.c
// Core idea: Build a runtime lock dependency graph and check for cycles (deadlocks)
//   On each lock: add an edge A→B in the graph
//   If an edge B→A already exists → emit a deadlock warning!
//
//   A deadlock doesn't need to actually occur! Lockdep can "predict" potential deadlocks

// "Lock classes" tracked by Lockdep:
//   - Locks of the same type (e.g., all inode→i_lock) → same class
//   - Locks taken in different functions → different classes (even if the type is the same)

// /proc/lockdep: complete lock dependency graph
// /proc/lockdep_stats: dependency graph statistics

Interpreting Lockdep Reports

Typical lockdep output:
  ======================================================
  WARNING: possible circular locking dependency detected
  ======================================================
  CPU 0: lock(A); ... lock(B);
  CPU 1: lock(B); ... lock(A);
  → Potential deadlock! Even if it hasn't occurred in actual execution yet

// Also includes:
//   - Which function and file acquired each lock
//   - Hold history for each lock since boot

Lockstat: Contention Statistics

// CONFIG_LOCK_STAT
// Records per-lock:
//   - wait time: how long it waited for the lock
//   - hold time: how long the lock was held (requires CONFIG_LOCK_STAT_HOLDLOCK)

// /proc/lock_stat:
//   Format: class_name    contended   total_wait_ns   max_wait_ns   min_wait_ns

KCSAN: Kernel Concurrency Sanitizer

// kernel/kcsan/
// Similar to user-space ThreadSanitizer (TSan)
// Detects data races at runtime: two CPUs accessing the same memory address simultaneously, with at least one being a write

// How it works:
//   1. For every memory access: record address + size + timestamp (watchpoint)
//   2. If another CPU accesses the same address without synchronization → report it

// Interpreting reports:
//   BUG: KCSAN: data-race in function_name (read/write of 4 bytes at addr)
//     CPU 0: write to x in func_a()
//     CPU 1: read from x in func_b()
//     → Access to x is not protected by a common lock → data race!

// Compile option: CONFIG_KCSAN=y
//   Kernels built with KCSAN are significantly slower (~10x), intended for debugging only

Hung Task Detector

Detection: Processes in TASK_UNINTERRUPTIBLE (D) state for more than 120 seconds
  → Usually caused by deadlocks, unreachable NFS, or driver bugs

Configuration:
  /proc/sys/kernel/hung_task_timeout_secs  (default 120)
  /proc/sys/kernel/hung_task_panic         (0=warn, 1=panic)

Output: Prints the kernel stack of the hung process (critical!)
  → If you see it waiting in mutex_lock() → find which process holds that lock

NMI Watchdog

Deadlock detection based on NMI (Non-Maskable Interrupt):
  Each CPU periodically receives NMIs → resetting per-CPU counters
  If a CPU hasn't received an NMI for a long time:
    → Likely due to disabling interrupts for too long (spin_lock_irqsave infinite loop)
    → Prints the stack backtrace of the hung CPU

How it works:
  perf event driver → overflow → NMI → watchdog overflow handler
  → Checks hrtimer time since last interrupt → if timeout → alarm

Other Debugging Tools

// CONFIG_DEBUG_ATOMIC_SLEEP:
//   Calling schedule() (or any function that might sleep) in an atomic context
//   → Immediately BUG()

// CONFIG_PROVE_LOCKING (enhancement for lockdep):
//   Every lock operation is checked by lockdep → higher runtime overhead
//   But it catches more incorrect lock usage

// CONFIG_DEBUG_SPINLOCK:
//   Checks for uninitialized spinlock usage, double unlock, etc.

// CONFIG_PROVE_RCU:
//   Checks for RCU readers calling rcu_dereference or sleeping outside critical sections

Quick Reference for Debugging Commands

# lockdep: dependency graph + statistics
cat /proc/lockdep        # Complete lock dependency graph (text, large)
cat /proc/lockdep_stats  # Statistics summary

# lockstat: contention
cat /proc/lock_stat | sort -k2 -n -r | head -20

# hung task: currently hung processes
echo t > /proc/sysrq-trigger   # Print kernel stacks for all processes (sysrq)
cat /proc/<pid>/stack           # Single-process kernel stack (5.14+)

# Data races (KCSAN, requires compile-time enablement)
dmesg | grep KCSAN

# Interrupt-disabled checks
dmesg | grep -i "sleeping function called from invalid context"

References

Kernel Documentation: Documentation/locking/lockdep-design.rst, Documentation/dev-tools/kcsan.rst
Source Code: kernel/locking/lockdep.c, kernel/kcsan/, kernel/watchdog.c, kernel/hung_task.c
LWN: "Lockdep", "The Kernel Concurrency Sanitizer"

Keywords: lockdep, lockstat, KCSAN, data race, hung task, NMI watchdog, DEBUG_ATOMIC_SLEEP, PROVE_LOCKING