Synchronization Debugging: Dependencies, Races, and Stalls

The same symptom of hanging can be caused by lock-order cycles, missed notifications, or simply the lock holder not getting CPU time. First, determine what progress is missing, then select the appropriate evidence.

Synchronization issues include at least deadlocks, data races, livelocks, long lock holding, and lost wakeups. While they may all manifest as "hanging," the required evidence differs. First, determine who has no progress and what they are waiting for, then choose the tool; do not attribute all alerts to "lock not released."

1. What Each Tool Answers

2. Why Lockdep Reports Before a Deadlock Occurs

When a task holds lock class A and acquires B, lockdep records A→B. If another path exhibits B→A, a potential cycle is formed even if they never actually deadlocked simultaneously; longer dependency chains also require analysis.

Lockdep also tracks usage states such as hardirq/softirq. For example, if a lock is used in an interrupt handler, and another path holds it while allowing that interrupt to preempt, it may cause a self-deadlock on the same CPU. Such context conflicts cannot be discovered solely by checking ABBA orders on two different CPUs.

Add each observed acquisition order to the graph one by one. Edges can come from different times; a closed loop already constitutes a sequence contradiction that needs investigation, without waiting for two threads to be stuck on-site simultaneously.

Lock Classes and Lock Instances

Lock classes are usually determined by initialization location/explicit class keys, rather than "all spinlocks are one class" or "different acquisition functions must be different classes." Locks in multiple objects of the same class can be grouped into one class; a lock in one object acquired in different functions does not automatically become different classes.

If the algorithm indeed acquires locks of the same class in a parent-child hierarchy simultaneously, you can express the proven order using appropriate subclass/nested APIs. However, adding subclasses only provides more information to the validator; it does not automatically make the actual order correct. Incorrect annotations may hide real problems.

The report provides the current acquisition location, existing dependencies, and related stacks to reconstruct the contradictory path; it is not a complete history recording of every lock hold since boot. See lockdep design.

3. How to Read a Lockdep Report

Reconstruct the dependencies according to the report, rather than just changing the last triggered location:

1. Identify the lock currently being acquired and the locks already held.
2. Follow existing dependencies to restore another or multiple opposite paths.
3. Confirm object instances, concurrency possibilities, and interrupt/preemption states.

These conditions collectively determine how the dependency cycle forms and also determine whether to change the lock order or object relationships.

Do not just change the last lock call that triggered the alert. Common fixes include unifying lock order, narrowing critical sections, phased operations, avoiding holding locks while waiting for callbacks, or redesigning object relationships. If adjusting hierarchy annotations, you must provide the basis for why objects held simultaneously must satisfy that hierarchy.

lockdep_assert_held() can turn the prerequisite "caller must hold a certain lock" for internal functions into a checkable contract. Long-term testing with no reports still requires confirming that the validator is actually enabled, has not stopped checking after an earlier error, and covers relevant errors and exit paths.

4. Lockstat: Look at Contention Count, Wait Time, and Hold Time Together

/proc/lock_stat reports lock-class statistics, including fields for acquisitions, contention, wait time, and hold time, after the corresponding configuration/execution control is enabled. Time units should be read from the version-specific header/documentation; do not fabricate a fixed-field nanosecond table and blindly sort by the second column.

Counts and durations must be combined; you cannot determine the root cause based on a single maximum number.

If measurements alter scheduling/timing, compare load performance before and after enabling. Sharding to reduce contention may also increase aggregate cost and consistency difficulty; do not solely pursue smaller lockstat numbers. See lockstat documentation.

5. KCSAN: Sampled Detection, Not Full Timestamps for Every Access

KCSAN uses compiler instrumentation and software watchpoints/delays to sample conflicting accesses; it does not build an exact full-history happens-before graph for all memory operations. Configuration can change sampling, delays, and handling of access types, so overhead should not be fixedly stated as ten times.

The address, size, read/write type, and two call stacks in the report are entry points for analysis. Two accesses not sharing a common lock do not necessarily indicate an error, as they may be synchronized via atomic/release protocols; conversely, the presence of a lock in the code does not mean every access is protected by it.

Markers like READ_ONCE/WRITE_ONCE, atomic, and data_race() must reflect the real protocol. Wrapping a report location with data_race() only expresses "this is a judged-allowed race"; it does not fix ordering, atomicity, or lifetime. Exceptions should explain why reading inconsistent values is harmless and why subsequent usage is safe.

For lock-free release problems, you can use litmus tests from the Linux Kernel Memory Model (LKMM) to analyze allowed results, then combine with KCSAN and stress testing. User-space C/C++ atomic model intuition cannot replace the kernel memory model without verification. See KCSAN and LKMM.

6. Atomic Context Sleep and Wait Protocols

Checks like CONFIG_DEBUG_ATOMIC_SLEEP usually report "sleeping function called from invalid context" via locations such as might_sleep. Such diagnostics do not uniformly equal immediate BUG/panic; specific handling is affected by the path and configuration.

Check held locks, preempt count, interrupt status, and call chain to confirm if APIs that might allocate/block are called in raw spinlock, hardirq, or other non-sleepable contexts. A specific allocation hitting a fast path and not blocking does not prove such a call is legal.

Lost wakeups do not necessarily produce lock dependency cycles: a thread might miss the only wakeup after checking the condition but before actually registering for wait. You should use correct protocols such as wait_event/completion, and verify timeouts, cancellation, and object lifetime after exit.

7. Hung Task, Softlockup, and Hardlockup

Hung task focuses on tasks that are in uninterruptible wait for a long time without corresponding progress. D state may be waiting for storage, network file systems, completions, or locks. The timeout threshold is configurable; 120 seconds is not an immutable constant across systems.

Softlockup indicates that the CPU has not allowed watchdog-related tasks to make scheduling progress for a long time, and it may still be able to handle interrupts. Long non-preemptible loops, scheduling starvation, etc., will affect it.

Hardlockup in common NMI/perf implementations is checked by whether normal timer interrupts are advancing, via NMIs that can still enter. It is not judged by "because normal interrupts are disabled, NMIs are also not received." Detector implementations vary by architecture, PMU, configuration, and virtualization support; VM pauses, etc., will also interfere with timing.

All three can appear simultaneously with RCU stall, but they express different missing progress. First, find repeated stacks and wait relationships, then judge the common root cause. See lockup watchdog.

8. Read-Only Check of Detection Capabilities and Thresholds

from pathlib import Path

names = ('hung_task_timeout_secs', 'hung_task_panic', 'watchdog',
         'watchdog_thresh', 'nmi_watchdog', 'softlockup_panic')
for name in names:
    path = Path('/proc/sys/kernel') / name
    try:
        print(name, path.read_text().strip())
    except OSError as exc:
        print(name, type(exc).__name__)
for path in ('/proc/lockdep_stats', '/proc/lock_stat'):
    try:
        with open(path) as source:
            head = ''.join(next(source, '') for _ in range(5))
        print(path, head.strip())
    except OSError as exc:
        print(path, type(exc).__name__)

State files may not exist because configuration is not enabled; permission denial does not equal non-existence of functionality. /proc/<pid>/stack requires corresponding configuration and access permissions; a single stack snapshot for one task is just that—a snapshot; associate with other tasks and repeated samples when necessary.

9. How to Verify After Fixing

Use the same configuration and context as the problem to first cover known trigger paths, then cover concurrent updates, error returns, timeouts, and exit cleanup. For lock-order issues, confirm dependencies no longer conflict; for race issues, confirm invariants and release/lifetime protocols hold; for performance issues, remeasure wait distributions and business latency.

Do not replace explanation by disabling detectors, raising timeouts, or adding annotations. Tool reports are evidence; final conclusions should land on specific invariants, lock orders, and object lifecycles.