Locking Mechanisms and Memory Ordering

A correct counter update does not imply that the related object is still alive; a snapshot retry does not mean that a bad access can be undone. Synchronization must address mutual exclusion, ordering, and lifetime simultaneously.

The goal of synchronization is not merely to prevent two CPUs from writing to the same address simultaneously. It also includes maintaining multi-field invariants, restricting object lifetimes, ensuring data publication order, and avoiding interrupt reentrancy on the same CPU. Before choosing a lock, list all readers/writers and their contexts.

For example, a queue consists of head, tail, and count. Changing count to atomic does not protect node links; a reader might see a new node before count is updated, or continue accessing a node after it has been freed. First, clearly define "when the queue is visible to readers, who can delete, and who is responsible for freeing."

1. Choose Based on Constraints, Not Arbitrary Weightings

"Read-heavy, write-light" is merely a statistical characteristic and does not automatically dictate the use of RCU or seqlock. Whether readers must see the latest value, whether they can sleep, whether retries are allowed, and whether objects must be held across critical sections all influence the choice.

2. What Spinlocks and Local Masking Do Separately

In non-PREEMPT_RT kernels, spin_lock() itself disables preemption to maintain the semantics of the corresponding critical section; it does not require the caller to disable preemption beforehand. However, it does not automatically mask local hardirqs.

Assume a task holds lock L, and subsequently, the interrupt handler on the same CPU also acquires L: the handler spins waiting for the task to release it, but the task cannot run before the handler returns, forming a self-deadlock loop. Use protocols like irqsave on the appropriate access side to avoid such reentrancy.

• spin_lock_irqsave() saves the local interrupt state, masks interrupts, and acquires the lock; irqrestore restores the original state.
• _irq() / unlock_irq do not save the original state; the caller must satisfy their entry state conventions.
• _bh() handles local bottom-half/softirq concurrency; this does not imply that hardirq or NMI are also masked.

Disabling local interrupts does not prevent other CPUs from accessing shared data; true cross-CPU mutual exclusion comes from locks. NMI is not controlled by standard interrupt masking; data used in NMI must be designed separately.

PREEMPT_RT Changes the Implementation and Semantics of spinlock_t

In RT kernels, the standard spinlock_t is based on a blocking real-time locking mechanism; do not assume preemption is disabled after acquisition. Its hardirq suffix is also not equivalent to local hard interrupt masking on non-RT systems. raw_spinlock_t retains strict spinning and corresponding atomic semantics, used in low-level scenarios where it is truly needed.

This is not an encouragement to replace all locks with raw spinlocks: excessively long non-preemptible sections harm real-time performance, and you cannot call the standard RT spinlock acquisition path (which may block) while holding a raw lock. See the Lock Types Documentation for type and nesting rules.

3. Contention Paths in qspinlock and Mutex

qspinlock combines compact lock state with queued waiting, including uncontended, pending, and queue paths; contenders can use MCS-style nodes to reduce the overhead of all CPUs constantly contending for the same lock word. It is not the case that "all contenders only ever touch their own cache line"; unlocking, head/tail handoffs still require shared state.

The implementation's bit layout is also configuration-dependent; read include/asm-generic/qspinlock_types.h. Do not arbitrarily draw locked/pending/tail as fixed ABIs by byte. Architectures and virtualization may also use different implementations.

The uncontended path of a mutex quickly acquires ownership; under contention, it can optimistically spin or sleep waiting, depending on conditions. Waking a waiter does not guarantee it runs on the CPU immediately; fairness, handoff, and owner state handling cannot be abbreviated to a single owner=0.

Holding a mutex allows sleeping in allowed contexts, but consider whether the sleep path requires the same lock; interfaces like mutex_lock_interruptible() can return without acquiring the lock, so the caller must check. A standard mutex does not equal an rt_mutex providing priority inheritance. See Mutex Design.

4. rwsem and percpu_rwsem Are Different Interfaces

rw_semaphore allows multiple readers or one writer; percpu_rw_semaphore is a specialized interface that leverages per-CPU read-side state and corresponding synchronization mechanisms to optimize certain read-heavy, write-light scenarios. It is not the case that a standard rwsem automatically becomes per-CPU in some version.

Read locks can still wait for write locks; a reader sleeping for a long time will block writers. The relationship between multiple readers and priority inheritance under RT requires special attention; do not promise strict wait-time upper bounds in arbitrary configurations.

If you attempt to upgrade directly from a read lock to a write lock, you may wait mutually with other upgraders. Release and re-check conditions according to interface support, acquire the write lock from the start, or design a specific conversion protocol.

5. Seqlock: Getting a Consistent Snapshot at the Cost of Potential Retries

When a writer enters, it advances the sequence to the writing state; when it exits, it advances it again; the reader copies data and then checks if the sequence has changed. write_seqlock() / unlock manage the sequence; callers must not manually seq++ between them.

Below is an example of a kernel C translation unit that can be compiled independently, without module registration. It demonstrates the invariant high = low + 1 for two scalars; the read result is only used after a successful check.

#include <linux/compiler.h>
#include <linux/seqlock.h>
#include <linux/types.h>

static DEFINE_SEQLOCK(range_lock);
static u32 range_low;
static u32 range_high = 1;

void kb_set_range(u32 low);
void kb_get_range(u32 *low, u32 *high);

void kb_set_range(u32 low)
{
    write_seqlock(&range_lock);
    WRITE_ONCE(range_low, low);
    WRITE_ONCE(range_high, low + 1); /* u32 wraparound is allowed in this example */
    write_sequnlock(&range_lock);
}

void kb_get_range(u32 *low, u32 *high)
{
    unsigned int seq;
    u32 a, b;
    do {
        seq = read_seqbegin(&range_lock);
        a = READ_ONCE(range_low);
        b = READ_ONCE(range_high);
    } while (read_seqretry(&range_lock, seq));
    *low = a;
    *high = b;
}

During the retry interval, do not perform irreversible external side effects based on temporary values.

This example assumes the access context matches standard seqlock; if interrupt readers can preempt writers, the writer also needs corresponding local masking, otherwise readers might wait for a writer that was preempted by themselves. Pure seqcount itself does not serialize multiple writers; there must be a separate write-side lock, satisfying rules such as non-preemptibility.

Read low=0, allow a writer to replace the pair with 10/11, then read high=11. Both loads can be valid individually while their combination is invalid. A retry must discard that combination, including any derived result. Do not perform irreversible external side effects before validation.

Snapshot Consistency and Pointer Lifetime Are Guaranteed Separately

Retries cannot fix illegal accesses that have already occurred. If a reader accesses a freed object via a pointer in the snapshot, and then discovers the sequence has changed, it is still a UAF (Use-After-Free). Pointer lifetime must be protected by RCU, reference counting, or other mechanisms. The read side must not perform non-repeatable external side effects in regions where retries are possible. See seqcount/seqlock.

6. Atomic Operations, Refcounts, and Ordering

atomic_read/set, non-returning RMW, returning RMW, and acquire/release/relaxed variants have different ordering guarantees; do not call all atomic operations full barriers. Failed conditional operations also require separate checks of return values and ordering semantics.

Object references should prefer appropriate interfaces like refcount_t, leveraging overflow/underflow protection and specific lifetime semantics; general counters do not automatically prevent "resurrection from zero" or safe acquisition of objects that may already be freed.

CAS (Compare-And-Swap) can atomically compare and replace a state, but it does not automatically eliminate the ABA problem: if a value goes through A→B→A, the comparison can still succeed. Lock-free structures often also require versioning, reclamation protocols, and strict ordering design. See atomic types.

7. Publishing Data Requires Matching Observation Relationships

An example of publishing once and not modifying the payload afterwards: the writer fills the payload first, then calls smp_store_release(&ready, 1); the reader reads the payload only after observing the published value via smp_load_acquire(&ready).

The effect of release/acquire comes from matching publication and observation, not from any two CPUs simply executing a barrier once to automatically synchronize. If ready is reused or the payload is overwritten in a loop, additional protocols are needed to prevent competition between old readers and new writers.

These operations are not a unified "cache flush," and single-access constraints do not independently establish complete publication relationships. See the Memory Barriers Documentation.

READ_ONCE/WRITE_ONCE constrain particular accesses and compiler transformations; a compiler barrier is not a CPU barrier. smp barriers, DMA ordering, and MMIO accessors address different domains. None should casually be described as “flush all caches.”

8. Waiting Must Cooperate with Conditions and Lifetimes

Wait queues use interfaces like wait_event to register waits and check conditions under the correct protocol; the waker updates the condition first, then notifies. Simply writing if (!ready) schedule() will lose notifications between the check and the sleep.

completion is suitable for one-time or controlled repeated completion events; a timeout return does not cancel asynchronous operations; the wait object and callback context must remain valid. When destroying work, first prevent new submissions, then synchronously cancel/wait, and finally free resources. Do not hold locks required by the completion path during waiting.

See Synchronization Debugging for how to verify these constraints.