Synchronization: choosing and using a cross-core primitive

Status: shipped. tk_spinlock (kernel/tk_spinlock.h) and tk_seqlock (kernel/tk_seqlock.h) are in the tree and HW-verified on AURIX TC275 (examples/sync_tricore); the blocking primitives (tk_mutex/tk_sem/tk_cond) predate them and are documented in isolation/kernel-owned-sync.md.

This document is the map: which primitive protects shared state from concurrent access, and why — because “cross-core” hides two regimes with different correct answers.

1. The fork that decides everything: two regimes

“Cross-core” means one of two very different things in this kernel, and the mechanism differs:

  • SMP — several cores under one kernel instance: one scheduler, one lock, one ready-queue. A thread can block and be woken on any core, because a single scheduler owns them all.

  • AMP — separate kernel instances (“islands”), made share-nothing at the hardware level by the C3-B supervisor fence (../decisions/0003-supervisor-fence-seam.md). There is no shared scheduler, so you fundamentally cannot block a thread on island A and wake it from island B. Islands coordinate only through memory they deliberately share.

This is not a quirk of our design — it is the general rule: SMP ⇒ shared scheduler ⇒ blocking locks work; AMP ⇒ share-nothing ⇒ spin or pass messages. Get this right first; the rest follows.

The regime is set by TK_CONFIG_MAX_KERNELS + how cores are assigned (see smp-model.md and ../decisions/0002-multi-instance-kernel.md). A single build can even contain both — an SMP island beside a disjoint AMP island.

2. The mechanism menu

Mechanism

Blocks or spins

ISR-safe

Within SMP

Across AMP islands

Use for

Where

IRQ-mask critical section (arch_irq_save)

neither

yes

no (one core only)

no

thread ↔ ISR on the same core

inline in kernel/board

tk_spinlock

spins

yes

yes

yes (lock in shared RAM)

short critical sections, any context

kernel/tk_spinlock.h

tk_mutex / tk_sem / tk_cond

blocks

no

yes

no (needs the scheduler)

long critical sections, thread context

kernel/tk_mutex.h …

tk_seqlock

writer never waits; reader retries

readers yes

yes

yes

read-mostly data, one writer

kernel/tk_seqlock.h

atomics (__atomic / __sync)

neither

yes

yes

yes (atomic-capable mem)

single-word counters/flags

compiler builtins

SPSC/MPSC ring (message passing)

neither

yes

yes

yes

producer → consumer, hand off ownership

sketches/amp/mailbox.h

Picking, in one breath: protecting a short mutation of shared state → tk_spinlock. Protecting a long thread-context section within one SMP instance → tk_mutex (it sleeps instead of burning a core). Read-mostly state with a single writer → tk_seqlock (readers never block anyone). A single word → a raw atomic, no lock at all. And often the best answer across an AMP boundary is not a lock at all but message passing (the mailbox): don’t share mutable state, hand ownership across a queue.

3. tk_spinlock — universal mutual exclusion

The lowest-level “one context at a time” primitive: a contender busy-waits on one atomic word. Because it involves no scheduler, it is the only mutual-exclusion tool that works across an AMP boundary and from an ISR. It is built on the arch atomic (arch_spin_* — tricore swap.w, risc_v AMO, RP2040 SIO, Cortex-M exclusive/MAILBOX; a no-op single-core) wrapped in the IRQ-save key discipline.

tk_spinlock_t lock = TK_SPINLOCK_INIT;               /* zero == unlocked; place in shared RAM for AMP */

tk_spin_key_t k = tk_spin_lock(&lock);               /* mask local IRQs, then spin-acquire */
    ... short critical section ...
tk_spin_unlock(&lock, k);                            /* release, then restore IRQs from the key */

if (tk_spin_trylock(&lock, &k)) {                    /* one attempt; 1 = got it (key set), 0 = busy */
    ... ; tk_spin_unlock(&lock, k);
}

Three rules, each load-bearing:

  1. IRQ-save (why the key). While a core holds the lock it masks its own interrupts, so it cannot be preempted mid-section on its own core — otherwise a higher-priority thread or ISR that also wants the lock would spin forever behind the preempted holder (self-deadlock / unbounded priority inversion). The returned key carries the prior interrupt state, so nesting different locks is safe (each level saves and restores its own). This is the Linux spin_lock_irqsave / Zephyr k_spinlock shape.

  2. Short critical sections only. Since the holder runs interrupts-off and peers burn cycles spinning, the body must be a few loads/stores — never a loop, a tk_print, or anything that can block. Long thread-context sections belong to tk_mutex.

  3. Placement for AMP. A cross-island lock cannot be a kernel-owned handle (whose island’s pool would own it?), so tk_spinlock_t is a plain value the app places itself — in memory both islands can reach (the fence leaves shared RAM open) and that supports a cross-core atomic. On AURIX that is the non-cached LMU (0xB0000000), not a core’s private DSPR (a remote-DSPR access is not a reliable atomic). On Cortex-M SMP it is ordinary shared SRAM. See §5.

The value-type (not kernel-owned-handle) choice is a deliberate departure from isolation/kernel-owned-sync.md: a mutex is kernel-owned so a confined thread cannot forge one, but a spinlock is just a word — the worst an app can do is deadlock itself, never make the kernel touch arbitrary memory — and a shared cross-island word has no kernel to own it. On a single-core build arch_spin_* is a no-op, so tk_spin_lock correctly collapses to a plain interrupt-masking critical section (thread ↔ ISR) at zero cross-core cost.

4. tk_seqlock — read-mostly consistency

When state is read far more often than written (status/telemetry, a shared clock or config), a lock that makes every reader serialize is wasteful. A seqlock lets the writer never wait and readers never block: a reader that races a write simply retries its snapshot. There is no contended lock word at all.

A sequence counter is even when the data is consistent and odd while a write is in flight:

tk_seqlock_t sl = TK_SEQLOCK_INIT;

/* writer — one at a time (single writer, or serialize writers with a tk_spinlock): */
tk_seq_write_begin(&sl);  a = ...; b = ...; sum = a + b;  tk_seq_write_end(&sl);

/* reader — lock-free; the body may run 2+ times, so keep it side-effect-free: */
uint32_t seq;
do {
    seq = tk_seq_read_begin(&sl);        /* spin out any in-progress (odd) write */
    ra = a; rb = b; rsum = sum;          /* copy a consistent snapshot */
} while (tk_seq_read_retry(&sl, seq));   /* retry if a write landed during the copy */

Contract: exactly one writer at a time; readers idempotent inside the retry loop; the protected fields are plain data the reader copies out (don’t chase a pointer a write may repoint mid-read). The begin/end pair carries acquire/release fences (__atomic_thread_fence) that order the seq bumps against the field accesses. Crucially, a seqlock needs no cross-core atomic — only ordered word loads/stores + barriers — so it works even on Cortex-M0 / ARMv6-M, where the spinlock needs the SIO hardware lock.

5. Placement across an AMP boundary

The C3-B fence denies each island the other’s private RAM arena but leaves general and explicitly-shared RAM open. A cross-island tk_spinlock / tk_seqlock (and its protected data) must live in memory that is (a) reachable by both islands and (b) coherent + atomic-capable across cores:

Target

Cross-island shared memory

Note

AURIX TC275

non-cached LMU, 0xB0000000 (lmuram_nc, 32 KiB)

a core’s private DSPR is not a reliable cross-core atomic

Cortex-M SMP (RP2040, MCXN947)

any shared SRAM

the arch lock uses the SIO HW spinlock / MAILBOX mutex

RISC-V SMP

any shared RAM

AMO on the shared word

examples/sync_tricore places its lock, seqlock, and shared struct at 0xB0000000 and has both islands pound on them: the spinlock-guarded counter comes out exactly 2·ITERS (no lost updates) and the seqlock reader sees zero torn snapshots over a million reads while catching hundreds of real read/write races — the two-line HW proof that both primitives hold under genuine cross-core contention.