ADR 0002 — Multi-instance kernel: un-globaling the singleton, AMP islands in scope (“one model, N instances”)

Status: Accepted (2026-08-19). Supersedes (narrowly): the “separate-kernel AMP out of scope” stance in smp-design.md §8 and the smp-model.md (Q3) “SMP + AMP” resolution — the scope limit only; every scheduler mechanic those docs settled is preserved.

Context. Q3 (smp-model.md, 2026-08-16) settled the SMP model as a giant-lock singleton — one struct tk_kernel, one g_sched_lock, per-core current/idle/quantum/ entering, affinity subsuming pinning, select-in-trap with the lock held only inside *_select. It deliberately put separate-kernel AMP out of scope for lack of proof, and modeled cross-core comms as “SMP + an optional mailbox,” explicitly “not a second scheduling model … without reopening the separate-kernel design.”

That deferral was evidence-gated, and the evidence now exists. Between 2026-08-18 and -08-19 separate-kernel AMP was HW-proven twice, brick-by-brick, on the user’s bench:

  • rp2040 1 + 1 (sketches/amp/rp2040/): two amp_kernel instances, one per M0+ core, disjoint owned_cores, sharing nothing but a mailbox.

  • AURIX TC275 “2 SMP + 1 aside” (sketches/amp/tc275/): one instance owning two cores (kSMP {CPU0,CPU1}, a real SMP domain) beside a disjoint one-core AMP island (kAside {CPU2}) — the exact shape this ADR generalizes.

plus the temp/multicore.c whiteboard (rounds 1–6). Both proofs kept the Tier-B port entry-point names identical to the real kernel’s (tk_kernel_switch_select, tk_kernel_preempt_select, tk_kernel_current_context, tk_kernel_start_this_core), so the un-globaling is confined to tk_kernel.c + its internal header — no arch/board edits.

The reshape’s layering invariant frames the whole change: the kernel depends only on arch (kernel_arch.h); board + HAL = SDK; the kernel is a guest. So an API reshape may touch the kernel↔arch contract and ripple into the example apps (acceptable, mechanical) but must not require SDK edits.

Decision

Generalize the kernel singleton into per-instance kernels resolved per physical core, bringing separate-kernel AMP into scope. One scheduling model, instantiated N times.

1. Un-global: what moves, what stays global-by-physical-core

today (singleton)

becomes

why

active_kernel (1 ptr)

g_core_kernel[ARCH_CORE_COUNT] + this_kernel()

the linchpin — a core reads its kernel by arch_core_id()

g_sched_lock (1)

k->lock (into struct tk_kernel)

each instance schedules independently

—

new k->owned_cores (bitmask), k->name

the one knob: 2-bit set = SMP, disjoint 1-bit = AMP island

g_sched_entering[], g_sched_irq[], g_quantum_remaining[]

stay global, indexed by arch_core_id()

physical-HW reentrancy facts — a core has one each regardless of which hat it wears

g_ticks_ms, g_time_base

now per-instance — k->ticks_ms, k->time_base in struct tk_kernel (review C4 superseded the original “stay global”)

§6

struct tk_kernel.current_thread[N], idle_index[N]

unchanged width [ARCH_CORE_COUNT]; an instance uses only the slots for cores it owns

preserves collapse (N=1 uses all)

2. The same scheduler model, N times — not a new model

Every Q3 mechanic is preserved per instance: giant lock, first-class affinity (subsumes pinning), per-core current/idle/quantum/entering, select-in-trap (lock only inside *_select, never across a switch), the g_sched_entering re-entry guard, phase-offset first slice, TTAS lock, ARCH_HAS_LAZY_CONTEXT ⇒ mandatory affinity. We do not introduce a second scheduling model — we replicate the proven one. This honors Q3’s actual concern; only the singleton / AMP-scope limit is lifted.

3. The collapse property (non-negotiable safety contract)

N = 1 instance owning all ARCH_CORE_COUNT cores == the Q3 singleton, byte-for-byte. tk_kernel_init() with no topology declared auto-registers one all-cores instance bound to every core, so this_kernel() returns it everywhere and every existing singleton call resolves unchanged. The multi-instance machinery is present but dormant until an app opts into a second instance. Consequence: existing arch ports and the SDK are untouched; only example apps may ripple. This is what lets the graft land without a regression sweep of the ports.

4. API surface: [rt] stays pointer-free, [setup] targets an instance

  • [rt] (yield/sleep/exit/self/join) keep the “no kernel pointer” promise — they resolve this_kernel() from the calling core.

  • [setup] calls that target a specific, not-yet-running instance gain a handle. As built, tk_kernel_define is an ALLOCATOR that returns an opaque handle from a kernel-owned pool — tk_kernel_t* tk_kernel_define(uint32_t owned_cores, const char* name). (The originally-sketched struct tk_kernel* k first arg was un-allocatable: struct tk_kernel is Tier-D opaque, so the app cannot supply storage.) Plus an instance-targeted thread create, tk_thread_create_in. The bare forms target the default instance → every existing app is source-compatible.

  • tk_kernel_start() stays arg-less. As built, g_core_kernel[] is bound in tk_kernel_init (every slot → the default all-cores instance) and rebound incrementally by each tk_kernel_define; tk_kernel_start sets nothing here — it arms the timer, releases secondaries, and starts this_kernel() on the boot core. The invariant “g_core_kernel[] fully populated before arch_smp_start()” still holds because init (and any define) precede start.

  • Fork 1 (settled): runtime registration is the primary mechanism (HW-proven twice); single-instance is the zero-config default; a compile-time/link-time topology table is later sugar over the runtime API, not the primary path.

  • Fork 2 (settled, option a): a thread handle resolves to its instance by validating against this_kernel() — zero-cost, matches the islands-share-nothing invariant. A TCB back-pointer is a later add iff a real cross-island join case ever appears.

5. Bring-up keeps the SoC glue untouched

g_core_kernel[] is fully populated before arch_smp_start() — as built, bound in tk_kernel_init (default all-cores) and rebound by tk_kernel_define, both of which run on the boot core before tk_kernel_start releases the secondaries. Each secondary then calls the unchanged tk_kernel_start_this_core(), this_kernel() already resolves, and no secondary entry (no SoC glue) needs a bind edit. The layering invariant actively shapes the graft.

6. Timebase: per-instance (revised by review C4)

Superseded. This ADR originally read: “g_ticks_ms stays a single global wall clock; only the timebase-owning core advances it. Per-instance clocks are out of scope.” Review C4 reversed that: a single global wall clock advanced by one core is a cross-island SPOF — a fault/halt on that core freezes every other island’s sleeps and timeouts. Per-instance clocks are now the model.

Each instance owns its own k->ticks_ms + k->time_base in struct tk_kernel, advanced only by its own k->ticker_core (the lowest core it owns). The four arch tick handlers now call tk_tick_advance() on every core’s tick; the kernel self-selects which core advances, per instance — so a halt on one island’s ticker core cannot stop another island’s clock. Het-SoC caveat: a SoC with a single shared hardware timer makes that timer a HW SPOF regardless — a board-level fact to document, not kernel-fixable; normal multicore has per-core timers (ARM SysTick, RISC-V per-hart CLINT). Collapse (N=1, one instance owning all cores): ticker_core == 0, only core 0 advances — byte-identical timing to the old global.

7. SoC / multicore glue belongs in a per-arch multicore tier, not boards/

arch_core_id / arch_spin_* / arch_ipi / arch_smp_start + the secondary entry are arch-contract code — but on some SoCs they are implemented with SoC peripherals (RP2040 SIO, AURIX GPSR) that vanilla CPU-arch lacks. On ARMv6-M this is stark: no LDREX/STREX, and WFE/SEV inter-core event semantics are IMPLEMENTATION-DEFINED, so the SIO hardware spinlock is the only atomic. Today boards/rp2040_pico/tk_smp_rp2040.c implements this contract and #includes tk_kernel.h — a board file reaching into the kernel (the invariant’s one leak, self-documented in the file).

Decision: such code moves to a per-arch multicore tier — arch/<cpu>/<mp>/<soc>/, e.g. arch/arm_v6m/<mp>/rp2040/. The umbrella word is config-neutral (mp / multicore, not amp — the same glue serves SMP too); the exact word is finalized at move-time. This is a decoupled follow-up refactor, not a graft prerequisite (the graft is pure tk_kernel.c). Relates to _legacy/docs/notes/smp-bringup-centralization.md.

8. Naming

  • Rename rule: rename a legacy name only when it is wrong about its layer or role (tk_board_log → tk_print: the console is an application choice, not board logging); keep a name that is already consistent with its tier’s convention and honest. Renames are driven by correctness, not modernization — each is an API break that churns the examples.

  • The one ergonomic exception: add tk_sleep_ms as a static inline forwarder over the canonical tk_thread_sleep_ms (no ABI symbol, no macro blast radius, -Werror clean, flattens to zero cost, UNPRIV syscall path untouched). Examples adopt the short form so it is what people copy; the canonical stays for the browsable tk_thread_* family. Guard rail: this is a deliberate singular exception — we do not alias tk_thread_yield/_exit/_join, so the exception cannot metastasize.

Consequences

  • One bitmask spans the whole space: SMP-all, SMP-subset (2-of-4), bare-core-0 + kernel-elsewhere, and disjoint AMP islands — chosen by owned_cores, no new model.

  • Staged, always-green sequencing:

    1. Un-global tk_kernel.c — collapse-verified on the existing single-instance boards (host / nRF52 / RV32-qemu / rp2040-SMP) before any second instance exists.

    2. Multi-instance enablement — registry, tk_kernel_define, per-instance init + per-instance idle-thread reservation (each instance reserves idles only for the cores it owns).

    3. SoC/multicore-tier refactor (§7) — independent.

    4. tk_sleep_ms alias (§8) — independent.

  • The AURIX↔rp2040 delta remains exactly ARCH_HAS_LAZY_CONTEXT (Q3’s payoff): the un-globaling adds no per-arch variance.

  • Supersedes narrowly: smp-design.md §8 and smp-model.md’s “SMP + AMP” section are superseded on the scope question only; their scheduler mechanics stand and are carried forward here.

Open / deferred

  • ~~The <mp>-tier umbrella word and the physical move of the rp2040 glue (§7).~~ DONE — umbrella word = mp; glue moved to arch/arm_v6m/mp/rp2040/ (the board fragment still pulls it into TK_BOARD_SRCS, so only that SoC compiles it).

  • Compile-time / link-time topology declaration as sugar over the runtime registry (§4).

  • Cross-island thread join / handle resolution beyond this_kernel() (§4, fork 2).

  • Per-instance timebase for a hosted third-party RTOS (§6).

  • Post-graft review (2026-08-20): docs/reviews/amp-smp-review-0001.md — a 5-lens as-built audit. Bucket A (lifecycle-phase guard, carve-idle release, owned_cores IPI filter, handle validation, this reconciliation, core-count assert) applied in the review cycle. Buckets B (error-reporting convention, _in API variants) and C (per-instance memory isolation, timebase SPOF, fault containment, privileged-tagging the linchpin globals) are the memory/safety-AMP session agenda.

Follow-through (north star — code lands in the staged order above)

  • Touched by the un-globaling: kernel/tk_kernel.c, kernel/tk_kernel_internal.h (struct tk_kernel gains owned_cores/name/lock; active_kernel → g_core_kernel[]). kernel/tk_kernel.h gains the tk_sleep_ms inline. No arch/ or boards/ edits for the graft itself.

  • Examples may need mechanical updates when the multi-instance surface + tk_sleep_ms land (the accepted, bounded trade-off).

  • This ADR is referenced by the pending corpus migration (it marks the superseded sections of smp-design.md / smp-model.md when those move into root docs/).