ADR 0004 — Per-island memory isolation: fencing kernel-owned RAM + a first-class isolation attribute

Status: Accepted (2026-08-24) — implemented (d5d4084 D2 core + this commit’s D1). Extends ADR 0003 (the supervisor-fence seam) and ADR 0002 (multi-instance kernels / AMP islands).

Context

ADR 0003 gave each AMP island a hardware fence that binds even its kernel — a privileged store into a neighbour island’s RAM faults. But the fence only covers the app arena an island declares with tk_kernel_set_ram(base, size). The kernel’s own per-instance state — struct tk_kernel, which holds that island’s tcbs[], scheduler bookkeeping, per-core idle_stack[][], and its spinlock — is not covered: s_kernel and s_extra_kernels[] all live in one shared .privileged_bss. So today a rogue island’s kernel can still scribble a neighbour’s TCBs or ready-state, and the freedom-from-interference claim holds for app RAM but not for kernel RAM (backlog D2; QC3-1(i)). sizeof(struct tk_kernel) is ≈ 3 KiB (SMP=2, 8 TCBs, 1 KiB idle stacks) — dominated by tcbs[] and idle_stack[][].

Separately (backlog D1), the mixed-criticality asymmetry “fence the safety island, leave the QM island shared” is largely achievable today via selective set_ram: only the safety island declares an arena, so a QM core is denied the safety arena while the QM arena stays shared. The residue is (a) a first-class, self-documenting way to declare that intent, and (b) the one thing the current seam cannot express — a trusted island whose core is fenced from nothing (a monitor that may read every island), because tk_instance_fence_enter denies all neighbour arenas unconditionally.

The load-bearing constraint (found while scoping): tk_kernel_liveness_check reads a neighbour instance’s k->ticks_ms and k->sched_progress directly — a deliberate cross-island stall-detection channel (ADR 0002 review L3). So D2 must NOT fence the whole struct tk_kernel; that would fault the monitor. The heartbeat fields are meant to be read across the boundary.

Decision

1. D2 — fence each island’s whole kernel struct; MIRROR the heartbeats to shared RAM

Two moves — fence the whole private struct (simple, tight), and relocate only the read channel the neighbour monitor needs.

Fence the whole struct, placed fence-granular. Each instance’s struct tk_kernel (its tcbs[], lock, current_thread[]/idle_index[], idle_stack[][], scheduler policy, fence_base/size, ticker_core, …) becomes a per-instance, fence-granular region — aligned + padded to the coarsest granule (4 KiB for the paging class; next pow2 ≥ size for PMSAv7 — 4 KiB at today’s ≈ 3 KiB struct). Realised by an aligned(FENCE_GRANULE) struct type and distinct per-instance objects (not one opaque s_extra_kernels[]) so each gets its own aligned slot = one deny-list entry.

Auto-deny. tk_instance_fence_enter(me), already building the neighbour deny-list, adds each neighbour’s struct region (&k, sizeof k rounded to the granule) alongside its app arena — the kernel knows these addresses, no app involvement. List grows to ≤ 2·(MAX_KERNELS−1). fence_enter reads neighbour metadata before arch_instance_fence_apply runs (unfenced), so list-building never trips the fence it installs.

Mirror the heartbeats. tk_kernel_liveness_check reads a neighbour’s ticks_ms + sched_progress — which the fence now denies. So the owning core write-throughs those two counters into a small shared array s_heartbeat[MAX_KERNELS] (ordinary .bss, never fenced) where it already updates them (the tick advance; the schedule/idle progress bump), and the monitor reads s_heartbeat[i] instead of the neighbour struct. This is the same “publish to a shared region” shape tk_introspect uses. The mirror is gated on isolation — at collapse (MAX_KERNELS==1 / isolation off) it does not exist and the struct is byte-identical; the counters stay canonical in the struct, the mirror is a redundant shadow only when fencing is on. The mirror is R/W-shared (the deny-list-of-holes seam cannot express read-only), so a rogue island could spoof its own heartbeat — acceptable for advisory stall detection; it still cannot touch a neighbour’s private scheduler state, which is the property that matters.

Preferred over splitting the struct (a shared prefix wastes a 4 KiB page; extracting the fields breaks the N=1 byte-identical collapse): the mirror keeps the struct whole and canonical, and the deny region is simply the whole struct.

2. D1 — the TRUSTED escape hatch (kernel-RAM fencing is unconditional)

Implementation refined the three-mode sketch to what earns its keep. Kernel-RAM fencing (§1) is unconditional when isolation is on — every island’s struct tk_kernel is mutually fenced, the strongest freedom-from-interference and the right safety default (you never want an island’s scheduler state corruptible to save a 4 KiB region). So the “SHARED vs ISOLATED” kernel distinction collapses: an island’s mixed-criticality “sharedness” is entirely about its app arena, already opt-in per island via set_ram (a QM island simply doesn’t call it; the asymmetry falls out — ADR 0003). That leaves ONE genuinely new capability the deny-list seam cannot express:

  • TRUSTED — tk_kernel_set_trusted(handle) marks an instance whose core installs no fence (empty deny-list), so a monitor/supervisor may read every island’s RAM directly (beyond the shared heartbeat mirror, which carries only the two liveness counters). Its OWN kernel RAM is still denied to other cores, so the monitor cannot be corrupted — TRUSTED loosens only the monitor’s own view. A one-line branch in tk_instance_fence_enter (if (me->trusted) return;), a trusted flag on the instance, both folded away at MAX_KERNELS==1.

(The SHARED/ISOLATED labels are dropped: with kernel RAM unconditional, “isolated” is the default and “shared” means “no set_ram”. A future shared-kernel island — MPU-budget-driven — could revive a per-instance opt-out; deferred, no use case today.)

3. Granularity, budget, degrade

  • Granule: paging = 4 KiB align + multiple; PMSAv7 = pow2 + natural alignment. The private-body region is placed to satisfy the coarsest.

  • Budget: ≤ 2·(MAX_KERNELS−1) deny entries. Paging arches refine within their table budget (fine for a few KiB regions); region-MPU arches spend one MPU region per entry — fail-CLOSED (ADR 0003 §5) if a particular core runs out. Realistic island counts (2–3) fit comfortably.

  • Degrade: unchanged from ADR 0003 §5 — loud at compile time, fail-closed at runtime.

4. Collapse safety (ADR 0002 §3, non-negotiable)

At TK_CONFIG_MAX_KERNELS == 1 or isolation off: no split (the shared prefix folds back so the single instance is byte-identical), no auto-deny, no mode field, no alignment padding. A single-instance build is byte-identical to one built without D2 — verified by the same collapse checks as ADR 0003.

Consequences

  • Freedom-from-interference becomes complete at the kernel level — a neighbour can no longer corrupt an island’s scheduler state/TCBs, only (harmlessly) its advisory heartbeat. The missing half of ADR 0003.

  • Cross-island monitoring is preserved by design (the heartbeat prefix stays shared), and the trusted island gives a monitor full read access when it needs more than heartbeats.

  • Per-instance placement + the struct split are the main churn — extracting/reordering the heartbeat fields touches now_ms / tick / schedule / idle / liveness; the private struct needs an aligned type and distinct per-instance objects instead of s_extra_kernels[].

  • Builds on Epic B — the seam is already uintptr_t, so a 64-bit port fences kernel RAM above 4 GiB without further seam work.

Verification

  • D2 — examples/amp_kernel_fence (arm_v8a virt SMP=2), through the real tk_kernel_define → tk_kernel_start path: a normal island’s store into the DEFAULT kernel struct FAULTS (Data Abort, FAR inside the fenced region), an ordinary global write succeeds. Regression: all four fence demos still PASS, amp_spawn (no isolation) unchanged, check_fence_matrix PASS, collapse clean.

  • D1 (TRUSTED) — examples/amp_trusted_monitor: a trusted monitor island reads the default kernel struct DIRECTLY (raw word + live ticks_ms) with no fault, where the normal island above faulted.

Resolved / remaining

  1. ~~Shared-heartbeat mechanism (reorder vs extract).~~ → the heartbeat mirror (§1).

  2. ~~TRUSTED write access.~~ → a trusted core skips the fence entirely (if (me->trusted) return;), so it gets R/W to every island; the seam can’t express read-only holes, and R/W is fine (the monitor is trusted).

  3. Region-MPU budget for > 2 isolated islands (≤ 2 deny entries each): paging arches refine within their table budget; region-MPU arches spend one MPU region per entry and fail-CLOSED over budget. Document the per-arch ceiling when a >2-island region-MPU build is attempted — untested today.

  4. ~~First verification arch.~~ → arm_v8a virt SMP=2 (both demos above).