SMP programming on Tuwa

Copyright (c) 2026 Muhammad Anisur Rahman. All rights reserved.

Tuwa schedules on multiple cores. This manual is about what changes for your code when it does, which is less than people expect in the API and more than they expect in the assumptions.

Availability: SMP runs on AArch64, x86-64, RV64 and RV32 — every port in this release. The archived ports are single-core.

What does not change

Every API in 08-api-manual.md works unchanged. LoadTask, TuwaSleep, messages, mutexes, signals — all of it behaves the same. You do not write a different program for SMP.

What does change

One thing, and everything else follows from it:

> Two of your tasks can now be executing at the same instant.

On one core, "task A runs, then task B runs" is guaranteed and a lot of code quietly depends on it. Two cores make that false. Nothing in the API changed; what changed is which of your assumptions were true by accident.

Disabling interrupts is no longer a lock

This is the big one. On a single core:


DisableSysInterrupt();
shared_counter++;             /* nothing else can run - true */
EnableSysInterrupt();

On four cores that protects you from this core's interrupts and from nothing else. Another core is running the same line simultaneously.

Use a mutex. If the section is short enough that a mutex feels heavy, it is short enough to be an atomic operation instead.

"Only one task uses this" needs checking

A static buffer inside a function, a module-level scratch variable, a "nobody else calls this" — each was safe because of the single-core interleaving, not because of anything you wrote down.

Priority no longer means strict ordering

On one core a priority-2 task cannot run while a priority-0 task is runnable. On four cores it can — there are four cores and the top-priority task only occupies one. Code that used priority as an implicit lock ("higher priority means I get there first") is broken by SMP.

Placing tasks on cores

By default the scheduler places tasks. When you need control:

To… Call
Create a task on a specific core TuwaLoadTaskOnCpu(cpu, ...)
Pin an existing task TuwaSetTaskAffinity(task, cpu)
Move a running task TuwaMigrateTask(task, cpu)
Ask where a task is TuwaTaskCpu(task)
Let the placement policy choose TuwaPlaceTask(task)

Reasons to pin, in rough order of how often they are the real reason:

  1. A device belongs to one core. Its interrupt arrives there; the task

    2. servicing it should be there too.

  2. Cache locality. A task working a large structure does better staying

    3. where the data is warm.

  3. Isolation. A hard-real-time task on its own core stops competing

    4. with everything else.

Reasons not to pin: a hunch that it will be faster. Measure first — TuwaCpuLoad and TuwaCpuTaskCount report per-core state.

Bringing the cores up

Boot is not symmetric. One core starts the kernel; the others are brought in afterwards.


primary core     boots, initialises the kernel, prepares the run queues,
                 the per-core scheduler state and the locks
                       |
                 TuwaSmpStartSecondaries()
                       |
secondary cores  TuwaSecondaryStart() -> enter the scheduler
                       |
                 TuwaSmpWaitAllOnline()   <- primary waits here

Everything up to TuwaSmpStartSecondaries() is preparation done by one core, so it needs no locking. That is deliberate: the locks protect structures that only become shared at the moment the secondaries start.

TuwaShowCpuInfo() reports what came up.

Self-tests

Two, and they are worth running on a new port before trusting anything:
TuwaSmpAtomicSelfTest() Do this core's atomics actually work?
TuwaSmpPlacementSelfTest() Does placement put tasks where it says?

The first is not paranoia. Atomics are the one thing a port can get subtly wrong in a way that works under light load and fails under contention — which is to say, works in testing and fails in the field.

A hazard found the hard way

The AArch64 ticket lock originally spun on a plain load plus WFE. It deadlocked. The fix is the exclusive-monitor idiom: the load must be exclusive, so the monitor arms and WFE is actually woken by the store that releases the lock. With a plain load nothing arms the monitor and the waiter sleeps forever.

Two things to take from it. First, if you write a lock on a new port, use the architecture's documented idiom exactly — the "obvious simplification" is how this class of bug arrives. Second, it only failed under real contention, which is why TuwaSmpAtomicSelfTest exists.

Porting SMP to a new core

If your architecture is one of the four supported, there is nothing to do. For a new one:

  1. Atomics first. Compare-and-swap or load-linked/store-conditional,

    2. and the memory barriers. Everything else rests on this.

  2. Secondary core entry. How a core is released from reset and where

    3. it starts. This is board work as much as CPU work.

  3. Per-core state. Each core needs its own current-task pointer and

    4. scheduler state.

  4. Inter-core interrupt, if the architecture has one.
  5. Run the self-tests.

Start from arm64 — it is the port the SMP substrate was built on.

Checklist for making existing code SMP-safe