rose-ash/plans/lib-guest-scheduler.md

# lib/guest/scheduler — extraction plan

Two distinct concurrency models — Erlang's addressed processes + mailboxes, and
Go's anonymous channels + goroutines — sit on the same underlying machinery:
a fork/yield/block/resume scheduler over CEK io-suspended continuations. This
plan captures that machinery as `lib/guest/scheduler/` so language N+1 with a
new concurrency model costs ~200 lines of model-specific code instead of
re-inventing the scheduler.

Reference: `plans/lib-guest.md` (parent — two-language rule, stratification),
`plans/erlang-on-sx.md` (first consumer, in production), Go-on-SX (second
consumer, see `plans/go-on-sx.md` once that lands).

**Branch:** `architecture`. SX files via `sx-tree` MCP only.

## Thesis

The substrate already provides what a scheduler needs: CEK io-suspension
(`make-cek-suspended`, `cek-resume`) gives suspendable execution; first-class
environments give each unit of execution its own scope; the trampolined
evaluator means we never blow the host stack. What every guest with concurrency
*re-implements* on top of this is the **fork/yield/block/resume protocol** —
the bookkeeping that decides which suspended computation runs next.

Two concrete consumers, two different concurrency vocabularies, sharing one
underlying scheduler, is the proof. If only Erlang lives on it, "scheduler kit"
is a euphemism for "Erlang scheduler with a Go skin." The two-language rule
is the gate.

## Current state (2026-05-26)

- **Erlang-on-SX** has the full pattern in production: 729/729 conformance,
  spawn/send/receive, selective receive, monitor/link, hot reload. The
  scheduler logic is currently coupled to Erlang-shaped concepts (PIDs,
  mailboxes, links) — extraction-blocking but not extraction-defeating.
- **Go-on-SX** does not exist yet. `plans/go-on-sx.md` is the umbrella plan
  (TBD); this scheduler plan is a sibling/dependency.
- **lib/guest/scheduler/** does not exist. The two-language rule blocks
  extraction until Go-on-SX independently implements its scheduler.

**Status: Phase 0 (Erlang shape capture).** No code change in this plan yet.

## Why the two models actually share a kit

The non-obvious claim is that Erlang processes and Go goroutines really do
share machinery beneath their different vocabularies. The mapping:

| Concept | Erlang | Go | Common kit name |
|---|---|---|---|
| Unit of execution | process (PID-addressed) | goroutine (anonymous) | **task** |
| Spawn | `spawn(Fun)` → PID | `go expr` → nothing | `task-spawn` |
| Block target | mailbox match | channel send/recv | `task-block` |
| Wake condition | message arrives | counterpart ready | `task-resume` predicate |
| Yield | `receive` with no match | channel blocked | scheduler hands off |
| Termination | exit reason → linked tasks | panic / return | task lifecycle |
| Selection | selective `receive` | `select` statement | both = "wait for any of N predicates" |

What the kit owns:
- The **task table** (token → suspended CEK continuation + status).
- The **runnable queue** + scheduling policy (round-robin v1; pluggable).
- The **block→resume protocol**: a blocked task registers a predicate; when
  any task changes state, blocked tasks are re-polled; first whose predicate
  fires becomes runnable.
- The **fairness/preemption budget** — gas per step before forced yield.

What each language owns:
- The semantics layer on top: Erlang's PID→task map + mailbox per task +
  selective-receive predicates; Go's channel value → blocked-task list per
  channel + send/recv pairing + select multiplexing.
- The language-visible API (`spawn`/`!`/`receive` vs `go`/`<-`/`select`).

This is exactly the lib/guest pattern: extract the dispatch skeleton, keep
the rules in the language layer.

## API surface (proposed — design only, not yet implemented)

```
(make-scheduler &key gas-per-step ;; default 1000
                     policy)      ;; :round-robin | :fifo
  -> scheduler-handle

(task-spawn sched body-thunk) -> task-token
  ;; body-thunk is a 0-arg fn whose body runs as the task.
  ;; Returns immediately; task is enqueued runnable.

(task-current sched) -> task-token
  ;; Inside a task, the token of the running task. Useful for self-reference.

(task-yield sched) -> nil
  ;; Voluntary yield. Caller is re-enqueued at the tail of runnable.

(task-block sched resume-predicate) -> any
  ;; Caller suspends. Predicate is (fn () -> resume-value-or-#f).
  ;; When predicate returns non-#f, caller resumes with that value.
  ;; Predicate is polled on every scheduler tick when there's nothing
  ;; obviously runnable. (Optimisation: language layer can wake explicitly —
  ;; see task-wake.)

(task-wake sched task) -> nil
  ;; Hint to the scheduler: re-poll this task's resume-predicate now.
  ;; Used by sender-side when a receiver might unblock.

(task-status sched task) -> :runnable | :blocked | :finished | :crashed

(task-result sched task) -> value | {:error reason}
  ;; After :finished or :crashed.

(scheduler-step sched) -> :ran | :idle | :all-done
  ;; Run at most gas-per-step instructions of one task. Caller drives the
  ;; loop.

(scheduler-run sched) -> nil
  ;; Run until :all-done. Equivalent to (until (= :all-done (scheduler-step
  ;; sched))).
```

Notes on the design:
- `task-block` with a resume-predicate is the universal blocking primitive.
  Erlang's `receive` is `(task-block sched (fn () (mailbox-match self pat)))`.
  Go's `<-ch` is `(task-block sched (fn () (channel-recv-ready ch)))`.
- `task-wake` is the optimisation: instead of polling every blocked task
  every step, the language layer wakes the specific task whose predicate
  is now likely true. v1 can omit it; performance work later.
- `gas-per-step` gives fairness without true preemption. Tasks that don't
  yield within their gas budget are force-yielded by the CEK loop. (CEK
  io-suspension already does this for IO; gas budget extends to plain
  instructions.)
- No priority/affinity in v1. Both Erlang and Go default to non-priority
  scheduling; specialised cases (Erlang's high-priority processes) are
  language-layer concerns.

## Build order — phases

This is a long-running plan paced against Go-on-SX. Phases are not loop-style
"one commit per phase" — they're milestone gates.

### Phase 0 — Erlang shape capture (doc-only) ⬜
- Read `lib/erlang/runtime.sx` scheduler code (currently coupled to Erlang
  vocabulary).
- Write a 1-page summary of what's actually a scheduler and what's actually
  Erlang. Identify the boundary.
- **Acceptance:** summary committed to this plan as a new section "Erlang
  scheduler shape (captured 2026-MM-DD)". No code change.
- **Output:** clear-eyed mental model. Without this, we'll merge Erlang's
  scheduler shape into the kit and pretend it generalises.

### Phase 1 — Go scheduler independent implementation ⬜
- During Go-on-SX, implement `lib/go/sched.sx` from scratch. Do NOT look at
  Erlang's scheduler while doing this. (Or read it once, then close it.)
- Pass Go's channel + goroutine + select conformance tests.
- **Acceptance:** Go scheduler green, lib/go/scoreboard.json includes scheduler
  tests, two-consumer rule now passable.
- **Output:** two independent, working implementations of the same idea.

### Phase 2 — Diff and proposed kit ⬜
- Side-by-side diff: Erlang's scheduler vs Go's scheduler. Where do they
  agree? Where does each have language-specific bookkeeping?
- The diff is the kit. Things in *both* go in `lib/guest/scheduler/`; things
  in only one stay in `lib/erlang/` or `lib/go/`.
- Draft `lib/guest/scheduler/api.sx` (signatures only, no body) reflecting the
  proposed surface.
- **Acceptance:** API draft circulated for review; agreement that the surface
  covers both consumers; no merge yet.

### Phase 3 — Implement `lib/guest/scheduler/` ⬜
- Implement the kit per the agreed API. New file(s) in `lib/guest/scheduler/`.
- The kit has its own tests in `lib/guest/scheduler/tests/` — agnostic of any
  particular language vocabulary.
- **Acceptance:** kit tests pass. Erlang and Go conformance scoreboards
  unchanged (the language implementations still use their own scheduler —
  we haven't refactored yet).

### Phase 4 — Refactor Erlang to use the kit ⬜
- `lib/erlang/runtime.sx` scheduler logic deleted; replaced with calls into
  `lib/guest/scheduler/`. Erlang's PID table, mailbox-per-PID, selective
  receive stay in `lib/erlang/`.
- **No-regression gate:** Erlang conformance holds at current pass count
  (currently 729/729). Hard requirement.
- **Acceptance:** Erlang scoreboard unchanged; `lib/erlang/runtime.sx`
  meaningfully smaller (the scheduler code is gone).

### Phase 5 — Refactor Go to use the kit ⬜
- Same exercise for Go. `lib/go/sched.sx` shrinks to channel/goroutine
  bookkeeping + delegation.
- **No-regression gate:** Go conformance scoreboard at its current pass
  count.
- **Acceptance:** Go scoreboard unchanged; `lib/go/sched.sx` meaningfully
  smaller.

### Phase 6 — Documentation + design-diary close ⬜
- Document `lib/guest/scheduler/` API in `lib/guest/README.md` (or wherever
  the lib/guest API index lives).
- Capture the chiselling diary: what *almost* went in the kit but ended up
  language-specific, and why. This is the load-bearing knowledge for the
  third consumer when it arrives.
- **Acceptance:** API documented; diary section added to this plan.

## Two-language rule — gating

**The rule is hard.** No code in `lib/guest/scheduler/` lands until BOTH
Phase 1 (Go independent) AND Phase 0 (Erlang capture) are complete AND a
review confirms the two implementations actually share machinery in a way
the kit captures.

If, during Phase 2 diff, we discover that the agreement is shallow (e.g.,
both have a runnable queue but the policies are fundamentally incompatible),
the **right outcome is to NOT extract**. Add a "rejected extraction" note to
this plan documenting what we learned and close it. That outcome is fine —
it tells us the two concurrency models aren't actually sister, which is a
real result.

## Open questions

- **Preemption.** v1 is cooperative; gas-per-step gives fairness but not
  hard preemption. Erlang BEAM does true preemption (reduction counting).
  Go uses async preemption (signal-driven since 1.14). Neither extreme fits
  cooperatively over CEK. Is gas-per-step + voluntary yield enough? Probably
  for v1; revisit if a guest needs hard real-time.
- **Priority/affinity.** Both Erlang and Go can run without it. Defer.
- **Distribution.** Erlang nodes, Go's distributed channels — both are
  language-specific layers on top of the local scheduler. Out of scope.
- **Cancellation.** Go has `context.Context`; Erlang has `exit/2`. Both
  bottom out at "deliver an exception to a task." Worth modelling? Probably
  as a kit primitive `(task-cancel sched task reason)` that delivers an
  exception via CEK exception machinery, language layer wraps it.
- **Third consumer.** If/when JS-on-SX gets a proper async/await + Promise
  scheduler, that'd be a great third consumer to validate the kit didn't
  over-fit to Erlang+Go.

## Progress log

_Newest first. Append one dated entry per milestone landed._

- 2026-05-27 — From Go-on-SX Phase 5 first slice: the channel
  primitive landed as closures-over-mutable-state in
  `lib/go/sched.sx`. Concrete shape:

  ```
  (list :go-chan SEND-FN RECV-FN CLOSED?-FN CLOSE!-FN)
  ```

  Each closure captures a shared `buf` (a mutable list) and `closed`
  flag (a let-bound boolean mutated via `set!`). Identity: two
  `make()` calls produce distinct closures, satisfying Go spec
  § Channel types' "distinct channels with same type" rule.

  **Design insight for the kit**: the channel-as-closure-bundle shape
  is the right scheduler-kit primitive — implementation-hide the
  buffer behind opaque accessor closures, so the underlying storage
  can be swapped (linked list → ring buffer → segmented array) without
  changing the API. Erlang's mailboxes will need the same trick.

  **v0 limitation logged**: no real preemption. SX doesn't expose
  first-class continuations to guest code, so v0 runs `go f()`
  synchronously and relies on the spawned goroutine completing before
  the main goroutine receives. Real concurrent semantics — blocking
  send on full buffer, blocking recv on empty — needs the
  scheduler kit to ship the suspension/resumption machinery (or for
  Phase 5b to bake CEK-style trampolining into the eval layer).

  Cross-ref: the `:select-case` uniform shape from the parser-side
  diary entry pairs with this — the kit's `sched-select` should
  accept a list of channel-op cases (built from the closures-over-
  state primitives logged here) and pick a ready one. Source:
  Go-on-SX commit landing `lib/go/sched.sx` first cut.

- 2026-05-27 — Follow-up from same Phase 2 work: **`select` AST shape**
  landed. Each case is `(list :select-case COMM-STMT BODY)` where
  COMM-STMT is one of `:send`, `:short-decl` (recv into new var),
  `:assign` (recv into existing var), or a bare receive expression
  `(:app (:var "<-") [chan])`. The shape is uniform across all four
  comm-stmt kinds — the kit's `sched-select` primitive should accept a
  list of cases each described by `(direction chan value-target?)` and
  let the kit's runtime pick a ready case. That uniformity is what
  makes a single kit primitive cover all four Go case shapes.

  Also: Go's `select` with `default` makes the multiplexer non-blocking;
  without default it blocks until a case is ready. The kit primitive
  should mirror this — present-or-absent default determines blocking
  semantics. Erlang's `receive ... after Timeout -> ...` is a similar
  pattern with a timeout case rather than default; the kit primitive
  should handle both as instances of "non-blocking-fallback case."
  Source: Go-on-SX commit `parse.sx — switch + select`.

- 2026-05-27 — From Go-on-SX Phase 2 (parser side, ahead of scheduler
  implementation): the **parsed AST shapes** for Go's concurrency
  primitives have landed and are worth recording before Phase 5 builds
  the scheduler.

  ```
  go EXPR              → (list :go EXPR)
  defer EXPR           → (list :defer EXPR)
  ch <- v              → (list :send CHAN VALUE)
  <-ch                 → (list :app (:var "<-") [CHAN])   ; unary recv
  for range COLL { }   → (list :range-for nil nil nil COLL BODY)
  for k, v := range C  → (list :range-for :short-decl KEY VAL COLL BODY)
  ```

  **Design insight for the kit**: the `:go` and `:defer` shapes are
  pleasingly minimal — both wrap a single expression. Erlang's
  `spawn(Mod, Fun, Args)` will produce something more elaborate; the
  scheduler kit primitive `(sched-spawn task)` should accept a thunk so
  both languages reduce to a uniform spawn API.

  The `:send` shape carries CHAN + VALUE — symmetric with channel-recv
  as the unary `<-` form. Once the scheduler has channel primitives,
  both shapes thunk-down to a single `(chan-op direction chan value)`
  abstraction.

  Range over channels (`for v := range ch`) is currently parsed as
  range-for with `coll = ch`; the scheduler kit will dispatch on the
  type of `coll` at execution time (channels yield via receive,
  collections via iteration). This dispatch is the right place for the
  scheduler kit to express the channel-receive ⇄ iteration polymorphism.
  Source: Go-on-SX commit `parse.sx — go/defer/send/range`.

- 2026-05-26 — Plan drafted. Phase 0 unstarted. Awaiting Go-on-SX to begin
  Phase 1.