# lib/guest/scheduler — extraction plan Two distinct concurrency models — Erlang's addressed processes + mailboxes, and Go's anonymous channels + goroutines — sit on the same underlying machinery: a fork/yield/block/resume scheduler over CEK io-suspended continuations. This plan captures that machinery as `lib/guest/scheduler/` so language N+1 with a new concurrency model costs ~200 lines of model-specific code instead of re-inventing the scheduler. Reference: `plans/lib-guest.md` (parent — two-language rule, stratification), `plans/erlang-on-sx.md` (first consumer, in production), Go-on-SX (second consumer, see `plans/go-on-sx.md` once that lands). **Branch:** `architecture`. SX files via `sx-tree` MCP only. ## Thesis The substrate already provides what a scheduler needs: CEK io-suspension (`make-cek-suspended`, `cek-resume`) gives suspendable execution; first-class environments give each unit of execution its own scope; the trampolined evaluator means we never blow the host stack. What every guest with concurrency *re-implements* on top of this is the **fork/yield/block/resume protocol** — the bookkeeping that decides which suspended computation runs next. Two concrete consumers, two different concurrency vocabularies, sharing one underlying scheduler, is the proof. If only Erlang lives on it, "scheduler kit" is a euphemism for "Erlang scheduler with a Go skin." The two-language rule is the gate. ## Current state (2026-05-26) - **Erlang-on-SX** has the full pattern in production: 729/729 conformance, spawn/send/receive, selective receive, monitor/link, hot reload. The scheduler logic is currently coupled to Erlang-shaped concepts (PIDs, mailboxes, links) — extraction-blocking but not extraction-defeating. - **Go-on-SX** does not exist yet. `plans/go-on-sx.md` is the umbrella plan (TBD); this scheduler plan is a sibling/dependency. - **lib/guest/scheduler/** does not exist. The two-language rule blocks extraction until Go-on-SX independently implements its scheduler. **Status: Phase 0 (Erlang shape capture).** No code change in this plan yet. ## Why the two models actually share a kit The non-obvious claim is that Erlang processes and Go goroutines really do share machinery beneath their different vocabularies. The mapping: | Concept | Erlang | Go | Common kit name | |---|---|---|---| | Unit of execution | process (PID-addressed) | goroutine (anonymous) | **task** | | Spawn | `spawn(Fun)` → PID | `go expr` → nothing | `task-spawn` | | Block target | mailbox match | channel send/recv | `task-block` | | Wake condition | message arrives | counterpart ready | `task-resume` predicate | | Yield | `receive` with no match | channel blocked | scheduler hands off | | Termination | exit reason → linked tasks | panic / return | task lifecycle | | Selection | selective `receive` | `select` statement | both = "wait for any of N predicates" | What the kit owns: - The **task table** (token → suspended CEK continuation + status). - The **runnable queue** + scheduling policy (round-robin v1; pluggable). - The **block→resume protocol**: a blocked task registers a predicate; when any task changes state, blocked tasks are re-polled; first whose predicate fires becomes runnable. - The **fairness/preemption budget** — gas per step before forced yield. What each language owns: - The semantics layer on top: Erlang's PID→task map + mailbox per task + selective-receive predicates; Go's channel value → blocked-task list per channel + send/recv pairing + select multiplexing. - The language-visible API (`spawn`/`!`/`receive` vs `go`/`<-`/`select`). This is exactly the lib/guest pattern: extract the dispatch skeleton, keep the rules in the language layer. ## API surface (proposed — design only, not yet implemented) ``` (make-scheduler &key gas-per-step ;; default 1000 policy) ;; :round-robin | :fifo -> scheduler-handle (task-spawn sched body-thunk) -> task-token ;; body-thunk is a 0-arg fn whose body runs as the task. ;; Returns immediately; task is enqueued runnable. (task-current sched) -> task-token ;; Inside a task, the token of the running task. Useful for self-reference. (task-yield sched) -> nil ;; Voluntary yield. Caller is re-enqueued at the tail of runnable. (task-block sched resume-predicate) -> any ;; Caller suspends. Predicate is (fn () -> resume-value-or-#f). ;; When predicate returns non-#f, caller resumes with that value. ;; Predicate is polled on every scheduler tick when there's nothing ;; obviously runnable. (Optimisation: language layer can wake explicitly — ;; see task-wake.) (task-wake sched task) -> nil ;; Hint to the scheduler: re-poll this task's resume-predicate now. ;; Used by sender-side when a receiver might unblock. (task-status sched task) -> :runnable | :blocked | :finished | :crashed (task-result sched task) -> value | {:error reason} ;; After :finished or :crashed. (scheduler-step sched) -> :ran | :idle | :all-done ;; Run at most gas-per-step instructions of one task. Caller drives the ;; loop. (scheduler-run sched) -> nil ;; Run until :all-done. Equivalent to (until (= :all-done (scheduler-step ;; sched))). ``` Notes on the design: - `task-block` with a resume-predicate is the universal blocking primitive. Erlang's `receive` is `(task-block sched (fn () (mailbox-match self pat)))`. Go's `<-ch` is `(task-block sched (fn () (channel-recv-ready ch)))`. - `task-wake` is the optimisation: instead of polling every blocked task every step, the language layer wakes the specific task whose predicate is now likely true. v1 can omit it; performance work later. - `gas-per-step` gives fairness without true preemption. Tasks that don't yield within their gas budget are force-yielded by the CEK loop. (CEK io-suspension already does this for IO; gas budget extends to plain instructions.) - No priority/affinity in v1. Both Erlang and Go default to non-priority scheduling; specialised cases (Erlang's high-priority processes) are language-layer concerns. ## Build order — phases This is a long-running plan paced against Go-on-SX. Phases are not loop-style "one commit per phase" — they're milestone gates. ### Phase 0 — Erlang shape capture (doc-only) ⬜ - Read `lib/erlang/runtime.sx` scheduler code (currently coupled to Erlang vocabulary). - Write a 1-page summary of what's actually a scheduler and what's actually Erlang. Identify the boundary. - **Acceptance:** summary committed to this plan as a new section "Erlang scheduler shape (captured 2026-MM-DD)". No code change. - **Output:** clear-eyed mental model. Without this, we'll merge Erlang's scheduler shape into the kit and pretend it generalises. ### Phase 1 — Go scheduler independent implementation ⬜ - During Go-on-SX, implement `lib/go/sched.sx` from scratch. Do NOT look at Erlang's scheduler while doing this. (Or read it once, then close it.) - Pass Go's channel + goroutine + select conformance tests. - **Acceptance:** Go scheduler green, lib/go/scoreboard.json includes scheduler tests, two-consumer rule now passable. - **Output:** two independent, working implementations of the same idea. ### Phase 2 — Diff and proposed kit ⬜ - Side-by-side diff: Erlang's scheduler vs Go's scheduler. Where do they agree? Where does each have language-specific bookkeeping? - The diff is the kit. Things in *both* go in `lib/guest/scheduler/`; things in only one stay in `lib/erlang/` or `lib/go/`. - Draft `lib/guest/scheduler/api.sx` (signatures only, no body) reflecting the proposed surface. - **Acceptance:** API draft circulated for review; agreement that the surface covers both consumers; no merge yet. ### Phase 3 — Implement `lib/guest/scheduler/` ⬜ - Implement the kit per the agreed API. New file(s) in `lib/guest/scheduler/`. - The kit has its own tests in `lib/guest/scheduler/tests/` — agnostic of any particular language vocabulary. - **Acceptance:** kit tests pass. Erlang and Go conformance scoreboards unchanged (the language implementations still use their own scheduler — we haven't refactored yet). ### Phase 4 — Refactor Erlang to use the kit ⬜ - `lib/erlang/runtime.sx` scheduler logic deleted; replaced with calls into `lib/guest/scheduler/`. Erlang's PID table, mailbox-per-PID, selective receive stay in `lib/erlang/`. - **No-regression gate:** Erlang conformance holds at current pass count (currently 729/729). Hard requirement. - **Acceptance:** Erlang scoreboard unchanged; `lib/erlang/runtime.sx` meaningfully smaller (the scheduler code is gone). ### Phase 5 — Refactor Go to use the kit ⬜ - Same exercise for Go. `lib/go/sched.sx` shrinks to channel/goroutine bookkeeping + delegation. - **No-regression gate:** Go conformance scoreboard at its current pass count. - **Acceptance:** Go scoreboard unchanged; `lib/go/sched.sx` meaningfully smaller. ### Phase 6 — Documentation + design-diary close ⬜ - Document `lib/guest/scheduler/` API in `lib/guest/README.md` (or wherever the lib/guest API index lives). - Capture the chiselling diary: what *almost* went in the kit but ended up language-specific, and why. This is the load-bearing knowledge for the third consumer when it arrives. - **Acceptance:** API documented; diary section added to this plan. ## Two-language rule — gating **The rule is hard.** No code in `lib/guest/scheduler/` lands until BOTH Phase 1 (Go independent) AND Phase 0 (Erlang capture) are complete AND a review confirms the two implementations actually share machinery in a way the kit captures. If, during Phase 2 diff, we discover that the agreement is shallow (e.g., both have a runnable queue but the policies are fundamentally incompatible), the **right outcome is to NOT extract**. Add a "rejected extraction" note to this plan documenting what we learned and close it. That outcome is fine — it tells us the two concurrency models aren't actually sister, which is a real result. ## Open questions - **Preemption.** v1 is cooperative; gas-per-step gives fairness but not hard preemption. Erlang BEAM does true preemption (reduction counting). Go uses async preemption (signal-driven since 1.14). Neither extreme fits cooperatively over CEK. Is gas-per-step + voluntary yield enough? Probably for v1; revisit if a guest needs hard real-time. - **Priority/affinity.** Both Erlang and Go can run without it. Defer. - **Distribution.** Erlang nodes, Go's distributed channels — both are language-specific layers on top of the local scheduler. Out of scope. - **Cancellation.** Go has `context.Context`; Erlang has `exit/2`. Both bottom out at "deliver an exception to a task." Worth modelling? Probably as a kit primitive `(task-cancel sched task reason)` that delivers an exception via CEK exception machinery, language layer wraps it. - **Third consumer.** If/when JS-on-SX gets a proper async/await + Promise scheduler, that'd be a great third consumer to validate the kit didn't over-fit to Erlang+Go. ## Progress log _Newest first. Append one dated entry per milestone landed._ - 2026-05-27 — Follow-up from same Phase 2 work: **`select` AST shape** landed. Each case is `(list :select-case COMM-STMT BODY)` where COMM-STMT is one of `:send`, `:short-decl` (recv into new var), `:assign` (recv into existing var), or a bare receive expression `(:app (:var "<-") [chan])`. The shape is uniform across all four comm-stmt kinds — the kit's `sched-select` primitive should accept a list of cases each described by `(direction chan value-target?)` and let the kit's runtime pick a ready case. That uniformity is what makes a single kit primitive cover all four Go case shapes. Also: Go's `select` with `default` makes the multiplexer non-blocking; without default it blocks until a case is ready. The kit primitive should mirror this — present-or-absent default determines blocking semantics. Erlang's `receive ... after Timeout -> ...` is a similar pattern with a timeout case rather than default; the kit primitive should handle both as instances of "non-blocking-fallback case." Source: Go-on-SX commit `parse.sx — switch + select`. - 2026-05-27 — From Go-on-SX Phase 2 (parser side, ahead of scheduler implementation): the **parsed AST shapes** for Go's concurrency primitives have landed and are worth recording before Phase 5 builds the scheduler. ``` go EXPR → (list :go EXPR) defer EXPR → (list :defer EXPR) ch <- v → (list :send CHAN VALUE) <-ch → (list :app (:var "<-") [CHAN]) ; unary recv for range COLL { } → (list :range-for nil nil nil COLL BODY) for k, v := range C → (list :range-for :short-decl KEY VAL COLL BODY) ``` **Design insight for the kit**: the `:go` and `:defer` shapes are pleasingly minimal — both wrap a single expression. Erlang's `spawn(Mod, Fun, Args)` will produce something more elaborate; the scheduler kit primitive `(sched-spawn task)` should accept a thunk so both languages reduce to a uniform spawn API. The `:send` shape carries CHAN + VALUE — symmetric with channel-recv as the unary `<-` form. Once the scheduler has channel primitives, both shapes thunk-down to a single `(chan-op direction chan value)` abstraction. Range over channels (`for v := range ch`) is currently parsed as range-for with `coll = ch`; the scheduler kit will dispatch on the type of `coll` at execution time (channels yield via receive, collections via iteration). This dispatch is the right place for the scheduler kit to express the channel-receive ⇄ iteration polymorphism. Source: Go-on-SX commit `parse.sx — go/defer/send/range`. - 2026-05-26 — Plan drafted. Phase 0 unstarted. Awaiting Go-on-SX to begin Phase 1.