plans: Go-on-SX + sister lib/guest extraction plans (scheduler, bidirectional types)

- go-on-sx.md: rewrite of 2026-04-26 draft to integrate lib/guest framework. Adds Phase 3 (independent bidirectional type checker — first static-typed guest), Phase 10 (extraction enabler), chisel discipline, conformance scoreboard model. Phases 1-2 now consume lib/guest/core lex+pratt+ast. - lib-guest-scheduler.md: NEW. Extraction plan for the fork/yield/block/ resume scheduler shared by Erlang (addressed processes + mailboxes) and Go (anonymous channels + goroutines). Two-language rule blocks extraction until both consumers independently work; rejected-extraction is a valid outcome. - lib-guest-static-types-bidirectional.md: NEW. Sister to lib/guest/hm.sx. Bidirectional checker kit (synth/check judgments, pluggable subtype + unify) for the languages HM doesn't fit — Go, Rust, TS, Swift, Kotlin, Scala 3, Hack. First consumer: Go-on-SX. Second TBD; recommendation TypeScript. The three plans cross-reference each other. Go-on-SX implements scheduler + checker independently of the kits; extraction is its own workstream once two consumers exist.
2026-05-26 20:54:22 +00:00
parent abde5fbac1
commit 0f7444e0d5
3 changed files with 896 additions and 112 deletions
--- a/plans/lib-guest-scheduler.md
+++ b/plans/lib-guest-scheduler.md
@@ -0,0 +1,235 @@
+# lib/guest/scheduler — extraction plan
+
+Two distinct concurrency models — Erlang's addressed processes + mailboxes, and
+Go's anonymous channels + goroutines — sit on the same underlying machinery:
+a fork/yield/block/resume scheduler over CEK io-suspended continuations. This
+plan captures that machinery as `lib/guest/scheduler/` so language N+1 with a
+new concurrency model costs ~200 lines of model-specific code instead of
+re-inventing the scheduler.
+
+Reference: `plans/lib-guest.md` (parent — two-language rule, stratification),
+`plans/erlang-on-sx.md` (first consumer, in production), Go-on-SX (second
+consumer, see `plans/go-on-sx.md` once that lands).
+
+**Branch:** `architecture`. SX files via `sx-tree` MCP only.
+
+## Thesis
+
+The substrate already provides what a scheduler needs: CEK io-suspension
+(`make-cek-suspended`, `cek-resume`) gives suspendable execution; first-class
+environments give each unit of execution its own scope; the trampolined
+evaluator means we never blow the host stack. What every guest with concurrency
+*re-implements* on top of this is the **fork/yield/block/resume protocol** —
+the bookkeeping that decides which suspended computation runs next.
+
+Two concrete consumers, two different concurrency vocabularies, sharing one
+underlying scheduler, is the proof. If only Erlang lives on it, "scheduler kit"
+is a euphemism for "Erlang scheduler with a Go skin." The two-language rule
+is the gate.
+
+## Current state (2026-05-26)
+
+- **Erlang-on-SX** has the full pattern in production: 729/729 conformance,
+  spawn/send/receive, selective receive, monitor/link, hot reload. The
+  scheduler logic is currently coupled to Erlang-shaped concepts (PIDs,
+  mailboxes, links) — extraction-blocking but not extraction-defeating.
+- **Go-on-SX** does not exist yet. `plans/go-on-sx.md` is the umbrella plan
+  (TBD); this scheduler plan is a sibling/dependency.
+- **lib/guest/scheduler/** does not exist. The two-language rule blocks
+  extraction until Go-on-SX independently implements its scheduler.
+
+**Status: Phase 0 (Erlang shape capture).** No code change in this plan yet.
+
+## Why the two models actually share a kit
+
+The non-obvious claim is that Erlang processes and Go goroutines really do
+share machinery beneath their different vocabularies. The mapping:
+
+| Concept | Erlang | Go | Common kit name |
+|---|---|---|---|
+| Unit of execution | process (PID-addressed) | goroutine (anonymous) | **task** |
+| Spawn | `spawn(Fun)` → PID | `go expr` → nothing | `task-spawn` |
+| Block target | mailbox match | channel send/recv | `task-block` |
+| Wake condition | message arrives | counterpart ready | `task-resume` predicate |
+| Yield | `receive` with no match | channel blocked | scheduler hands off |
+| Termination | exit reason → linked tasks | panic / return | task lifecycle |
+| Selection | selective `receive` | `select` statement | both = "wait for any of N predicates" |
+
+What the kit owns:
+- The **task table** (token → suspended CEK continuation + status).
+- The **runnable queue** + scheduling policy (round-robin v1; pluggable).
+- The **block→resume protocol**: a blocked task registers a predicate; when
+  any task changes state, blocked tasks are re-polled; first whose predicate
+  fires becomes runnable.
+- The **fairness/preemption budget** — gas per step before forced yield.
+
+What each language owns:
+- The semantics layer on top: Erlang's PID→task map + mailbox per task +
+  selective-receive predicates; Go's channel value → blocked-task list per
+  channel + send/recv pairing + select multiplexing.
+- The language-visible API (`spawn`/`!`/`receive` vs `go`/`<-`/`select`).
+
+This is exactly the lib/guest pattern: extract the dispatch skeleton, keep
+the rules in the language layer.
+
+## API surface (proposed — design only, not yet implemented)
+
+```
+(make-scheduler &key gas-per-step ;; default 1000
+                     policy)      ;; :round-robin | :fifo
+  -> scheduler-handle
+
+(task-spawn sched body-thunk) -> task-token
+  ;; body-thunk is a 0-arg fn whose body runs as the task.
+  ;; Returns immediately; task is enqueued runnable.
+
+(task-current sched) -> task-token
+  ;; Inside a task, the token of the running task. Useful for self-reference.
+
+(task-yield sched) -> nil
+  ;; Voluntary yield. Caller is re-enqueued at the tail of runnable.
+
+(task-block sched resume-predicate) -> any
+  ;; Caller suspends. Predicate is (fn () -> resume-value-or-#f).
+  ;; When predicate returns non-#f, caller resumes with that value.
+  ;; Predicate is polled on every scheduler tick when there's nothing
+  ;; obviously runnable. (Optimisation: language layer can wake explicitly —
+  ;; see task-wake.)
+
+(task-wake sched task) -> nil
+  ;; Hint to the scheduler: re-poll this task's resume-predicate now.
+  ;; Used by sender-side when a receiver might unblock.
+
+(task-status sched task) -> :runnable | :blocked | :finished | :crashed
+
+(task-result sched task) -> value | {:error reason}
+  ;; After :finished or :crashed.
+
+(scheduler-step sched) -> :ran | :idle | :all-done
+  ;; Run at most gas-per-step instructions of one task. Caller drives the
+  ;; loop.
+
+(scheduler-run sched) -> nil
+  ;; Run until :all-done. Equivalent to (until (= :all-done (scheduler-step
+  ;; sched))).
+```
+
+Notes on the design:
+- `task-block` with a resume-predicate is the universal blocking primitive.
+  Erlang's `receive` is `(task-block sched (fn () (mailbox-match self pat)))`.
+  Go's `<-ch` is `(task-block sched (fn () (channel-recv-ready ch)))`.
+- `task-wake` is the optimisation: instead of polling every blocked task
+  every step, the language layer wakes the specific task whose predicate
+  is now likely true. v1 can omit it; performance work later.
+- `gas-per-step` gives fairness without true preemption. Tasks that don't
+  yield within their gas budget are force-yielded by the CEK loop. (CEK
+  io-suspension already does this for IO; gas budget extends to plain
+  instructions.)
+- No priority/affinity in v1. Both Erlang and Go default to non-priority
+  scheduling; specialised cases (Erlang's high-priority processes) are
+  language-layer concerns.
+
+## Build order — phases
+
+This is a long-running plan paced against Go-on-SX. Phases are not loop-style
+"one commit per phase" — they're milestone gates.
+
+### Phase 0 — Erlang shape capture (doc-only) ⬜
+- Read `lib/erlang/runtime.sx` scheduler code (currently coupled to Erlang
+  vocabulary).
+- Write a 1-page summary of what's actually a scheduler and what's actually
+  Erlang. Identify the boundary.
+- **Acceptance:** summary committed to this plan as a new section "Erlang
+  scheduler shape (captured 2026-MM-DD)". No code change.
+- **Output:** clear-eyed mental model. Without this, we'll merge Erlang's
+  scheduler shape into the kit and pretend it generalises.
+
+### Phase 1 — Go scheduler independent implementation ⬜
+- During Go-on-SX, implement `lib/go/sched.sx` from scratch. Do NOT look at
+  Erlang's scheduler while doing this. (Or read it once, then close it.)
+- Pass Go's channel + goroutine + select conformance tests.
+- **Acceptance:** Go scheduler green, lib/go/scoreboard.json includes scheduler
+  tests, two-consumer rule now passable.
+- **Output:** two independent, working implementations of the same idea.
+
+### Phase 2 — Diff and proposed kit ⬜
+- Side-by-side diff: Erlang's scheduler vs Go's scheduler. Where do they
+  agree? Where does each have language-specific bookkeeping?
+- The diff is the kit. Things in *both* go in `lib/guest/scheduler/`; things
+  in only one stay in `lib/erlang/` or `lib/go/`.
+- Draft `lib/guest/scheduler/api.sx` (signatures only, no body) reflecting the
+  proposed surface.
+- **Acceptance:** API draft circulated for review; agreement that the surface
+  covers both consumers; no merge yet.
+
+### Phase 3 — Implement `lib/guest/scheduler/` ⬜
+- Implement the kit per the agreed API. New file(s) in `lib/guest/scheduler/`.
+- The kit has its own tests in `lib/guest/scheduler/tests/` — agnostic of any
+  particular language vocabulary.
+- **Acceptance:** kit tests pass. Erlang and Go conformance scoreboards
+  unchanged (the language implementations still use their own scheduler —
+  we haven't refactored yet).
+
+### Phase 4 — Refactor Erlang to use the kit ⬜
+- `lib/erlang/runtime.sx` scheduler logic deleted; replaced with calls into
+  `lib/guest/scheduler/`. Erlang's PID table, mailbox-per-PID, selective
+  receive stay in `lib/erlang/`.
+- **No-regression gate:** Erlang conformance holds at current pass count
+  (currently 729/729). Hard requirement.
+- **Acceptance:** Erlang scoreboard unchanged; `lib/erlang/runtime.sx`
+  meaningfully smaller (the scheduler code is gone).
+
+### Phase 5 — Refactor Go to use the kit ⬜
+- Same exercise for Go. `lib/go/sched.sx` shrinks to channel/goroutine
+  bookkeeping + delegation.
+- **No-regression gate:** Go conformance scoreboard at its current pass
+  count.
+- **Acceptance:** Go scoreboard unchanged; `lib/go/sched.sx` meaningfully
+  smaller.
+
+### Phase 6 — Documentation + design-diary close ⬜
+- Document `lib/guest/scheduler/` API in `lib/guest/README.md` (or wherever
+  the lib/guest API index lives).
+- Capture the chiselling diary: what *almost* went in the kit but ended up
+  language-specific, and why. This is the load-bearing knowledge for the
+  third consumer when it arrives.
+- **Acceptance:** API documented; diary section added to this plan.
+
+## Two-language rule — gating
+
+**The rule is hard.** No code in `lib/guest/scheduler/` lands until BOTH
+Phase 1 (Go independent) AND Phase 0 (Erlang capture) are complete AND a
+review confirms the two implementations actually share machinery in a way
+the kit captures.
+
+If, during Phase 2 diff, we discover that the agreement is shallow (e.g.,
+both have a runnable queue but the policies are fundamentally incompatible),
+the **right outcome is to NOT extract**. Add a "rejected extraction" note to
+this plan documenting what we learned and close it. That outcome is fine —
+it tells us the two concurrency models aren't actually sister, which is a
+real result.
+
+## Open questions
+
+- **Preemption.** v1 is cooperative; gas-per-step gives fairness but not
+  hard preemption. Erlang BEAM does true preemption (reduction counting).
+  Go uses async preemption (signal-driven since 1.14). Neither extreme fits
+  cooperatively over CEK. Is gas-per-step + voluntary yield enough? Probably
+  for v1; revisit if a guest needs hard real-time.
+- **Priority/affinity.** Both Erlang and Go can run without it. Defer.
+- **Distribution.** Erlang nodes, Go's distributed channels — both are
+  language-specific layers on top of the local scheduler. Out of scope.
+- **Cancellation.** Go has `context.Context`; Erlang has `exit/2`. Both
+  bottom out at "deliver an exception to a task." Worth modelling? Probably
+  as a kit primitive `(task-cancel sched task reason)` that delivers an
+  exception via CEK exception machinery, language layer wraps it.
+- **Third consumer.** If/when JS-on-SX gets a proper async/await + Promise
+  scheduler, that'd be a great third consumer to validate the kit didn't
+  over-fit to Erlang+Go.
+
+## Progress log
+
+_Newest first. Append one dated entry per milestone landed._
+
+- 2026-05-26 — Plan drafted. Phase 0 unstarted. Awaiting Go-on-SX to begin
+  Phase 1.