plans: Go-on-SX + sister lib/guest extraction plans (scheduler, bidirectional types)
- go-on-sx.md: rewrite of 2026-04-26 draft to integrate lib/guest framework. Adds Phase 3 (independent bidirectional type checker — first static-typed guest), Phase 10 (extraction enabler), chisel discipline, conformance scoreboard model. Phases 1-2 now consume lib/guest/core lex+pratt+ast. - lib-guest-scheduler.md: NEW. Extraction plan for the fork/yield/block/ resume scheduler shared by Erlang (addressed processes + mailboxes) and Go (anonymous channels + goroutines). Two-language rule blocks extraction until both consumers independently work; rejected-extraction is a valid outcome. - lib-guest-static-types-bidirectional.md: NEW. Sister to lib/guest/hm.sx. Bidirectional checker kit (synth/check judgments, pluggable subtype + unify) for the languages HM doesn't fit — Go, Rust, TS, Swift, Kotlin, Scala 3, Hack. First consumer: Go-on-SX. Second TBD; recommendation TypeScript. The three plans cross-reference each other. Go-on-SX implements scheduler + checker independently of the kits; extraction is its own workstream once two consumers exist.
This commit is contained in:
@@ -1,24 +1,64 @@
|
||||
# Go-on-SX: Go on the CEK/VM
|
||||
# Go-on-SX — Go as an SX guest language
|
||||
|
||||
Compile Go source to SX AST; the existing CEK evaluator runs it. The unique angle: Go's
|
||||
goroutines and channels map cleanly onto SX's IO suspension machinery (`perform`/`cek-resume`)
|
||||
— a goroutine is a `cek-step-loop` running in a cooperative scheduler, a channel send/receive
|
||||
is a `perform` that suspends until the other end is ready.
|
||||
Port Go to SX as the **first static-typed, bidirectional-checked guest** in
|
||||
the rose-ash language family. Goal isn't a production Go compiler; it's to
|
||||
prove the substrate from a paradigm angle the existing eleven guests don't
|
||||
cover, and to chisel out the lib/guest kits that statically-typed guests N+1
|
||||
and N+2 will need.
|
||||
|
||||
End-state goal: **core Go programs running**, including goroutines, channels, defer/panic/recover,
|
||||
interfaces, and structs. Not a full Go compiler — no generics, no CGo, no full stdlib — but
|
||||
a faithful runtime for idiomatic Go concurrent programs.
|
||||
Reference:
|
||||
- `plans/lib-guest.md` — parent, chiselling discipline, two-language rule.
|
||||
- `plans/lib-guest-scheduler.md` — sister kit; Go's scheduler pairs with
|
||||
Erlang's. Extraction gated on this loop reaching Phase 5.
|
||||
- `plans/lib-guest-static-types-bidirectional.md` — sister kit; Go's
|
||||
checker pairs with a TBD second consumer. Extraction gated on this loop
|
||||
reaching Phase 3.
|
||||
- `plans/erlang-on-sx.md` — reference implementation for paradigm-port:
|
||||
process model, BIF registry, hot reload, VM bytecode opcodes.
|
||||
|
||||
## Ground rules
|
||||
**Branch:** `loops/go` (loop-style workstream once kicked off). SX files via
|
||||
`sx-tree` MCP only.
|
||||
|
||||
- **Scope:** only touch `lib/go/**` and `plans/go-on-sx.md`. Do **not** edit `spec/`,
|
||||
`hosts/`, `shared/`, or other `lib/<lang>/`.
|
||||
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
|
||||
- **SX files:** use `sx-tree` MCP tools only.
|
||||
- **Architecture:** Go source → Go AST → SX AST. No standalone Go evaluator.
|
||||
- **Concurrency model:** cooperative, not preemptive. Goroutines yield at channel ops and
|
||||
`time.Sleep`. A round-robin scheduler in SX drives them.
|
||||
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
|
||||
## Thesis — why Go
|
||||
|
||||
Eleven guests already live in `lib/`: apl, common-lisp, datalog, erlang,
|
||||
forth, haskell, hyperscript, js, kernel, lua, minikanren, ocaml, prolog,
|
||||
ruby, scheme, smalltalk, tcl. Every one is either **dynamically typed**
|
||||
(most) or **HM-inferred** (haskell, ocaml). None exercise:
|
||||
|
||||
1. **Bidirectional static type checking** — annotation-driven, locally-
|
||||
inferred, the dominant paradigm of modern statically-typed languages.
|
||||
2. **Anonymous-channel concurrency** — Go's `chan` and `select`. Erlang has
|
||||
addressed processes + mailboxes; Go has anonymous values + structural
|
||||
pairing. Two different vocabularies for the same underlying scheduler
|
||||
machinery.
|
||||
3. **Structural interfaces** — `io.Reader` is "anything with this method
|
||||
signature", not a declared subtype relationship. Different from Haskell
|
||||
typeclasses (nominal), different from Lua duck typing (no declaration).
|
||||
|
||||
These three together make Go an unusually high-value port for proving SX.
|
||||
If SX can host Go cleanly, it can host the next decade of mainstream
|
||||
statically-typed languages (Rust, TS, Swift, Kotlin, Scala 3, Hack) because
|
||||
they share these three properties.
|
||||
|
||||
Like Erlang-on-SX validated the actor model on the substrate, Go-on-SX
|
||||
validates the goroutine model + bidirectional types.
|
||||
|
||||
## Non-goals (deliberate)
|
||||
|
||||
Out of scope. Reject feature requests for these without further consideration:
|
||||
|
||||
- **`unsafe` package.** Memory mucking. Skip entirely.
|
||||
- **CGo.** C interop. Out of scope at every level.
|
||||
- **Full `reflect`.** Provide enough for `fmt.Println` to render values;
|
||||
reject the rest.
|
||||
- **Build tags, modules, vendoring.** Treat source as monolithic. One
|
||||
package per file, no real import resolution.
|
||||
- **Production performance.** Conformance tests pass; benchmarks don't.
|
||||
- **Garbage collection tuning.** SX's GC is what you get.
|
||||
- **Race detector, escape analysis, inlining.** Out of scope.
|
||||
- **`os`, `net/http`, full stdlib.** Provide a deliberately small slice
|
||||
(Phase 8 below).
|
||||
|
||||
## Architecture sketch
|
||||
|
||||
@@ -26,113 +66,330 @@ a faithful runtime for idiomatic Go concurrent programs.
|
||||
Go source text
|
||||
│
|
||||
▼
|
||||
lib/go/tokenizer.sx — Go tokens: keywords, idents, string/rune/number literals,
|
||||
│ operators, semicolon insertion rules
|
||||
lib/go/lex.sx — tokens; ASI; literals; operators
|
||||
│ (consumes lib/guest/core/lex.sx)
|
||||
▼
|
||||
lib/go/parser.sx — Go AST: package, import, var, const, type, func, struct,
|
||||
│ interface, goroutine, channel ops, defer, select, for range
|
||||
lib/go/parse.sx — AST: package/import/var/const/type/func/struct/
|
||||
│ interface; expressions; statements
|
||||
│ (consumes lib/guest/core/pratt.sx + ast.sx)
|
||||
▼
|
||||
lib/go/transpile.sx — Go AST → SX AST
|
||||
│
|
||||
lib/go/types.sx — bidirectional type checker. Synth + check judgments;
|
||||
│ structural interface satisfaction; pluggable subtype
|
||||
│ (INDEPENDENT — no lib/guest/static-types-bidirectional
|
||||
│ yet; this loop builds the first consumer)
|
||||
▼
|
||||
lib/go/runtime.sx — goroutine scheduler, channel primitives, defer stack,
|
||||
│ panic/recover, interface dispatch, slice/map ops
|
||||
lib/go/eval.sx — tree-walk evaluator on CEK. Variables as mutable cells;
|
||||
│ slices = (length, capacity, backing-vector); maps =
|
||||
│ SX dict; defer stack per frame.
|
||||
▼
|
||||
CEK / VM
|
||||
lib/go/sched.sx — goroutine scheduler + channels + select
|
||||
│ (INDEPENDENT — no lib/guest/scheduler yet; this loop
|
||||
│ builds the first consumer)
|
||||
▼
|
||||
lib/go/std/ — minimal stdlib slice (fmt, strings, strconv, sync,
|
||||
time, errors)
|
||||
```
|
||||
|
||||
Key semantic mappings:
|
||||
- `go fn()` → spawn new coroutine (SX coroutine primitive, Phase 4 of primitives)
|
||||
- `ch <- v` (send) → `perform` that suspends until receiver ready; scheduler picks next goroutine
|
||||
- `v := <-ch` (receive) → `perform` that suspends until sender ready
|
||||
- `select { case ... }` → scheduler checks all channel readiness, picks first ready
|
||||
- `defer fn()` → push onto a per-goroutine defer stack; run on return/panic
|
||||
- `panic(v)` → `raise` the value; `recover()` catches it in deferred function
|
||||
- `interface{}` → any SX value (duck typed)
|
||||
- `struct { ... }` → SX hash table with field names as keys
|
||||
- `slice` → SX vector with length + capacity metadata
|
||||
- `map[K]V` → SX mutable hash table (Phase 10 of primitives)
|
||||
Semantic mappings (operational):
|
||||
- `go fn(args)` → `task-spawn` on the local scheduler.
|
||||
- `ch <- v` → `task-block` with predicate "receiver waiting on ch".
|
||||
- `v := <-ch` → `task-block` with predicate "sender waiting on ch".
|
||||
- `select { case ... }` → `task-block` with predicate "any case ready".
|
||||
- `defer fn()` → push thunk onto per-frame defer stack; runs LIFO on
|
||||
return or panic.
|
||||
- `panic(v)` → raise SX exception; deferred fns run while unwinding.
|
||||
- `recover()` → CEK exception capture inside a deferred fn.
|
||||
- `interface{T}` → type-check matches structurally against T's method
|
||||
set; at runtime, the value carries its concrete-type metadata.
|
||||
- `struct{...}` → SX dict + type tag; methods are functions in the type's
|
||||
method table.
|
||||
- `*T` (pointer) → mutable cell (Common Lisp port did the same).
|
||||
- `[]T` (slice) → triple (length, capacity, backing-vector).
|
||||
- `map[K]V` → SX dict; iteration order spec-undefined (v1 = sorted for
|
||||
determinism — programs that depend on indeterminism fail loudly, which
|
||||
is a feature not a bug).
|
||||
|
||||
## Roadmap
|
||||
## Conformance scoreboard
|
||||
|
||||
### Phase 1 — tokenizer + parser
|
||||
- [ ] Tokenizer: keywords (`package`, `import`, `func`, `var`, `const`, `type`, `struct`,
|
||||
`interface`, `go`, `chan`, `select`, `defer`, `return`, `if`, `else`, `for`, `range`,
|
||||
`switch`, `case`, `default`, `break`, `continue`, `goto`, `fallthrough`, `map`,
|
||||
`make`, `new`, `nil`, `true`, `false`), automatic semicolon insertion, string literals
|
||||
(interpreted + raw `` `...` ``), rune literals `'a'`, number literals (int, float, hex,
|
||||
octal, binary, complex), operators, slices `[:]`
|
||||
- [ ] Parser: package clause, imports, top-level `func`/`var`/`const`/`type`; function
|
||||
bodies: short variable decl `:=`, assignments, `if`/`else`, `for`/`range`, `switch`,
|
||||
`return`, struct literals, slice literals, map literals, composite literals, type
|
||||
assertions `v.(T)`, method calls `v.Method(args)`, goroutine `go`, channel ops
|
||||
`<-ch`, `ch <- v`, `defer`, `select`
|
||||
- [ ] Tests in `lib/go/tests/parse.sx`
|
||||
Following `lib/erlang/scoreboard.json` precedent. Add
|
||||
`lib/go/scoreboard.json` on first iteration; populate as suites land.
|
||||
Suites planned:
|
||||
|
||||
### Phase 2 — transpile: basic Go (no goroutines)
|
||||
- [ ] `go-eval-ast` entry
|
||||
- [ ] Arithmetic, string ops, comparison, boolean
|
||||
- [ ] Variables, short decl, assignment, multiple assignment
|
||||
- [ ] `if`/`else if`/`else`
|
||||
- [ ] `for` (C-style), `for range` over slice/map/string
|
||||
- [ ] Functions: named + anonymous, multiple return values (SX multiple values, Phase 8)
|
||||
- [ ] Structs → SX hash tables; field access `.field`; struct literals `T{f: v}`
|
||||
- [ ] Slices → SX vectors; `len`, `cap`, `append`, `copy`, slice expressions `s[a:b]`
|
||||
- [ ] Maps → SX hash tables; `make(map[K]V)`, `m[k]`, `m[k] = v`, `delete(m, k)`,
|
||||
comma-ok `v, ok := m[k]`
|
||||
- [ ] Pointers — modelled as single-element mutable vectors; `&x` creates wrapper, `*p` dereferences
|
||||
- [ ] `fmt.Println`/`fmt.Printf`/`fmt.Sprintf` → SX IO perform (print)
|
||||
- [ ] 40+ eval tests in `lib/go/tests/eval.sx`
|
||||
| Suite | Tests target | What it covers |
|
||||
|---|---|---|
|
||||
| `lex` | 50+ | Keywords, operators, literals, ASI |
|
||||
| `parse` | 80+ | All statement & expression shapes |
|
||||
| `types` | 90+ | Synth, check, interface satisfaction, generics |
|
||||
| `eval` | 100+ | Tree-walk over typed AST |
|
||||
| `runtime` | 60+ | Goroutines, channels, select, close |
|
||||
| `stdlib` | 40+ | fmt, strings, strconv, sync, time, errors |
|
||||
| `e2e` | 10+ | Complete representative programs |
|
||||
|
||||
### Phase 3 — defer / panic / recover
|
||||
- [ ] Defer stack per function frame — SX list of thunks, run LIFO on return
|
||||
- [ ] `defer` statement pushes thunk; transpiler wraps function body in try/finally equivalent
|
||||
- [ ] `panic(v)` → `raise` with Go panic wrapper
|
||||
- [ ] `recover()` → catches panic value inside a deferred function; returns nil otherwise
|
||||
- [ ] Panic propagation across call stack until recovered or fatal
|
||||
- [ ] Tests: defer ordering, panic/recover, panic in goroutine without recover
|
||||
## Phasing — one feature per commit
|
||||
|
||||
### Phase 4 — goroutines + channels
|
||||
- [ ] Coroutine-based goroutine type using SX coroutine primitive (Phase 4 of primitives)
|
||||
- [ ] Round-robin scheduler in `lib/go/runtime.sx`: maintains run queue, steps each
|
||||
goroutine one turn at a time, suspends at channel ops
|
||||
- [ ] Unbuffered channels: `make(chan T)` → rendezvous point; send suspends until receive
|
||||
and vice versa. Implemented as a pair of waiting queues + `cek-resume`.
|
||||
- [ ] Buffered channels: `make(chan T, n)` → circular buffer; send only blocks when full,
|
||||
receive only blocks when empty
|
||||
- [ ] `close(ch)` — mark channel closed; receivers drain then get zero value + `false`
|
||||
- [ ] `select` — scheduler inspects all cases, picks a ready one (random if multiple),
|
||||
blocks if none ready until at least one becomes ready
|
||||
- [ ] `go fn(args)` — spawns new goroutine on run queue
|
||||
- [ ] `time.Sleep(d)` — yields current goroutine, re-queues after d milliseconds
|
||||
(simulated with IO perform timer)
|
||||
- [ ] Tests: ping-pong, fan-out, fan-in, select with default, range over channel
|
||||
Loop-style. Each phase: implement → test → commit → tick `[ ]` → append
|
||||
Progress-log line → push `origin/loops/go`.
|
||||
|
||||
### Phase 5 — interfaces
|
||||
- [ ] Interface type → SX dict `{:type "T" :methods {...}}` dispatch table
|
||||
- [ ] `interface{}` / `any` → any SX value (already implicit)
|
||||
- [ ] Type assertion `v.(T)` → check `:type` field, panic if mismatch
|
||||
- [ ] Type switch `switch v.(type) { case T: ... }` → dispatches on `:type`
|
||||
- [ ] Method sets — structs implement interfaces implicitly if they have the right methods
|
||||
- [ ] Value vs pointer receivers — pointer receiver gets the mutable vector wrapper
|
||||
- [ ] Built-in interfaces: `error` (`Error() string`), `Stringer` (`String() string`)
|
||||
- [ ] Tests: interface satisfaction, type assertion, type switch, error interface
|
||||
### Phase 1 — Tokenizer (`lib/go/lex.sx`) ⬜
|
||||
- Consume `lib/guest/core/lex.sx`. Tag the chisel note `consumes-lex`.
|
||||
- Keywords (25), operators + punctuation (47 distinct), identifiers,
|
||||
literals (int / float / imaginary / rune / string with raw + interpreted
|
||||
variants), comments.
|
||||
- **Automatic semicolon insertion** — the one tricky bit. Newline becomes
|
||||
`;` after identifier/literal/`)`/`]`/`}` per Go spec § Semicolons. Build
|
||||
it into the tokenizer, not the parser.
|
||||
- Tests: golden-token streams for every keyword/operator/literal kind +
|
||||
ASI edge cases.
|
||||
- **Acceptance:** lex/ suite at 50+ tests.
|
||||
|
||||
### Phase 6 — standard library subset
|
||||
- [ ] `fmt` — `Println`, `Printf`, `Sprintf`, `Fprintf`, `Errorf`, `Stringer` dispatch
|
||||
- [ ] `strings` — `Contains`, `HasPrefix`, `HasSuffix`, `Split`, `Join`, `TrimSpace`,
|
||||
`ToUpper`, `ToLower`, `Replace`, `Index`, `Count`, `Repeat`
|
||||
- [ ] `strconv` — `Itoa`, `Atoi`, `FormatFloat`, `ParseFloat`, `ParseInt`, `FormatInt`
|
||||
- [ ] `math` — full surface via SX math primitives (Phase 15)
|
||||
- [ ] `sort` — `sort.Slice`, `sort.Ints`, `sort.Strings`
|
||||
- [ ] `errors` — `errors.New`, `errors.Is`, `errors.As`
|
||||
- [ ] `sync` — `sync.Mutex` (cooperative — just a boolean flag + goroutine queue),
|
||||
`sync.WaitGroup`, `sync.Once`
|
||||
- [ ] `io` — `io.Reader`/`io.Writer` interfaces; `io.ReadAll`; `strings.NewReader`
|
||||
### Phase 2 — Parser (`lib/go/parse.sx`) ⬜
|
||||
- Consume `lib/guest/core/pratt.sx` + `lib/guest/core/ast.sx`. Chisel notes
|
||||
`consumes-pratt consumes-ast`.
|
||||
- Grammar coverage:
|
||||
- Declarations: `package`, `import`, `var`, `const`, `type`, `func`
|
||||
- Types: basic, slice `[]T`, array `[N]T`, map `map[K]V`, chan `chan T`,
|
||||
func `func(...)...`, struct, interface, pointer `*T`
|
||||
- Expressions: literals, identifier, call, index `[]`, slice `[a:b]`,
|
||||
type assertion `v.(T)`, operators
|
||||
- Statements: `if`/`else`, `for` (C-style + range), `switch`, `select`,
|
||||
`return`, `defer`, `go`, `break`/`continue`, assign, short-decl `:=`,
|
||||
send `ch <- v`, recv `<-ch`
|
||||
- Output: SX-shaped AST per `lib/guest/core/ast.sx` conventions.
|
||||
- Tests: round-trip parse of hello world, fibonacci, FizzBuzz, goroutine
|
||||
ping-pong, struct + method.
|
||||
- **Acceptance:** parse/ suite at 80+ tests.
|
||||
|
||||
### Phase 7 — full conformance target
|
||||
- [ ] Vendor a Go test suite or hand-build 100+ program tests in `lib/go/tests/programs/`
|
||||
- [ ] Drive scoreboard
|
||||
### Phase 3 — Bidirectional type checker, MVP (`lib/go/types.sx`) ⬜
|
||||
- **Independent implementation.** Do NOT use lib/guest/static-types-
|
||||
bidirectional/ — that kit doesn't exist yet and depends on this work
|
||||
for its design. See `plans/lib-guest-static-types-bidirectional.md`.
|
||||
- Synth + check judgments. Context as a value (per-block scope).
|
||||
- Coverage MVP: declared-type variables, function signatures (params +
|
||||
returns), call type-checking, simple composite types (slice, map, chan
|
||||
element), interface satisfaction (structural match against method sets),
|
||||
short variable declaration `:=` (synth from RHS).
|
||||
- **Untyped constants.** `42` has type `untyped int` until contextualised;
|
||||
this is the canonical pitfall (see Gotchas below).
|
||||
- Defer: generics (Phase 7), full conversion rules.
|
||||
- Tests: positive (type-correct programs check) + negative (mismatched
|
||||
types fail with informative errors carrying AST paths).
|
||||
- **Acceptance:** types/ suite at 60+ tests. Chisel note `shapes-static-
|
||||
types-bidirectional` — append a paragraph to the sister plan's design
|
||||
diary describing what synth/check shape emerged.
|
||||
|
||||
### Phase 4 — Tree-walk evaluator (`lib/go/eval.sx`) ⬜
|
||||
- AST-walking interpreter over CEK. Each Go statement maps to one step
|
||||
function (precedent: `step-sf-if` etc. in spec/evaluator.sx).
|
||||
- Variables: mutable cells. Pointer semantics: `&x` returns the cell,
|
||||
`*p` dereferences.
|
||||
- Slices: triple (length, capacity, backing-vector). `append` honours
|
||||
capacity-grow per spec.
|
||||
- Maps: SX dict + key-type metadata.
|
||||
- Structs: SX dict + type tag. Methods looked up via type's method table.
|
||||
- Functions: closures over enclosing scope; multiple return values.
|
||||
- Channels: stub (Phase 5 wires them).
|
||||
- Tests: arithmetic, control flow, recursion, closures, slices, maps,
|
||||
structs, methods, pointer semantics, multiple-return.
|
||||
- **Acceptance:** eval/ suite at 80+ tests. No concurrency yet.
|
||||
|
||||
### Phase 5 — Goroutines + channels + select (`lib/go/sched.sx`) ⬜
|
||||
- **Independent implementation.** Do NOT use lib/guest/scheduler/ — that
|
||||
kit doesn't exist yet and depends on this work for its design. See
|
||||
`plans/lib-guest-scheduler.md`.
|
||||
- `go expr` — spawn a goroutine; returns nothing.
|
||||
- `chan T` — `make(chan T)` creates an unbuffered channel; `make(chan T,n)`
|
||||
creates a buffered channel (Phase 5b — defer buffer to a sub-phase).
|
||||
- `<-ch` — receive (blocks until sender ready).
|
||||
- `ch <- v` — send (blocks until receiver ready for unbuffered, or buffer
|
||||
has room for buffered).
|
||||
- `select { case ... }` — non-deterministic multiplexing; `default` makes
|
||||
it non-blocking.
|
||||
- `close(ch)` — closes channel. Receive on closed → zero value + ok=false.
|
||||
- Tests: ping-pong, fan-out/fan-in, work queue, select with default,
|
||||
select with timeout (via a `time.After`-like stub), close semantics,
|
||||
range over channel.
|
||||
- **Acceptance:** runtime/ suite at 40+ tests. Chisel note `shapes-
|
||||
scheduler` — append a paragraph to the sister plan's design diary
|
||||
describing what task-spawn/block/wake/yield shape emerged.
|
||||
|
||||
### Phase 5b — Buffered channels + select fairness ⬜
|
||||
- Buffered: send blocks only when buffer full; recv only when empty.
|
||||
- `select` random case ordering (spec mandates pseudo-random; v1 uses a
|
||||
fixed seed for determinism with a `runtime`-package knob to randomise).
|
||||
- Tests: buffer-full blocking, buffer-empty blocking, select fairness
|
||||
over many iterations.
|
||||
- **Acceptance:** runtime/ +20 tests.
|
||||
|
||||
### Phase 6 — `defer` + panic/recover ⬜
|
||||
- Defer stack per function frame; runs LIFO on return (normal or panic).
|
||||
- `panic(v)` unwinds frames running deferreds; `recover()` inside a
|
||||
deferred fn captures the panic value and stops unwinding.
|
||||
- Goroutine panic propagation: a panicking goroutine that doesn't recover
|
||||
crashes the whole program (honour Go spec, or document divergence).
|
||||
- Tests: defer order (LIFO), defer + named-return mutation, panic/recover,
|
||||
panic across goroutines, defer in a loop (push per iter, run on fn
|
||||
return — common bug).
|
||||
- **Acceptance:** eval/ +20 tests.
|
||||
|
||||
### Phase 7 — Generics (Go 1.18+) ⬜
|
||||
- Type parameters with constraints (type sets: `interface{ int | float64
|
||||
}`, `comparable`, `any`).
|
||||
- Type inference at call sites — basic; the full Go inference algorithm
|
||||
is notoriously complex. Implement enough for common cases; document
|
||||
limitations in a Blockers section below.
|
||||
- Tests: generic function (`func Map[T, U any](xs []T, f func(T) U) []U`),
|
||||
generic data structure (linked list), constrained type param.
|
||||
- **Acceptance:** types/ +30 tests.
|
||||
|
||||
### Phase 8 — Minimal stdlib (`lib/go/std/`) ⬜
|
||||
- Implement just what's needed for representative programs:
|
||||
- `fmt` — `Println`, `Printf`, `Sprintf`, `Fprintf`, `Errorf`,
|
||||
`Stringer` dispatch. Verbs: `%d %s %v %t %f %T %+v`.
|
||||
- `strings` — `Contains`, `HasPrefix`, `HasSuffix`, `Split`, `Join`,
|
||||
`TrimSpace`, `ToUpper`, `ToLower`, `Replace`, `Index`, `Count`,
|
||||
`Repeat`, `NewReader`.
|
||||
- `strconv` — `Itoa`, `Atoi`, `FormatFloat`, `ParseFloat`, `ParseInt`,
|
||||
`FormatInt`.
|
||||
- `errors` — `New`, `Is`, `As`, `Unwrap`.
|
||||
- `sync` — `Mutex` (cooperative — flag + waiter queue), `WaitGroup`,
|
||||
`Once`, `RWMutex`.
|
||||
- `time` — `Now`, `Since`, `After` (channel-returning timer), `Sleep`,
|
||||
`Duration`, `Time`.
|
||||
- `io` — `Reader`/`Writer` interfaces; `ReadAll`; `Copy`.
|
||||
- `sort` — `Slice`, `Ints`, `Strings`.
|
||||
- Tests: round-trip Itoa/Atoi, fmt verb coverage, sync.WaitGroup with
|
||||
goroutines, time.After in a select, sort.Slice with custom less fn.
|
||||
- **Acceptance:** stdlib/ suite at 40+ tests.
|
||||
|
||||
### Phase 9 — End-to-end programs ⬜
|
||||
- Complete programs from canonical sources (gopl.io, "concurrency
|
||||
patterns" talk examples) running end-to-end:
|
||||
- Concurrent prime sieve
|
||||
- HTTP-ish ping-pong over stubbed transport
|
||||
- Word frequency counter
|
||||
- Pipeline (channel chain)
|
||||
- Producer/consumer with sync.WaitGroup
|
||||
- "Bounded parallelism" pattern (worker pool over a job channel)
|
||||
- **Acceptance:** e2e/ suite at 10+ tests, all passing.
|
||||
|
||||
### Phase 10 — lib/guest extraction enabler ⬜
|
||||
- Now that Go has lex+parse+types+eval+sched, sister plans are unblocked
|
||||
on the Go side. This phase is **doc-only** in `loops/go`:
|
||||
- Cross-reference `plans/lib-guest-scheduler.md` — mark its Phase 1
|
||||
(Go scheduler independent) as complete from Go's side.
|
||||
- Cross-reference `plans/lib-guest-static-types-bidirectional.md` —
|
||||
mark its Phase 1 as complete from Go's side.
|
||||
- Update the chiselling diary in each sister plan with the actual
|
||||
Go-side surface that emerged.
|
||||
- **Acceptance:** sister plans cross-referenced + diaries updated. No
|
||||
new Go code.
|
||||
|
||||
### Phase 11 — VM bytecode opcodes (deferred, optional) ⬜
|
||||
- Following Erlang-on-SX Phase 10 precedent: identify hot paths in the
|
||||
tree-walk evaluator, define Go-specific bytecode opcodes, compile hot
|
||||
fns through them. Substantial work; only justified if Go programs
|
||||
exercise enough volume that performance starts mattering.
|
||||
- **Acceptance:** TBD on demand.
|
||||
|
||||
## Ground rules (loop-style)
|
||||
|
||||
- **Scope:** only `lib/go/**` and this plan. Do not touch `spec/`,
|
||||
`hosts/`, `shared/`, `lib/guest/**` (read-only consumer at this phase),
|
||||
or other `lib/<lang>/`.
|
||||
- **Consume `lib/guest/core/`** for lex/parse/ast/match/layout. Hand-
|
||||
rolling defeats the chiselling goal.
|
||||
- **Do NOT extract into `lib/guest/scheduler/` or `lib/guest/static-
|
||||
types-bidirectional/` from this loop.** Those extractions are gated on
|
||||
two consumers AND the discipline of writing each consumer
|
||||
independently. Extraction is its own workstream after Go and the
|
||||
second consumer both exist.
|
||||
- **Substrate gaps** → Blockers entry with minimal repro. Don't fix the
|
||||
substrate from this loop. Belongs to `sx-improvements.md`.
|
||||
- **NEVER call `sx_build` without timeout awareness** — 600s watchdog.
|
||||
- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after every edit.
|
||||
- **Worktree:** branch `loops/go`, push `origin/loops/go`. Never `main`,
|
||||
never `architecture`.
|
||||
- **Commit granularity:** one feature per commit. Short factual messages:
|
||||
`go: parse short-decl + 6 tests [consumes-pratt]`. Chisel note at end
|
||||
in brackets.
|
||||
- **Plan file:** update Progress log + tick boxes every commit.
|
||||
- **If blocked** for two iterations on the same issue, add to Blockers
|
||||
and move on. Phases 1-4 are sequential; Phases 5-8 are largely
|
||||
independent once 4 lands.
|
||||
|
||||
## Chisel discipline (per parent lib-guest plan)
|
||||
|
||||
Every commit ends its message with a chisel note in brackets:
|
||||
|
||||
- `[consumes-X]` — used `lib/guest/X` kit.
|
||||
- `[shapes-scheduler]` / `[shapes-static-types-bidirectional]` — revealed
|
||||
something about what the sister lib-guest kits should look like. Add a
|
||||
paragraph to the relevant sister plan's design diary.
|
||||
- `[proposes-Y]` — revealed a gap in another existing kit. Blockers entry
|
||||
in the kit's plan.
|
||||
- `[nothing]` — pure Go work that didn't touch substrate or lib/guest
|
||||
story. Acceptable; if it shows up twice in a row, stop and reflect.
|
||||
|
||||
## Go-specific gotchas
|
||||
|
||||
- **ASI (automatic semicolon insertion).** Newline becomes `;` after
|
||||
identifier/literal/`)`/`]`/`}`. Build into the tokenizer; the Go spec's
|
||||
"Semicolons" section is unusually precise — follow it literally.
|
||||
- **Untyped constants.** `42` has type `untyped int` until used in a
|
||||
context that forces a type. The canonical example: `var x float64 = 42
|
||||
/ 7` — must compute as `untyped int / untyped int = 6` then convert to
|
||||
`float64 = 6.0`. Wrong: float-coercing eagerly gives 6.0 prematurely.
|
||||
Wrong: integer-truncating after coercion gives `5.something`. Test it.
|
||||
- **Methods vs functions.** `func (r Receiver) Method()` is a method
|
||||
bound to a type; `func Function(r Receiver)` is just a function.
|
||||
Methods on pointer-receivers vs value-receivers have asymmetric
|
||||
satisfaction in interfaces — pointer-receiver methods are NOT in the
|
||||
value's method set for interface satisfaction.
|
||||
- **Interface satisfaction is structural and silent.** Type satisfies an
|
||||
interface if its method set contains all the interface's methods.
|
||||
Lazy check: at every point a value flows into an interface-typed slot.
|
||||
- **Channels are first-class values.** Pass them, store them, send them
|
||||
through other channels. Each channel has identity.
|
||||
- **`select` with `default`** = non-blocking. Without `default`, blocks
|
||||
until a case is ready.
|
||||
- **`nil` is typed.** `var x *int` makes x a `(*int)(nil)`. Comparison
|
||||
`x == nil` works on typed nil; but `var i interface{} = (*int)(nil); i
|
||||
== nil` is `false` — i holds a typed-nil-of-type-`*int`, not untyped
|
||||
nil. The classic Go footgun. Test it.
|
||||
- **Goroutine panic propagation.** A panicking goroutine that doesn't
|
||||
recover crashes the whole program. Implement faithfully or document
|
||||
divergence.
|
||||
- **`defer` in a loop.** Each iteration pushes; they all run on function
|
||||
return. Common bug; tests should cover.
|
||||
- **Iteration order of maps.** Spec: unspecified. v1 = sorted by SX-
|
||||
canonical key order for determinism; document that programs depending
|
||||
on iteration order are not Go-conformant. Add a `runtime`-package knob
|
||||
to enable randomisation later.
|
||||
|
||||
## Style
|
||||
|
||||
- No comments in `.sx` unless non-obvious. Cite Go spec sections inline
|
||||
for non-obvious decisions (Go's spec is rigorous; citations work).
|
||||
- No new planning docs — update this plan inline.
|
||||
- One feature per iteration. Commit. Log. Push. Next.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Module/import model.** Go has packages and import paths. Probably
|
||||
model "package" as one or more `.sx` files in a directory, no real
|
||||
import resolution against a remote module graph. Decide in Phase 2.
|
||||
2. **Goroutine identity.** Spec says goroutines have no identity; the
|
||||
scheduler does internally. Expose to user code? No (not Go). Expose
|
||||
for debugging? Yes via a `runtime`-package stub.
|
||||
3. **Error handling: panic-as-exception vs explicit error returns.** Go
|
||||
strongly prefers explicit errors. Stdlib stubs follow that: `strconv.
|
||||
Atoi("x")` returns `(0, err)`, not panic.
|
||||
4. **Memory model.** Go has a happens-before model for atomics + channel
|
||||
ops. SX runtime is single-threaded under the scheduler — every channel
|
||||
op is a synchronization point automatically. Don't model relaxed
|
||||
memory; document the simplification.
|
||||
5. **Iteration order of maps.** Already addressed in Gotchas; flagged
|
||||
here as a known divergence from spec.
|
||||
|
||||
## Blockers
|
||||
|
||||
@@ -140,6 +397,11 @@ _(none yet)_
|
||||
|
||||
## Progress log
|
||||
|
||||
_Newest first._
|
||||
_Newest first. Append one dated entry per commit._
|
||||
|
||||
_(awaiting phase 1)_
|
||||
- 2026-05-26 — Plan rewritten to integrate the lib/guest framework
|
||||
(chiselling discipline, sister plans for scheduler + bidirectional
|
||||
types, type-checker phase added, conformance scoreboard model adopted).
|
||||
Original 2026-04-26 draft preserved in git history. Loop not yet
|
||||
kicked off; Phase 1 (tokenizer) is the first iteration when this loop
|
||||
spins up.
|
||||
|
||||
235
plans/lib-guest-scheduler.md
Normal file
235
plans/lib-guest-scheduler.md
Normal file
@@ -0,0 +1,235 @@
|
||||
# lib/guest/scheduler — extraction plan
|
||||
|
||||
Two distinct concurrency models — Erlang's addressed processes + mailboxes, and
|
||||
Go's anonymous channels + goroutines — sit on the same underlying machinery:
|
||||
a fork/yield/block/resume scheduler over CEK io-suspended continuations. This
|
||||
plan captures that machinery as `lib/guest/scheduler/` so language N+1 with a
|
||||
new concurrency model costs ~200 lines of model-specific code instead of
|
||||
re-inventing the scheduler.
|
||||
|
||||
Reference: `plans/lib-guest.md` (parent — two-language rule, stratification),
|
||||
`plans/erlang-on-sx.md` (first consumer, in production), Go-on-SX (second
|
||||
consumer, see `plans/go-on-sx.md` once that lands).
|
||||
|
||||
**Branch:** `architecture`. SX files via `sx-tree` MCP only.
|
||||
|
||||
## Thesis
|
||||
|
||||
The substrate already provides what a scheduler needs: CEK io-suspension
|
||||
(`make-cek-suspended`, `cek-resume`) gives suspendable execution; first-class
|
||||
environments give each unit of execution its own scope; the trampolined
|
||||
evaluator means we never blow the host stack. What every guest with concurrency
|
||||
*re-implements* on top of this is the **fork/yield/block/resume protocol** —
|
||||
the bookkeeping that decides which suspended computation runs next.
|
||||
|
||||
Two concrete consumers, two different concurrency vocabularies, sharing one
|
||||
underlying scheduler, is the proof. If only Erlang lives on it, "scheduler kit"
|
||||
is a euphemism for "Erlang scheduler with a Go skin." The two-language rule
|
||||
is the gate.
|
||||
|
||||
## Current state (2026-05-26)
|
||||
|
||||
- **Erlang-on-SX** has the full pattern in production: 729/729 conformance,
|
||||
spawn/send/receive, selective receive, monitor/link, hot reload. The
|
||||
scheduler logic is currently coupled to Erlang-shaped concepts (PIDs,
|
||||
mailboxes, links) — extraction-blocking but not extraction-defeating.
|
||||
- **Go-on-SX** does not exist yet. `plans/go-on-sx.md` is the umbrella plan
|
||||
(TBD); this scheduler plan is a sibling/dependency.
|
||||
- **lib/guest/scheduler/** does not exist. The two-language rule blocks
|
||||
extraction until Go-on-SX independently implements its scheduler.
|
||||
|
||||
**Status: Phase 0 (Erlang shape capture).** No code change in this plan yet.
|
||||
|
||||
## Why the two models actually share a kit
|
||||
|
||||
The non-obvious claim is that Erlang processes and Go goroutines really do
|
||||
share machinery beneath their different vocabularies. The mapping:
|
||||
|
||||
| Concept | Erlang | Go | Common kit name |
|
||||
|---|---|---|---|
|
||||
| Unit of execution | process (PID-addressed) | goroutine (anonymous) | **task** |
|
||||
| Spawn | `spawn(Fun)` → PID | `go expr` → nothing | `task-spawn` |
|
||||
| Block target | mailbox match | channel send/recv | `task-block` |
|
||||
| Wake condition | message arrives | counterpart ready | `task-resume` predicate |
|
||||
| Yield | `receive` with no match | channel blocked | scheduler hands off |
|
||||
| Termination | exit reason → linked tasks | panic / return | task lifecycle |
|
||||
| Selection | selective `receive` | `select` statement | both = "wait for any of N predicates" |
|
||||
|
||||
What the kit owns:
|
||||
- The **task table** (token → suspended CEK continuation + status).
|
||||
- The **runnable queue** + scheduling policy (round-robin v1; pluggable).
|
||||
- The **block→resume protocol**: a blocked task registers a predicate; when
|
||||
any task changes state, blocked tasks are re-polled; first whose predicate
|
||||
fires becomes runnable.
|
||||
- The **fairness/preemption budget** — gas per step before forced yield.
|
||||
|
||||
What each language owns:
|
||||
- The semantics layer on top: Erlang's PID→task map + mailbox per task +
|
||||
selective-receive predicates; Go's channel value → blocked-task list per
|
||||
channel + send/recv pairing + select multiplexing.
|
||||
- The language-visible API (`spawn`/`!`/`receive` vs `go`/`<-`/`select`).
|
||||
|
||||
This is exactly the lib/guest pattern: extract the dispatch skeleton, keep
|
||||
the rules in the language layer.
|
||||
|
||||
## API surface (proposed — design only, not yet implemented)
|
||||
|
||||
```
|
||||
(make-scheduler &key gas-per-step ;; default 1000
|
||||
policy) ;; :round-robin | :fifo
|
||||
-> scheduler-handle
|
||||
|
||||
(task-spawn sched body-thunk) -> task-token
|
||||
;; body-thunk is a 0-arg fn whose body runs as the task.
|
||||
;; Returns immediately; task is enqueued runnable.
|
||||
|
||||
(task-current sched) -> task-token
|
||||
;; Inside a task, the token of the running task. Useful for self-reference.
|
||||
|
||||
(task-yield sched) -> nil
|
||||
;; Voluntary yield. Caller is re-enqueued at the tail of runnable.
|
||||
|
||||
(task-block sched resume-predicate) -> any
|
||||
;; Caller suspends. Predicate is (fn () -> resume-value-or-#f).
|
||||
;; When predicate returns non-#f, caller resumes with that value.
|
||||
;; Predicate is polled on every scheduler tick when there's nothing
|
||||
;; obviously runnable. (Optimisation: language layer can wake explicitly —
|
||||
;; see task-wake.)
|
||||
|
||||
(task-wake sched task) -> nil
|
||||
;; Hint to the scheduler: re-poll this task's resume-predicate now.
|
||||
;; Used by sender-side when a receiver might unblock.
|
||||
|
||||
(task-status sched task) -> :runnable | :blocked | :finished | :crashed
|
||||
|
||||
(task-result sched task) -> value | {:error reason}
|
||||
;; After :finished or :crashed.
|
||||
|
||||
(scheduler-step sched) -> :ran | :idle | :all-done
|
||||
;; Run at most gas-per-step instructions of one task. Caller drives the
|
||||
;; loop.
|
||||
|
||||
(scheduler-run sched) -> nil
|
||||
;; Run until :all-done. Equivalent to (until (= :all-done (scheduler-step
|
||||
;; sched))).
|
||||
```
|
||||
|
||||
Notes on the design:
|
||||
- `task-block` with a resume-predicate is the universal blocking primitive.
|
||||
Erlang's `receive` is `(task-block sched (fn () (mailbox-match self pat)))`.
|
||||
Go's `<-ch` is `(task-block sched (fn () (channel-recv-ready ch)))`.
|
||||
- `task-wake` is the optimisation: instead of polling every blocked task
|
||||
every step, the language layer wakes the specific task whose predicate
|
||||
is now likely true. v1 can omit it; performance work later.
|
||||
- `gas-per-step` gives fairness without true preemption. Tasks that don't
|
||||
yield within their gas budget are force-yielded by the CEK loop. (CEK
|
||||
io-suspension already does this for IO; gas budget extends to plain
|
||||
instructions.)
|
||||
- No priority/affinity in v1. Both Erlang and Go default to non-priority
|
||||
scheduling; specialised cases (Erlang's high-priority processes) are
|
||||
language-layer concerns.
|
||||
|
||||
## Build order — phases
|
||||
|
||||
This is a long-running plan paced against Go-on-SX. Phases are not loop-style
|
||||
"one commit per phase" — they're milestone gates.
|
||||
|
||||
### Phase 0 — Erlang shape capture (doc-only) ⬜
|
||||
- Read `lib/erlang/runtime.sx` scheduler code (currently coupled to Erlang
|
||||
vocabulary).
|
||||
- Write a 1-page summary of what's actually a scheduler and what's actually
|
||||
Erlang. Identify the boundary.
|
||||
- **Acceptance:** summary committed to this plan as a new section "Erlang
|
||||
scheduler shape (captured 2026-MM-DD)". No code change.
|
||||
- **Output:** clear-eyed mental model. Without this, we'll merge Erlang's
|
||||
scheduler shape into the kit and pretend it generalises.
|
||||
|
||||
### Phase 1 — Go scheduler independent implementation ⬜
|
||||
- During Go-on-SX, implement `lib/go/sched.sx` from scratch. Do NOT look at
|
||||
Erlang's scheduler while doing this. (Or read it once, then close it.)
|
||||
- Pass Go's channel + goroutine + select conformance tests.
|
||||
- **Acceptance:** Go scheduler green, lib/go/scoreboard.json includes scheduler
|
||||
tests, two-consumer rule now passable.
|
||||
- **Output:** two independent, working implementations of the same idea.
|
||||
|
||||
### Phase 2 — Diff and proposed kit ⬜
|
||||
- Side-by-side diff: Erlang's scheduler vs Go's scheduler. Where do they
|
||||
agree? Where does each have language-specific bookkeeping?
|
||||
- The diff is the kit. Things in *both* go in `lib/guest/scheduler/`; things
|
||||
in only one stay in `lib/erlang/` or `lib/go/`.
|
||||
- Draft `lib/guest/scheduler/api.sx` (signatures only, no body) reflecting the
|
||||
proposed surface.
|
||||
- **Acceptance:** API draft circulated for review; agreement that the surface
|
||||
covers both consumers; no merge yet.
|
||||
|
||||
### Phase 3 — Implement `lib/guest/scheduler/` ⬜
|
||||
- Implement the kit per the agreed API. New file(s) in `lib/guest/scheduler/`.
|
||||
- The kit has its own tests in `lib/guest/scheduler/tests/` — agnostic of any
|
||||
particular language vocabulary.
|
||||
- **Acceptance:** kit tests pass. Erlang and Go conformance scoreboards
|
||||
unchanged (the language implementations still use their own scheduler —
|
||||
we haven't refactored yet).
|
||||
|
||||
### Phase 4 — Refactor Erlang to use the kit ⬜
|
||||
- `lib/erlang/runtime.sx` scheduler logic deleted; replaced with calls into
|
||||
`lib/guest/scheduler/`. Erlang's PID table, mailbox-per-PID, selective
|
||||
receive stay in `lib/erlang/`.
|
||||
- **No-regression gate:** Erlang conformance holds at current pass count
|
||||
(currently 729/729). Hard requirement.
|
||||
- **Acceptance:** Erlang scoreboard unchanged; `lib/erlang/runtime.sx`
|
||||
meaningfully smaller (the scheduler code is gone).
|
||||
|
||||
### Phase 5 — Refactor Go to use the kit ⬜
|
||||
- Same exercise for Go. `lib/go/sched.sx` shrinks to channel/goroutine
|
||||
bookkeeping + delegation.
|
||||
- **No-regression gate:** Go conformance scoreboard at its current pass
|
||||
count.
|
||||
- **Acceptance:** Go scoreboard unchanged; `lib/go/sched.sx` meaningfully
|
||||
smaller.
|
||||
|
||||
### Phase 6 — Documentation + design-diary close ⬜
|
||||
- Document `lib/guest/scheduler/` API in `lib/guest/README.md` (or wherever
|
||||
the lib/guest API index lives).
|
||||
- Capture the chiselling diary: what *almost* went in the kit but ended up
|
||||
language-specific, and why. This is the load-bearing knowledge for the
|
||||
third consumer when it arrives.
|
||||
- **Acceptance:** API documented; diary section added to this plan.
|
||||
|
||||
## Two-language rule — gating
|
||||
|
||||
**The rule is hard.** No code in `lib/guest/scheduler/` lands until BOTH
|
||||
Phase 1 (Go independent) AND Phase 0 (Erlang capture) are complete AND a
|
||||
review confirms the two implementations actually share machinery in a way
|
||||
the kit captures.
|
||||
|
||||
If, during Phase 2 diff, we discover that the agreement is shallow (e.g.,
|
||||
both have a runnable queue but the policies are fundamentally incompatible),
|
||||
the **right outcome is to NOT extract**. Add a "rejected extraction" note to
|
||||
this plan documenting what we learned and close it. That outcome is fine —
|
||||
it tells us the two concurrency models aren't actually sister, which is a
|
||||
real result.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Preemption.** v1 is cooperative; gas-per-step gives fairness but not
|
||||
hard preemption. Erlang BEAM does true preemption (reduction counting).
|
||||
Go uses async preemption (signal-driven since 1.14). Neither extreme fits
|
||||
cooperatively over CEK. Is gas-per-step + voluntary yield enough? Probably
|
||||
for v1; revisit if a guest needs hard real-time.
|
||||
- **Priority/affinity.** Both Erlang and Go can run without it. Defer.
|
||||
- **Distribution.** Erlang nodes, Go's distributed channels — both are
|
||||
language-specific layers on top of the local scheduler. Out of scope.
|
||||
- **Cancellation.** Go has `context.Context`; Erlang has `exit/2`. Both
|
||||
bottom out at "deliver an exception to a task." Worth modelling? Probably
|
||||
as a kit primitive `(task-cancel sched task reason)` that delivers an
|
||||
exception via CEK exception machinery, language layer wraps it.
|
||||
- **Third consumer.** If/when JS-on-SX gets a proper async/await + Promise
|
||||
scheduler, that'd be a great third consumer to validate the kit didn't
|
||||
over-fit to Erlang+Go.
|
||||
|
||||
## Progress log
|
||||
|
||||
_Newest first. Append one dated entry per milestone landed._
|
||||
|
||||
- 2026-05-26 — Plan drafted. Phase 0 unstarted. Awaiting Go-on-SX to begin
|
||||
Phase 1.
|
||||
287
plans/lib-guest-static-types-bidirectional.md
Normal file
287
plans/lib-guest-static-types-bidirectional.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# lib/guest/static-types-bidirectional — design-diary plan
|
||||
|
||||
Capture the dispatch skeleton of bidirectional type checking
|
||||
(synthesis/checking judgments, context as a value, pluggable subtyping and
|
||||
unification) as `lib/guest/static-types-bidirectional/`, so static-typed
|
||||
guest languages that aren't Hindley-Milner-inferred cost ~300 lines of
|
||||
language-specific rules instead of re-inventing the checker plumbing.
|
||||
|
||||
Reference: `plans/lib-guest.md` (parent — two-language rule, stratification),
|
||||
`lib/guest/hm.sx` (sister module — full Hindley-Milner for inference-heavy
|
||||
languages like Haskell-on-SX), Go-on-SX (planned first consumer), TBD second
|
||||
consumer.
|
||||
|
||||
**Branch:** `architecture`. SX files via `sx-tree` MCP only.
|
||||
|
||||
## Thesis
|
||||
|
||||
`lib/guest/hm.sx` covers languages where the user writes few type annotations
|
||||
and the checker infers the rest globally (Haskell-on-SX, an eventual ML port,
|
||||
a typed-Scheme-with-Damas-Milner). But most modern statically-typed languages
|
||||
in actual production — Go, Rust, Swift, TypeScript, Kotlin, Scala 3, Hack —
|
||||
do **bidirectional checking instead**: declarations carry annotations, locals
|
||||
are inferred from immediate context, return types thread inwards from call
|
||||
sites. This isn't a weaker form of HM; it's a different design that scales
|
||||
better to mutation, subtyping, ad-hoc polymorphism, and gradual typing —
|
||||
none of which HM handles cleanly.
|
||||
|
||||
If `lib/guest/` is going to credibly host the next decade of statically-typed
|
||||
languages, it needs a bidirectional kit alongside `hm.sx`. They're sisters,
|
||||
not rivals.
|
||||
|
||||
**This plan is a design diary, not an implementation queue.** The two-language
|
||||
rule blocks extraction until two consumers exist. Go-on-SX is the first; the
|
||||
second is TBD. Until then, this plan documents what the API surface *should*
|
||||
be based on a single consumer, openly acknowledging that the second consumer
|
||||
will revise it.
|
||||
|
||||
## Current state (2026-05-26)
|
||||
|
||||
- `lib/guest/hm.sx` exists, used by Haskell-on-SX. 180 lines. The HM kit is
|
||||
the sister extraction this plan complements.
|
||||
- No bidirectional kit anywhere in `lib/guest/`.
|
||||
- Go-on-SX does not exist yet. When it does, `lib/go/types.sx` will be the
|
||||
first consumer.
|
||||
- Second consumer is unidentified. Most likely candidates, in order:
|
||||
1. **TypeScript-on-SX** — purely structural, gradual typing, the most-
|
||||
popular bidirectional language alive. Natural pair.
|
||||
2. **Rust-on-SX** — bidirectional with substantial extras (lifetimes,
|
||||
traits, borrow checking). Heavyweight; lifetimes don't go in this kit.
|
||||
3. **Typed Racket subset** — if anyone ports it. Bidirectional + gradual.
|
||||
4. **Hack / Flow / Python-with-types** — same shape.
|
||||
|
||||
**Status: Phase 0 (literature survey).** No code in this plan yet.
|
||||
|
||||
## Why bidirectional, not HM (for the languages that need it)
|
||||
|
||||
Five reasons HM doesn't fit these languages:
|
||||
|
||||
1. **Subtyping.** HM unification requires equality of types; subtyping
|
||||
requires a different judgment (`t ⊑ u`). Go's `interface{}` accepts any
|
||||
concrete type that satisfies it — subtyping, not unification.
|
||||
2. **Mutation.** HM's let-polymorphism interacts pathologically with
|
||||
mutable references (the value restriction). Go, Rust, TS all have
|
||||
first-class mutation and need rules that handle it directly.
|
||||
3. **Annotations as ground truth.** Bidirectional treats declared types as
|
||||
*given*, then propagates them. HM treats every type as a variable to be
|
||||
solved. For languages where annotations are expected, bidirectional is
|
||||
the natural shape.
|
||||
4. **Generics with constraints.** Go's type parameters carry constraints
|
||||
(`type T comparable`); Rust has trait bounds. HM has typeclasses but
|
||||
they're orthogonal to its constraint solver. Bidirectional weaves
|
||||
constraints into the checking rules naturally.
|
||||
5. **Gradual typing.** TS's `any`, Hack's pessimistic mode, Python's
|
||||
`Any` — gradual checking is built on bidirectional's "check or skip"
|
||||
distinction. HM either checks or it doesn't.
|
||||
|
||||
These languages collectively are the majority of new statically-typed code.
|
||||
Hosting them on lib/guest at all requires the bidirectional shape.
|
||||
|
||||
## API surface (proposed — design only, will revise with second consumer)
|
||||
|
||||
```
|
||||
;; --- judgments ---
|
||||
|
||||
(synth ctx expr) -> {:type T} | {:error msg}
|
||||
;; "expr synthesises type T in context ctx."
|
||||
;; Used at function calls (arg types known), let bindings, literals.
|
||||
|
||||
(check ctx expr expected-type) -> :ok | {:error msg}
|
||||
;; "expr checks against expected-type in context ctx."
|
||||
;; Used in function bodies (return type known), arguments (param type known),
|
||||
;; assignments (LHS type known).
|
||||
|
||||
;; --- context ---
|
||||
|
||||
(make-ctx) -> ctx
|
||||
(ctx-extend ctx name type) -> ctx ;; functional update
|
||||
(ctx-lookup ctx name) -> type | nil
|
||||
|
||||
;; --- pluggable rules ---
|
||||
|
||||
(register-synth-rule! kit ast-tag synth-fn) -> nil
|
||||
;; ast-tag: a keyword identifying the AST node shape (eg. :call :let :lit-int)
|
||||
;; synth-fn: (ctx node) -> {:type T} | {:error msg}
|
||||
|
||||
(register-check-rule! kit ast-tag check-fn) -> nil
|
||||
;; check-fn: (ctx node expected-type) -> :ok | {:error msg}
|
||||
|
||||
(register-type-equiv! kit pred) -> nil
|
||||
;; pred: (t1 t2) -> bool. The "are these types compatible" predicate.
|
||||
;; For Go: structural-interface-match-or-equal.
|
||||
;; For TS: structural-equality-with-any-bidirectional-bottom.
|
||||
;; For Rust: nominal equality + trait obligations.
|
||||
|
||||
(register-subtype! kit pred) -> nil
|
||||
;; pred: (sub super) -> bool. Optional; defaults to type-equiv.
|
||||
;; Go has no subtyping between concrete types but interface satisfaction
|
||||
;; is morally subtyping. TS has structural subtyping properly.
|
||||
|
||||
(register-unify! kit unifier) -> nil
|
||||
;; Optional; for type-variable resolution (generics).
|
||||
;; unifier: (t1 t2 subst) -> {:subst s'} | {:error msg}
|
||||
|
||||
;; --- driver ---
|
||||
|
||||
(make-kit) -> kit
|
||||
(check-program kit ctx program) -> {:ok ctx'} | {:error msg path-to-error}
|
||||
```
|
||||
|
||||
Design notes:
|
||||
- **The kit dispatches on AST tags**, which is what makes it pluggable. Each
|
||||
language registers rules for its node types. There's no hardcoded set of
|
||||
expression shapes in the kit.
|
||||
- **Synth and check are mutually recursive.** Inside a synth-rule for `call`,
|
||||
the rule synthesises the function's type, then `check`s each argument
|
||||
against the corresponding parameter type. Inside a check-rule for `lambda`,
|
||||
the rule pulls argument types from the expected function type and
|
||||
`synth`s the body. This pingponging is the bidirectional core.
|
||||
- **Pluggable type-equiv + subtype + unify** is the three-knob shape. Pierce
|
||||
& Turner ("Local Type Inference") and Dunfield & Krishnaswami ("Sound and
|
||||
Complete Bidirectional Typechecking") both factor it this way.
|
||||
- **No type variables in the core API.** Generics handling is a kit
|
||||
*extension*: when a language registers a `unify` predicate, the kit
|
||||
threads a substitution through synth/check. Languages without generics
|
||||
(early Go) leave it null.
|
||||
- **Errors carry a path.** `{:error msg path}` where path is a list of AST
|
||||
tags leading to the failure. Good error messages are why bidirectional is
|
||||
practical; the kit must support them.
|
||||
|
||||
## What's NOT in the kit (language-layer concerns)
|
||||
|
||||
Per the chiselling discipline, the kit is the dispatch skeleton; rules stay
|
||||
in the language. Specifically:
|
||||
|
||||
- **The literal type table.** Go's `42` is `untyped int` until contextualised;
|
||||
TS's `42` is the literal type `42`. Each language ships its own.
|
||||
- **Specific subtyping rules.** Go's interface satisfaction is recursive
|
||||
structural matching against method sets. TS's depends on object property
|
||||
satisfaction. Each language ships its own predicate.
|
||||
- **Generics constraint solving.** Go's type-set-based constraints, Rust's
|
||||
trait bounds, TS's conditional types — each is non-trivial and language-
|
||||
specific. The kit threads a substitution; the language defines what's in
|
||||
it.
|
||||
- **Effects, lifetimes, ownership.** Rust's borrow checker is not a type
|
||||
checker in the bidirectional-kit sense — it's a separate dataflow pass.
|
||||
Out of scope.
|
||||
- **Gradual fallback.** TS's `any` lets unchecked code coexist with checked
|
||||
code. The kit supports this via "check returns :ok on a sentinel any-type"
|
||||
but the sentinel is registered by the language.
|
||||
|
||||
## Build order — phases
|
||||
|
||||
### Phase 0 — Literature survey + Go's type system specifics ⬜
|
||||
- Read: Pierce & Turner "Local Type Inference" (2000); Dunfield & Krishnaswami
|
||||
"Sound and Complete Bidirectional Typechecking for Higher-Rank Polymorphism"
|
||||
(2013, 2019 revision); the Go language spec § "Types" + "Expressions".
|
||||
- Survey how Rust / TS / Kotlin / Scala 3 implement bidirectional in practice
|
||||
(their compilers are open source). Note where they diverge.
|
||||
- Output: a short summary section "Bidirectional design space (captured
|
||||
2026-MM-DD)" appended to this plan. Specifically: list every place
|
||||
language implementations diverge, so we can predict which divergences will
|
||||
show up between Go and the second consumer.
|
||||
- **Acceptance:** survey committed to this plan. No code.
|
||||
|
||||
### Phase 1 — Go independent implementation ⬜
|
||||
- During Go-on-SX, implement `lib/go/types.sx` from scratch. Do not write
|
||||
with extraction in mind — write the simplest Go-specific bidirectional
|
||||
checker.
|
||||
- Hit Go's distinctive type-system features: untyped constants, interface
|
||||
satisfaction (structural), generics (Go 1.18 type parameters with type-set
|
||||
constraints — defer this if scope explodes).
|
||||
- Pass Go's type-checker conformance tests.
|
||||
- **Acceptance:** Go conformance scoreboard includes type-checker tests, all
|
||||
passing.
|
||||
- **Output:** one consumer. Two-language rule still not met; no extraction.
|
||||
|
||||
### Phase 2 — Pick + start the second consumer ⬜
|
||||
- Decide between TS, Rust-subset, or typed-Scheme-subset. Recommendation:
|
||||
**TypeScript** — most-different from Go (gradual, structural everywhere),
|
||||
testing the kit's range maximally. Rust's lifetime/borrow machinery isn't
|
||||
part of this kit, so a Rust port wouldn't actually exercise the kit very
|
||||
hard.
|
||||
- Implement just enough of the second language to type-check a non-trivial
|
||||
function. Don't port the whole language; port the type checker.
|
||||
- **Acceptance:** second consumer's type checker green on its small slice.
|
||||
|
||||
### Phase 3 — Diff and proposed kit ⬜
|
||||
- Side-by-side: Go's checker vs the second consumer's checker. Where do they
|
||||
agree (the kit). Where does each diverge (the language).
|
||||
- Draft `lib/guest/static-types-bidirectional/api.sx` (signatures only).
|
||||
- Compare against the API sketch in this plan. The API WILL change at this
|
||||
step; that's the whole point of having two consumers.
|
||||
- **Acceptance:** revised API committed to this plan; agreement that both
|
||||
consumers can adopt it.
|
||||
|
||||
### Phase 4 — Implement the kit ⬜
|
||||
- `lib/guest/static-types-bidirectional/` with the agreed API. Kit tests in
|
||||
`lib/guest/static-types-bidirectional/tests/` — using a minimal "toy"
|
||||
language (synth-rule for `:int`, check-rule for `:lambda`) to verify the
|
||||
dispatch skeleton works.
|
||||
- **Acceptance:** kit tests pass; both consumers' scoreboards still green
|
||||
with their own implementations.
|
||||
|
||||
### Phase 5 — Refactor both consumers to use the kit ⬜
|
||||
- Go: `lib/go/types.sx` becomes a thin layer over the kit — registers Go's
|
||||
synth/check/equiv rules, calls `check-program`. Lifecycle code shrinks.
|
||||
- Second consumer: same exercise.
|
||||
- **No-regression gate:** both consumers' conformance scoreboards unchanged.
|
||||
- **Acceptance:** both `lib/<lang>/types.sx` files meaningfully smaller; kit
|
||||
is doing real work.
|
||||
|
||||
### Phase 6 — Documentation + chiselling diary ⬜
|
||||
- Document the API in lib/guest's README index.
|
||||
- Diary section in this plan: what we considered putting in the kit but
|
||||
ended up keeping language-specific, and why.
|
||||
- **Acceptance:** documentation present; diary captured.
|
||||
|
||||
## Two-language rule — gating
|
||||
|
||||
Same as `lib-guest-scheduler.md`. The kit does not exist until both consumers
|
||||
independently work AND we've reviewed the diff AND we believe the shared
|
||||
skeleton is real. Rejected-extraction is a valid outcome.
|
||||
|
||||
## Relationship to `lib/guest/hm.sx`
|
||||
|
||||
Sister modules, not rivals. Some languages will use HM (full inference,
|
||||
let-polymorphism); some will use bidirectional (annotation-driven, subtyping-
|
||||
friendly). Some might use both — Scala-on-SX, hypothetically, has local-type-
|
||||
inference in expressions and global-HM-style constraint solving in implicit
|
||||
resolution. The kit boundaries are:
|
||||
|
||||
- `hm.sx` — unification-based, whole-expression inference. Damas-Milner core.
|
||||
Best for: ML family, Haskell, OCaml subset, Standard ML.
|
||||
- `static-types-bidirectional/` — synth/check judgments, pluggable equiv +
|
||||
subtype. Best for: Go, Rust, TS, Kotlin, Swift, Scala 3, Hack.
|
||||
|
||||
A language can call into both: bidirectional for the surface, HM-style
|
||||
unification inside generics resolution. That's actually how Scala 3 works.
|
||||
The kits compose; design accordingly.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Variance.** Go has none; TS has covariant/contravariant/bivariant; Rust
|
||||
has variance markers per type parameter. Does the kit need a variance
|
||||
predicate as a fourth pluggable knob? Probably yes, but defer until the
|
||||
second consumer forces the question.
|
||||
- **Effect tracking.** Some bidirectional checkers (Koka, Eff, certain
|
||||
capability-effect TS variants) track effects in types. Out of scope for
|
||||
v1; the kit must not actively prevent it though.
|
||||
- **Refinement types.** TS has narrowing (`typeof x === "string"` refines
|
||||
`x` to `string`); Hack and Flow are similar. These layer above the kit
|
||||
(the kit's `check` returns a refined context as part of `:ok`). Sketch
|
||||
this in Phase 3 if TS is the second consumer.
|
||||
- **Error recovery.** Real-world type checkers don't halt on first error;
|
||||
they recover and continue to surface as many errors as possible. The kit
|
||||
needs an error-accumulation mode. Design it in Phase 4.
|
||||
- **Performance.** For toy languages, naive synth/check is fine. For Go-
|
||||
sized programs, the checker has to be memoised on synthesised types of
|
||||
subexpressions. Not a v1 concern; flag if it bites.
|
||||
|
||||
## Progress log
|
||||
|
||||
_Newest first. Append one dated entry per milestone landed._
|
||||
|
||||
- 2026-05-26 — Plan drafted as design diary. Phase 0 unstarted. Gated on
|
||||
Go-on-SX (first consumer) and a TBD second consumer (recommendation:
|
||||
TypeScript). No code yet — kit cannot exist before two consumers do.
|
||||
Reference in New Issue
Block a user