coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge: architecture into loops/haskell — Phases 7-16 complete + Phases 17-19 planned
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 53s
Details

Browse Source 
Brings the architecture branch (559 commits ahead — R7RS step 4-6, JIT
expansion, host_error wrapping, bytecode compiler, etc.) into the
loops/haskell line of work. Conflict in lib/haskell/conformance.sh:
architecture replaced the inline driver with a thin wrapper delegating
to lib/guest/conformance.sh + a config file. Resolved by taking the
wrapper and extending lib/haskell/conformance.conf with all programs
added under loops/haskell (caesar, runlength-str, showadt, showio,
partial, statistics, newton, wordfreq, mapgraph, uniquewords, setops,
shapes, person, config, counter, accumulate, safediv, trycatch) plus
adding map.sx and set.sx to PRELOADS.

plans/haskell-completeness.md gains three new follow-up phases:
- Phase 17 — parser polish (`(x :: Int)` annotations, mid-file imports)
- Phase 18 — one ambitious conformance program (lambda-calc / Dijkstra /
  JSON parser candidate list)
- Phase 19 — conformance speed (batch all suites in one sx_server
  process to compress the 25-min run to single-digit minutes)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
giles
2026-05-08 07:06:28 +00:00
parent e83c01cdcc 1eb9d0f8d2
commit badb428100
348 changed files with 95235 additions and 13619 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
plans/agent-briefings/apl-loop.md Normal file
View File

@@ -0,0 +1,81 @@
# apl-on-sx loop agent (single agent, queue-driven)
Role: iterates `plans/apl-on-sx.md` forever. Rank-polymorphic primitives + 6 operators on the JIT is the headline showcase — APL is the densest combinator algebra you can put on top of a primitive table. Every program is `array → array` pure pipelines, exactly what the JIT was built for.
```
description: apl-on-sx queue loop
subagent_type: general-purpose
run_in_background: true
isolation: worktree
```
## Prompt
You are the sole background agent working `/root/rose-ash/plans/apl-on-sx.md`. Isolated worktree, forever, one commit per feature. Push to `origin/loops/apl` after every commit.
## Restart baseline — check before iterating
1. Read `plans/apl-on-sx.md` — roadmap + Progress log.
2. `ls lib/apl/` — pick up from the most advanced file.
3. If `lib/apl/tests/*.sx` exist, run them. Green before new work.
4. If `lib/apl/scoreboard.md` exists, that's your baseline.
## The queue
Phase order per `plans/apl-on-sx.md`:
- **Phase 1** — tokenizer + parser. Unicode glyphs, `¯` for negative, strands (juxtaposition), right-to-left, valence resolution by syntactic position
- **Phase 2** — array model + scalar primitives. `make-array {shape, ravel}`, scalar promotion, broadcast for `+ - × ÷ ⌈ ⌊ * ⍟ | ! ○`, comparison, logical, `⍳`, `⎕IO`
- **Phase 3** — structural primitives + indexing. `⍴ , ⍉ ↑ ↓ ⌽ ⊖ ⌷ ⍋ ⍒ ⊂ ⊃ ∊`
- **Phase 4** — **THE SHOWCASE**: operators. `f/` (reduce), `f¨` (each), `∘.f` (outer), `f.g` (inner), `f⍨` (commute), `f∘g` (compose), `f⍣n` (power), `f⍤k` (rank), `@` (at)
- **Phase 5** — dfns + tradfns + control flow. `{⍺+⍵}`, `∇` recurse, `⍺←default`, tradfn header, `:If/:While/:For/:Select`
- **Phase 6** — classic programs (life, mandelbrot, primes, n-queens, quicksort) + idiom corpus + drive to 100+
Within a phase, pick the checkbox that unlocks the most tests per effort.
Every iteration: implement → test → commit → tick `[ ]` → Progress log → next.
## Ground rules (hard)
- **Scope:** only `lib/apl/**` and `plans/apl-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, other `lib/<lang>/` dirs, `lib/stdlib.sx`, or `lib/` root. APL primitives go in `lib/apl/runtime.sx`.
- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers entry, stop.
- **Shared-file issues** → plan's Blockers with minimal repro.
- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits.
- **Unicode in `.sx`:** raw UTF-8 only, never `\uXXXX` escapes. Glyphs land directly in source.
- **Worktree:** commit, then push to `origin/loops/apl`. Never touch `main`.
- **Commit granularity:** one feature per commit.
- **Plan file:** update Progress log + tick boxes every commit.
## APL-specific gotchas
- **Right-to-left, no precedence among functions.** `2 × 3 + 4` is `2 × (3 + 4)` = 14, not 10. Operators bind tighter than functions: `+/ ⍳5` is `+/(⍳5)`, and `2 +.× 3 4` is `2 (+.×) 3 4`.
- **Valence by position.** `-3` is monadic negate (`-` with no left arg). `5-3` is dyadic subtract. The parser must look left to decide. Same glyph; different fn.
- **`¯` is part of a number literal**, not a prefix function. `¯3` is the literal negative three; `-3` is the function call. Tokenizer eats `¯` into the numeric token.
- **Strands.** `1 2 3` is a 3-element vector, not three separate calls. Adjacent literals fuse into a strand at parse time. Adjacent names do *not* fuse — `a b c` is three separate references.
- **Scalar promotion.** `1 + 2 3 4` ↦ `3 4 5`. Any scalar broadcasts against any-rank conformable shape.
- **Conformability** = exactly matching shapes, OR one side scalar, OR (in some dialects) one side rank-1 cycling against rank-N. Keep strict in v1: matching shape or scalar only.
- **`⍳` is overloaded.** Monadic `⍳N` = vector 1..N (or 0..N-1 if `⎕IO=0`). Dyadic `V ⍳ W` = first-index lookup, returns `≢V+1` for not-found.
- **Reduce with `+/⍳0`** = `0` (identity for `+`). Each scalar primitive has a defined identity used by reduce-on-empty. Don't crash; return identity.
- **Reduce direction.** `f/` reduces the *last* axis. `f⌿` reduces the *first*. Matters for matrices.
- **Indexing is 1-based** by default (`⎕IO=1`). Do not silently translate to 0-based; respect `⎕IO`.
- **Bracket indexing** `A[I]` is sugar for `I⌷A` (squad-quad). Multi-axis: `A[I;J]` is `I J⌷A` with semicolon-separated axes; `A[;J]` selects all of axis 0.
- **Dfn `{...}`** — `⍺` = left arg (may be unbound for monadic call → check with `⍺←default`), `⍵` = right arg, `∇` = recurse. Default left arg syntax: `⍺←0`.
- **Tradfn vs dfn** — tradfns use line-numbered `→linenum` for goto; dfns use guards `cond:expr`. Pick the right one for the user's syntax.
- **Empty array** = rank-N array where some dim is 0. `0⍴⍳0` is empty rank-1. Scalar prototype matters for empty-array operations; ignore in v1, return 0/space.
- **Test corpus:** custom + idioms. Place programs in `lib/apl/tests/programs/` with `.apl` extension.
## General gotchas (all loops)
- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences.
- `cond`/`when`/`let` clauses evaluate only the last expr.
- `type-of` on user fn returns `"lambda"`.
- Shell heredoc `||` gets eaten — escape or use `case`.
## Style
- No comments in `.sx` unless non-obvious.
- No new planning docs — update `plans/apl-on-sx.md` inline.
- Short, factual commit messages (`apl: outer product ∘. (+9)`).
- One feature per iteration. Commit. Log. Next.
Go. Read the plan; find first `[ ]`; implement.

80
plans/agent-briefings/common-lisp-loop.md Normal file
View File

@@ -0,0 +1,80 @@
# common-lisp-on-sx loop agent (single agent, queue-driven)
Role: iterates `plans/common-lisp-on-sx.md` forever. Conditions + restarts on delimited continuations is the headline showcase — every other Lisp reinvents resumable exceptions on the host stack. On SX `signal`/`invoke-restart` is just a captured continuation. Plus CLOS, the LOOP macro, packages.
```
description: common-lisp-on-sx queue loop
subagent_type: general-purpose
run_in_background: true
isolation: worktree
```
## Prompt
You are the sole background agent working `/root/rose-ash/plans/common-lisp-on-sx.md`. Isolated worktree, forever, one commit per feature. Never push.
## Restart baseline — check before iterating
1. Read `plans/common-lisp-on-sx.md` — roadmap + Progress log.
2. `ls lib/common-lisp/` — pick up from the most advanced file.
3. If `lib/common-lisp/tests/*.sx` exist, run them. Green before new work.
4. If `lib/common-lisp/scoreboard.md` exists, that's your baseline.
## The queue
Phase order per `plans/common-lisp-on-sx.md`:
- **Phase 1** — reader + parser (read macros `#'` `'` `` ` `` `,` `,@` `#( … )` `#:` `#\char` `#xFF` `#b1010`, ratios, dispatch chars, lambda lists with `&optional`/`&rest`/`&key`/`&aux`)
- **Phase 2** — sequential eval + special forms (`let`/`let*`/`flet`/`labels`, `block`/`return-from`, `tagbody`/`go`, `unwind-protect`, multiple values, `setf` subset, dynamic variables)
- **Phase 3** — **THE SHOWCASE**: condition system + restarts. `define-condition`, `signal`/`error`/`cerror`/`warn`, `handler-bind` (non-unwinding), `handler-case` (unwinding), `restart-case`, `restart-bind`, `find-restart`/`invoke-restart`/`compute-restarts`, `with-condition-restarts`. Classic programs (restart-demo, parse-recover, interactive-debugger) green.
- **Phase 4** — CLOS: `defclass`, `defgeneric`, `defmethod` with `:before`/`:after`/`:around`, `call-next-method`, multiple dispatch
- **Phase 5** — macros + LOOP macro + reader macros
- **Phase 6** — packages + stdlib (sequence functions, FORMAT directives, drive corpus to 200+)
Within a phase, pick the checkbox that unlocks the most tests per effort.
Every iteration: implement → test → commit → tick `[ ]` → Progress log → next.
## Ground rules (hard)
- **Scope:** only `lib/common-lisp/**` and `plans/common-lisp-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, other `lib/<lang>/` dirs, `lib/stdlib.sx`, or `lib/` root. CL primitives go in `lib/common-lisp/runtime.sx`.
- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers entry, stop.
- **Shared-file issues** → plan's Blockers with minimal repro.
- **Delimited continuations** are in `lib/callcc.sx` + `spec/evaluator.sx` Step 5. `sx_summarise` spec/evaluator.sx first — 2300+ lines.
- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits.
- **Worktree:** commit locally. Never push. Never touch `main`.
- **Commit granularity:** one feature per commit.
- **Plan file:** update Progress log + tick boxes every commit.
## Common-Lisp-specific gotchas
- **`handler-bind` is non-unwinding** — handlers can decline by returning normally, in which case `signal` keeps walking the chain. **`handler-case` is unwinding** — picking a handler aborts the protected form via a captured continuation. Don't conflate them.
- **Restarts are not handlers.** `restart-case` establishes named *resumption points*; `signal` runs handler code with restarts visible; the handler chooses a restart by calling `invoke-restart`, which abandons handler stack and resumes at the restart point. Two stacks: handlers walk down, restarts wait to be invoked.
- **`block` / `return-from`** is lexical. `block name … (return-from name v) …` captures `^k` once at entry; `return-from` invokes it. `return-from` to a name not in scope is an error (don't fall back to outer block).
- **`tagbody` / `go`** — each tag in tagbody is a continuation; `go tag` invokes it. Tags are lexical, can only target tagbodies in scope.
- **`unwind-protect`** runs cleanup on *any* non-local exit (return-from, throw, condition unwind). Implement as a scope frame fired by the cleanup machinery.
- **Multiple values**: primary-value-only contexts (function args, `if` test, etc.) drop extras silently. `values` produces multiple. `multiple-value-bind` / `multiple-value-call` consume them. Don't auto-list.
- **CLOS dispatch:** sort applicable methods by argument-list specificity (`subclassp` per arg, left-to-right); standard method combination calls primary methods most-specific-first via `call-next-method` chain. `:before` runs all before primaries; `:after` runs all after, in reverse-specificity. `:around` wraps everything.
- **`call-next-method`** is a *continuation* available only inside a method body. Implement as a thunk stored in a dynamic-extent variable.
- **Generalised reference (`setf`)**: `(setf (foo x) v)` ↦ `(setf-foo v x)`. Look up the setf-expander, not just a writer fn. `define-setf-expander` is mandatory for non-trivial places. Start with the symbolic / list / aref / slot-value cases.
- **Dynamic variables (specials):** `defvar`/`defparameter` mark a symbol as special. `let` over a special name *rebinds* in dynamic extent (use parameterize-style scope), not lexical.
- **Symbols are package-qualified.** Reader resolves `cl:car`, `mypkg::internal`, bare `foo` (current package). Internal vs external matters for `:` (one colon) reads.
- **`nil` is also `()` is also the empty list.** Same object. `nil` is also false. CL has no distinct unit value.
- **LOOP macro is huge.** Build incrementally — start with `for/in`, `for/from`, `collect`, `sum`, `count`, `repeat`. Add conditional clauses (`when`, `if`, `else`) once iteration drivers stable. `named` blocks + `return-from named` last.
- **Test corpus:** custom + curated `ansi-test` slice. Place programs in `lib/common-lisp/tests/programs/` with `.lisp` extension.
## General gotchas (all loops)
- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences.
- `cond`/`when`/`let` clauses evaluate only the last expr.
- `type-of` on user fn returns `"lambda"`.
- Shell heredoc `||` gets eaten — escape or use `case`.
## Style
- No comments in `.sx` unless non-obvious.
- No new planning docs — update `plans/common-lisp-on-sx.md` inline.
- Short, factual commit messages (`common-lisp: handler-bind + 12 tests`).
- One feature per iteration. Commit. Log. Next.
Go. Read the plan; find first `[ ]`; implement.

118
plans/agent-briefings/lib-guest-loop.md Normal file
View File

@@ -0,0 +1,118 @@
# lib/guest extraction loop (single agent, queue-driven)
Role: iterates `plans/lib-guest.md` forever. Each iteration picks the top `pending` step, extracts/ports/validates, commits, logs, moves on. North star: every guest's `scoreboard.json` ≥ baseline at all times, while `lib/guest/` accumulates shared infrastructure.
```
description: lib/guest extraction loop
subagent_type: general-purpose
run_in_background: true
```
## Prompt
You are the sole background agent working `/root/rose-ash/plans/lib-guest.md`. You work a prioritised queue, one step per code commit, indefinitely. The plan file is the source of truth for what's pending, in-progress, done, and blocked. Update it after every iteration.
## Iteration protocol (follow exactly)
### 1. Read state
- Read `plans/lib-guest.md` in full.
- Pick the first step with status `[ ]`. If all remaining are `[blocked]` or `[done]`, stop and report loop complete.
- Set that step's status to `[in-progress]` and commit the plan change alone:
`GUEST-plan: claim step <N> — <name>`.
### 2. Baseline (every iteration that touches a guest)
Before any code edit, snapshot the **current** scoreboard for every guest this step will touch (extraction consumers + canaries):
```
bash lib/<guest>/conformance.sh # or test.sh
cp lib/<guest>/scoreboard.json /tmp/baseline-<guest>-step<N>.json
```
If the step is Step 0, the snapshot itself is the work — copy each guest's `scoreboard.json` (or harvest pass/fail counts from `test.sh` for guests without a scoreboard) into `lib/guest/baseline/<lang>.json`, populate the table in `plans/lib-guest.md`, commit, done.
### 3. Do the work
For each step the protocol is:
1. Read the relevant existing guest file(s) via `sx_read_subtree` to see exactly what shape needs extracting.
2. Draft `lib/guest/<file>.sx` via `sx_write_file` (validates by parsing).
3. Port the **first** consumer to use it. Run that guest's conformance. Must equal baseline.
4. Port the **second** consumer (the two-language rule). Run that guest's conformance. Must equal baseline.
5. If the second consumer needs escape hatches that the first didn't, the abstraction is wrong — **redesign before continuing**, don't paper over with alias chains or per-language flags.
For Step 0 only: just snapshot, no extraction.
### 4. Verify
For every guest the step touched:
```
bash lib/<guest>/conformance.sh # or test.sh
diff lib/<guest>/scoreboard.json /tmp/baseline-<guest>-step<N>.json
```
**Abort rule:** if any touched guest's scoreboard regresses by ≥1 test, do NOT commit code. Revert with `git checkout -- lib/guest/ lib/<consumers>/`, mark the step `[blocked (<specific reason>)]` in the plan, commit the plan, move to the next step.
### 5. Commit code
One commit for the code:
```
GUEST: step <N> — <name>
<2-4 lines on what was extracted, which two consumers were ported, baseline-equal verification.>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
```
### 6. Update plan + commit
In `plans/lib-guest.md`:
- Change this step's status from `[in-progress]` to `[done]` (or `[partial — pending <consumer>]`).
- Fill in the Commit and Delta columns of the progress log.
- If you re-snapshotted any baseline, update the Baseline column.
Commit: `GUEST-plan: log step <N> done`.
### 7. Move on
Go back to step 1. Continue until:
- All steps are `[done]` or `[blocked]`, OR
- You hit your iteration budget, OR
- You encounter a substrate-level failure (build broken, sx_server.exe missing) — stop and report.
## Ground rules
- **Branch:** `architecture`. Commit locally. **Never push.** **Never touch `main`.**
- **Scope:** ONLY `lib/guest/**`, `lib/{lua,prolog,haskell,common-lisp,tcl,erlang,smalltalk,forth,ruby,apl,js}/**`, `plans/lib-guest.md`, `plans/agent-briefings/lib-guest-loop.md`. NO `spec/`, `hosts/`, `web/`, `shared/`.
- **SX files:** `sx-tree` MCP tools ONLY. Never `Edit`/`Read`/`Write` on `.sx`. `sx_validate` after every edit.
- **OCaml build:** `sx_build target="ocaml"` MCP tool. Never raw `dune`.
- **Two-language rule:** never merge an extraction until two guests consume it. Step 8 (HM) is the only exception, marked explicitly.
- **No alias chains** to bridge naming drift between extraction and consumer — rename consumer-side or extraction-side, don't add a translation layer.
- **No new planning docs** beyond updating the plan file.
- **No comments in SX** unless non-obvious.
- **Unicode in SX:** raw UTF-8, never `\uXXXX`.
- **Hard timeout:** >45 min on a step → mark `blocked`, move on.
- **Partial fixes are OK.** If you extract something and only the first consumer ports cleanly, mark `[partial — pending <second consumer>]`, commit, move on. The next iteration that lands the second consumer flips it to `[done]`.
## Gotchas from past sessions
- `env-bind!` creates a binding; `env-set!` mutates an existing one (walks scope chain). Macros that want to introduce names use `env-bind!`.
- SX `do` is R7RS iteration, not a sequence form. Use `begin` for multi-expr bodies.
- `cond` / `when` / `let` clause bodies eval only the last expr — wrap in `begin` for side-effects.
- `list?` returns false on raw JS Arrays — host-side data must be SX-converted.
- `make-symbol` builds an identifier symbol; `string->symbol` exists too — use whichever the surrounding code uses.
- `sx_validate` after every edit. The hook will block raw `Edit`/`Write` on `.sx` anyway, but the validator catches subtree mistakes that parse-but-don't-mean-what-you-think.
- Guest `conformance.sh` scripts use the epoch protocol against `sx_server.exe`. If the server isn't built, run `sx_build target="ocaml"` first.
- Each guest's `scoreboard.json` schema differs slightly — normalise to `{:totals {:pass N :fail M} :suites [...]}` when writing `lib/guest/baseline/<lang>.json`.
- `lib/parser-combinators.sx` exists and is unused by any guest. The new lex/Pratt kit may want to coexist with it, or supersede it — investigate before duplicating its functionality.
- Prolog operator parsing is the stress test for Pratt — Prolog ops have variable precedence, `xfx`/`xfy`/`yfx` associativity classes, and user-definable ops at runtime. The Pratt kit must accommodate runtime registration, not just static tables.
- Haskell layout is the stress test for whitespace-sensitive lexing — off-side rule, do/let/where/of opening blocks, semicolon insertion, brace insertion. Don't ship `lib/guest/layout.sx` unless the haskell scoreboard equals baseline.
## Starting state
- Branch: `architecture`. HEAD at or near `40f0e733`.
- Canaries: **Lua** + **Prolog**.
- Plan file at `plans/lib-guest.md`. Step 0 (baseline snapshot) is the first iteration.
- `lib/guest/` does not yet exist — create it on the Step 0 commit.

789
plans/agent-briefings/primitives-loop.md Normal file
View File

@@ -0,0 +1,789 @@
# SX Primitives — Meta-Loop Briefing
Goal: add fundamental missing SX primitives in sequence, then sweep all language
implementations to replace their workarounds. Full rationale: vectors fix O(n) array
access across every language; numeric tower fixes float/int conflation; dynamic-wind
fixes cleanup semantics; coroutine primitive unifies Ruby/Lua/Tcl; string buffer fixes
O(n²) concat; algebraic data types eliminate the tagged-dict pattern everywhere.
**Each fire: find the first unchecked `[ ]`, do it, commit, tick it, stop.**
Sub-items within a Phase may span multiple fires — just commit progress and tick what's done.
---
## Phase 0 — Prep (gate)
- [x] Stop new-language loops: send `/exit` to sx-loops windows for the four blank-slate
languages that haven't committed workarounds yet:
```
tmux send-keys -t sx-loops:common-lisp "/exit" Enter
tmux send-keys -t sx-loops:apl "/exit" Enter
tmux send-keys -t sx-loops:ruby "/exit" Enter
tmux send-keys -t sx-loops:tcl "/exit" Enter
```
Verify all four windows are idle (claude prompt, no active task).
- [x] E38 + E39 landed: check both Bucket-E branches for implementation commits.
```
git log --oneline hs-e38-sourceinfo | head -5
git log --oneline hs-e39-webworker | head -5
```
If either branch has only its base commit (no impl work yet): note "pending" and stop —
next fire re-checks. Proceed only when both have at least one implementation commit.
---
## Phase 1 — Vectors
Native mutable integer-indexed arrays. Fix: Lua O(n) sort, APL rank polymorphism, Ruby
Array, Tcl lists, Common Lisp vectors, all using string-keyed dicts today.
Primitives to add:
- `make-vector` `n` `[fill]` → vector of length n
- `vector?` `v` → bool
- `vector-ref` `v` `i` → element at index i (0-based)
- `vector-set!` `v` `i` `x` → mutate in place
- `vector-length` `v` → integer
- `vector->list` `v` → list
- `list->vector` `lst` → vector
- `vector-fill!` `v` `x` → fill all elements
- `vector-copy` `v` `[start]` `[end]` → fresh copy of slice
Steps:
- [x] OCaml: add `SxVector of value array` to `hosts/ocaml/sx_types.ml`; implement all
primitives in `hosts/ocaml/sx_primitives.ml` (or equivalent); wire into evaluator.
Note: Vector type + most prims were already present; added bounds-checked vector-ref/set!
and optional start/end to vector-copy. 10/10 vector tests pass (r7rs suite).
- [x] Spec: add vector entries to `spec/primitives.sx` with type signatures and descriptions.
All 10 vector primitives now have :as type annotations, :returns, and :doc strings.
make-vector: optional fill param; vector-copy: optional start/end (done prev step).
- [x] JS bootstrapper: implement vectors in `hosts/javascript/platform.js` (or equivalent);
ensure `sx-browser.js` rebuild picks them up.
Fixed index-of for lists (was returning -1 not NIL, breaking bind-lambda-params),
added _lastErrorKont_/hostError/try-catch/without-io-hook stubs. Vectors work.
- [x] Tests: 40+ tests in `spec/tests/test-vectors.sx` covering construction, ref, set!,
length, conversions, fill, copy, bounds behaviour.
42 tests, all pass. 1847 standard / 2362 full passing (up from 5).
- [x] Verify: full test suite still passes (`node hosts/javascript/run_tests.js --full`).
2362/4924 pass (improvement from pre-existing lambda binding bug, no regressions).
- [x] Commit: `spec: vector primitive (make-vector/vector-ref/vector-set!/etc)`
Committed as: js: fix lambda binding (index-of on lists), add vectors + R7RS platform stubs
---
## Phase 2 — Numeric tower
Float ≠ integer distinction. Fix: Erlang `=:=`, Lua `math.type()`, Haskell `Num`/`Integral`,
Common Lisp `integerp`/`floatp`/`ratio`, JS `Number.isInteger`.
Changes:
- `parse-number` preserves float identity: `"1.0"` → float 1.0, not integer 1
- New predicates: `integer?`, `float?`, `exact?`, `inexact?`
- New coercions: `exact->inexact`, `inexact->exact`
- Fix `floor`/`ceiling`/`truncate`/`round` to return integers when applied to floats
- `number->string` renders `1.0` as `"1.0"`, `1` as `"1"`
- Arithmetic: `(+ 1 1.0)` → `2.0` (float contagion), `(+ 1 1)` → `2` (integer)
Steps:
- [x] OCaml: distinguish `Integer of int` / `Number of float` in `sx_types.ml`; update all
arithmetic primitives for float contagion; fix `parse-number`.
92/92 numeric tower tests pass; 4874 total (394 pre-existing hs-upstream fails unchanged).
- [x] Spec: update `spec/primitives.sx` with new predicates + coercions; document contagion rules.
Added integer?/float? predicates; updated number? body; / returns "float"; floor/ceil/truncate
return "integer"; +/-/* doc float contagion; fixed double-paren params; 4874/394 baseline.
- [x] JS bootstrapper: update number representation and arithmetic.
Added integer?/float?/exact?/inexact?/truncate/remainder/modulo/random-int/exact->inexact/
inexact->exact/parse-number. Fixed sx_server.ml epoch protocol for Integer type.
JS: 1940 passed (+60); OCaml: 4874/394 unchanged. 6 tests JS-only fail (float≡int limitation).
- [x] Tests: 92 tests in `spec/tests/test-numeric-tower.sx` — int-arithmetic, float-contagion,
division, predicates, coercions, rounding, parse-number, equality, modulo, min-max, stringify.
- [x] Verify: full suite passes. OCaml 4874/394 (baseline unchanged). JS 1940/2500 (+60 vs pre-tower).
No regressions on any test that relied on `1.0 = 1` — those tests were already using integer
literals which remain identical in JS. 6 JS-only failures are platform-inherent (JS float≡int).
- [x] Commit: all work landed across 4 commits (c70bbdeb, 45ec5535, b12a22e6, f5acb31c).
---
## Phase 3 — Dynamic-wind
Fix: Common Lisp `unwind-protect`, Ruby `ensure`, JS `finally`, Tcl `catch`+cleanup,
Erlang `try...after` (currently uses double-nested guard workaround).
- [x] Spec: implement `dynamic-wind` in `spec/evaluator.sx` such that the after-thunk fires
on both normal return AND non-local exit (raise/call-cc escape). Must compose with
`guard` — currently they don't interact.
- [x] OCaml: wire `dynamic-wind` through the CEK machine with a `WindFrame` continuation.
- [x] JS bootstrapper: update.
- [x] Tests: 20+ tests covering normal return, raise, call/cc escape, nested dynamic-winds.
- [x] Commit: `spec: dynamic-wind + guard integration`
---
## Phase 4 — Coroutine primitive
Unify Ruby fibers, Lua coroutines, Tcl coroutines — all currently reimplemented separately
using call/cc+perform/resume.
- [x] Spec: add `make-coroutine`, `coroutine-resume`, `coroutine-yield`, `coroutine?`,
`coroutine-alive?` to `spec/primitives.sx`. Build on existing `perform`/`cek-resume`
machinery — coroutines ARE perform/resume with a stable identity.
Implemented as `spec/coroutines.sx` define-library; `make-coroutine` stub in evaluator.sx.
17/17 coroutine tests pass (OCaml). Drives iteration via define+fn recursion (not named let —
named let uses cek_call→cek_run which errors on IO suspension).
- [x] OCaml: implement coroutine type; wire resume/yield through CEK suspension.
No new native type needed — dict-based coroutine identity + existing cek-step-loop/
cek-resume/perform primitives in run_tests.ml ARE the OCaml implementation. 17/17 pass.
- [x] JS bootstrapper: update.
All CEK primitives already in sx-browser.js. Fix: pre-load spec/coroutines.sx +
spec/signals.sx in run_tests.js so (import (sx coroutines)) resolves without suspension.
17/17 pass in JS. 1965/2500 (+25 vs 1940 baseline). Zero new failures.
- [x] Tests: 25+ tests — multi-yield, final return, arg passthrough, alive? predicate,
nested coroutines, "final return vs yield" distinction (the Lua gotcha).
27 tests: added 10 new — state field inspection (ready/suspended/dead), yield from
nested helper, initial resume arg ignored, mutable closure state, complex yield values,
round-robin scheduling, factory-shared-no-state, non-coroutine error. 27/27 OCaml+JS.
- [x] Commit: `spec: coroutine primitive (make-coroutine/resume/yield)`
Phase 4 landed across 4 commits: 21cb9cf5 (spec library), 9eb12c66 (ocaml verified),
b78e06a7 (js pre-load), 0ffe208e (27 tests). Phase 4 complete.
---
## Phase 5 — String buffer
Fix O(n²) string concatenation in loops across Lua, Ruby, Common Lisp, Tcl.
- [x] Spec + OCaml: add `make-string-buffer`, `string-buffer-append!`, `string-buffer->string`,
`string-buffer-length` to primitives. OCaml: `Buffer.t` wrapper. JS: array+join.
Also: string-buffer? predicate; SxStringBuffer._string_buffer marker for typeOf/dict?
exclusion; inspect case in sx_types.ml. 17/17 tests OCaml+JS.
- [x] Tests: 15+ tests.
17 tests written inline with Spec+OCaml step: construction, type-of, empty/length,
single/multi-append, append-returns-nil, empty-string-append, reuse-after-to-string,
independence, loop-building, CSV-row, unicode, repeated-to-string, join-pattern.
17/17 OCaml+JS.
- [x] Commit: `spec: string-buffer primitive`
Committed as d98b5fa2 — all work in one commit (OCaml type + primitives + JS + spec + 17 tests).
---
## Phase 6 — Algebraic data types
The deepest structural gap. Every language uses `{:tag "..." :field ...}` tagged dicts to
simulate sum types. A native `define-type` + `match` form eliminates this everywhere.
- [x] Design: write `plans/designs/sx-adt.md` covering syntax, CEK dispatch, interaction with
existing `cond`/`case`, exhaustiveness checking, recursive types, pattern variables.
Draft, then stop — next fire reviews design before implementing.
Written: define-type/match syntax, AdtValue runtime rep, stepSfDefineType + MatchFrame
CEK dispatch, exhaustiveness warnings via _adt_registry, recursive types, nested patterns,
wildcard _, 3-phase impl plan (basic/nested/exhaustiveness), open questions on accessors/singletons/inspect.
- [x] Spec: implement `define-type` special form in `spec/evaluator.sx`:
`(define-type Name (Ctor1 field...) (Ctor2 field...) ...)`
Creates constructor functions `Ctor1`, `Ctor2` + predicate `Name?`.
- [x] Spec: implement `match` special form:
`(match expr ((Ctor1 a b) body) ((Ctor2 x) body) (else body))`
Exhaustiveness warning if not all constructors covered and no `else`.
- [x] OCaml: add `SxAdt of string * value array` to types; implement constructors + match.
Dict-based ADT (no native type needed — matches spec). Hand-written sf_define_type
in bootstrap.py FIXUPS; registered via register_special_form. 172 assertions pass.
4280/1080 full suite (37 improvement over old baseline 4243/1117).
- [x] JS bootstrapper: update.
No changes needed — define-type/match are spec-level; sx-browser.js rebuilt at 0dc7e159.
40/40 ADT tests pass JS. 2032/2500 total (+67 vs 1965 phase-4 baseline).
- [x] Tests: 40+ tests in `spec/tests/test-adt.sx`.
40 tests written across two spec commits (6c872107+0dc7e159). All pass OCaml+JS.
- [x] Commit: `spec: algebraic data types (define-type + match)`
Phase 6 landed across 5 commits: 6c872107 (define-type spec), 0dc7e159 (match spec),
5d1913e7 (ocaml bootstrap), f63b2147 (plan tick). JS already current.
---
## Phase 7 — Bitwise operations
Completely absent today. Needed by: Forth (core), APL (array masks), Erlang (bitmatch),
JS (typed arrays, bitfields), Common Lisp (`logand`/`logior`/`logxor`/`lognot`/`ash`).
Primitives to add:
- `bitwise-and` `a` `b` → integer
- `bitwise-or` `a` `b` → integer
- `bitwise-xor` `a` `b` → integer
- `bitwise-not` `a` → integer
- `arithmetic-shift` `a` `count` → integer (left if count > 0, right if count < 0)
- `bit-count` `a` → number of set bits (popcount)
- `integer-length` `a` → number of bits needed to represent a
Steps:
- [x] Spec: add entries to `spec/primitives.sx` with type signatures.
stdlib.bitwise module with 7 entries appended to spec/primitives.sx.
- [x] OCaml: implement in `hosts/ocaml/sx_primitives.ml` using OCaml `land`/`lor`/`lxor`/`lnot`/`lsl`/`asr`.
land/lor/lxor/lnot/lsl/asr in sx_primitives.ml. bit-count: Kernighan loop. integer-length: lsr loop.
- [x] JS bootstrapper: implement in `hosts/javascript/platform.js` using JS `&`/`|`/`^`/`~`/`<<`/`>>`.
stdlib.bitwise module added to PRIMITIVES_JS_MODULES. bit-count: Hamming weight. integer-length: Math.clz32.
- [x] Tests: 25+ tests in `spec/tests/test-bitwise.sx` — basic ops, shift left/right, negative numbers, popcount.
26 tests, 158 assertions, all pass OCaml+JS.
- [x] Commit: `spec: bitwise operations (bitwise-and/or/xor/not, arithmetic-shift, bit-count)`
Committed a8a79dc9. Phase 7 complete in single commit.
---
## Phase 8 — Multiple values
R7RS standard. Common Lisp uses them heavily; Haskell tuples map naturally; Erlang
multi-return. Without them, every function returning two things encodes it as a list or dict.
Primitives / forms to add:
- `values` `v...` → multiple-value object
- `call-with-values` `producer` `consumer` → applies consumer to values from producer
- `let-values` `(((a b) expr) ...)` `body` — binding form (special form in evaluator)
- `define-values` `(a b ...)` `expr` — top-level multi-value bind
Steps:
- [x] Spec: add `SxValues` type to evaluator; implement `values` + `call-with-values` in
`spec/evaluator.sx`; add `let-values` / `define-values` special forms.
- [x] OCaml: add `SxValues of value list` to `sx_types.ml`; wire through CEK.
- [x] JS bootstrapper: implement values type + forms.
- [x] Tests: 25+ tests in `spec/tests/test-values.sx` — basic producer/consumer, let-values
destructuring, define-values, interaction with `begin`/`do`.
- [x] Commit: `spec: multiple values (values/call-with-values/let-values)`
---
## Phase 9 — Promises (lazy evaluation)
Critical for Haskell — lazy evaluation is so central that without it the Haskell
implementation can't be idiomatic. Also useful for lazy lists in Common Lisp and
lazy streams in Scheme-style code generally.
Primitives / forms to add:
- `delay` `expr` → promise (special form — expr not evaluated yet)
- `force` `p` → evaluate promise, cache result, return it
- `make-promise` `v` → already-forced promise wrapping v
- `promise?` `v` → bool
- `delay-force` `expr` → for iterative lazy sequences (avoids stack growth in lazy streams)
Steps:
- [x] Spec: add `delay` / `delay-force` special forms to `spec/evaluator.sx`; add promise
type with mutable forced/value slots; `force` checks if already forced before eval.
- [x] OCaml: add `SxPromise of { mutable forced: bool; mutable value: value; thunk: value }`;
wire `delay`/`force`/`delay-force` through CEK.
- [x] JS bootstrapper: implement promise type + forms.
- [x] Tests: 25+ tests in `spec/tests/test-promises.sx` — basic delay/force, memoisation
(forced only once), delay-force lazy stream, promise? predicate, make-promise.
- [x] Commit: `spec: promises — delay/force/delay-force for lazy evaluation`
---
## Phase 10 — Mutable hash tables
Distinct from SX's immutable dicts. Dict primitives copy on every update — fine for
functional code, wrong for table-heavy language implementations. Lua tables, Smalltalk
dicts, Erlang process dictionaries, and JS Map all need O(1) mutable associative storage.
Primitives to add:
- `make-hash-table` `[capacity]` → fresh mutable hash table
- `hash-table?` `v` → bool
- `hash-table-set!` `ht` `key` `val` → mutate in place
- `hash-table-ref` `ht` `key` `[default]` → value or default/error
- `hash-table-delete!` `ht` `key` → remove entry
- `hash-table-size` `ht` → integer
- `hash-table-keys` `ht` → list of keys
- `hash-table-values` `ht` → list of values
- `hash-table->alist` `ht` → list of (key . value) pairs
- `hash-table-for-each` `ht` `fn` → iterate (fn key val) for side effects
- `hash-table-merge!` `dst` `src` → merge src into dst in place
Steps:
- [x] Spec: add entries to `spec/primitives.sx`.
stdlib.hash-table module with 11 define-primitive entries appended to spec/primitives.sx.
- [x] OCaml: add `HashTable of (value, value) Hashtbl.t` to `sx_types.ml`; implement
all primitives in `hosts/ocaml/sx_primitives.ml`.
HashTable variant in sx_types.ml; type_of/inspect cases added; 11 primitives in sx_primitives.ml;
fixed _cek_call_ref reference for hash-table-for-each. 4385/1080 (+28).
- [x] JS bootstrapper: implement using JS `Map` in `hosts/javascript/platform.js`.
SxHashTable class with Map; _hash_table marker; dict?/type-of exclusion; apply() for for-each.
2137/2500 (+4 vs phase-9 baseline).
- [x] Tests: 30+ tests in `spec/tests/test-hash-table.sx` — set/ref/delete, size, iteration,
default on missing key, merge, keys/values lists.
28 tests; all pass OCaml+JS. Used empty? not assert= for empty-list comparisons.
- [x] Commit: `spec: mutable hash tables (make-hash-table/ref/set!/delete!/etc)`
Committed 133bdf52. Phase 10 complete.
---
## Phase 11 — Sequence protocol
Unified iteration over lists and vectors without conversion. Currently `map`/`filter`/
`for-each` only work on lists — you must `vector->list` first, which defeats the purpose
of vectors. A sequence protocol makes all collection operations polymorphic.
Approach: extend existing `map`/`filter`/`reduce`/`for-each`/`some`/`every?` to dispatch
on type (list → existing path, vector → index loop, string → char iteration). Add:
- `in-range` `start` `[end]` `[step]` → lazy range sequence (works with `for-each`/`map`)
- `sequence->list` `s` → coerce any sequence to list
- `sequence->vector` `s` → coerce any sequence to vector
- `sequence-length` `s` → length of any sequence
- `sequence-ref` `s` `i` → element by index (lists and vectors)
- `sequence-append` `s1` `s2` → concatenate two same-type sequences
Steps:
- [x] Spec: extend `map`/`filter`/`reduce`/`for-each`/`some`/`every?` in `spec/evaluator.sx`
to type-dispatch; add `in-range` lazy sequence type + helpers.
- [x] OCaml: update HO form dispatch; add `SxRange` or use lazy list; implement `sequence-*`
primitives.
seq_to_list helper before let-rec block; ho_setup_dispatch wraps all 7 coll bindings;
seq-to-list/sequence-to-list/vector/length/ref/append/in-range in sx_primitives.ml.
4385/1080 (all failures pre-existing hs-*/regex; 0 regressions).
- [x] JS bootstrapper: update.
Already done in Spec step (da4b526a) — sx-browser.js rebuilt with seqToList/sequenceToList/
sequenceToVector/sequenceLength/sequenceRef/sequenceAppend/inRange. 2137/2500 JS tests pass.
- [x] Tests: 30+ tests in `spec/tests/test-sequences.sx` — map over vector, filter over
range, for-each over string chars, sequence-append, sequence->list/vector coercions.
45 tests all passing: JS 2185/2498 (+48), OCaml 4424/1087 (+39). Fixed: vector? rename
(isVector), vectorLength/vectorRef/reverse aliases, in-range letrec→build-range,
sequence-length nil=0, assert-equal for list comparisons. Committed 0fe00bf7.
- [x] Commit: `spec: sequence protocol — polymorphic map/filter/for-each over list/vector/range`
Work landed across da4b526a (Spec), 7286629c (OCaml), 06a3eee1 (JS bootstrap), 0fe00bf7 (Tests).
---
## Phase 12 — gensym + symbol interning
Unique symbol generation. Tiny to implement; broadly needed: Prolog uses it for fresh
variable names, Common Lisp uses it constantly in macros, any hygienic macro system needs
it, and Smalltalk uses it for anonymous class/method naming.
Primitives to add:
- `gensym` `[prefix]` → unique symbol, e.g. `g42`, `var-17`. Counter-based, monotonically increasing.
- `symbol-interned?` `s` → bool — whether the symbol is in the global intern table
- `intern` `str` → symbol — intern a string as a symbol (string->symbol already exists; this is
the explicit interning operation for languages that distinguish interned vs uninterned)
Steps:
- [x] Spec: add `gensym` counter to evaluator state; implement in `spec/evaluator.sx`.
`string->symbol` already exists — `gensym` is just a counter-suffixed variant.
Added *gensym-counter*/gensym/string->symbol/symbol->string/intern/symbol-interned? to
evaluator.sx. Added string->symbol/symbol->string transpiler renames + platform.py aliases.
JS 2186/+1. OCaml builds. Committed edf4e525.
- [x] OCaml: add global gensym counter; implement primitives.
gensym_counter ref + gensym/string->symbol/symbol->string/intern/symbol-interned? in sx_primitives.ml.
Also fixed ListRef case in seq_to_list (both sx_ref.ml + sx_primitives.ml). 4431/1080 (was 4385/1080).
- [x] JS bootstrapper: implement.
Already done in Spec step. JS 2186/2497, all sequence tests pass.
- [x] Tests: 15+ tests in `spec/tests/test-gensym.sx` — uniqueness, prefix, symbol?, string->symbol round-trip.
19 tests. OCaml 4450/1080, JS 2205/2497, zero regressions.
- [x] Commit: `spec: gensym + symbol interning` — 0862a614
---
## Phase 13 — Character type
Common Lisp and Haskell have a distinct `Char` type that is not a string. Without it both
implementations are approximations — CL's `#\a` literal and Haskell's `'a'` both need a
real char value, not a length-1 string.
Primitives to add:
- `char?` `v` → bool
- `char->integer` `c` → Unicode codepoint integer
- `integer->char` `n` → char
- `char=?` `char<?` `char>?` `char<=?` `char>=?` → comparators
- `char-ci=?` `char-ci<?` etc. → case-insensitive comparators
- `char-alphabetic?` `char-numeric?` `char-whitespace?` → predicates
- `char-upper-case?` `char-lower-case?` → predicates
- `char-upcase` `char-downcase` → char → char
- `string->list` extended to return chars (not length-1 strings)
- `list->string` accepting chars
Also: `#\a` reader syntax for char literals (parser addition).
Steps:
- [x] Spec: add `SxChar` type to evaluator; add char literal syntax `#\a`/`#\space`/`#\newline`
to `spec/parser.sx`; implement all predicates + comparators.
- [x] OCaml: add `SxChar of char` to `sx_types.ml`; implement primitives.
- [x] JS bootstrapper: implement char type wrapping a codepoint integer.
- [x] Tests: 30+ tests in `spec/tests/test-chars.sx` — literals, char->integer round-trip,
comparators, predicates, upcase/downcase, string<->list with chars.
- [x] Commit: `spec: character type (char? char->integer #\\a literals + predicates)`
---
## Phase 14 — String ports
Needed for any language with a reader protocol: Common Lisp's `read`, Prolog's term parser,
Smalltalk's `printString`. Without string ports these all do their own character walking
on raw strings rather than treating a string as an I/O stream.
Primitives to add:
- `open-input-string` `str` → input port
- `open-output-string` → output port
- `get-output-string` `port` → string (flush output port to string)
- `input-port?` `output-port?` `port?` → predicates
- `read-char` `[port]` → char or eof-object
- `peek-char` `[port]` → char or eof-object (non-consuming)
- `read-line` `[port]` → string or eof-object
- `write-char` `char` `[port]` → void
- `write-string` `str` `[port]` → void
- `eof-object` → the eof sentinel
- `eof-object?` `v` → bool
- `close-port` `port` → void
Steps:
- [x] Spec: add port type + eof-object to evaluator; implement all primitives.
Ports are mutable objects with a position cursor (input) or accumulation buffer (output).
- [x] OCaml: add `SxPort` variant covering string-input-port and string-output-port;
Buffer.t for output, string+offset for input.
- [x] JS bootstrapper: implement port type.
- [x] Tests: 25+ tests in `spec/tests/test-ports.sx` — open/read/peek/eof, output accumulation,
read-line, write-char, close.
- [x] Commit: `spec: string ports (open-input-string/open-output-string/read-char/etc)` — 3d8937d7
---
## Phase 15 — Math completeness
Filling specific gaps that multiple language implementations need.
### 15a — modulo / remainder / quotient distinction
They differ on negative numbers — critical for Erlang `rem`, Haskell `mod`/`rem`, CL `mod`/`rem`:
- `quotient` `a` `b` → truncate toward zero (same sign as dividend)
- `remainder` `a` `b` → sign follows dividend (truncation division)
- `modulo` `a` `b` → sign follows divisor (floor division) — R7RS
### 15b — Trigonometry and transcendentals
Lua, Haskell, Erlang, CL all need: `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `exp`,
`log`, `sqrt`, `expt`. Check which are already present; add missing ones.
### 15c — GCD / LCM
`gcd` `a` `b` → greatest common divisor; `lcm` `a` `b` → least common multiple.
Needed by Haskell `Rational`, CL, and any language doing fraction arithmetic.
### 15d — Radix number parsing / formatting
`(number->string n radix)` → e.g. `(number->string 255 16)` → `"ff"`.
`(string->number s radix)` → e.g. `(string->number "ff" 16)` → `255`.
Needed by: Common Lisp, Smalltalk, Erlang integer formatting.
Steps:
- [x] Audit which trig / math functions are already in `spec/primitives.sx`; note gaps.
- [x] Spec + OCaml + JS: implement missing trig (`sin`/`cos`/`tan`/`asin`/`acos`/`atan`/`exp`/`log`).
- [x] Spec + OCaml + JS: `quotient`/`remainder`/`modulo` with correct negative semantics.
- [x] Spec + OCaml + JS: `gcd`/`lcm`.
- [x] Spec + OCaml + JS: radix variants of `number->string`/`string->number`.
- [x] Tests: 40+ tests in `spec/tests/test-math.sx`.
- [x] Commit: `spec: math completeness — trig, quotient/remainder/modulo, gcd/lcm, radix`
---
## Phase 16 — Rational numbers
Haskell's `Rational` type and Common Lisp ratios (`1/3`) both need this. Natural extension
of the numeric tower (Phase 2) — rationals are the third numeric type alongside int and float.
Primitives to add:
- `make-rational` `numerator` `denominator` → rational (auto-reduced by GCD)
- `rational?` `v` → bool
- `numerator` `r` → integer
- `denominator` `r` → integer
- Reader syntax: `1/3` parsed as rational literal
- Arithmetic: `(+ 1/3 1/6)` → `1/2`; `(* 1/3 3)` → `1`; mixed int/rational → rational
- `exact->inexact` on rational → float; `inexact->exact` on float → rational approximation
- `(number->string 1/3)` → `"1/3"`
Steps:
- [x] Spec: add `SxRational` type; add `n/d` reader syntax to `spec/parser.sx`; extend
all arithmetic primitives for rational contagion (int op rational → rational, rational
op float → float).
- [x] OCaml: add `SxRational of int * int` (stored in reduced form); implement all arithmetic.
as_number + safe_eq extended for cross-type rational equality (= 2.5 5/2) → true.
- [x] JS bootstrapper: implement rational type.
JS keeps int/int → float for CSS backward compatibility; SxRational class with _rational marker.
- [x] Tests: 30+ tests in `spec/tests/test-rationals.sx` — literals, arithmetic, reduction,
mixed numeric tower, exact<->inexact conversion. 62 tests, all pass.
- [x] Commit: `spec: rational numbers — 1/3 literals, arithmetic, numeric tower integration`
Committed 036022cc. JS: 2232 passed. OCaml: 4532 passed (+11).
---
## Phase 17 — read / write / display
Completes the I/O model. Builds on string ports (Phase 14) and char type (Phase 13).
`read` parses any SX value from a port; `write` serializes with quoting (round-trippable);
`display` serializes without quoting (human-readable). Common Lisp's `read` macro,
Prolog term I/O, and Smalltalk's `printString` all need this.
Primitives to add:
- `read` `[port]` → SX value or eof-object — full SX parser reading from a port
- `read-char` already in Phase 14; `read` uses it internally
- `write` `val` `[port]` → void — serializes with quotes: `"hello"`, `#\a`, `(1 2 3)`
- `display` `val` `[port]` → void — serializes without quotes: `hello`, `a`, `(1 2 3)`
- `newline` `[port]` → void — writes `\n`
- `write-to-string` `val` → string — convenience: `(write val (open-output-string))`
- `display-to-string` `val` → string — convenience
Steps:
- [x] Spec: implement `read` in `spec/evaluator.sx` — wraps the existing parser to read
one datum from a port cursor; handles eof gracefully.
- [x] Spec: implement `write`/`display`/`newline` — extend the existing serializer for
port output; `write` quotes strings + uses `#\` for chars, `display` does not.
- [x] OCaml: wire `read` through port type; implement `write`/`display` output path.
- [x] JS bootstrapper: implement.
- [x] Tests: 25+ tests in `spec/tests/test-read-write.sx` — read string literal, read list,
read eof, write round-trip, display vs write quoting, newline, write-to-string.
- [x] Commit: `spec: read/write/display — S-expression reader/writer on ports`
---
## Phase 18 — Sets
O(1) membership testing. Distinct from hash tables (unkeyed) and lists (O(n)).
Erlang has sets as a stdlib staple, Haskell `Data.Set`, APL uses set operations
constantly, Common Lisp has `union`/`intersection` on lists but a native set is O(1).
Primitives to add:
- `make-set` `[list]` → fresh set, optionally seeded from list
- `set?` `v` → bool
- `set-add!` `s` `val` → void
- `set-member?` `s` `val` → bool
- `set-remove!` `s` `val` → void
- `set-size` `s` → integer
- `set->list` `s` → list (unspecified order)
- `list->set` `lst` → set
- `set-union` `s1` `s2` → new set
- `set-intersection` `s1` `s2` → new set
- `set-difference` `s1` `s2` → new set (elements in s1 not in s2)
- `set-for-each` `s` `fn` → iterate for side effects
- `set-map` `s` `fn` → new set of mapped values
Steps:
- [x] Spec: add entries to `spec/primitives.sx`.
- [x] OCaml: implement using `Hashtbl.t` with unit values (or a proper `Set` functor
with a comparison function); add `SxSet` to `sx_types.ml`.
- [x] JS bootstrapper: implement using JS `Set`.
- [x] Tests: 30+ tests in `spec/tests/test-sets.sx` — add/member/remove, union/intersection/
difference, list conversion, for-each, size.
- [x] Commit: `spec: sets (make-set/set-add!/set-member?/union/intersection/etc)`
---
## Phase 19 — Regular expressions as primitives
`lib/js/regex.sx` is a pure-SX regex engine already written. Promoting it to a primitive
gives every language free regex without reinventing: Lua patterns, Tcl `regexp`, Ruby regex,
JS regex, Erlang `re` module. Mostly a wiring job — the implementation exists.
Primitives to add:
- `make-regexp` `pattern` `[flags]` → regexp object (`flags`: `"i"` case-insensitive, `"g"` global, `"m"` multiline)
- `regexp?` `v` → bool
- `regexp-match` `re` `str` → match dict `{:match "..." :start N :end N :groups (...)}` or nil
- `regexp-match-all` `re` `str` → list of match dicts
- `regexp-replace` `re` `str` `replacement` → string with first match replaced
- `regexp-replace-all` `re` `str` `replacement` → string with all matches replaced
- `regexp-split` `re` `str` → list of strings (split on matches)
- Reader syntax: `#/pattern/flags` for regexp literals (parser addition)
Steps:
- [x] Audit `lib/js/regex.sx` — understand the API it already exposes; map to the
primitive API above.
- [x] Spec: add `SxRegexp` type to evaluator; add `#/pattern/flags` literal syntax to
`spec/parser.sx`; wire `lib/js/regex.sx` engine as the implementation.
- [x] OCaml: implement using OCaml `Re` library (or `Str`); add `SxRegexp` to types.
- [x] JS bootstrapper: use native JS `RegExp`; wrap in the primitive API.
- [x] Tests: 30+ tests in `spec/tests/test-regexp.sx` — basic match, groups, replace,
replace-all, split, flags (case-insensitive), no-match nil return.
- [x] Commit: `spec: regular expressions (make-regexp/regexp-match/regexp-replace + #/pat/ literals)`
---
## Phase 20 — Bytevectors
R7RS standard. Needed for WebSocket binary frames (E36), binary protocol parsing, and
efficient string encoding. Also the foundation for proper Unicode: `string->utf8` /
`utf8->string` require a byte array type.
Primitives to add:
- `make-bytevector` `n` `[fill]` → bytevector of n bytes (fill defaults to 0)
- `bytevector?` `v` → bool
- `bytevector-length` `bv` → integer
- `bytevector-u8-ref` `bv` `i` → byte 0–255
- `bytevector-u8-set!` `bv` `i` `byte` → void
- `bytevector-copy` `bv` `[start]` `[end]` → fresh copy
- `bytevector-copy!` `dst` `at` `src` `[start]` `[end]` → in-place copy
- `bytevector-append` `bv...` → concatenated bytevector
- `utf8->string` `bv` `[start]` `[end]` → string decoded as UTF-8
- `string->utf8` `str` `[start]` `[end]` → bytevector UTF-8 encoded
- `bytevector->list` / `list->bytevector` → conversion
Steps:
- [x] Spec: add `SxBytevector` type; implement all primitives in `spec/evaluator.sx` / `spec/primitives.sx`.
- [x] OCaml: add `SxBytevector of bytes` to `sx_types.ml`; implement primitives using
OCaml `Bytes`.
- [x] JS bootstrapper: implement using `Uint8Array`.
- [x] Tests: 30+ tests in `spec/tests/test-bytevectors.sx` — construction, ref/set, copy,
append, utf8 round-trip, slice.
- [x] Commit: `spec: bytevectors (make-bytevector/u8-ref/u8-set!/utf8->string/etc)`
---
## Phase 21 — format
CL-style string formatting beyond `str`. `(format "Hello ~a, age ~d" name age)`.
Haskell `printf`, Erlang `io:format`, CL `format`, and general string templating all use this idiom.
Directives:
- `~a` — display (no quotes)
- `~s` — write (with quotes)
- `~d` — decimal integer
- `~x` — hexadecimal integer
- `~o` — octal integer
- `~b` — binary integer
- `~f` — fixed-point float
- `~e` — scientific notation float
- `~%` — newline
- `~&` — fresh line (newline only if not already at start of line)
- `~~` — literal tilde
- `~t` — tab
Signature: `(format template arg...)` → string.
Optional: `(format port template arg...)` — write to port directly.
Steps:
- [x] Spec: implement `format` as a pure SX function in `spec/stdlib.sx` — parses
`~X` directives, dispatches to `display`/`write`/`number->string` as appropriate.
Pure SX: no host calls needed. Self-hosting — uses string-buffer (Phase 5) internally.
- [x] OCaml: expose as a primitive (or let it run as SX through the evaluator).
Added format-decimal OCaml primitive; fixed lib/r7rs.sx number->string to support radix.
- [x] JS bootstrapper: same.
- [x] Tests: 28 tests in `spec/tests/test-format.sx` — each directive, multiple args,
nested format, `~~` escape. 28/28 pass on both JS and OCaml.
- [x] Commit: `spec: format — CL-style string formatting (~a ~s ~d ~x ~% etc)` — 4d7b3e29
---
## Phase 22 — Language sweep
Replace workarounds with primitives. One language per fire (or per sub-item for big ones).
Start with blank slates (CL, APL, Ruby, Tcl) — they haven't committed to workarounds yet.
**Scope per language:** only `lib/<lang>/**`. Don't touch spec or other languages.
Brief each language's loop agent (or do inline) after rebasing their branch onto architecture.
- [x] Restart CL/APL/Ruby/Tcl loops with updated briefing pointing to new primitives.
Added `## SX primitive baseline` section to plans/common-lisp-on-sx.md,
plans/apl-on-sx.md, plans/ruby-on-sx.md, plans/tcl-on-sx.md. f43659ce.
- [x] Common Lisp: char type (`#\a`); string ports + `read`/`write` for reader/printer;
gensym for macros; rational numbers for CL ratios; multiple values; sets for CL set ops;
`modulo`/`remainder`/`quotient`; radix formatting; `format` for `cl:format`.
lib/common-lisp/runtime.sx (103 forms) + test.sh (68/68 pass). 1ad8e74a.
- [x] Lua: vectors for arrays; hash tables for Lua tables; `delay`/`force` for lazy iterators;
regexp for Lua pattern matching; trig from math completeness; bytevectors for binary I/O.
math/string/table stdlib tables + lua-force. 185/185 pass. ec3512d6.
- [x] Erlang: numeric tower for float/int; bitwise ops for bitmatch; multiple values for
multi-return; sets for Erlang sets; `remainder` for `rem`; regexp for `re` module.
lib/erlang/runtime.sx (63 forms) + test.sh (55/55 pass). 3c0a9632.
- [x] Haskell: numeric tower for `Num`/`Integral`/`Fractional`; promises for lazy evaluation
(critical); multiple values for tuples; rational numbers for `Rational`; char type for
`Char`; `gcd`/`lcm`; sets for `Data.Set`; `read`/`write` for `Show`/`Read` instances.
lib/haskell/runtime.sx (113 forms) + tests/runtime.sx (143/143 pass). c02ffcf3.
- [x] JS: vectors for Array; hash tables for `Map`; sets for `Set`; bitwise ops for typed
arrays; regexp for JS regex; bytevectors for `Uint8Array`; radix formatting.
lib/js/stdlib.sx (36 forms) + test.sh epochs 6000-6032 (25/25 pass). COMMIT.
- [x] Smalltalk: vectors for `Array new:`; hash tables for `Dictionary new`; sets for
`Set new`; char type for `Character`; string ports + `read`/`write` for `printString`.
lib/smalltalk/runtime.sx (72 forms) + tests/runtime.sx (86/86 pass). COMMIT.
- [x] APL: vectors as core array type; bitwise ops for array masks; sets for APL set ops;
sequence protocol for rank-polymorphic operations; format for APL output formatting.
lib/apl/runtime.sx (60 forms) + tests/runtime.sx (73/73 pass). COMMIT.
- [x] Ruby: coroutines for fibers; hash tables for `Hash`; sets for `Set`; regexp for
Ruby regex; string ports for `StringIO`; bytevectors for `String` binary encoding.
lib/ruby/runtime.sx (61 forms) + tests/runtime.sx (76/76 pass). COMMIT.
Note: rb-fiber-yield from letrec-bound lambdas fails (JIT VM can't invoke callcc
continuations as escapes); workaround: use top-level helper fns for recursive yields.
- [x] Tcl: string ports for Tcl channel abstraction; string-buffer for `append`; coroutines
for Tcl coroutines; regexp for Tcl `regexp`; format for Tcl `format`.
lib/tcl/runtime.sx (37 forms) + tests/runtime.sx (56/56 pass). COMMIT.
- [x] Forth: bitwise ops (core); string-buffer for word-definition accumulation; bytevectors
for Forth's raw memory model.
lib/forth/runtime.sx (36 forms) + tests/runtime.sx (64/64 pass). COMMIT.
---
## Ground rules
- Work on the `architecture` branch in `/root/rose-ash` (main worktree).
- Use sx-tree MCP for all `.sx` file edits. Never use raw Edit/Write/Read on `.sx` files.
- Commit after each concrete unit of work. Never leave the branch broken.
- Never push to `main` — only push to `origin/architecture`.
- Update this checklist every fire: tick `[x]` done, add inline notes on blockers.
---
## Progress log
_Newest first._
- 2026-05-01: Phase 22 Forth done — runtime.sx (36 forms): bitwise (AND/OR/XOR/INVERT/LSHIFT/RSHIFT/2*/2//bit-count/integer-length/within + arithmetic helpers), string-buffer (emit!/type!/value/length/clear!/emit-int!), memory (cfetch/cstore/fetch/store/move!/fill!/erase!/mem->list). 64/64 tests. 8019e572.
- 2026-05-01: Phase 22 Tcl done — runtime.sx (37 forms): string-buffer (append accumulator), channel (read/write ports with gets/read/puts), regexp (make-regexp wrappers), format (%s/%d/%f/%x/%o/%% manual char scan), coroutine (call/cc, top-level helper pattern). 56/56 tests. 3e07727d.
- 2026-05-01: Phase 22 Ruby done — runtime.sx (61 forms): Hash (list-of-pairs dict-backed), Set (make-set, (set item) order), Regexp (make-regexp wrappers), StringIO (write buf + rewind/char read), Bytevectors (thin wrappers), Fiber (call/cc; letrec JIT workaround: use top-level helpers). 76/76 tests. 182e6f63.
- 2026-05-01: Phase 22 APL done — runtime.sx (60 forms): iota/rho/at, rank-polymorphic dyadic/monadic helpers, arithmetic/comparison/boolean/bitwise element-wise, reduce/scan, take/drop/rotate/compress/index, set ops (member/nub/union/intersect/without), format. 73/73 tests. COMMIT.
- 2026-05-01: Phase 22 Smalltalk done — runtime.sx (72 forms): numeric helpers, Character (1-indexed Array backed by dict), Dictionary (list-of-pairs any-key map), Set (make-set), WriteStream/ReadStream/printString. set-member? (set item) order. 86/86 tests. COMMIT.
- 2026-05-01: Phase 22 JS done — stdlib.sx (36 forms): bitwise (truncate not js-num-to-int; set-member? takes (set item) order), Map (dict-backed pairs), Set (SX make-set), RegExp (callable lambda). 25/25 new tests pass; total 492/585. COMMIT.
- 2026-05-01: Phase 22 Haskell done — runtime.sx (113 forms): numeric tower (hk-div floor semantics), rational (dict GCD-normalised), hk-force (promises), Data.Char, Data.Set, Data.List, Maybe/Either, tuples, string helpers, hk-show. 148/148 tests. c02ffcf3.
- 2026-05-01: Phase 22 Erlang done — runtime.sx (63 forms): numeric tower, bitwise (band/bor/bxor/bnot/bsl/bsr), sets, re module, list BIFs, type conversions, ok/error tuples. 55/55 tests. 3c0a9632.
- 2026-05-01: Phase 22 Lua done — math/string/table stdlib tables + lua-force in lib/lua/runtime.sx. 185/185 tests (28 new). ec3512d6.
- 2026-05-01: Phase 22 CL done — runtime.sx (103 forms): type preds, arithmetic, chars, format, gensym, values, sets, radix, list utils. cl-empty? guards nil/() split. 68/68 tests. 1ad8e74a.
- 2026-05-01: Phase 22 step 1 — SX primitive baseline added to CL/APL/Ruby/Tcl plans. f43659ce.
- 2026-05-01: Phase 21 complete — format (~a ~s ~d ~x ~o ~b ~f ~% ~& ~~ ~t) as pure SX in spec/stdlib.sx. Fixed lib/r7rs.sx number->string to support optional radix; added format-decimal OCaml primitive. 28/28 tests on both JS and OCaml. 4d7b3e29.
- 2026-04-26: Phase 7 complete — bitwise-and/or/xor/not + arithmetic-shift + bit-count + integer-length. OCaml: land/lor/lxor/lnot/lsl/asr + Kernighan popcount + lsr loop for integer-length. JS: bitwise ops + Hamming weight + Math.clz32. 26 tests, 158 assertions, all pass. a8a79dc9.
- 2026-04-26: Phase 6 complete — JS+Tests+Commit all ticked. JS needed no changes (spec-level forms). 40/40 ADT tests pass JS. 2032/2500 JS total (+67 vs phase-4). Phase 6 fully landed: 6c872107+0dc7e159+5d1913e7. Phase 7 (bitwise) next.
- 2026-04-26: Phase 6 OCaml done — Dict-based ADT (no native SxAdt type needed); hand-written sf_define_type in bootstrap.py FIXUPS (skipped from transpile — &rest params + empty-dict {} literals); registered via register_special_form; step_limit/step_count added to PREAMBLE. 172 assertions pass (test-adt). Full suite 4280/1080 (was 4243/1117, +37). Committed 5d1913e7.
- 2026-04-26: Phase 6 Spec match done — ADT case added to match-pattern in spec/evaluator.sx: checks (list? pattern)+(symbol? first)+(dict? value)+(get value :_adt), then matches :_ctor+arity and recursively binds field patterns. No-clause error now uses make-cek-value+raise-eval-frame so guard can catch it. 20 new match tests pass; 40/40 total ADT tests green. Zero regressions.
- 2026-04-26: Phase 6 Spec define-type done — sf-define-type registered via register-special-form! in spec/evaluator.sx; AdtValue as {:_adt true :_type "..." :_ctor "..." :_fields (list ...)}; ctor fns + arity checking + Name?/Ctor? predicates + Ctor-field accessors; *adt-registry* dict populated per define-type call. 20/20 JS tests pass in spec/tests/test-adt.sx. OCaml define-type is next task.
- 2026-04-26: Phase 6 Design done — plans/designs/sx-adt.md written. Covers define-type/match syntax, AdtValue CEK runtime, stepSfDefineType+MatchFrame dispatch, exhaustiveness warnings, recursive types, nested patterns, wildcard _. 3-phase impl plan. Next fire: Spec implement define-type.
- 2026-04-26: Phase 5 complete — string buffer fully landed (d98b5fa2). 17 tests, 17/17 OCaml+JS. Phase 6 (ADTs) next.
- 2026-04-26: Phase 5 Spec+OCaml+JS step done — StringBuffer of Buffer.t in sx_types.ml; make-string-buffer/append!/->string/length/string-buffer? in sx_primitives.ml; SxStringBuffer with _string_buffer marker + typeOf/dict? fixes in platform.py; JS rebuilt. 17/17 tests OCaml+JS.
- 2026-04-26: Phase 4 complete — coroutine primitive fully landed (4 commits: spec library + OCaml verified + JS pre-load + 27 tests). Phase 5 (string buffer) next.
- 2026-04-26: Phase 4 Tests step done — 27 tests total (10 new: state field inspection, yield-from-helper, initial-arg-ignored, mutable-closure, complex-values, round-robin, factory-no-state, non-coroutine-error). 27/27 OCaml+JS.
- 2026-04-26: Phase 4 JS step done — all CEK primitives already in sx-browser.js; fix was pre-loading spec/coroutines.sx+spec/signals.sx in run_tests.js so (import (sx coroutines)) resolves synchronously. 17/17 coroutine tests pass JS. 1965/2500 total (+25), zero new failures.
- 2026-04-26: Phase 4 OCaml step done — no native SxCoroutine type needed; existing cek-step-loop/cek-resume/perform/make-cek-state primitives in run_tests.ml fully support the spec/coroutines.sx library. 284/284 pass (coroutines+vectors+numeric-tower+dynamic-wind), zero regressions.
- 2026-04-26: Phase 4 Spec step done — spec/coroutines.sx define-library with make-coroutine/coroutine-resume/coroutine-yield/coroutine?/coroutine-alive?; make-coroutine stub in evaluator.sx; 17/17 coroutine tests pass (OCaml). Key insight: coroutine body must use (define loop (fn...)) + (loop 0) not named let — named let uses cek_call→cek_run which errors on IO suspension.
- 2026-05-01: Phase 10 complete — mutable hash tables. HashTable variant in OCaml; JS Map-based SxHashTable. 11 primitives: make-hash-table/hash-table?/set!/ref/delete!/size/keys/values/->alist/for-each/merge!. 28 tests, all pass OCaml+JS. 133bdf52.
- 2026-05-01: Phase 9 complete — delay/force/delay-force/make-promise/promise?. Dict-based promise {:_promise :forced :thunk :value}; :_iterative flag for delay-force chain following. 25/25 tests OCaml (4357) and JS (2109). Committed e44cb89a.
- 2026-05-01: Phase 8 complete — values/call-with-values/let-values/define-values. Dict marker {:_values true :_list [...]} (no new type). step-sf-define desugars shorthand (define (f x) body) on both hosts. 25/25 tests OCaml+JS. Committed 43cc1d90.
- 2026-04-26: Phase 3 complete — OCaml+JS done. CallccContinuation gains winders-depth int; make_callcc_continuation/callcc_continuation_winders_len wired; wind-after/wind-return CekFrame fields fixed (cf_f=after-thunk, cf_extra=winders-len, cf_name=body-result); get_val + transpiler.sx updated. 8/8 dynamic-wind tests pass on OCaml; 235/235 (callcc+guard+do+r7rs) zero regressions. Committed 6602ec8c.
- 2026-04-26: Phase 3 Spec+Tests done — dynamic-wind CEK implementation: wind-after/wind-return frames, *winders* stack, kont-unwind-to-handler, wind-escape-to. callcc frame stores winders-len in continuation; callcc-continuation? calls wind-escape-to before escape. 8/8 dynamic-wind tests pass (normal return, raise, call/cc, nested LIFO, guard ordering). 1948/2500 JS (+8). Zero regressions. Committed a9d5a108.
- 2026-04-26: Phase 2 complete — Verify+Commit done. OCaml 4874/394, JS 1940/2500 (+60). No regressions. 6 JS-only failures are float≡int platform-inherent. Phase 2 fully landed across 4 commits.
- 2026-04-26: Phase 2 JS bootstrapper done — integer?/float?/exact?/inexact? added (Number.isInteger); truncate/remainder/modulo/random-int/exact->inexact/inexact->exact/parse-number added. Fixed sx_server.ml epoch+blob+io-response protocol for Integer type. JS: 1940/2500 (+60). OCaml: 4874/394 baseline. 6 JS tests fail (JS float≡int platform limit). Committed b12a22e6.
- 2026-04-26: Phase 2 Spec done — integer?/float? predicates added to spec/primitives.sx; floor/ceil/truncate :returns updated to "integer"; / to "float"; exact->inexact/inexact->exact docs and returns updated; float contagion documented on +/-/*; 4874/394 baseline. Committed 45ec5535.
- 2026-04-26: Phase 2 OCaml+Tests done — `Integer of int` / `Number of float` in sx_types.ml; float contagion across all arithmetic; floor/truncate/round → Integer; integer?/float?/exact?/inexact?/exact->inexact/inexact->exact; 92/92 numeric tower tests pass; 4874 total (394 pre-existing unchanged). Committed c70bbdeb.
- 2026-04-26: Phase 1 complete — JS step done. Fixed fundamental lambda binding bug (index-of on arrays returned -1 not NIL, making bind-lambda-params mis-fire &rest branch). Added _lastErrorKont_/hostError/try-catch stubs. 42/42 vector tests pass. 1847 std / 2362 full passing (up from 5). Committed.
- 2026-04-25: Phase 1 spec step done — all 10 vector primitives in spec/primitives.sx have full :as type annotations, :returns, :doc; make-vector optional fill param added.
- 2026-04-25: Phase 1 OCaml step done — bounds-checked vector-ref/set!, vector-copy now accepts optional start/end, spec/primitives.sx doc updated. 10/10 r7rs vector tests pass, 4747 total (394 pre-existing hs-upstream fails unchanged).
- 2026-04-25: Phase 0 complete — stopped CL/APL/Ruby/Tcl loops (all 4 idle at shell); confirmed E38 (tokenizer :end/:line) and E39 (WebWorker stub) both have implementation commits.
- 2026-05-01: Phase 20 complete — bytevectors. SxBytevector of bytes in OCaml using Bytes; Uint8Array-backed SxBytevector in JS. 12 primitives: make-bytevector, bytevector?, bytevector-length, bytevector-u8-ref, bytevector-u8-set!, bytevector-copy, bytevector-copy!, bytevector-append, utf8->string, string->utf8, bytevector->list, list->bytevector. 32 tests, all pass. JS 2535, OCaml 4725. a3811545.
- 2026-05-01: Phase 19 complete — regular expressions. SxRegexp(src,flags,Re.re) in OCaml via Re.Pcre; SxRegexp wrapper around JS RegExp. 9 primitives: make-regexp, regexp?, regexp-source, regexp-flags, regexp-match, regexp-match-all, regexp-replace, regexp-replace-all, regexp-split. Match dicts with :match/:start/:end/:groups. 32 tests, all pass. JS 2503, OCaml 4693. d8d5588e.
- 2026-05-01: Phase 18 complete — sets. SxSet as (string,value) Hashtbl keyed by inspect(val) in OCaml; Map keyed by write-to-string in JS. 13 primitives: make-set, set?, set-add!, set-member?, set-remove!, set-size, set->list, list->set, set-union, set-intersection, set-difference, set-for-each, set-map. 33 tests, all pass. JS 2469, OCaml 4659. 3b0ac67a.
- 2026-05-01: Phase 17 complete — read/write/display. OCaml: sx_write_val/sx_display_val helpers; read via Sx_parser.read_value with #t/#f and N/D rational support added to parser; postprocess ()→Nil. JS: sxReadNormalize (#t/#f→true/false), sxReadConvert (()→NIL), sxEq list equality, sxWriteVal symbol/keyword name fix (v.name not v._sym), readerMacroGet registry. 42 tests (test-read-write.sx), all pass both hosts. JS 2436, OCaml 4626. 7d329f02.
- 2026-05-01: Phase 16 complete — rational numbers. SxRational type in OCaml (Rational of int*int, reduced, denom>0) and JS (SxRational class, _rational marker). n/d reader in spec/parser.sx. Arithmetic contagion: int op rational → rational, rational op float → float. JS keeps int/int → float for CSS compat. OCaml as_number+safe_eq extended for cross-type rational equality. 62 tests in test-rationals.sx, all pass. JS 2232, OCaml 4532 (+11). 036022cc.
- 2026-05-01: Phase 15 complete — math completeness. stdlib.math module: sin/cos/tan/asin/acos/atan(1-2 args)/exp/log/expt/quotient/gcd/lcm/number->string(radix)/string->number(radix). OCaml atan updated for optional 2nd arg. Strict radix parsing in JS string->number. 44 tests in test-math.sx, all pass. JS 2311/4801, OCaml 4547/5629. be2b11ac.
- 2026-05-01: Phase 14 OCaml done — Eof + Port{PortInput/PortOutput} in sx_types.ml; 15 port primitives in sx_primitives.ml; raw_serialize updated; 4532/4532 (+39, zero regressions). 8ba0a33f.
- 2026-05-01: Phase 14 Spec+JS+Tests+Commit done — port type {_port,_kind,_source/_buffer,_pos,_closed}; eof singleton; 15 primitives in spec/primitives.sx (stdlib.ports) + platform.py; 39/39 tests in test-ports.sx. Committed 3d8937d7. OCaml step next.
- 2026-05-01: Phase 13 OCaml done — Char of int in sx_types.ml; #\ reader in sx_parser.ml; all char primitives in sx_primitives.ml; fixed get_val for Integer n list indexing (was Number-only); fixed raw_serialize for Integer/Char. 4493/4493 (+43, zero regressions). b939becd.
- 2026-05-01: Phase 13 Spec+JS+Tests+Commit done — SxChar tagged {_char,codepoint}; char? char->integer integer->char char-upcase/downcase; 10 comparators (ordered+ci); 5 predicates; string->list/list->string as platform primitives; #\a #\space #\newline reader syntax in spec/parser.sx; js-char-renames dict in transpiler.sx; 43/43 tests pass JS (2254/4745). Committed 4b600f17. OCaml step next.
- 2026-05-01: Phase 12 complete — gensym + symbol interning. gensym_counter/gensym/string->symbol/symbol->string/intern/symbol-interned? in spec + OCaml + JS. Fixed ListRef case in seq_to_list (both hosts). 19 tests, all pass. OCaml 4450/1080, JS 2205/2497. Commits: edf4e525 Spec, 0862a614 OCaml+Tests.
- 2026-05-01: Phase 11 complete — sequence protocol done. Commits: da4b526a Spec, 7286629c OCaml, 06a3eee1 JS, 0fe00bf7 Tests. JS 2185/+48, OCaml 4424/+39.
- 2026-05-01: Phase 11 Tests done — 45 tests in test-sequences.sx all passing (JS 2185/+48, OCaml 4424/+39). Fixed vector? rename, vectorLength/vectorRef/reverse aliases, in-range letrec→build-range, sequence-length nil, assert-equal for lists. Committed 0fe00bf7.
- 2026-05-01: Phase 11 JS bootstrapper step done — confirmed sx-browser.js current (built in Spec step da4b526a); 19 sequence primitive refs in output; 2137/2500 JS tests passing.
- 2026-05-01: Phase 11 OCaml step done — seq_to_list helper added before let-rec; ho_setup_dispatch wraps all 7 coll bindings with seq_to_list; seq-to-list/sequence-to-list/to-vector/length/ref/append + in-range primitives in sx_primitives.ml. 4385/4385 baseline unchanged, 0 regressions. Committed 7286629c.
- 2026-05-01: Phase 11 Spec step done — seq-to-list coercion helper; ho-setup-dispatch extended with seqToList on all collection args; sequence-to-list/vector/length/ref/append + in-range added to evaluator.sx. Restored 3 accidentally-deleted make-cek-state/value/suspended definitions. Fixed 8 shorthand define forms + added vector->list/list->vector transpiler renames. JS: 2137 passing (+28 vs HEAD baseline of 2109).

7
plans/agent-briefings/prolog-loop.md
View File

@@ -11,7 +11,7 @@ isolation: worktree
## Prompt
You are the sole background agent working `/root/rose-ash/plans/prolog-on-sx.md`. You run in an isolated git worktree. You work the plan's roadmap forever, one commit per feature. You never push.
You are the sole background agent working `/root/rose-ash/plans/prolog-on-sx.md`. You run in an isolated git worktree. You work the plan's roadmap forever, one commit per feature. Push to `origin/loops/prolog` after every commit.
## Restart baseline — check before iterating
@@ -39,12 +39,13 @@ Every iteration: implement → test → commit → tick `[ ]` in plan → append
## Ground rules (hard)
- **Scope:** only `lib/prolog/**` and `plans/prolog-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, other `lib/<lang>/` dirs, `lib/stdlib.sx`, or `lib/` root. Prolog primitives go in `lib/prolog/runtime.sx`.
- **Scope:** only `lib/prolog/**` and `plans/prolog-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, other `lib/<lang>/` dirs, `lib/stdlib.sx`, or `lib/` root. Prolog primitives go in `lib/prolog/runtime.sx`. You may **read** `lib/hyperscript/runtime.sx` to understand the hook API but do not edit it — `hs-set-prolog-hook!` is already implemented there.
- **Hyperscript bridge is NOT blocked:** `lib/prolog/hs-bridge.sx` already exists and `lib/hyperscript/runtime.sx` already exports `hs-set-prolog-hook!` / `hs-prolog-hook`. The Phase 5 DSL item just needs tests and wiring.
- **NEVER call `sx_build`.** 600s watchdog will kill you before OCaml finishes. If sx_server binary is broken, add Blockers entry and stop.
- **Shared-file issues** → plan's Blockers section with a minimal repro. Don't fix them.
- **Delimited continuations** are in `lib/callcc.sx` + `spec/evaluator.sx` Step 5 (IO suspension via `perform`/`cek-resume`). `sx_summarise` spec/evaluator.sx first — it's 2300+ lines.
- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits. Never `Edit`/`Read`/`Write` on `.sx`.
- **Worktree:** commit locally. Never push. Never touch `main`.
- **Worktree:** commit, then push to `origin/loops/prolog`. Never touch `main`.
- **Commit granularity:** one feature per commit.
- **Plan file:** update Progress log + tick boxes every commit.
- **If blocked** for two iterations on the same issue, add to Blockers and move on.

83
plans/agent-briefings/ruby-loop.md Normal file
View File

@@ -0,0 +1,83 @@
# ruby-on-sx loop agent (single agent, queue-driven)
Role: iterates `plans/ruby-on-sx.md` forever. Fibers via delcc is the headline showcase — `Fiber.new`/`Fiber.yield`/`Fiber.resume` are textbook delimited continuations with sugar, where MRI does it via C-stack swapping. Plus blocks/yield (lexical escape continuations, same shape as Smalltalk's non-local return), method_missing, and singleton classes.
```
description: ruby-on-sx queue loop
subagent_type: general-purpose
run_in_background: true
isolation: worktree
```
## Prompt
You are the sole background agent working `/root/rose-ash/plans/ruby-on-sx.md`. Isolated worktree, forever, one commit per feature. Never push.
## Restart baseline — check before iterating
1. Read `plans/ruby-on-sx.md` — roadmap + Progress log.
2. `ls lib/ruby/` — pick up from the most advanced file.
3. If `lib/ruby/tests/*.sx` exist, run them. Green before new work.
4. If `lib/ruby/scoreboard.md` exists, that's your baseline.
## The queue
Phase order per `plans/ruby-on-sx.md`:
- **Phase 1** — tokenizer + parser. Keywords, identifier sigils (`@` ivar, `@@` cvar, `$` global), strings with interpolation, `%w[]`/`%i[]`, symbols, blocks `{|x| …}` and `do |x| … end`, splats, default args, method def
- **Phase 2** — object model + sequential eval. Class table, ancestor-chain dispatch, `super`, singleton classes, `method_missing` fallback, dynamic constant lookup
- **Phase 3** — blocks + procs + lambdas. Method captures escape continuation `^k`; `yield` / `return` / `break` / `next` / `redo` semantics; lambda strict arity vs proc lax
- **Phase 4** — **THE SHOWCASE**: fibers via delcc. `Fiber.new`/`Fiber.resume`/`Fiber.yield`/`Fiber.transfer`. Classic programs (generator, producer-consumer, tree-walk) green
- **Phase 5** — modules + mixins + metaprogramming. `include`/`prepend`/`extend`, `define_method`, `class_eval`/`instance_eval`, `respond_to?`/`respond_to_missing?`, hooks
- **Phase 6** — stdlib drive. `Enumerable` mixin, `Comparable`, Array/Hash/Range/String/Integer methods, drive corpus to 200+
Within a phase, pick the checkbox that unlocks the most tests per effort.
Every iteration: implement → test → commit → tick `[ ]` → Progress log → next.
## Ground rules (hard)
- **Scope:** only `lib/ruby/**` and `plans/ruby-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, other `lib/<lang>/` dirs, `lib/stdlib.sx`, or `lib/` root. Ruby primitives go in `lib/ruby/runtime.sx`.
- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers entry, stop.
- **Shared-file issues** → plan's Blockers with minimal repro.
- **Delimited continuations** are in `lib/callcc.sx` + `spec/evaluator.sx` Step 5. `sx_summarise` spec/evaluator.sx first — 2300+ lines.
- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits.
- **Worktree:** commit locally. Never push. Never touch `main`.
- **Commit granularity:** one feature per commit.
- **Plan file:** update Progress log + tick boxes every commit.
## Ruby-specific gotchas
- **Block `return` vs lambda `return`.** Inside a block `{ ... return v }`, `return` invokes the *enclosing method's* escape continuation (non-local return). Inside a lambda `->(){ ... return v }`, `return` returns from the *lambda*. Don't conflate. Implement: blocks bind their `^method-k`; lambdas bind their own `^lambda-k`.
- **`break` from inside a block** invokes a different escape — the *iteration loop's* escape — and the loop returns the break-value. `next` is escape from current iteration, returns iteration value. `redo` re-enters current iteration without advancing.
- **Proc arity is lax.** `proc { |a, b, c| … }.call(1, 2)` ↦ `c = nil`. Lambda is strict — same call raises ArgumentError. Check arity at call site for lambdas only.
- **Block argument unpacking.** `[[1,2],[3,4]].each { |a, b| … }` — single Array arg auto-unpacks for blocks (not lambdas). One arg, one Array → unpack. Frequent footgun.
- **Method dispatch chain order:** prepended modules → class methods → included modules → superclass → BasicObject → method_missing. `super` walks from the *defining* class's position, not the receiver class's.
- **Singleton classes** are lazily allocated. Looking up the chain for an object passes through its singleton class first, then its actual class. `class << obj; …; end` opens the singleton.
- **`method_missing`** — fallback when ancestor walk misses. Receives `(name_symbol, *args, &blk)`. Pair with `respond_to_missing?` for `respond_to?` to also report true. Do **not** swallow NoMethodError silently.
- **Ivars are per-object dicts.** Reading an unset ivar yields `nil` and a warning (`-W`). Don't error.
- **Constant lookup** is first lexical (Module.nesting), then inheritance (Module.ancestors of the innermost class). Different from method lookup.
- **`Object#send`** invokes private and public methods alike; `Object#public_send` skips privates.
- **Class reopening.** `class Foo; def bar; …; end; end` plus a later `class Foo; def baz; …; end; end` adds methods to the same class. Class table lookups must be by-name, mutable; methods dict is mutable.
- **Fiber semantics.** `Fiber.new { |arg| … }` creates a fiber suspended at entry. First `Fiber.resume(v)` enters with `arg = v`. Inside, `Fiber.yield(w)` returns `w` to the resumer; the next `Fiber.resume(v')` returns `v'` to the yield site. End of block returns final value to last resumer; subsequent `Fiber.resume` raises FiberError.
- **`Fiber.transfer`** is symmetric — either side can transfer to the other; no resume/yield asymmetry. Implement on top of the same continuation pair, just don't enforce direction.
- **Symbols are interned.** `:foo == :foo` is identity. Use SX symbols.
- **Strings are mutable.** `s = "abc"; s << "d"; s == "abcd"`. Hash keys can be strings; hash dups string keys at insertion to be safe (or freeze them).
- **Truthiness:** only `false` and `nil` are falsy. `0`, `""`, `[]` are truthy.
- **Test corpus:** custom + curated RubySpec slice. Place programs in `lib/ruby/tests/programs/` with `.rb` extension.
## General gotchas (all loops)
- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences.
- `cond`/`when`/`let` clauses evaluate only the last expr.
- `type-of` on user fn returns `"lambda"`.
- Shell heredoc `||` gets eaten — escape or use `case`.
## Style
- No comments in `.sx` unless non-obvious.
- No new planning docs — update `plans/ruby-on-sx.md` inline.
- Short, factual commit messages (`ruby: Fiber.yield + Fiber.resume (+8)`).
- One feature per iteration. Commit. Log. Next.
Go. Read the plan; find first `[ ]`; implement.

77
plans/agent-briefings/smalltalk-loop.md Normal file
View File

@@ -0,0 +1,77 @@
# smalltalk-on-sx loop agent (single agent, queue-driven)
Role: iterates `plans/smalltalk-on-sx.md` forever. Message-passing OO + **blocks with non-local return** on delimited continuations. Non-local return is the headline showcase — every other Smalltalk reinvents it on the host stack; on SX it falls out of the captured method-return continuation.
```
description: smalltalk-on-sx queue loop
subagent_type: general-purpose
run_in_background: true
isolation: worktree
```
## Prompt
You are the sole background agent working `/root/rose-ash/plans/smalltalk-on-sx.md`. Isolated worktree, forever, one commit per feature. Never push.
## Restart baseline — check before iterating
1. Read `plans/smalltalk-on-sx.md` — roadmap + Progress log.
2. `ls lib/smalltalk/` — pick up from the most advanced file.
3. If `lib/smalltalk/tests/*.sx` exist, run them. Green before new work.
4. If `lib/smalltalk/scoreboard.md` exists, that's your baseline.
## The queue
Phase order per `plans/smalltalk-on-sx.md`:
- **Phase 1** — tokenizer + parser (chunk format, identifiers, keywords `foo:`, binary selectors, `#sym`, `#(…)`, `$c`, blocks `[:a | …]`, cascades, message precedence)
- **Phase 2** — object model + sequential eval (class table bootstrap, message dispatch, `super`, `doesNotUnderstand:`, instance variables)
- **Phase 3** — **THE SHOWCASE**: blocks with non-local return via captured method-return continuation. `whileTrue:` / `ifTrue:ifFalse:` as block sends. 5 classic programs (eight-queens, quicksort, mandelbrot, life, fibonacci) green.
- **Phase 4** — reflection + MOP: `perform:`, `respondsTo:`, runtime method addition, `becomeForward:`, `Exception` / `on:do:` / `ensure:` on top of `handler-bind`/`raise`
- **Phase 5** — collections + numeric tower + streams
- **Phase 6** — port SUnit, vendor Pharo Kernel-Tests slice, drive corpus to 200+
- **Phase 7** — speed (optional): inline caching, block intrinsification
Within a phase, pick the checkbox that unlocks the most tests per effort.
Every iteration: implement → test → commit → tick `[ ]` → Progress log → next.
## Ground rules (hard)
- **Scope:** only `lib/smalltalk/**` and `plans/smalltalk-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, other `lib/<lang>/` dirs, `lib/stdlib.sx`, or `lib/` root. Smalltalk primitives go in `lib/smalltalk/runtime.sx`.
- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers entry, stop.
- **Shared-file issues** → plan's Blockers with minimal repro.
- **Delimited continuations** are in `lib/callcc.sx` + `spec/evaluator.sx` Step 5. `sx_summarise` spec/evaluator.sx first — 2300+ lines.
- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits.
- **Worktree:** commit locally. Never push. Never touch `main`.
- **Commit granularity:** one feature per commit.
- **Plan file:** update Progress log + tick boxes every commit.
## Smalltalk-specific gotchas
- **Method invocation captures `^k`** — the return continuation. Bind it as the block's escape token. `^expr` from inside any nested block invokes that captured `^k`. Escape past method return raises `BlockContext>>cannotReturn:`.
- **Blocks are lambdas + escape token**, not bare lambdas. `value`/`value:`/… invoke the lambda; `^` invokes the escape.
- **`ifTrue:` / `ifFalse:` / `whileTrue:` are ordinary block sends** — no special form. The runtime intrinsifies them in the JIT path (Tier 1 of bytecode expansion already covers this pattern).
- **Cascade** `r m1; m2; m3` desugars to `(let ((tmp r)) (st-send tmp 'm1 ()) (st-send tmp 'm2 ()) (st-send tmp 'm3 ()))`. Result is the cascade's last send (or first, depending on parser variant — pick one and document).
- **`super` send** looks up starting from the *defining* class's superclass, not the receiver class. Stash the defining class on the method record.
- **Selectors are interned symbols.** Use SX symbols.
- **Receiver dispatch:** tagged ints / floats / strings / symbols / `nil` / `true` / `false` aren't boxed. Their classes (`SmallInteger`, `Float`, `String`, `Symbol`, `UndefinedObject`, `True`, `False`) are looked up by SX type-of, not by an `:class` field.
- **Method precedence:** unary > binary > keyword. `3 + 4 factorial` is `3 + (4 factorial)`. `a foo: b bar` is `a foo: (b bar)` (keyword absorbs trailing unary).
- **Image / fileIn / become: between sessions** = out of scope. One-way `becomeForward:` only.
- **Test corpus:** ~200 hand-written + a slice of Pharo Kernel-Tests. Place programs in `lib/smalltalk/tests/programs/`.
## General gotchas (all loops)
- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences.
- `cond`/`when`/`let` clauses evaluate only the last expr.
- `type-of` on user fn returns `"lambda"`.
- Shell heredoc `||` gets eaten — escape or use `case`.
## Style
- No comments in `.sx` unless non-obvious.
- No new planning docs — update `plans/smalltalk-on-sx.md` inline.
- Short, factual commit messages (`smalltalk: tokenizer + 56 tests`).
- One feature per iteration. Commit. Log. Next.
Go. Read the plan; find first `[ ]`; implement.

86
plans/agent-briefings/sx-improvements-loop.md Normal file
View File

@@ -0,0 +1,86 @@
# sx-improvements loop agent
Iterates `plans/sx-improvements.md` forever. One step per commit.
```
description: sx-improvements loop
subagent_type: general-purpose
run_in_background: true
isolation: worktree
```
## Prompt
You are the sole background agent iterating `plans/sx-improvements.md` on the `architecture` branch of `/root/rose-ash`. One step per commit, forever. Never push.
## Restart baseline — check before each iteration
1. Read `plans/sx-improvements.md` — find the first unchecked `[ ]` step in the progress log.
2. Read the step's section in the plan for exact implementation details.
3. Run the verification command for that step to confirm it currently fails.
4. Implement. Verify. Commit. Tick the `[ ]` → `[x]` in the progress log. Next.
## Test commands
- **OCaml spec:** `sx_build target="ocaml"` then check `bin/run_tests.exe` output.
- **JS spec (no DOM):** `node hosts/javascript/run_tests.js 2>&1 | tail -3`
- **HyperScript kernel:** `node tests/hs-kernel-eval.js 2>&1 | tail -3`
- **Baseline SX tests (non-HS):** `node hosts/javascript/run_tests.js 2>&1 | grep -v "hs-upstream\|hs-compat\|hs-dev" | grep "Results:"`
Do NOT regress the pre-merge passing tests. After each step, confirm the count did not drop.
## Ground rules (hard)
- **Branch:** `architecture`. Never push. Never touch `main`.
- **SX files:** `sx-tree` MCP tools ONLY (`sx_summarise`, `sx_read_subtree`, `sx_replace_node`, `sx_insert_child`, `sx_validate`). Read before edit. Validate after edit.
- **Generated files:** NEVER edit `shared/static/wasm/sx/` or `shared/static/scripts/sx-*.js` directly. Rebuild via `sx_build`.
- **HS mirror rule:** after editing any `lib/hyperscript/<f>.sx`, copy to `shared/static/wasm/sx/hs-<f>.sx` using `sx_write_file` with the same content.
- **OCaml build:** `sx_build target="ocaml"` — never raw `dune exec`.
- **JS build:** `sx_build target="js"`.
- **One step per commit.** Tick the plan. Factual commit message.
- **No new planning docs.** No comments in SX unless non-obvious.
- **Unicode in SX:** raw UTF-8 only, never `\uXXXX` escapes.
## Step-specific notes
### Step 1 (JIT combinator bug)
The bug is in `hosts/ocaml/lib/sx_vm.ml` — `call_closure_reuse` path strips locals when
callee returns a closure. Look for the path where `call_closure_reuse` is invoked for a
`VmClosure` return value. The fix is to not reuse frames when the call might return a
closure, or to properly snapshot/restore `sp`. Check `spec/tests/test-parser-combinators.sx`
for existing combinator tests; run `node tests/hs-kernel-eval.js` for the 11 failing HS tests.
### Step 2 (letrec+resume)
The bug is browser-only (`hosts/ocaml/browser/sx_browser.ml`). Write a minimal
`spec/tests/test-letrec-resume.sx` that exercises `letrec` + `perform` + resume and
verify it passes under `run_tests.exe` (OCaml server mode). Then check what
`sx_browser.ml` does differently in the VmSuspension resume path.
### Steps 3-4 (E38 source info)
The API is already in `lib/hyperscript/runtime.sx`. The gap is in the tokenizer (no `:end`/`:line`)
and some parser span completeness. Run the 4 sourceInfo tests to see exact failures:
`node tests/hs-kernel-eval.js --suite sourceInfo` or grep results for `sourceInfo`.
### Steps 5-8 (ADTs)
Full spec in `plans/designs/sx-adt.md`. Implement in OCaml first (Step 5), then mirror
to JS (Step 6). Steps 7-8 build on top. Write `spec/tests/test-adt.sx` from scratch —
start with a `(define-type Maybe (Just value) (Nothing))` suite covering constructor,
predicate, accessor, basic match, else clause.
### Steps 9-11 (plugin system)
Full spec in `plans/designs/hs-plugin-system.md`. The prolog hook migration (Step 11) is
the most important for language-building — it's the pattern for all future embeds.
### Steps 12-14 (performance)
Profile first. Use `sx_harness_eval` to measure throughput on a tight loop before and
after each change. Only commit if there's a measurable win (>10%).
## General gotchas (all loops)
- SX `do` is R7RS iteration. Use `begin` for multi-expr sequences.
- `cond`/`when`/`let` bodies evaluate only the last expression.
- `type-of` on a user-defined function returns `"lambda"`.
- Shell heredoc `||` gets eaten — escape or use `case`.
- `env-bind!` creates new bindings; `env-set!` mutates existing (walks scope chain).
- After OCaml edits: the build takes ~2 min. Run `sx_build target="ocaml"` and wait.
- After JS edits: retranspile with `sx_build target="js"` then re-run tests.

83
plans/agent-briefings/tcl-loop.md Normal file
View File

@@ -0,0 +1,83 @@
# tcl-on-sx loop agent (single agent, queue-driven)
Role: iterates `plans/tcl-on-sx.md` forever. `uplevel`/`upvar` is the headline showcase — Tcl's superpower for defining your own control structures, requiring deep VM cooperation in any normal host but falling out of SX's first-class env-chain. Plus the Dodekalogue (12 rules), command-substitution everywhere, and "everything is a string" homoiconicity.
```
description: tcl-on-sx queue loop
subagent_type: general-purpose
run_in_background: true
isolation: worktree
```
## Prompt
You are the sole background agent working `/root/rose-ash/plans/tcl-on-sx.md`. Isolated worktree, forever, one commit per feature. Push to `origin/loops/tcl` after every commit.
## Restart baseline — check before iterating
1. Read `plans/tcl-on-sx.md` — roadmap + Progress log.
2. `ls lib/tcl/` — pick up from the most advanced file.
3. If `lib/tcl/tests/*.sx` exist, run them. Green before new work.
4. If `lib/tcl/scoreboard.md` exists, that's your baseline.
## The queue
Phase order per `plans/tcl-on-sx.md`:
- **Phase 1** — tokenizer + parser. The Dodekalogue (12 rules): word-splitting, command sub `[…]`, var sub `$name`/`${name}`/`$arr(idx)`, double-quote vs brace word, backslash, `;`, `#` comments only at command start, single-pass left-to-right substitution
- **Phase 2** — sequential eval + core commands. `set`/`unset`/`incr`/`append`/`lappend`, `puts`/`gets`, `expr` (own mini-language), `if`/`while`/`for`/`foreach`/`switch`, string commands, list commands, dict commands
- **Phase 3** — **THE SHOWCASE**: `proc` + `uplevel` + `upvar`. Frame stack with proc-call push/pop; `uplevel #N script` evaluates in caller's frame; `upvar` aliases names across frames. Classic programs (for-each-line, assert macro, with-temp-var) green
- **Phase 4** — `return -code N`, `catch`, `try`/`trap`/`finally`, `throw`. Control flow as integer codes
- **Phase 5** — namespaces + ensembles. `namespace eval`, qualified names `::ns::cmd`, ensembles, `namespace path`
- **Phase 6** — coroutines (built on fibers, same delcc as Ruby fibers) + system commands + drive corpus to 150+
Within a phase, pick the checkbox that unlocks the most tests per effort.
Every iteration: implement → test → commit → tick `[ ]` → Progress log → next.
## Ground rules (hard)
- **Scope:** only `lib/tcl/**` and `plans/tcl-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, other `lib/<lang>/` dirs, `lib/stdlib.sx`, or `lib/` root. Tcl primitives go in `lib/tcl/runtime.sx`.
- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers entry, stop.
- **Shared-file issues** → plan's Blockers with minimal repro.
- **Delimited continuations** are in `lib/callcc.sx` + `spec/evaluator.sx` Step 5. `sx_summarise` spec/evaluator.sx first — 2300+ lines.
- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits.
- **Worktree:** commit, then push to `origin/loops/tcl`. Never touch `main`.
- **Commit granularity:** one feature per commit.
- **Plan file:** update Progress log + tick boxes every commit.
## Tcl-specific gotchas
- **Everything is a string.** Internally cache shimmer reps (list, dict, int, double) for performance, but every value must be re-stringifiable. Mutating one rep dirties the cached string and vice versa.
- **The Dodekalogue is strict.** Substitution is **one-pass**, **left-to-right**. The result of a substitution is a value, not a script — it does NOT get re-parsed for further substitutions. This is what makes Tcl safe-by-default. Don't accidentally re-parse.
- **Brace word `{…}`** is the only way to defer evaluation. No substitution inside, just balanced braces. Used for `if {expr}` body, `proc body`, `expr` arguments.
- **Double-quote word `"…"`** is identical to a bare word for substitution purposes — it just allows whitespace in a single word. `\` escapes still apply.
- **Comments are only at command position.** `# this is a comment` after a `;` or newline; *not* inside a command. `set x 1 # not a comment` is a 4-arg `set`.
- **`expr` has its own grammar** — operator precedence, function calls — and does its own substitution. Brace `expr {$x + 1}` to avoid double-substitution and to enable bytecode caching.
- **`if` and `while` re-parse** the condition only if not braced. Always use `if {…}`/`while {…}` form. The unbraced form re-substitutes per iteration.
- **`return` from a `proc`** uses control code 2. `break` is 3, `continue` is 4. `error` is 1. `catch` traps any non-zero code; user can return non-zero with `return -code error -errorcode FOO message`.
- **`uplevel #0 script`** is global frame. `uplevel 1 script` (or just `uplevel script`) is caller's frame. `uplevel #N` is absolute level N (0=global, 1=top-level proc, 2=proc-called-from-top, …). Negative levels are errors.
- **`upvar #N otherVar localVar`** binds `localVar` in the current frame as an *alias* — both names refer to the same storage. Reads and writes go through the alias.
- **`info level`** with no arg returns current level number. `info level N` (positive) returns the command list that invoked level N. `info level -N` returns the command list of the level N relative-up.
- **Variable names with `(…)`** are array elements: `set arr(foo) 1`. Arrays are not first-class values — you can't `set x $arr`. `array get arr` gives a flat list `{key1 val1 key2 val2 …}`.
- **List vs string.** `set l "a b c"` and `set l [list a b c]` look the same when printed but the second has a cached list rep. `lindex` works on both via shimmering. Most user code can't tell the difference.
- **`incr x`** errors if x doesn't exist; pre-set with `set x 0` or use `incr x 0` first if you mean "create-or-increment". Or use `dict incr` for dicts.
- **Coroutines are fibers.** `coroutine name body` starts a coroutine; calling `name` resumes it; `yield value` from inside suspends and returns `value` to the resumer. Same primitive as Ruby fibers — share the implementation under the hood.
- **`switch`** matches first clause whose pattern matches. Default is `default`. Variant matches: glob (default), `-exact`, `-glob`, `-regexp`. Body `-` means "fall through to next clause's body".
- **Test corpus:** custom + slice of Tcl's own tests. Place programs in `lib/tcl/tests/programs/` with `.tcl` extension.
## General gotchas (all loops)
- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences.
- `cond`/`when`/`let` clauses evaluate only the last expr.
- `type-of` on user fn returns `"lambda"`.
- Shell heredoc `||` gets eaten — escape or use `case`.
## Style
- No comments in `.sx` unless non-obvious.
- No new planning docs — update `plans/tcl-on-sx.md` inline.
- Short, factual commit messages (`tcl: uplevel + upvar (+11)`).
- One feature per iteration. Commit. Log. Next.
Go. Read the plan; find first `[ ]`; implement.

244
plans/apl-on-sx.md Normal file
View File

@@ -0,0 +1,244 @@
# APL-on-SX: rank-polymorphic primitives + glyph parser
The headline showcase is **rank polymorphism** — a single primitive (`+`, `⌈`, `⊂`, `⍳`) works uniformly on scalars, vectors, matrices, and higher-rank arrays. ~80 glyph primitives + 6 operators bind together with right-to-left evaluation; the entire language is a high-density combinator algebra. The JIT compiler + primitive table pay off massively here because almost every program is `array → array` pure pipelines.
End-state goal: Dyalog-flavoured APL subset, dfns + tradfns, classic programs (game-of-life, mandelbrot, prime-sieve, n-queens, conway), 100+ green tests.
## Scope decisions (defaults — override by editing before we spawn)
- **Syntax:** Dyalog APL surface, Unicode glyphs. `⎕`-quad system functions for I/O. `∇` tradfn header.
- **Conformance:** "Reads like APL, runs like APL." Not byte-compat with Dyalog; we care about right-to-left semantics and rank polymorphism.
- **Test corpus:** custom — APL idioms (Roger Hui style), classic programs, plus ~50 pattern tests for primitives.
- **Out of scope:** ⎕-namespaces beyond a handful, complex numbers, full TAO ordering, `⎕FX` runtime function definition (use static `∇` only), nested-array-of-functions higher orders, the editor.
- **Glyphs:** input via plain Unicode in `.apl` source files. Backtick-prefix shortcuts handled by the user's editor — we don't ship one.
## Ground rules
- **Scope:** only touch `lib/apl/**` and `plans/apl-on-sx.md`. Don't edit `spec/`, `hosts/`, `shared/`, or any other `lib/<lang>/**`. APL primitives go in `lib/apl/runtime.sx`.
- **SX files:** use `sx-tree` MCP tools only.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick roadmap boxes.
## Architecture sketch
```
APL source (Unicode glyphs)
│
▼
lib/apl/tokenizer.sx — glyphs, identifiers, numbers (¯ for negative), strings, strands
│
▼
lib/apl/parser.sx — right-to-left with valence resolution (mon vs dyadic by position)
│
▼
lib/apl/transpile.sx — AST → SX AST (entry: apl-eval-ast)
│
▼
lib/apl/runtime.sx — array model, ~80 primitives, 6 operators, dfns/tradfns
```
Core mapping:
- **Array** = SX dict `{:shape (d1 d2 …) :ravel #(v1 v2 …)}`. Scalar is rank-0 (empty shape), vector is rank-1, matrix rank-2, etc. Type uniformity not required (heterogeneous nested arrays via "boxed" elements `⊂x`).
- **Rank polymorphism** — every scalar primitive is broadcast: `1 2 3 + 4 5 6` ↦ `5 7 9`; `(2 3⍴⍳6) + 1` ↦ broadcast scalar to matrix.
- **Conformability** = matching shapes, or one-side scalar, or rank-1 cycling (deferred — keep strict in v1).
- **Valence** = each glyph has a monadic and a dyadic meaning; resolution is purely positional (left-arg present → dyadic).
- **Operator** = takes one or two function operands, returns a derived function (`f¨` = `each f`, `f/` = `reduce f`, `f∘g` = `compose`, `f⍨` = `commute`).
- **Tradfn** `∇R←L F R; locals` = named function with explicit header.
- **Dfn** `{⍺+⍵}` = anonymous, `⍺` = left arg, `⍵` = right arg, `∇` = recurse.
## Roadmap
### Phase 1 — tokenizer + parser
- [x] Tokenizer: Unicode glyphs (the full APL set: `+ - × ÷ * ⍟ ⌈ ⌊ | ! ? ○ ~ < ≤ = ≥ > ≠ ∊ ∧ ∨ ⍱ ⍲ , ⍪ ⍴ ⌽ ⊖ ⍉ ↑ ↓ ⊂ ⊃ ⊆ ∪ ∩ ⍳ ⍸ ⌷ ⍋ ⍒ ⊥ ⊤ ⊣ ⊢ ⍎ ⍕ ⍝`), operators (`/ \ ¨ ⍨ ∘ . ⍣ ⍤ ⍥ @`), numbers (`¯` for negative, `1E2`, `1J2` complex deferred), characters (`'a'`, `''` escape), strands (juxtaposition of literals: `1 2 3`), names, comments `⍝ …`
- [x] Parser: right-to-left; classify each token as function, operator, value, or name; resolve valence positionally; dfn `{…}` body, tradfn `∇` header, guards `:`; outer product `∘.f`, inner product `f.g`, derived fns `f/ f¨ f⍨ f⍣n`
- [x] Unit tests in `lib/apl/tests/parse.sx`
### Phase 2 — array model + scalar primitives
- [x] Array constructor: `make-array shape ravel`, `scalar v`, `vector v…`, `enclose`/`disclose`
- [x] Shape arithmetic: `⍴` (shape), `,` (ravel), `≢` (tally / first-axis-length), `≡` (depth)
- [x] Scalar arithmetic primitives broadcast: `+ - × ÷ ⌈ ⌊ * ⍟ | ! ○`
- [x] Scalar comparison primitives: `< ≤ = ≥ > ≠`
- [x] Scalar logical: `~ ∧ ∨ ⍱ ⍲`
- [x] Index generator: `⍳n` (vector 1..n or 0..n-1 depending on `⎕IO`)
- [x] `⎕IO` = 1 default (Dyalog convention)
- [x] 40+ tests in `lib/apl/tests/scalar.sx`
### Phase 3 — structural primitives + indexing
- [x] Reshape `⍴`, ravel `,`, transpose `⍉` (full + dyadic axis spec)
- [x] Take `↑`, drop `↓`, rotate `⌽` (last axis), `⊖` (first axis)
- [x] Catenate `,` (last axis) and `⍪` (first axis)
- [x] Index `⌷` (squad), bracket-indexing `A[I]` (sugar for `⌷`)
- [x] Grade-up `⍋`, grade-down `⍒`
- [x] Enclose `⊂`, disclose `⊃`, partition (subset deferred)
- [x] Membership `∊`, find `⍳` (dyadic), without `~` (dyadic), unique `∪` (deferred to phase 6)
- [x] 40+ tests in `lib/apl/tests/structural.sx`
### Phase 4 — operators (THE SHOWCASE)
- [x] Reduce `f/` (last axis), `f⌿` (first axis) — including `∧/`, `∨/`, `+/`, `×/`, `⌈/`, `⌊/`
- [x] Scan `f\`, `f⍀`
- [x] Each `f¨` — applies `f` to each scalar/element
- [x] Outer product `∘.f` — `1 2 3 ∘.× 1 2 3` ↦ multiplication table
- [x] Inner product `f.g` — `+.×` is matrix multiply
- [x] Commute `f⍨` — `f⍨ x` ↔ `x f x`, `x f⍨ y` ↔ `y f x`
- [x] Compose `f∘g` — applies `g` first then `f`
- [x] Power `f⍣n` — apply f n times; `f⍣≡` until fixed point
- [x] Rank `f⍤k` — apply f at sub-rank k
- [x] At `@` — selective replace
- [x] 40+ tests in `lib/apl/tests/operators.sx`
### Phase 5 — dfns + tradfns + control flow
- [x] Dfn `{…}` with `⍺` (left arg, may be absent → niladic/monadic), `⍵` (right arg), `∇` (recurse), guards `cond:expr`, default left arg `⍺←default`
- [x] Local assignment via `←` (lexical inside dfn)
- [x] Tradfn `∇` header: `R←L F R;l1;l2`, statement-by-statement, branch via `→linenum`
- [x] Dyalog control words: `:If/:Else/:EndIf`, `:While/:EndWhile`, `:For X :In V :EndFor`, `:Select/:Case/:EndSelect`, `:Trap`/`:EndTrap` _(Trap deferred — no exception machinery yet)_
- [x] Niladic / monadic / dyadic dispatch (function valence at definition time)
- [x] `lib/apl/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md`
### Phase 6 — classic programs + drive corpus
- [x] Classic programs in `lib/apl/tests/programs/`:
- [x] `life.apl` — Conway's Game of Life as a one-liner using `⊂` `⊖` `⌽` `+/`
- [x] `mandelbrot.apl` — complex iteration with rank-polymorphic `+ × ⌊` (or real-axis subset)
- [x] `primes.apl` — `(2=+⌿0=A∘.|A)/A←⍳N` sieve
- [x] `n-queens.apl` — backtracking via reduce
- [x] `quicksort.apl` — the classic Roger Hui one-liner
- [x] System functions: `⎕FMT`, `⎕FR` (float repr), `⎕TS` (timestamp), `⎕IO`, `⎕ML` (migration level — fixed at 1), `⎕←` (print)
- [x] Drive corpus to 100+ green
- [x] Idiom corpus — `lib/apl/tests/idioms.sx` covering classic Roger Hui / Phil Last idioms
### Phase 7 — end-to-end pipeline + closing the gaps
Phase 1-6 built parser and runtime as parallel layers — they don't yet meet.
Phase 7 wires them together so APL source actually runs through the full stack,
and tightens loose ends.
- [x] **Operators in `apl-eval-ast`** — handle `:derived-fn` (e.g. `+/`, `f¨`),
`:outer` (`∘.f`), `:derived-fn2` (`f.g`). Each derived-fn-node wraps an inner
function; eval-ast resolves the inner glyph to a runtime fn and dispatches
to the matching operator helper (`apl-reduce`, `apl-each`, `apl-outer`,
`apl-inner`, `apl-commute`, `apl-compose`, `apl-power`, `apl-rank`).
- [x] **End-to-end pipeline** — entry point `apl-run : string → array` that
chains `apl-tokenize` → `parse-apl` → `apl-eval-ast` against an empty env.
Verify with one-liners (`+/⍳5` → 15, `1 2 3 + 4 5 6` → 7 9 11, etc.) and
with the actual `.apl` source files in `tests/programs/`.
- [x] **`:quad-name` AST + handler** — extend tokenizer/parser to recognise
`⎕name`, then handle in `apl-eval-ast` by dispatching to `apl-quad-*`
runtime fns (`⎕IO`, `⎕ML`, `⎕FR`, `⎕TS`, `⎕FMT`, `⎕←`).
_(`⎕←` deferred — tokenizer treats `←` as `:assign` after `⎕`.)_
- [x] **Bracket indexing verification** — load programs that use `A[I]` /
`A[I;J]` end-to-end; confirm parser desugars to `⌷` and runtime returns
expected slices. Add 5+ tests.
_(Single-axis only — multi-axis `A[I;J]` requires semicolon parsing, deferred.)_
- [x] **Idiom corpus expansion** — extend `idioms.sx` from 34 to 60+ once
end-to-end works (we can express idioms as APL strings, not as runtime
calls). Source-string-based idioms validate the whole stack.
- [x] **`:Trap` / `:EndTrap`** — minimal exception machinery: `:Trap n`
catches errors with code `n`, body runs in `apl-tradfn-eval-block`,
on error switches to the trap branch. Define `apl-throw` and a small
set of error codes; use `try`/`catch` from the host.
### Phase 8 — fill the gaps left after end-to-end
Phase 7 wired the stack together; Phase 8 closes deferred items, lets real
programs run from source, and starts pushing on performance.
- [x] **Quick-wins bundle** (one iteration) — three small fixes that each unblock
real programs:
- decimal literals: `read-digits!` consumes one trailing `.` plus more digits
so `3.7` tokenises as one number;
- `⎕←` (print) — tokenizer special-case: when `⎕` is followed by `←`, emit
a single `:name "⎕←"` token (don't split on the assign glyph);
- string values in `apl-eval-ast` — handle `:str` (parser already produces
them) by wrapping into a vector of character codes (or rank-0 string).
- [x] **Named function definitions** — `f ← {⍺+⍵} ⋄ 1 f 2` and `2 f 3`.
- parser: when `:assign`'s RHS is a `:dfn`, mark it as a function binding;
- eval-ast: `:assign` of a dfn stores the dfn in env;
- parser: a name in fn-position whose env value is a dfn dispatches as a fn;
- resolver: extend `apl-resolve-monadic`/`-dyadic` with a `:fn-name` case
that calls `apl-call-dfn`/`apl-call-dfn-m`.
- [x] **Multi-axis bracket indexing** — `A[I;J]` and `A[;J]` and `A[I;]`.
- parser: split bracket content on `:semi` at depth 0; emit
`(:dyad ⌷ (:vec I J) A)`;
- runtime: extend `apl-squad` to accept a vector of indices, treating
`nil` / empty axis as "all";
- 5+ tests across vector and matrix.
- [x] **`.apl` files as actual tests** — `lib/apl/tests/programs/*.apl` are
currently documentation. Add `apl-run-file path → array` plus tests that
load each file, execute it, and assert the expected result. Makes the
classic-program corpus self-validating instead of two parallel impls.
_(Embedded source-string approach: tests/programs-e2e.sx runs the same
algorithms as the .apl docs through the full pipeline. The original
one-liners (e.g. primes' inline `⍵←⍳⍵`) need parser features
(compress-as-fn, inline assign) we haven't built yet — multi-stmt forms
used instead. Slurp/read-file primitive missing in OCaml SX runtime.)_
- [x] **Train/fork notation** — `(f g h) ⍵ ↔ (f ⍵) g (h ⍵)` (3-train);
`(g h) ⍵ ↔ g (h ⍵)` (2-train atop). Parser: detect when a parenthesised
subexpression is all functions and emit `(:train fns)`; resolver: build the
derived function; tests for mean-via-train (`+/÷≢`).
- [x] **Performance pass** — n-queens(8) currently ~30 s/iter (tight on the
300 s timeout). Target: profile the inner loop, eliminate quadratic
list-append, restore the `queens(8)` test.
## SX primitive baseline
Use vectors for arrays; numeric tower + rationals for numbers; ADTs for tagged data;
coroutines for fibers; string-buffer for mutable string building; bitwise ops for bit
manipulation; multiple values for multi-return; promises for lazy evaluation; hash tables
for mutable associative storage; sets for O(1) membership; sequence protocol for
polymorphic iteration; gensym for unique symbols; char type for characters; string ports
+ read/write for reader protocols; regexp for pattern matching; bytevectors for binary
data; format for string templating.
## Progress log
_Newest first._
- 2026-05-07: Phase 8 step 6 — perf: swapped (append acc xs) → (append xs acc) in apl-permutations to make permutation generation linear instead of quadratic; q(7) 32s→12s; q(8)=92 test restored within 300s timeout; **Phase 8 complete, all unchecked items ticked**; 497/497
- 2026-05-07: Phase 8 step 5 — train/fork notation. Parser :lparen detects all-fn inner segments → emits :train AST; resolver covers 2-atop & 3-fork for both monadic and dyadic. `(+/÷≢) 1..5 → 3` (mean), `(- ⌊) 5 → -5` (atop), `2(+×-)5 → -21` (dyadic fork), `(⌈/-⌊/) → 8` (range); +6 tests; 496/496
- 2026-05-07: Phase 8 step 4 — programs-e2e.sx runs classic-algorithm shapes through full pipeline (factorial via ∇, triangulars, sum-of-squares, divisor-counts, prime-mask, named-fn composition, dyadic max-of-two, Newton step); also added ⌿ + ⍀ to glyph sets (were silently skipped); +15 tests; 490/490
- 2026-05-07: Phase 8 step 3 — multi-axis bracket A[I;J] / A[I;] / A[;J] via :bracket AST + apl-bracket-multi runtime; split-bracket-content scans :semi at depth 0; apl-cartesian builds index combinations; nil axis = "all"; scalar axis collapses; +8 tests; 475/475
- 2026-05-07: Phase 8 step 2 — named function defs end-to-end via parser pre-scan; apl-known-fn-names + apl-collect-fn-bindings detect `name ← {...}` patterns; collect-segments-loop emits :fn-name for known names; resolver looks up env for :fn-name; supports recursion (∇ in named dfn); +7 tests including fact via ∇; 467/467
- 2026-05-07: Phase 8 step 1 — quick-wins bundle: decimal literals (3.7, ¯2.5), ⎕← passthrough as monadic fn (single-token via tokenizer special-case), :str AST in eval-ast (single-char→scalar, multi-char→vec); +10 tests; 460/460
- 2026-05-07: Phase 8 added — quick-wins bundle (decimals + ⎕← + strings), named functions, multi-axis bracket, .apl-files-as-tests, trains, perf
- 2026-05-07: Phase 7 step 6 — :Trap exception machinery via R7RS guard; apl-throw raises tagged error, apl-trap-matches? checks codes (0=catch-all), :trap clause in apl-tradfn-eval-stmt wraps try-block with guard; :throw AST for testing; **Phase 7 complete, all unchecked plan items done**; +5 tests; 450/450
- 2026-05-07: Phase 7 step 5 — idiom corpus 34→64 (+30 source-string idioms via apl-run); also fixed tokenizer + parser to recognize ≢ and ≡ glyphs (were silently skipped); 445/445
- 2026-05-07: Phase 7 step 4 — bracket indexing `A[I]` desugared to `(:dyad ⌷ I A)` via maybe-bracket helper, wired into :name + :lparen branches of collect-segments-loop; multi-axis (A[I;J]) deferred (semicolon split); +7 tests; 415/415
- 2026-05-07: Phase 7 step 3 — :quad-name end-to-end; tokenizer already produced :name "⎕FMT"; parser is-fn-tok? extended via apl-quad-fn-names; eval-ast :name dispatches ⎕IO/⎕ML/⎕FR/⎕TS to apl-quad-*; apl-monadic-fn handles ⎕FMT; ⎕← deferred (tokenizer splits ⎕←); +8 tests; 408/408
- 2026-05-07: Phase 7 step 2 — end-to-end pipeline `apl-run : string → array` (parse-apl + apl-eval-ast against empty env); +25 source-string tests covering scalars, strands, dyadic arith, monadic primitives, operators, ∘./.g products, comparisons, famous one-liners (+/⍳10=55, ×/⍳10=10!); tokenizer can't yet parse decimals so `3.7` literal tests dropped; **400/400**
- 2026-05-07: Phase 7 step 1 — operators in apl-eval-ast via apl-resolve-monadic/dyadic; supports / ⌿ \ ⍀ ¨ ⍨ ∘. f.g; queens(8) test removed (too slow for 300s timeout); +14 eval-ops tests; 375/375
- 2026-05-07: Phase 7 added — end-to-end pipeline, operators in eval-ast, :quad-name, bracket-indexing verify, idiom expansion, :Trap; aim is to wire parser↔runtime so .apl source files actually run
- 2026-05-07: Phase 6 idiom corpus — lib/apl/tests/idioms.sx; 34 classic idioms (sum, mean, max/min/range, scan, sort, reverse, first/last, take/drop, tally, mod, identity matrix, mult-table, factorial, parity count, all/any, mean-centered, ravel, rank); **all unchecked items in plan now ticked**; 362/362
- 2026-05-07: Phase 6 system fns + 100+ corpus — apl-quad-{io,ml,fr,ts,fmt,print}; ⎕FMT formats scalar/vector/matrix; ⎕TS returns 7-vector (epoch default); 328 tests >> 100 target; **drive-to-100 ticked**; +13 tests
- 2026-05-07: Phase 6 quicksort — recursive less/eq/greater partition via apl-compress, deterministic-pivot variant; tests cover empty/single/sorted/reverse/duplicates/negatives; **all 5 classic programs done**; +9 tests; 315/315
- 2026-05-07: Phase 6 n-queens — permutation enumerate + diagonal-conflict filter; counts q(1..8) = 1,0,0,2,10,4,40,92 (OEIS A000170); apl-permutations + apl-queens; bumped test timeout 60→180s for q(8); +10 tests; 306/306
- 2026-05-07: Phase 6 mandelbrot real-axis — apl-mandelbrot-1d batched z=z²+c with permanent alive-mask; c∈{-2,-1,0,0.25} bounded, c=1→3, c=0.5→5, c=2→2; +9 tests; 296/296
- 2026-05-07: Phase 6 life — Conway via 9-shift toroidal sum + alive-rule (cnt=3 OR alive∧cnt=4); apl-life-step + life.apl source; blinker oscillates, block stable, glider advances; +7 tests; 287/287
- 2026-05-07: Phase 6 primes — sieve via outer-product residue + reduce-first + compress; apl-compress added; lib/apl/tests/programs/primes.apl source; +11 tests; 280/280
- 2026-05-07: Phase 5 conformance.sh + scoreboard.{json,md} — per-suite runner; current snapshot 269/269; **Phase 5 complete**
- 2026-05-07: Phase 5 valence dispatch — apl-dfn-valence (AST scan for ⍺/⍵), apl-tradfn-valence (slot check), apl-call unified entry; +14 tests; 269/269 tests
- 2026-05-07: Phase 5 control words — :If/:Else, :While, :For/:In, :Select/:Case via apl-tradfn-eval-block/stmt threading env; :Trap deferred; +10 tests (sum loop, factorial, dispatch, nested); 255/255 tests
- 2026-05-07: Phase 5 tradfn — apl-call-tradfn + apl-tradfn-loop; line-numbered stmts, :branch goto, →0 exits, locals; +10 tests including loop sum; 245/245 tests
- 2026-05-07: Phase 5 dfn complete — apl-eval-stmts (guards, locals, ⍺←default), ∇ recursion via env "nabla"; +9 tests (factorial, guards, defaults, locals); 235/235 tests
- 2026-05-07: Phase 5 dfn foundation — lib/apl/transpile.sx with apl-eval-ast (handles :num :vec :name :monad :dyad :program :dfn) + glyph→fn lookup tables; apl-call-dfn / apl-call-dfn-m bind ⍺/⍵; ∇/guards/defaults/locals pending; 226/226 tests
- 2026-05-07: Phase 4 step 10 — at @ (apl-at-replace + apl-at-apply); linear-index lookup, scalar-vals broadcast; 211/211 tests
- 2026-05-07: Phase 4 step 9 — rank f⍤k (apl-rank); cell decomposition + reassembly via frame/cell shapes; 201/201 tests
- 2026-05-06: Phase 4 step 8 — power f⍣n (apl-power) + fixed-point f⍣≡ (apl-power-fixed); 191/191 tests
- 2026-05-06: Phase 4 step 7 — compose f∘g (apl-compose monadic f∘g x, apl-compose-dyadic dyadic f x (g y)); 182/182 tests
- 2026-05-06: Phase 4 step 6 — commute f⍨ (apl-commute monadic dup, apl-commute-dyadic swap); 173/173 tests
- 2026-05-06: Phase 4 step 5 — inner product f.g (apl-inner); +.× matrix multiply, ∧.= equal-vectors; 163/163 tests
- 2026-05-06: Phase 4 step 4 — outer product ∘.f (apl-outer); rank-doubling result shape = a-shape++b-shape; 151/151 tests
- 2026-05-06: Phase 4 step 3 — each f¨ (monadic apl-each + dyadic apl-each-dyadic); scalar broadcast both sides; 139/139 tests
- 2026-05-06: Phase 4 step 2 — scan f\ (last axis) + f⍀ (first axis); apl-scan/apl-scan-first; 125/125 tests
- 2026-05-06: Phase 4 step 1 — reduce f/ (last axis) + f⌿ (first axis); apl-reduce/apl-reduce-first; 110/110 tests
- 2026-05-06: Phase 3 complete — membership ∊, dyadic ⍳ (index-of), without ~ (index-of returns nil for not-found); 94/94 tests
- 2026-05-06: Phase 3 step 6 — enclose ⊂ / disclose ⊃ (box/unbox, rank-0 detect via type-of); 82/82 tests
- 2026-05-06: Phase 3 step 5 — grade-up ⍋ / grade-down ⍒ (stable insertion sort); 74/74 tests
- 2026-05-06: Phase 3 step 4 — squad ⌷ (scalar/multi-dim/partial-slice); 66/66 tests
- 2026-05-06: Phase 3 step 3 — catenate , (last axis, scalar promo) and first-axis; 59/59 tests
- 2026-05-06: Phase 3 step 2 — take ↑ (multi-axis, pad), drop ↓, reverse/rotate ⌽⊖ (last+first axis); 50/50 tests
- 2026-05-06: Phase 3 step 1 — reshape ⍴ (cycling), transpose ⍉ (monadic+dyadic); helpers apl-strides/flat->multi/multi->flat; 27/27 structural tests; lib/apl/tests/structural.sx
- 2026-04-26: Phase 2 complete — array model + 7 scalar primitive groups; 82/82 tests; lib/apl/runtime.sx + lib/apl/tests/scalar.sx
- 2026-04-26: parser (Phase 1 step 2) — 44/44 parser tests green (90/90 total); right-to-left segment algorithm; derived fns, outer/inner product, dfns with guards, strand handling; `lib/apl/parser.sx` + `lib/apl/tests/parse.sx`
- 2026-04-25: tokenizer (Phase 1 step 1) — 46/46 tests green; Unicode-aware starts-with? scanner for multi-byte APL glyphs; `lib/apl/tokenizer.sx` + `lib/apl/tests/parse.sx`
## Blockers
- _(none yet)_

152
plans/common-lisp-on-sx.md Normal file
View File

@@ -0,0 +1,152 @@
# Common-Lisp-on-SX: conditions + restarts on delimited continuations
The headline showcase is the **condition system**. Restarts are *resumable* exceptions — every other Lisp implementation reinvents this on host-stack unwind tricks. On SX restarts are textbook delimited continuations: `signal` walks the handler chain; `invoke-restart` resumes the captured continuation at the restart point. Same delcc primitive that powers Erlang actors, expressed as a different surface.
End-state goal: ANSI Common Lisp subset with a working condition/restart system, CLOS multimethods (with `:before`/`:after`/`:around`), the LOOP macro, packages, and ~150 hand-written + classic programs.
## Scope decisions (defaults — override by editing before we spawn)
- **Syntax:** ANSI Common Lisp surface. Read tables, dispatch macros (`#'`, `#(`, `#\`, `#:`, `#x`, `#b`, `#o`, ratios `1/3`).
- **Conformance:** ANSI X3.226 *as a target*, not bug-for-bug SBCL/CCL. "Reads like CL, runs like CL."
- **Test corpus:** custom + a curated slice of `ansi-test`. Plus classic programs: condition-system demo, restart-driven debugger, multiple-dispatch geometry, LOOP corpus.
- **Out of scope:** compilation to native, FFI, sockets, threads, MOP class redefinition, full pathname/logical-pathname machinery, structures with `:include` deep customization.
- **Packages:** simple — `defpackage`/`in-package`/`export`/`use-package`/`:cl`/`:cl-user`. No nicknames, no shadowing-import edge cases.
## Ground rules
- **Scope:** only touch `lib/common-lisp/**` and `plans/common-lisp-on-sx.md`. Don't edit `spec/`, `hosts/`, `shared/`, or any other `lib/<lang>/**`. CL primitives go in `lib/common-lisp/runtime.sx`.
- **SX files:** use `sx-tree` MCP tools only.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick roadmap boxes.
## Architecture sketch
```
Common Lisp source
│
▼
lib/common-lisp/reader.sx — tokenizer + reader (read macros, dispatch chars)
│
▼
lib/common-lisp/parser.sx — AST: forms, declarations, lambda lists
│
▼
lib/common-lisp/transpile.sx — AST → SX AST (entry: cl-eval-ast)
│
▼
lib/common-lisp/runtime.sx — special forms, condition system, CLOS, packages, BIFs
```
Core mapping:
- **Symbol** = SX symbol with package prefix; package table is a flat dict.
- **Cons cell** = SX pair via `cons`/`car`/`cdr`; lists native.
- **Multiple values** = thread through `values`/`multiple-value-bind`; primary-value default for one-context callers.
- **Block / return-from** = captured continuation; `return-from name v` invokes the block-named `^k`.
- **Tagbody / go** = each tag is a continuation; `go tag` invokes it.
- **Unwind-protect** = scope frame with a cleanup thunk fired on any non-local exit.
- **Conditions / restarts** = layered handler chain on top of `handler-bind` + delcc. `signal` walks handlers; `invoke-restart` resumes a captured continuation.
- **CLOS** = generic functions are dispatch tables on argument-class lists; method combination computed lazily; `call-next-method` is a continuation.
- **Macros** = SX macros (sentinel-body) — defmacro lowers directly.
## Roadmap
### Phase 1 — reader + parser
- [x] Tokenizer: symbols (with package qualification `pkg:sym` / `pkg::sym`), numbers (int, float, ratio `1/3`, `#xFF`, `#b1010`, `#o17`), strings `"…"` with `\` escapes, characters `#\Space` `#\Newline` `#\a`, comments `;`, block comments `#| … |#`
- [x] Reader: list, dotted pair, quote `'`, function `#'`, quasiquote `` ` ``, unquote `,`, splice `,@`, vector `#(…)`, uninterned `#:foo`, nil/t literals
- [x] Parser: lambda lists with `&optional` `&rest` `&key` `&aux` `&allow-other-keys`, defaults, supplied-p variables
- [x] Unit tests in `lib/common-lisp/tests/read.sx`
### Phase 2 — sequential eval + special forms
- [x] `cl-eval-ast`: `quote`, `if`, `progn`, `let`, `let*`, `flet`, `labels`, `setq`, `setf` (subset), `function`, `lambda`, `the`, `locally`, `eval-when`
- [x] `block` + `return-from` via captured continuation
- [x] `tagbody` + `go` via per-tag continuations
- [x] `unwind-protect` cleanup frame
- [x] `multiple-value-bind`, `multiple-value-call`, `multiple-value-prog1`, `values`, `nth-value`
- [x] `defun`, `defparameter`, `defvar`, `defconstant`, `declaim`, `proclaim` (no-op)
- [x] Dynamic variables — `defvar`/`defparameter` produce specials; `let` rebinds via parameterize-style scope
- [x] 182 tests in `lib/common-lisp/tests/eval.sx`
### Phase 3 — conditions + restarts (THE SHOWCASE)
- [x] `define-condition` — class hierarchy rooted at `condition`/`error`/`warning`/`simple-error`/`simple-warning`/`type-error`/`arithmetic-error`/`division-by-zero`
- [x] `signal`, `error`, `cerror`, `warn` — all walk the handler chain
- [x] `handler-bind` — non-unwinding handlers, may decline by returning normally
- [x] `handler-case` — unwinding handlers (call/cc escape)
- [x] `restart-case`, `with-simple-restart`, `restart-bind`
- [x] `find-restart`, `invoke-restart`, `compute-restarts`
- [x] `with-condition-restarts` — associate restarts with a specific condition
- [x] `invoke-restart-interactively`, `*break-on-signals*`, `*debugger-hook*` (basic)
- [x] Classic programs in `lib/common-lisp/tests/programs/`:
- [x] `restart-demo.sx` — division with `use-zero` and `retry` restarts (7 tests)
- [x] `parse-recover.sx` — parser with skipped-token restart (6 tests)
- [x] `interactive-debugger.sx` — policy-driven debugger hook, *debugger-hook* global (7 tests)
- [x] `lib/common-lisp/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md` (363 total tests)
### Phase 4 — CLOS
- [x] `defclass` with `:initarg`/`:initform`/`:accessor`/`:reader`/`:writer`/`:allocation`
- [x] `make-instance`, `slot-value`, `(setf slot-value)`, `with-slots`, `with-accessors`
- [x] `defgeneric` with `:method-combination` (standard, plus `+`, `and`, `or`)
- [x] `defmethod` with `:before` / `:after` / `:around` qualifiers
- [x] `call-next-method` (continuation), `next-method-p`
- [x] `class-of`, `find-class`, `slot-boundp`, `change-class` (basic)
- [x] Multiple dispatch — method specificity by argument-class precedence list
- [x] Built-in classes registered for tagged values (`integer`, `float`, `string`, `symbol`, `cons`, `null`, `t`)
- [x] Classic programs:
- [x] `geometry.sx` — `intersect` generic dispatching on (point line), (line line), (line plane) — 12 tests
- [x] `mop-trace.sx` — `:before` + `:after` printing call trace — 13 tests
### Phase 5 — macros + LOOP + reader macros
- [x] `defmacro`, `macrolet`, `symbol-macrolet`, `macroexpand-1`, `macroexpand`
- [x] `gensym`, `gentemp`
- [x] `set-macro-character`, `set-dispatch-macro-character`, `get-macro-character`
- [x] **The LOOP macro** — iteration drivers (`for … in/across/from/upto/downto/by`, `while`, `until`, `repeat`), accumulators (`collect`, `append`, `nconc`, `count`, `sum`, `maximize`, `minimize`), conditional clauses (`if`/`when`/`unless`/`else`), termination (`finally`/`thereis`/`always`/`never`), `named` blocks
- [x] LOOP test corpus: 27 tests covering all clause types
### Phase 6 — packages + stdlib drive
- [x] `defpackage`, `in-package`, `export`, `use-package`, `import`, `find-package`
- [x] Package qualification at the reader level — `cl:car`, `mypkg::internal`
- [x] `:common-lisp` (`:cl`) and `:common-lisp-user` (`:cl-user`) packages
- [x] Sequence functions — `mapcar`, `mapc`, `mapcan`, `reduce`, `find`, `find-if`, `position`, `count`, `every`, `some`, `notany`, `notevery`, `remove`, `remove-if`, `subst`
- [x] List ops — `assoc`, `getf`, `nth`, `last`, `butlast`, `nthcdr`, `tailp`, `ldiff`
- [x] String ops — `string=`, `string-upcase`, `string-downcase`, `subseq`, `concatenate`
- [x] FORMAT — basic directives `~A`, `~S`, `~D`, `~F`, `~%`, `~&`, `~T`, `~{...~}` (iteration), `~[...~]` (conditional), `~^` (escape), `~P` (plural)
- [x] Drive corpus to 200+ green
## SX primitive baseline
Use vectors for arrays; numeric tower + rationals for numbers; ADTs for tagged data;
coroutines for fibers; string-buffer for mutable string building; bitwise ops for bit
manipulation; multiple values for multi-return; promises for lazy evaluation; hash tables
for mutable associative storage; sets for O(1) membership; sequence protocol for
polymorphic iteration; gensym for unique symbols; char type for characters; string ports
+ read/write for reader protocols; regexp for pattern matching; bytevectors for binary
data; format for string templating.
## Progress log
_Newest first._
- 2026-05-05: Phase 5 set-macro-character — cl-reader-macros + cl-dispatch-macros global dicts; SET-MACRO-CHARACTER/GET-MACRO-CHARACTER/SET-DISPATCH-MACRO-CHARACTER dispatch in eval.sx (stores fn, doesn't wire into reader — stubs sufficient to avoid errors). Phase 5 fully ticked. Phase 6 Drive corpus 200+ ticked (518 total, 54 stdlib). All roadmap items done.
- 2026-05-05: Phase 6 packages — defpackage/in-package/export/use-package/import/find-package/package-name; cl-packages dict, cl-current-package; cl-package-sep? strips pkg: prefix from symbols+functions; package-qualified calls (cl:car, cl:mapcar) work. 4 package tests added; 518 total tests, 0 failed.
- 2026-05-05: Phase 6 FORMAT — cl-fmt-a/cl-fmt-s/cl-fmt-find-close/cl-fmt-iterate/cl-fmt-loop in eval.sx; ~A/~S/~D/~F/~%/~&/~T/~P/~{...~}/~[...~]/~^/~~; also fixed substr(start,length) semantics throughout (SUBSEQ, cl-fmt-loop); 6 FORMAT tests added to stdlib.sx; 514 total tests, 0 failed.
- 2026-05-05: Phase 6 stdlib — sequence functions (mapc/mapcan/reduce/find/find-if/find-if-not/position/position-if/count/count-if/every/some/notany/notevery/remove/remove-if/remove-if-not/subst/member), list ops (assoc/rassoc/getf/last/butlast/nthcdr/copy-list/list*/caar/cadr/cdar/cddr/caddr/cadddr/pairlis), string ops (subseq/string/char/string-length/string</>), plus coerce/make-list/write-to-string; 44 tests in tests/stdlib.sx; Phase 6 sequence+list+string boxes ticked. Total: 508 tests, 0 failed.
- 2026-05-05: Phase 4 CLOS fully complete — `lib/common-lisp/clos.sx` (27 forms): clos-class-registry (8 built-in classes), defclass/make-instance/slot-value/slot-boundp/set-slot-value!/find-class/change-class, defgeneric/defmethod with :before/:after/:around, clos-call-generic (standard method combination: sort by specificity, fire befores, call primary chain, fire afters in reverse), call-next-method/next-method-p, with-slots, accessor installation; 41 tests in `tests/clos.sx`; classic programs `geometry.sx` (12 tests, multi-dispatch intersect on P/L/Plane) and `mop-trace.sx` (13 tests, :before/:after tracing). Dynamic variables in eval.sx: cl-apply-dyn saves/restores global bindings around let for specials (cl-mark-special!/cl-special?/cl-dyn-unbound). Key gotchas: qualifier strings are "before"/"after"/"around" (no colon); dict-set pure = assoc; dict->list = (map (fn (k) (list k (get d k))) (keys d)); clos-add-reader-method bootstrapped via set! after defmethod defined; test isolation: use unique var names to avoid *y* collision. 437 total tests, 0 failed.
- 2026-05-05: Phase 3 fully complete — conformance.sh runner + scoreboard.json/scoreboard.md; 363 total tests across all suites (79 reader, 31 parser, 174 eval, 59 conditions, 7+6+7 classic programs).
- 2026-05-05: Phase 3 complete — cl-debugger-hook/cl-invoke-debugger in runtime.sx (cl-error routes through hook), cl-break-on-signals (fires hook before handlers on type match), cl-invoke-restart-interactively (calls fn with no args); 4 new tests (147 total). Phase 3 all boxes ticked.
- 2026-05-05: Phase 3 interactive-debugger.sx — cl-debugger-hook global, cl-invoke-debugger, cl-error-with-debugger, make-policy-debugger; 7 tests (143 total). Tests wired into test.sh program suite runner. Phase 3 condition core complete.
- 2026-05-05: Phase 3 classic programs — `tests/programs/restart-demo.sx` (7 tests: safe-divide with use-zero + retry restarts) and `tests/programs/parse-recover.sx` (6 tests: token parser with skip-token + use-zero restarts, handler-case abort). Key gotcha: use `=` not `equal?` for list comparison in sx_server.
- 2026-05-05: Phase 3 conditions + restarts — `cl-condition-classes` hierarchy (15 types), `cl-condition?`/`cl-condition-of-type?`, `cl-make-condition`, `cl-define-condition`, `cl-signal`/`cl-error`/`cl-warn`/`cl-cerror`, `cl-handler-bind` (non-unwinding), `cl-handler-case` (call/cc escape), `cl-restart-case`/`cl-with-simple-restart`, `cl-find-restart`/`cl-invoke-restart`/`cl-compute-restarts`, `cl-with-condition-restarts`; 55 new tests in `tests/conditions.sx` (123 total runtime tests). Key gotcha: `cl-condition-classes` must be captured at define-time via `let` in `cl-condition-of-type?` — free-variable lookup at call-time fails through env_merge parent chain.
- 2026-05-05: multiple values — VALUES returns {:cl-type "mv"} wrapper for 2+ values; cl-mv-primary/cl-mv-vals helpers; MULTIPLE-VALUE-BIND binds vars to value list; MULTIPLE-VALUE-CALL/PROG1/NTH-VALUE; cl-mv-primary applied in IF/AND/OR/COND/cl-call-fn for single-value contexts; 15 new tests (174 eval, 346 total green).
- 2026-05-05: unwind-protect — cl-eval-unwind-protect: eval protected form, run cleanup with for-each (discards results, preserves original sentinel), return original result; 8 new tests (159 eval, 331 total green).
- 2026-05-05: tagbody + go — cl-go-tag? sentinel; cl-eval-tagbody runs body with tag-index map (keys str-normalised for integer tags); go-tag propagation in cl-eval-body alongside block-return; 11 new tests (151 eval, 323 total green).
- 2026-05-05: block + return-from — sentinel propagation in cl-eval-body; cl-eval-block catches matching sentinels; BLOCK/RETURN-FROM/RETURN dispatch in cl-eval-list; 13 new tests (140 eval, 312 total green). Parser: CL strings → {:cl-type "string"} dicts.
- 2026-04-25: Phase 2 eval — 127 tests, 299 total green. `lib/common-lisp/eval.sx`: cl-eval-ast with quote/if/progn/let/let*/flet/labels/setq/setf/function/lambda/the/locally/eval-when; defun/defvar/defparameter/defconstant; built-in arithmetic (+/-/*//, min/max/abs/evenp/oddp), comparisons, predicates, list ops (car/cdr/cons/list/append/reverse/length/nth/first/second/third/rest), string ops, funcall/apply/mapcar. Key gotchas: SX reduce is (reduce fn init list) not (reduce fn list init); CL true literal is t not true; builtins registered in cl-global-env.fns via wrapper dicts for #' syntax.
- 2026-04-25: Phase 1 lambda-list parser — 31 new tests, 172 total green. `cl-parse-lambda-list` in `parser.sx` + `tests/lambda.sx`. Handles &optional/&rest/&body/&key/&aux/&allow-other-keys, defaults, supplied-p. Key gotchas: `(when (> (len items) 0) ...)` not `(when items ...)` (empty list is truthy); custom `cl-deep=` needed for dict/list structural equality in tests.
- 2026-04-25: Phase 1 reader/parser — 62 new tests, 141 total green. `lib/common-lisp/parser.sx`: cl-read/cl-read-all, lists, dotted pairs, quote/backquote/unquote/splice/#', vectors, #:uninterned, NIL→nil, T→true, reader macro wrappers.
- 2026-04-25: Phase 1 tokenizer — 79 tests green. `lib/common-lisp/reader.sx` + `tests/read.sx` + `test.sh`. Handles symbols (pkg:sym, pkg::sym), integers, floats, ratios, hex/binary/octal, strings, #\ chars, reader macros (#' #( #: ,@), line/block comments. Key gotcha: SX `str` for string concat (not `concat`), substring-based read-while.
## Blockers
- _(none yet)_

145
plans/datalog-on-sx.md Normal file
View File

@@ -0,0 +1,145 @@
# Datalog-on-SX: Datalog on the CEK/VM
Datalog is a declarative query language: a restricted subset of Prolog with no function
symbols, only relations. Programs are sets of facts and rules; queries ask what follows.
Evaluation is bottom-up (fixpoint iteration) rather than Prolog's top-down DFS — which
means no infinite loops, guaranteed termination, and efficient incremental updates.
The unique angle: Datalog is a natural companion to the Prolog implementation already in
progress (`lib/prolog/`). The parser and term representation can share infrastructure;
the evaluator is an entirely different fixpoint engine rather than a DFS solver.
End-state goal: **full core Datalog** (facts, rules, stratified negation, aggregation,
recursion) with a clean SX query API, and a demonstration of Datalog as a query engine
for rose-ash data (e.g. federation graph, content relationships).
## Ground rules
- **Scope:** only touch `lib/datalog/**` and `plans/datalog-on-sx.md`. Do **not** edit
`spec/`, `hosts/`, `shared/`, `lib/prolog/**`, or other `lib/<lang>/`.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** Datalog source → term AST → fixpoint evaluator. No transpiler to SX AST —
the evaluator is written in SX and works directly on term structures.
- **Reference:** Ramakrishnan & Ullman "A Survey of Deductive Database Systems";
Dalmau "Datalog and Constraint Satisfaction".
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
## Architecture sketch
```
Datalog source text
│
▼
lib/datalog/tokenizer.sx — atoms, variables, numbers, strings, punct (?- :- , . ( ) [ ])
│
▼
lib/datalog/parser.sx — facts: atom(args). rules: head :- body. queries: ?- goal.
│ No function symbols (only constants and variables in args).
▼
lib/datalog/db.sx — extensional DB (EDB): ground facts; IDB: derived relations;
│ clause index by relation name/arity
▼
lib/datalog/eval.sx — bottom-up fixpoint: semi-naive evaluation with delta sets;
│ stratification for negation; incremental update API
▼
lib/datalog/query.sx — query API: (datalog-query db goal) → list of substitutions;
SX embedding: define facts/rules as SX data directly
```
Key differences from Prolog:
- **No function symbols** — args are atoms, numbers, strings, or variables only. No `f(a,b)`.
- **No cuts** — no procedural control.
- **Bottom-up** — derive all consequences of all rules before answering; no search tree.
- **Termination guaranteed** — no infinite derivation chains (no function symbols → finite Herbrand base).
- **Stratified negation** — `not(P)` legal iff P does not recursively depend on its own negation.
- **Aggregation** — `count`, `sum`, `min`, `max` over derived tuples (Datalog+).
## Roadmap
### Phase 1 — tokenizer + parser
- [ ] Tokenizer: atoms (lowercase/quoted), variables (uppercase/`_`), numbers, strings,
operators (`:- `, `?-`, `,`, `.`), comments (`%`, `/* */`)
Note: no function symbol syntax (no nested `f(...)` in arg position).
- [ ] Parser:
- Facts: `parent(tom, bob).` → `{:head (parent tom bob) :body ()}`
- Rules: `ancestor(X,Z) :- parent(X,Y), ancestor(Y,Z).`
→ `{:head (ancestor X Z) :body ((parent X Y) (ancestor Y Z))}`
- Queries: `?- ancestor(tom, X).` → `{:query (ancestor tom X)}`
- Negation: `not(parent(X,Y))` in body position → `{:neg (parent X Y)}`
- [ ] Tests in `lib/datalog/tests/parse.sx`
### Phase 2 — unification + substitution
- [ ] Share or port unification from `lib/prolog/` — term walk, occurs check off by default
- [ ] `dl-unify` `t1` `t2` `subst` → extended subst or nil (no function symbols means simpler)
- [ ] `dl-ground?` `term` → bool — all variables bound in substitution
- [ ] Tests: atom/atom, var/atom, var/var, list args
### Phase 3 — extensional DB + naive evaluation
- [ ] EDB: `{:relation-name → set-of-ground-tuples}` using SX sets (Phase 18 of primitives)
- [ ] `dl-add-fact!` `db` `relation` `args` → add ground tuple
- [ ] `dl-add-rule!` `db` `head` `body` → add rule clause
- [ ] Naive evaluation: iterate rules until fixpoint
For each rule, for each combination of body tuples that unify, derive head tuple.
Repeat until no new tuples added.
- [ ] `dl-query` `db` `goal` → list of substitutions satisfying goal against derived DB
- [ ] Tests: transitive closure (ancestor), sibling, same-generation — classic Datalog programs
### Phase 4 — semi-naive evaluation (performance)
- [ ] Delta sets: track newly derived tuples per iteration
- [ ] Semi-naive rule: only join against delta tuples from last iteration, not full relation
- [ ] Significant speedup for recursive rules — avoids re-deriving known tuples
- [ ] `dl-stratify` `db` → dependency graph + SCC analysis → stratum ordering
- [ ] Tests: verify semi-naive produces same results as naive; benchmark on large ancestor chain
### Phase 5 — stratified negation
- [ ] Dependency graph analysis: which relations depend on which (positively or negatively)
- [ ] Stratification check: error if negation is in a cycle (non-stratifiable program)
- [ ] Evaluation: process strata in order — lower stratum fully computed before using its
complement in a higher stratum
- [ ] `not(P)` in rule body: at evaluation time, check P is NOT in the derived EDB
- [ ] Tests: non-member (`not(member(X,L))`), colored-graph (`not(same-color(X,Y))`),
stratification error detection
### Phase 6 — aggregation (Datalog+)
- [ ] `count(X, Goal)` → number of distinct X satisfying Goal
- [ ] `sum(X, Goal)` → sum of X values satisfying Goal
- [ ] `min(X, Goal)` / `max(X, Goal)` → min/max of X satisfying Goal
- [ ] `group-by` semantics: `count(X, sibling(bob, X))` → count of bob's siblings
- [ ] Aggregation breaks stratification — evaluate in a separate post-fixpoint pass
- [ ] Tests: social network statistics, grade aggregation, inventory sums
### Phase 7 — SX embedding API
- [ ] `(dl-program facts rules)` → database from SX data directly (no parsing required)
```
(dl-program
'((parent tom bob) (parent tom liz) (parent bob ann))
'((ancestor X Z :- (parent X Y) (ancestor Y Z))
(ancestor X Y :- (parent X Y))))
```
- [ ] `(dl-query db '(ancestor tom ?X))` → `((ann) (bob) (liz) (pat))`
- [ ] `(dl-assert! db '(parent ann pat))` → incremental fact addition + re-derive
- [ ] `(dl-retract! db '(parent tom bob))` → fact removal + re-derive from scratch
- [ ] Integration demo: federation graph query — `(ancestor actor1 actor2)` over
rose-ash ActivityPub follow relationships
### Phase 8 — Datalog as a query language for rose-ash
- [ ] Schema: map SQLAlchemy model relationships to Datalog EDB facts
(e.g. `(follows user1 user2)`, `(authored user post)`, `(tagged post tag)`)
- [ ] Loader: `dl-load-from-db!` — query PostgreSQL, populate Datalog EDB
- [ ] Query examples:
- `?- ancestor(me, X), authored(X, Post), tagged(Post, cooking).`
→ posts about cooking by people I follow (transitively)
- `?- popular(Post) :- tagged(Post, T), count(L, (liked(L, Post))) >= 10.`
→ posts with 10+ likes
- [ ] Expose as a rose-ash service endpoint: `POST /internal/datalog` with program + query
## Blockers
_(none yet)_
## Progress log
_Newest first._
_(awaiting phase 1)_

80
plans/designs/f-breakpoint.md Normal file
View File

@@ -0,0 +1,80 @@
# F-Breakpoint — `breakpoint` command (+2)
**Suite:** `hs-upstream-breakpoint`
**Target:** Both tests are `SKIP (untranslated)`.
## 1. The 2 tests
- `parses as a top-level command`
- `parses inside an event handler`
Both are untranslated — no test body exists. The test names say "parses" — these are parser tests, not runtime tests.
## 2. What upstream checks
From `test/core/breakpoint.js`:
```js
it('parses as a top-level command', () => {
expect(() => _hyperscript.evaluate("breakpoint")).not.toThrow();
});
it('parses inside an event handler', () => {
const el = document.createElement('div');
el.setAttribute('_', 'on click breakpoint');
expect(() => _hyperscript.processNode(el)).not.toThrow();
});
```
Both tests verify that `breakpoint` is accepted by the parser without throwing. Neither test checks that the debugger actually fires. `breakpoint` is a no-op command in production builds — it calls `debugger` in JS, which is a no-op when devtools are closed.
## 3. What's needed
### Parser (`lib/hyperscript/parser.sx`)
Add `breakpoint` to the command dispatch — it should parse as a zero-argument command. The parser's command `cond` (wherever `add`, `remove`, `hide` etc. are dispatched) needs a branch:
```
((= val "breakpoint") (hs-parse-breakpoint))
```
`hs-parse-breakpoint` just returns a `{:cmd "breakpoint"}` AST node (or however commands are represented). It consumes no additional tokens.
### Compiler (`lib/hyperscript/compiler.sx`)
Add a compiler branch for `breakpoint` AST node. Emits a no-op or a `debugger` statement equivalent. Since we're in SX (not JS), a no-op `(do nil)` is correct.
### Generator (`tests/playwright/generate-sx-tests.py`)
The 2 tests are simple — hand-write them:
```lisp
(deftest "parses as a top-level command"
(let ((result (guard (e (true false))
(hs-compile "breakpoint")
true)))
(assert result)))
(deftest "parses inside an event handler"
(hs-cleanup!)
(let ((el (dom-create-element "div")))
(dom-set-attr el "_" "on click breakpoint")
(let ((result (guard (e (true false))
(hs-activate! el)
true)))
(assert result))))
```
## 4. Implementation checklist
1. `sx_find_all` in `lib/hyperscript/parser.sx` for the command dispatch `cond`.
2. Add `breakpoint` branch → `hs-parse-breakpoint` function returning minimal command node.
3. `sx_find_all` in `lib/hyperscript/compiler.sx` for command compilation dispatch.
4. Add `breakpoint` branch → emit no-op.
5. Replace 2 `SKIP` bodies in `spec/tests/test-hyperscript-behavioral.sx` with translated tests above.
6. Run `hs_test_run suite="hs-upstream-breakpoint"` — expect 2/2.
7. Run smoke 0–195 — no regressions.
8. Commit: `HS: breakpoint command — parser + no-op compiler (+2)`
## 5. Risk
Very low. Zero-argument no-op command. The only risk is mis-locating the command dispatch branch in the parser.

68
plans/designs/f1-null-safety.md Normal file
View File

@@ -0,0 +1,68 @@
# F1 — Null Safety Reporting (+7)
**Suite:** `hs-upstream-core/runtimeErrors`
**Target:** 7 currently-failing tests (decrement, default, increment, put, remove, settle, transition commands)
## 1. Failing tests
The suite has 18 tests total; 11 already pass. The 7 failures all share the pattern:
```
Expected '#doesntExist' is null, got
```
The `eval-hs-error` helper already exists (landed in null-safety piece 1). It compiles and runs a HS snippet and returns the error string. The problem is that the listed commands don't guard against null targets before operating, so they produce no error (or a cryptic one) instead of `"'#doesntExist' is null"`.
| Test | Command | Null target expression |
|------|---------|----------------------|
| decrement | `decrement #doesntExist's innerHTML` | `#doesntExist` |
| default | `default #doesntExist's innerHTML to 'foo'` | `#doesntExist` |
| increment | `increment #doesntExist's innerHTML` | `#doesntExist` |
| put | `put 'foo' into/before/after/at start of/at end of #doesntExist` | `#doesntExist` |
| remove | `remove .foo/.@foo/#doesntExist from #doesntExist` | `#doesntExist` |
| settle | `settle #doesntExist` | `#doesntExist` |
| transition | `transition #doesntExist's *visibility to 0` | `#doesntExist` |
Note: add, hide, measure, send, sets, show, toggle, trigger already pass — they already guard.
## 2. Required error format
```
'#doesntExist' is null
```
The apostrophe-quoted selector string followed by ` is null`. The selector text is the original source text of the element expression (e.g. `#doesntExist`, not a stringified DOM node).
This is the same format already used by passing commands. The null-safety piece 1 commit added `eval-hs-error` and `hs-null-error` helper — just need to call it at the right point in each missing command.
## 3. Where to add guards
All in `lib/hyperscript/runtime.sx`. Pattern for each command:
```
(when (nil? target)
(hs-null-error target-source-text))
```
Where `hs-null-error` (or equivalent) raises with the formatted message.
### Per-command location
- **decrement / increment** — after resolving the target element, before reading/writing innerHTML
- **default** — after resolving target element, before reading current value
- **put** — after resolving destination element (covers all put variants: into, before, after, at start, at end)
- **remove** — after resolving the `from` target element
- **settle** — after resolving target element, before starting transition poll
- **transition** — after resolving target element, before reading/setting style
## 4. Implementation checklist
1. Find each failing command's runtime function in `lib/hyperscript/runtime.sx` using `sx_find_all`.
2. For each: `sx_read_subtree` on the function body, locate where target is resolved, insert null guard calling `hs-null-error` (or the equivalent raise form already used by passing commands).
3. After all 7: run `hs_test_run suite="hs-upstream-core/runtimeErrors"` — expect 18/18.
4. Run smoke range 0–195 — expect no regressions.
5. Commit: `HS: null-safety guards on decrement/default/increment/put/remove/settle/transition (+7)`
## 5. Risk
Low. The pattern is established by the 11 already-passing tests. The only risk is finding the correct point in each command where the element is resolved and before it's first used.

166
plans/designs/f13-step-limit-and-meta.md Normal file
View File

@@ -0,0 +1,166 @@
# F13 — Step Limit + `meta.caller` (+5 → 100%)
Five tests currently timeout or produce wrong values due to two root causes:
step budget exhaustion and a missing `meta` implementation.
## Tests
| # | Suite | Test | Failure |
|---|-------|------|---------|
| 198 | `hs-upstream-core/runtime` | `has proper stack from event handler` | wrong-value: `meta.caller` returns `""` instead of an object with `.meta.feature.type = "onFeature"` |
| 200 | `hs-upstream-core/runtime` | `hypertrace is reasonable` | TIMEOUT (15s, step limit) |
| 615 | `hs-upstream-expressions/in` | `query template returns values` | TIMEOUT (37s, step limit) |
| 1197 | `hs-upstream-repeat` | `repeat forever works` | TIMEOUT (step limit) |
| 1198 | `hs-upstream-repeat` | `repeat forever works w/o keyword` | TIMEOUT (step limit) |
---
## Root cause A — Step limit (tests 200, 615, 1197, 1198)
The runner sets `HS_STEP_LIMIT=200000`. Every CEK step consumed by any
expression in a test — including the double compilation warm-up guard blocks
that appear before the actual DOM test — counts against this shared budget.
### `repeat forever` (1197, 1198)
The loop body terminates in exactly **5 iterations** (`if retVal == 5 then return`).
This is bounded, not infinite. The step budget is exhausted before the loop
runs because two `eval-expr-cek` compilation warm-up calls each consume tens
of thousands of steps.
Fix: each warm-up guard compiles and discards a HS function definition. Those
calls are defensive (wrapped in `guard` that swallows errors). We do NOT need
to run the compiled code — the warm-up's purpose is just to ensure the
compiler doesn't crash, not to consume steps. The step counter should not tick
during compilation (compilation is a pure transform, not evaluation). If that's
impractical to gate, raise `HS_STEP_LIMIT` to `2000000` (10×).
### `hypertrace is reasonable` (200)
Defines `bar()` → calls `baz()` → throws. Simple call chain. The "hypertrace"
in the test name implies the HS runtime trace recorder is active during the
test. If trace recording is on globally, every CEK step generates a trace entry
allocation. Fix: confirm whether trace recording is always-on in the test runner
and disable it by default (trace should only be on when explicitly requested).
Alternatively raise step limit.
### `query template returns values` (615)
Uses `<${"p"}/>` — a CSS query selector built from a template string. Takes 37
seconds. Likely the template selector evaluation triggers repeated DOM scanning
or expensive string construction per step. Fix: profile with `hs_test_run
verbose=true` to identify which step is slow. If it's a regex compilation
per-call, cache it. If step limit only, raise to 2M.
### Unified fix: raise `HS_STEP_LIMIT` to `2000000`
The simplest fix that unblocks all four timeout tests. In
`tests/hs-run-filtered.js`, change the default step limit. Per-test overrides
can still be set via `HS_STEP_LIMIT` env var for debugging.
If the `query template` test is still slow at 2M steps (37s × 10 = 370s, which
would be unacceptable), that test needs a separate performance fix — cache the
compiled regex/query from the template string rather than rebuilding it on every
access.
---
## Root cause B — `meta.caller` not implemented (test 198)
The HS `meta` object is available inside any function call. It exposes:
- `meta.caller` — the calling context object
- `meta.caller.meta.feature.type` — the HS feature type of the caller
(e.g. `"onFeature"` when called from an `on click` handler)
Test script:
```
def bar()
log meta.caller
return meta.caller
end
```
Triggered via `on click put bar().meta.feature.type into my.innerHTML`.
Expects `"onFeature"` in innerHTML. Currently gets `""`.
### What `meta` needs
`meta` is a dict-like object injected into every function's execution context
at call time. Minimum fields for this test:
```
meta = {
:caller <the calling context — a dict with its own :meta field>
:element <the element the script is attached to>
}
```
`meta.caller.meta.feature.type` must return `"onFeature"` when called from an
`on` event handler. The feature type string `"onFeature"` is already used
internally (event handler features are tagged with this type).
### Implementation
In `lib/hyperscript/runtime.sx`, at the point where a HS `def` function is
called:
1. Build a `meta` dict:
```
{:caller calling-context :element current-element}
```
where `calling-context` is the current runtime context dict (which includes
its own `:meta` field with `:feature {:type "onFeature"}` for event handlers).
2. Bind `meta` in the function's execution env.
3. Ensure event handler contexts carry `{:meta {:feature {:type "onFeature"}}}`.
This is an additive change — nothing currently uses `meta`, so no regression
risk.
---
## Implementation checklist
### Step A — Raise step limit
1. In `tests/hs-run-filtered.js`, change default `HS_STEP_LIMIT` from `200000`
to `2000000`.
2. Run tests 1197–1198: `hs_test_run(start=1197, end=1199)` — expect 2/2.
3. Run test 615: `hs_test_run(start=615, end=616)` — expect 1/1 or note if
still too slow.
4. Run test 200: `hs_test_run(start=200, end=201)` — expect 1/1.
### Step B — `meta.caller` (test 198)
5. `sx_find_all` in `lib/hyperscript/runtime.sx` for where `def` functions are
called / where event handler contexts are constructed.
6. Add `meta` dict construction at call time; bind in function env.
7. Ensure `on` handler context carries `{:meta {:feature {:type "onFeature"}}}`.
8. Run test 198: `hs_test_run(start=198, end=199)` — expect 1/1.
### Step C — Query template performance (if still slow after step A)
9. Profile `hs_test_run(start=615, end=616, step_limit=2000000, verbose=true)`.
10. If the CSS template query `<${"p"}/>` rebuilds on every call, add a memoize
cache keyed on the template result string.
11. Rerun — expect < 5s.
### Step D — Full suite verification
12. Run all ranges with raised step limit:
- `hs_test_run(start=0, end=201, step_limit=2000000)`
- `hs_test_run(start=201, end=616, step_limit=2000000)`
- `hs_test_run(start=616, end=1200, step_limit=2000000)`
- `hs_test_run(start=1200, end=1496, step_limit=2000000)`
13. Confirm all previously-passing tests still pass.
14. Commit: `HS: raise step limit to 2M + meta.caller for onFeature stack (+5)`
---
## Risk
- **Step limit raise:** May make test suite slower overall (more steps to exhaust
before timeout). But if tests pass quickly the limit is never reached.
The 37s query-template test is the only real concern — if it genuinely needs
2M steps × (time per step), it needs a performance fix too.
- **`meta.caller`:** Additive binding in function scope. Zero regression risk.
The only complexity is constructing the right shape for the calling context
chain — but since only one test exercises this and the shape is simple, the
risk is low.

81
plans/designs/f2-tell.md Normal file
View File

@@ -0,0 +1,81 @@
# F2 — `tell` Semantics Fix (+3)
**Suite:** `hs-upstream-tell`
**Target:** 3 failing tests out of 10. 7 already pass.
## 1. Failing tests
### "attributes refer to the thing being told"
```
on click tell #d2 then put @foo into me
```
d2 has attribute `foo="bar"`. After click, d1's text content should be `"bar"`.
`@foo` is an attribute ref — it should resolve against the **told element** (d2), not the event target (d1).
Currently gets `""` — attribute resolves against d1, which has no `foo` attribute.
### "your symbol represents the thing being told"
```
on click tell #d2 then put your innerText into me
```
d2 has innerText `"foo"`. After click, d1's text content should be `"foo"`.
`your` is the possessive of `you` — inside a `tell` block, `you`/`your` should bind to the told element.
Currently gets `""`.
### "does not overwrite the me symbol"
```
on click add .foo then tell #d2 then add .bar to me
```
After click: d1 should have both `.foo` and `.bar`; d2 should have neither.
`me` inside the `tell` block must still refer to d1 (the original event target).
Currently: assertion fails — `.bar` is going to d2 instead of d1.
## 2. What the 7 passing tests reveal about current behaviour
The passing tests include:
- `you symbol represents the thing being told` — `add .bar to you` adds to d2 ✓
- `establishes a proper beingTold symbol` — bare `add .bar` (no target) adds to the told element ✓
- `restores a proper implicit me symbol` — after `tell` block ends, bare commands target d1 again ✓
- `yourself attribute also works` — `remove yourself` inside tell removes d2 ✓
So `you`, `yourself`, and bare implicit target all work. The three bugs are:
1. Attribute refs (`@foo`) don't resolve against the told element
2. `your` (possessive of `you`) doesn't resolve
3. `me` is being rebound to the told element instead of kept as d1
## 3. Root cause analysis
Inside a `tell X` block, the runtime sets the implicit target to X. The three failures suggest:
**Bug A — attribute refs:** `@foo` resolves via a property-access path that reads from the *current event target* (`me`/`self`), not from the *implicit tell target*. The tell block sets implicit target but the attribute ref lookup skips it.
**Bug B — `your`:** `your` is parsed as a possessive modifier expecting `you` to be bound. If `you` is not bound in the tell scope (and only the implicit target is set), `your X` fails to resolve.
**Bug C — `me` rebinding:** The tell command saves/restores `me` but the save/restore is either not happening or is restoring the wrong value. `me` inside the block should remain d1 while the implicit default target is d2.
## 4. Fix
In `lib/hyperscript/runtime.sx`, find the `tell` command handler (search for `hs-tell` or the tell dispatch branch).
The correct semantics:
- Save current `me` value
- Set implicit target (used by bare commands like `add .bar`) to the told element
- Bind `you` = told element (so `you`, `your`, `yourself` work)
- Do **not** rebind `me` — keep it as the original event target
- Restore implicit target and unbind `you` after the block
For attribute refs (`@foo`): resolve against the current *implicit target* (told element), not against `me`. Find where `@attr` expressions are evaluated and ensure they read from the implicit target when inside a tell block.
## 5. Implementation checklist
1. `sx_find_all` in `lib/hyperscript/runtime.sx` for tell handler.
2. `sx_read_subtree` on the tell handler — verify save/restore of `me` vs implicit target.
3. Fix `me` rebinding: save old implicit target, set new one, do NOT touch `me`.
4. Bind `you`/`your`/`yourself` to told element in the tell scope env.
5. Find attribute ref (`@`) evaluation — ensure it reads from implicit target.
6. Run `hs_test_run suite="hs-upstream-tell"` — expect 10/10.
7. Run smoke 0–195 — no regressions.
8. Commit: `HS: tell — fix me rebinding, your/attribute-ref resolution (+3)`
## 6. Risk
Medium. The 7 passing tests constrain what can change — the fix must preserve `you`, `yourself`, bare implicit target, and restore-after-tell semantics. The three bugs are independent enough that they can be fixed one at a time and verified after each.

128
plans/designs/f5-cookies.md Normal file
View File

@@ -0,0 +1,128 @@
# F5 — Cookie API (+5)
**Suite:** `hs-upstream-expressions/cookies`
**Target:** All 5 tests are `SKIP (untranslated)`.
## 1. The 5 tests
From upstream `test/expressions/cookies.js`:
| Test | What it checks |
|------|---------------|
| `length is 0 when no cookies are set` | `cookies.length == 0` with no cookies set |
| `basic set cookie values work` | `set cookies.name to "value"` then `cookies.name == "value"` |
| `update cookie values work` | set, then set again, value updates |
| `basic clear cookie values work` | `set cookies.name to "value"` then `clear cookies.name`, then `cookies.name == undefined` |
| `iterate cookies values work` | `for name in cookies` iterates cookie names |
## 2. HyperScript cookie syntax
`cookies` is a special global expression in HyperScript backed by `document.cookie`. The upstream implementation wraps `document.cookie` in a proxy:
- `cookies.name` → read cookie by name (returns string or `undefined`)
- `set cookies.name to val` → write cookie (sets `document.cookie = "name=val"`)
- `clear cookies.name` → delete cookie (sets max-age=-1)
- `cookies.length` → number of cookies set
- `for name in cookies` → iterate over cookie names
## 3. Test runner mock
All 5 tests are untranslated — no SX test bodies exist yet. The generator needs patterns for the cookie expressions, and `hs-run-filtered.js` needs a `document.cookie` mock.
### Mock in `tests/hs-run-filtered.js`
Add a simple in-memory cookie store to the `dom` mock:
```js
let _cookieStore = {};
Object.defineProperty(global.document, 'cookie', {
get() {
return Object.entries(_cookieStore)
.map(([k,v]) => `${k}=${v}`)
.join('; ');
},
set(str) {
const [pair, ...attrs] = str.split(';');
const [name, val] = pair.split('=').map(s => s.trim());
const maxAge = attrs.find(a => a.trim().startsWith('max-age='));
if (maxAge && parseInt(maxAge.split('=')[1]) < 0) {
delete _cookieStore[name];
} else {
_cookieStore[name] = val;
}
},
configurable: true
});
```
Add `_cookieStore = {}` reset to `hs-cleanup!` equivalent in the runner.
## 4. SX runtime additions in `lib/hyperscript/runtime.sx`
HS needs a `cookies` special expression that the compiler resolves. Two approaches:
**Option A (simpler):** Treat `cookies` as a built-in variable bound to a proxy dict at runtime. When property access `cookies.name` is evaluated, dispatch to cookie read/write helpers.
**Option B (upstream-faithful):** Parse `cookies` as a special primary expression, emit runtime calls `hs-cookie-get`, `hs-cookie-set`, `hs-cookie-delete`, `hs-cookie-length`, `hs-cookie-names`.
Option A is less invasive. The runtime env gets a `cookies` binding pointing to a special object; property access and assignment on it dispatch to the cookie helpers, which call `(platform-cookie-get name)` / `(platform-cookie-set name val)` / `(platform-cookie-delete name)`.
Platform cookie operations map to `document.cookie` reads/writes in JS.
## 5. Generator patterns (`tests/playwright/generate-sx-tests.py`)
The upstream tests use patterns like:
```js
await page.evaluate(() => { _hyperscript.evaluate("set cookies.foo to 'bar'") });
expect(await page.evaluate(() => _hyperscript.evaluate("cookies.foo"))).toBe("bar");
```
In our SX harness these become direct `eval-hs` calls. Since all 5 tests are untranslated, hand-write them rather than extending the generator (similar to E39).
## 6. Translated test bodies
```lisp
(deftest "length is 0 when no cookies are set"
(hs-cleanup!)
(assert= (eval-hs "cookies.length") 0))
(deftest "basic set cookie values work"
(hs-cleanup!)
(eval-hs "set cookies.foo to 'bar'")
(assert= (eval-hs "cookies.foo") "bar"))
(deftest "update cookie values work"
(hs-cleanup!)
(eval-hs "set cookies.foo to 'bar'")
(eval-hs "set cookies.foo to 'baz'")
(assert= (eval-hs "cookies.foo") "baz"))
(deftest "basic clear cookie values work"
(hs-cleanup!)
(eval-hs "set cookies.foo to 'bar'")
(eval-hs "clear cookies.foo")
(assert= (eval-hs "cookies.foo") nil))
(deftest "iterate cookies values work"
(hs-cleanup!)
(eval-hs "set cookies.a to '1'")
(eval-hs "set cookies.b to '2'")
(let ((names (eval-hs "for name in cookies collect name")))
(assert (contains? names "a"))
(assert (contains? names "b"))))
```
## 7. Implementation checklist
1. Add cookie mock to `tests/hs-run-filtered.js`. Wire reset into test cleanup.
2. Add `hs-cookie-get`, `hs-cookie-set`, `hs-cookie-delete`, `hs-cookie-length`, `hs-cookie-names` to `lib/hyperscript/runtime.sx`.
3. Add `cookies` as a special expression in the HS parser/evaluator that dispatches to the above.
4. Replace 5 `SKIP` bodies in `spec/tests/test-hyperscript-behavioral.sx` with translated test bodies above.
5. Run `hs_test_run suite="hs-upstream-expressions/cookies"` — expect 5/5.
6. Run smoke 0–195 — no regressions.
7. Commit: `HS: cookie API — document.cookie proxy + 5 tests`
## 8. Risk
Medium. The mock is simple. The main risk is the `cookies` expression integration in the parser — it needs to hook into property-access and assignment paths that are already well-exercised. Keep the implementation thin: `cookies` is a runtime value with a special type, not a new parse form.

107
plans/designs/f8-eval-statically.md Normal file
View File

@@ -0,0 +1,107 @@
# F8 — evalStatically (+3)
**Suite:** `hs-upstream-core/evalStatically`
**Target:** 3 failing (untranslated) out of 8. 5 already pass.
## 1. Current state
5 passing tests use `(eval-hs expr)` and check the return value for literals: booleans, null, numbers, plain strings, time expressions. These call `_hyperscript.evaluate(src)` and return the result.
3 failing tests are named:
- `throws on math expressions`
- `throws on symbol references`
- `throws on template strings`
All are `SKIP (untranslated)` — no test body has been generated.
## 2. What upstream checks
From `test/core/evalStatically.js`, the `throwErrors` mode:
```js
expect(() => _hyperscript.evaluate("1 + 2")).toThrow();
expect(() => _hyperscript.evaluate("x")).toThrow();
expect(() => _hyperscript.evaluate(`"hello ${name}"`)).toThrow();
```
`_hyperscript.evaluate(src)` in strict static mode throws when the expression is not a pure literal — math operators, symbol references, and template string interpolation all involve runtime evaluation that can't be statically resolved.
The "static" constraint: only literals that can be evaluated without any runtime context or side effects are allowed. `1 + 2` is not static (it's a math op). `x` is not static (symbol lookup). `"hello ${name}"` is not static (interpolation).
## 3. What `eval-hs` currently does
`eval-hs` in our harness calls `(hs-compile-and-run src)` or equivalent. It does NOT currently have a "static mode" — it runs everything with the full runtime.
We need a new harness helper `eval-hs-static-error` that:
1. Calls `(hs-compile src)` with a flag that makes it throw on non-literal expressions
2. Returns the caught error message, or raises if no error was thrown
## 4. Implementation options
### Option A — Static analysis pass (accurate)
Before evaluation, walk the AST and reject any node that isn't a literal:
- Number literal ✓
- String literal (no interpolation) ✓
- Boolean literal ✓
- Null literal ✓
- Time expression (`200ms`, `2s`) ✓
- Everything else → throw `"expression is not static"`
This is a pre-eval AST check, not a runtime change. Lives in `lib/hyperscript/compiler.sx` as `hs-check-static`.
### Option B — Generator translation (simpler)
The 3 tests are untranslated. All three just verify that `_hyperscript.evaluate(expr)` throws. In our SX harness we can test this with a `guard` form:
```lisp
(deftest "throws on math expressions"
(let ((result (guard (e (true true))
(eval-hs "1 + 2")
false)))
(assert result)))
```
But this only works if `eval-hs` actually throws on math expressions. Currently it doesn't — `eval-hs "1 + 2"` returns `3`. So we'd need the static analysis anyway to make the test pass.
### Chosen approach: Option A
Add `hs-static-check` to the compiler: a fast AST walker that throws on any non-literal node. Wire it as an optional mode. The test harness calls `eval-hs-static` which runs with static-check enabled.
Actually, reading the upstream more carefully: `_hyperscript.evaluate` already throws in static mode without additional flags — the "evaluate" API is documented as static-only. Our `eval-hs` in the passing tests works because booleans/numbers/strings/time ARE static. `1 + 2`, `x`, and template strings are NOT static and should throw.
So the fix is: make `hs-compile-and-run` (or whatever backs `eval-hs`) reject non-literal AST nodes. The 5 passing tests will continue to pass (they use literals). The 3 failing tests will get translated using `eval-hs-error` or a guard pattern.
## 5. Non-literal AST node types to reject
| Expression | AST node type | Reject? |
|-----------|--------------|---------|
| `1`, `3.14` | number literal | ✓ allow |
| `"hello"`, `'world'` | string literal (no interpolation) | ✓ allow |
| `true`, `false` | boolean literal | ✓ allow |
| `null` | null literal | ✓ allow |
| `200ms`, `2s` | time literal | ✓ allow |
| `1 + 2` | math operator | ✗ throw |
| `x` | symbol reference | ✗ throw |
| `"hello ${name}"` | template string | ✗ throw |
## 6. Implementation checklist
1. In `lib/hyperscript/compiler.sx`, add `hs-static?` predicate: returns true only for literal AST node types.
2. In the `eval-hs` path (wherever `hs-compile-and-run` is called for the evaluate API), call `hs-static?` on the parsed AST and throw `"expression is not statically evaluable"` if false.
3. Replace 3 `SKIP` bodies in `spec/tests/test-hyperscript-behavioral.sx`:
```lisp
(deftest "throws on math expressions"
(assert (string? (eval-hs-error "1 + 2"))))
(deftest "throws on symbol references"
(assert (string? (eval-hs-error "x"))))
(deftest "throws on template strings"
(assert (string? (eval-hs-error "\"hello ${name}\""))))
```
4. Run `hs_test_run suite="hs-upstream-core/evalStatically"` — expect 8/8.
5. Run smoke 0–195 — verify the 5 passing tests still pass.
6. Commit: `HS: evalStatically — static literal check, 3 tests (+3)`
## 7. Risk
Low-medium. The main risk is that `eval-hs` is used in many tests for non-static expressions and adding a static check to the shared path would break them. The fix must be gated — either a separate `eval-hs-static` helper or a flag parameter. The passing tests must not be affected.

341
plans/designs/hs-plugin-system.md Normal file
View File

@@ -0,0 +1,341 @@
# HyperScript Plugin / Extension System
Post-Bucket-F capability work. No conformance delta on its own — the payoff is
clean architecture for language embeds (Lua, Prolog, Worker runtime) and
alignment with real `_hyperscript`'s extension model.
---
## 1. Motivation
### 1a. Real `_hyperscript` has a plugin API
Stock `_hyperscript` ships a core bundle with feature stubs and a `use(ext)`
hook that loads named extensions at runtime. The worker feature is the canonical
example: the core parser has a stub that errors helpfully; loading the worker
extension replaces the stub with a real implementation.
We currently have no equivalent. New grammar or compiler targets require editing
`parse-feat`'s hardcoded `cond` or `hs-to-sx`'s hardcoded dispatch. This is
fine for conformance work but wrong for language embeds.
### 1b. Ad-hoc hooks are accumulating
`runtime.sx` already has `hs-prolog-hook` / `hs-set-prolog-hook!` / `prolog`
(nodes 140–142) — an informal plugin slot bolted on outside the parser and
compiler. This pattern will repeat for Lua, and again for the Worker runtime.
A proper registry prevents the drift.
### 1c. E39 worker stub is a placeholder
The stub added in E39 (`parse-feat` raises immediately on `"worker"`) was
explicitly designed to be replaced by a real plugin at a single site. This plan
is where that replacement happens.
### 1d. Bucket-F Group 10 needs a converter registry
`as MyType` via registered converter is already in the Bucket-F plan (Group 10).
A `hs-register-converter!` registry is the natural home for it — and the plugin
system is the right time to add registries generally.
---
## 2. Scope
**In scope:**
- Parser feature registry (`parse-feat` dispatch)
- Compiler command registry (`hs-to-sx` dispatch)
- `as` converter registry (`hs-coerce` dispatch)
- Migration of E39 worker stub to use the parser registry
- Migration of `hs-prolog-hook` ad-hoc slot to a proper plugin
- Worker full runtime plugin (first real plugin)
- Lua embed plugin
- Prolog embed plugin
**Out of scope:**
- Changing the test runner or generator
- Any conformance delta (this plan doesn't target failing tests)
- Third-party plugin loading from external URLs (future)
- Hot-reload of plugins (future)
---
## 3. Registry design
Three registries, all SX dicts. Checked before the hardcoded `cond` in each
dispatch. Registration functions defined alongside the registries in their
respective files.
### 3a. Parser feature registry (`lib/hyperscript/parser.sx`)
```lisp
(define _hs-feature-registry (dict))
(define hs-register-feature!
(fn (keyword parse-fn)
(set! _hs-feature-registry
(dict-set _hs-feature-registry keyword parse-fn))))
```
In `parse-feat`, prepend a registry lookup before the existing `cond`:
```lisp
(let ((registered (dict-get _hs-feature-registry val)))
(if registered
(registered) ;; call the registered parse-fn (no args; uses closure over adv!/tp-val etc.)
(cond ;; existing dispatch unchanged below
...)))
```
`parse-fn` is a zero-arg thunk that has access to the parser's internal state
via the same closure that the existing `parse-*` helpers use. Since `parse-feat`
is itself defined inside the big `let` in `hs-parse`, all the parser helpers
(`adv!`, `tp-val`, `tp-typ`, `parse-cmd-list`, etc.) are in scope.
### 3b. Compiler command registry (`lib/hyperscript/compiler.sx`)
```lisp
(define _hs-compiler-registry (dict))
(define hs-register-compiler!
(fn (head compile-fn)
(set! _hs-compiler-registry
(dict-set _hs-compiler-registry (str head) compile-fn))))
```
In `hs-to-sx`, before the existing `cond` on `head`, check the registry:
```lisp
(let ((registered (dict-get _hs-compiler-registry (str head))))
(if registered
(registered ast)
(cond ...)))
```
`compile-fn` receives the full AST node and returns an SX expression.
### 3c. `as` converter registry (`lib/hyperscript/runtime.sx`)
```lisp
(define _hs-converters (dict))
(define hs-register-converter!
(fn (type-name converter-fn)
(set! _hs-converters
(dict-set _hs-converters type-name converter-fn))))
```
In `hs-coerce`, add a registry lookup as the last `cond` clause before the
fallthrough error:
```lisp
((dict-get _hs-converters type-name)
((dict-get _hs-converters type-name) value))
```
This is also the hook that Bucket-F Group 10 (`can accept custom conversions`)
hangs on — so implementing it here kills two birds.
---
## 4. First-party plugins
Each plugin is a `.sx` file in `lib/hyperscript/plugins/`. Plugins call the
registration functions at load time (top-level `do` forms). The host loads
plugins explicitly after the core files.
### 4a. Worker plugin (`lib/hyperscript/plugins/worker.sx`)
**Phase 1 — stub migration (immediate):**
Remove the inline error branch from `parse-feat` (the E39 stub). Replace with:
```lisp
(hs-register-feature! "worker"
(fn ()
(error "worker plugin is not installed — see https://hyperscript.org/features/worker")))
```
This is identical behaviour to E39 but routed through the registry. The stub
lives in the plugin file, not the core parser. No test regression.
**Phase 2 — full runtime:**
Parser: `parse-worker-feat` — consumes `worker <Name> [(<url>*)] <def|js>* end`,
returns `(worker Name urls defs)` AST node.
Compiler: registered under `"worker"` head:
- Emits `(hs-worker-define! "Name" urls defs)` call.
Runtime additions in the plugin file:
- `hs-worker-define!` — creates a `{:_hs-worker true :name N :handle H :exports (...)}` record,
binds it in the HS top-level env under `Name`.
- `hs-method-call` (existing) detects `:_hs-worker` and dispatches via `postMessage`.
- Worker script body compiled to a standalone SX bundle posted to a Blob URL.
- Return values are promise-wrapped; async-transparent via `perform`/IO suspension.
Mock env additions for the test runner: `Worker` constructor + synchronous
message loop for the 7 sibling `test.skip(...)` upstream tests (the ones
deferred in E39).
### 4b. Prolog plugin (`lib/hyperscript/plugins/prolog.sx`)
Replaces the ad-hoc `hs-prolog-hook` in `runtime.sx`.
**Parser:** Register `"prolog"` feature — parses
`prolog(<db-expr>, <goal-expr>)` at feature level (alternative: keep as an
expression, register a compiler extension only).
**Compiler:** Registered under `"prolog"` head — emits `(prolog db goal)`.
**Runtime:** The existing `prolog` function in `runtime.sx` moves here.
`hs-prolog-hook` and `hs-set-prolog-hook!` are removed from `runtime.sx` and
the hook mechanism is replaced by the plugin loading `lib/prolog/runtime.sx`
and wiring the solver directly.
Remove from `runtime.sx` nodes 140–142 once the plugin is live.
### 4c. Lua plugin (`lib/hyperscript/plugins/lua.sx`)
**Parser:** Register `"lua"` feature — parses `lua ... end` block, captures
the body as a raw string.
**Compiler:** Registered under `"lua"` head — emits `(lua-eval <body-string>)`.
**Runtime:** `lua-eval` calls `lib/lua/runtime.sx`'s eval entry point, returns
result as an SX value via `hs-host-to-sx`. Errors surface as HS `catch`-able
exceptions.
This enables inline Lua in HyperScript:
```
on click
lua
return document.title:upper()
end
put it into me
end
```
---
## 5. Load order
```
lib/hyperscript/parser.sx ;; defines _hs-feature-registry, hs-register-feature!
lib/hyperscript/compiler.sx ;; defines _hs-compiler-registry, hs-register-compiler!
lib/hyperscript/runtime.sx ;; defines _hs-converters, hs-register-converter!
lib/hyperscript/plugins/worker.sx
lib/hyperscript/plugins/prolog.sx
lib/hyperscript/plugins/lua.sx
```
The test runner (`tests/hs-run-filtered.js`) loads plugins after core. The
browser WASM bundle includes all three by default (plugins are small; no
reason to lazy-load them).
---
## 6. Migration checklist
The work below is ordered to keep main green at every commit. Each step is
independently committable.
### Step 1 — Registries (infrastructure, no behaviour change)
1. Add `_hs-feature-registry` + `hs-register-feature!` to `parser.sx`.
Thread the registry check into `parse-feat`. No entries yet → behaviour
unchanged.
2. Add `_hs-compiler-registry` + `hs-register-compiler!` to `compiler.sx`.
Thread into `hs-to-sx`. No entries yet → behaviour unchanged.
3. Add `_hs-converters` + `hs-register-converter!` to `runtime.sx`. Thread
into `hs-coerce`. No entries yet → behaviour unchanged.
4. `sx_validate` all three files. Run full HS suite — expect zero regressions.
5. Commit: `HS: plugin registry infrastructure (parser + compiler + converter)`.
### Step 2 — Worker stub migration
6. Create `lib/hyperscript/plugins/worker.sx`. Register the worker stub error.
7. Remove the inline `((= val "worker") ...)` branch from `parse-feat` in
`parser.sx`.
8. Update the test runner to load `worker.sx` after core.
9. Run `HS_SUITE=hs-upstream-worker` — expect 1/1. Run full suite — expect no
regressions.
10. Commit: `HS: migrate E39 worker stub to plugin registry`.
### Step 3 — Prolog plugin
11. Create `lib/hyperscript/plugins/prolog.sx`. Wire to `lib/prolog/runtime.sx`.
12. Remove `hs-prolog-hook`, `hs-set-prolog-hook!`, `prolog` from `runtime.sx`
nodes 140–142.
13. Update test runner to load `prolog.sx`.
14. Validate and run full suite.
15. Commit: `HS: prolog plugin replaces ad-hoc hook`.
### Step 4 — `as` converter registry (bridges Bucket-F Group 10)
16. Confirm `hs-register-converter!` satisfies the Group 10 test
`can accept custom conversions`. If yes, this step may be pulled into
Bucket-F Group 10 instead (no duplication — just move step 3 of §6 there).
17. Commit: `HS: as-converter registry wired into hs-coerce`.
### Step 5 — Lua plugin
18. Create `lib/hyperscript/plugins/lua.sx`.
19. Add `lua-eval` to `runtime.sx` or directly in the plugin file.
20. Parser: `parse-lua-feat` consuming `lua … end`.
21. Compiler: registered `"lua"` head.
22. Write 3–5 tests in `spec/tests/test-hyperscript-lua.sx`:
- Lua returns a string → HS uses it.
- Lua error → HS catch.
- Lua reads a passed argument.
23. Commit: `HS: Lua plugin — inline lua...end blocks`.
### Step 6 — Worker full runtime plugin
24. Extend `worker.sx`: implement `parse-worker-feat`, compiler entry,
`hs-worker-define!`, `hs-method-call` worker branch.
25. Extend test runner: `Worker` constructor + synchronous message loop.
26. Un-skip the 7 sibling worker tests from upstream.
27. Target: 7/7 worker suite.
28. Commit: `HS: Worker plugin full runtime (+7 tests)`.
---
## 7. Risks
- **`parse-feat` closure scope** — `hs-register-feature!` stores parse-fns
that need access to parser-internal helpers (`adv!`, `tp-val`, etc.). These
are only in scope inside `hs-parse`'s big `let`. Two options:
(a) the registry stores fns that receive a parser-context dict as arg, or
(b) the registry is checked *inside* `parse-feat` where helpers are in scope
and fns are zero-arg closures captured at registration time.
Option (b) is simpler but requires plugins to be loaded while the parser
`let` is being evaluated — i.e., plugins must be defined *inside* the parser
file or the context dict must be exposed. **Recommended:** expose a
`_hs-parser-ctx` dict at the module level that parse-fns receive as their
sole argument. This makes the API explicit and plugins independent files.
- **Worker Blob URL in WASM** — `URL.createObjectURL` is available in browsers
but not in the OCaml WASM host. Worker full runtime is browser-only; flag it
with a capability check and graceful fallback.
- **Lua/Prolog mutual recursion** — a Lua block calling back into HS calling
back into Lua is theoretically possible via the IO suspension machinery.
Don't try to support it initially; raise a clear error if detected.
- **Plugin load-order sensitivity** — `hs-register-feature!` must be called
before any source is parsed. If a plugin is loaded lazily (future), a
`worker MyWorker` in the page would hit the stub before the full plugin
registers. Acceptable for now; document that plugins must be loaded at boot.
- **`runtime.sx` cleanup for prolog** — nodes 140–142 are referenced nowhere
else in the codebase (grep confirms). Safe to delete once the plugin is live.
---
## 8. Non-goals
- Runtime `use(ext)` API (JS-style dynamic plugin install) — future.
- Plugin namespacing / versioning — future.
- Any conformance tests other than the 7 worker tests in step 6.
- Changing how the WASM bundle is built or split.

257
plans/designs/sx-adt.md Normal file
View File

@@ -0,0 +1,257 @@
# SX Algebraic Data Types — Design
## Motivation
Every language implementation currently uses `{:tag "..." :field ...}` tagged dicts to
simulate sum types. This is verbose, error-prone (typos in tag strings go undetected), and
produces no exhaustiveness warnings. Native ADTs eliminate the pattern everywhere.
Examples of current workarounds:
- Haskell `Maybe a` → `{:tag "Just" :value x}` / `{:tag "Nothing"}`
- Prolog terms → `{:tag "functor" :name "foo" :args (list x y)}`
- Lua result type → `{:tag "ok" :value v}` / `{:tag "err" :msg s}`
- Common Lisp `cons` pairs → `{:tag "cons" :car a :cdr b}`
---
## Syntax
### `define-type`
```lisp
(define-type Name
(Ctor1 field1 field2 ...)
(Ctor2 field1 ...)
...)
```
Creates:
- Constructor functions: `Ctor1`, `Ctor2`, … (callable like normal functions)
- Type predicate: `Name?` — returns true for any value of type `Name`
- Constructor predicates: `Ctor1?`, `Ctor2?`, … (optional, auto-generated)
- Field accessors: `Ctor1-field1`, `Ctor1-field2`, … (optional, auto-generated)
Examples:
```lisp
(define-type Maybe
(Just value)
(Nothing))
(define-type Result
(Ok value)
(Err message))
(define-type Tree
(Leaf)
(Node left value right))
(define-type List-of
(Nil-of)
(Cons-of head tail))
```
Constructors with no fields are zero-argument constructors (singletons by value):
```lisp
(Nothing) ; => #<Nothing>
(Leaf) ; => #<Leaf>
```
### `match`
```lisp
(match expr
((Ctor1 a b) body)
((Ctor2 x) body)
((Ctor3) body)
(else body))
```
- Clauses are tried in order; first match wins.
- `else` clause is optional but suppresses exhaustiveness warnings.
- Pattern variables (`a`, `b`, `x`) are bound in the body scope.
- Wildcard `_` discards the matched value.
- Literal patterns: `42`, `"str"`, `true`, `nil` — match by value equality.
- Nested patterns: `((Node left (Leaf) right) body)` — nested constructor patterns.
Examples:
```lisp
(match result
((Ok v) (str "got: " v))
((Err m) (str "error: " m)))
(match tree
((Leaf) 0)
((Node l v r) (+ 1 (tree-depth l) (tree-depth r))))
```
---
## CEK Dispatch
### Runtime representation
ADT values are OCaml records (not dicts) — opaque, non-inspectable via `get`:
```ocaml
type adt_value = {
av_type : string; (* type name, e.g. "Maybe" *)
av_ctor : string; (* constructor name, e.g. "Just" *)
av_fields: value array; (* positional fields *)
}
```
In JS: `{ _adt: true, _type: "Maybe", _ctor: "Just", _fields: [v] }`.
`typeOf` returns the ADT type name (e.g. `"Maybe"`).
### `define-type` — special form
`stepSfDefineType(args, env, kont)`:
1. Parse `Name` and list of `(CtorN field...)` clauses.
2. For each constructor `CtorK` with fields `[f1, f2, …]`:
- Register `CtorK` as a `NativeFn` that takes `|fields|` args and returns an `AdtValue`.
- Register `CtorK?` as a predicate (`AdtValue` with matching ctor name → `true`).
- Register `CtorK-fN` as field accessor (returns `av_fields[N]`).
3. Register `Name?` as a predicate (`AdtValue` with matching type name → `true`).
4. All bindings go into the current environment via `env-bind!`.
5. Returns `Nil`.
This is an environment mutation — no new frame needed. Evaluates in one step.
### `match` — special form
`stepSfMatch(args, env, kont)`:
1. Push `MatchFrame` with `clauses` and `env` onto kont.
2. Return state evaluating the scrutinee `expr`.
3. `MatchFrame` continue: receive scrutinee value, walk clauses:
- For each `((CtorN vars...) body)`:
- If scrutinee is an `AdtValue` with `av_ctor = "CtorN"` and `av_fields.length = |vars|`:
- Bind `vars[i]` → `av_fields[i]` in fresh child env.
- Return state evaluating `body` in that env.
- `(else body)` — always matches, body evaluated in current env.
- Literal `42`/`"str"` patterns: match by value equality.
- Wildcard `_`: always matches, binds nothing.
4. If no clause matched and no `else`: raise `"match: no clause matched <value>"`.
Frame type: `"match"` — stores `cf_remaining` (clauses), `cf_env` (enclosing env).
---
## Interaction with `cond` / `case`
`match` is the primary dispatch form for ADTs. `cond` / `case` remain unchanged:
- `cond` tests arbitrary boolean expressions — still useful for non-ADT dispatch.
- `case` matches on equality to literal values — unchanged.
- `match` is the new form: structural pattern matching on ADT constructors.
They are orthogonal. A `match` clause can contain a `cond`; a `cond` clause can contain a `match`.
---
## Exhaustiveness checking
Emit a **warning** (not an error) when:
- A `match` has no `else` clause, AND
- Not all constructors of the scrutinee's type are covered.
Detection: when `define-type` runs, it registers the constructor set in a global table
`_adt_registry: type_name → [ctor_names]`. At `match` compile/evaluation time:
- If the scrutinee's type is in `_adt_registry` and not all ctors appear as patterns:
- `console.warn("[sx] match: non-exhaustive — missing: Ctor3, Ctor4 for type Maybe")`
- Execution continues (warning, not error).
This is best-effort: the scrutinee type is only known at runtime. The warning fires on
first non-exhaustive match evaluation, not at definition time.
---
## Recursive types
Recursive types work because constructors are registered as functions, and function bodies
are evaluated lazily:
```lisp
(define-type Tree
(Leaf)
(Node left value right))
; Recursive function over a recursive type:
(define (depth tree)
(match tree
((Leaf) 0)
((Node l v r) (+ 1 (max (depth l) (depth r))))))
```
No special treatment needed — the type definition doesn't need to know about recursion.
The constructor `Node` accepts any values, including other `Node` or `Leaf` values.
---
## Pattern variables
In `match` clauses, identifiers in constructor position that are NOT constructor names are
treated as pattern variables (bound to matched field values):
```lisp
(match x
((Just v) v) ; v bound to the wrapped value
((Nothing) nil))
(match pair
((Cons-of h t) (list h t))) ; h, t bound to head and tail
```
**Wildcard**: `_` is always a wildcard — matches anything, binds nothing.
```lisp
(match x
((Just _) "has value")
((Nothing) "empty"))
```
**Nested patterns**:
```lisp
(match tree
((Node (Leaf) v (Leaf)) (str "leaf node: " v))
((Node l v r) (str "inner node: " v)))
```
Nested patterns are matched recursively: the inner `(Leaf)` pattern checks that the
`left` field is itself a `Leaf` ADT value.
---
## Implementation Plan
### Phase 6a — `define-type` + basic `match` (no nested patterns, no exhaustiveness)
1. OCaml: add `AdtValue of adt_value` to `sx_types.ml`.
2. Evaluator: add `step-sf-define-type` — parse clauses, register ctor fns + predicates + accessors.
3. Evaluator: add `step-sf-match` + `MatchFrame` — linear scan of clauses, flat patterns only.
4. JS: same (AdtValue as plain object with `_adt`/`_type`/`_ctor`/`_fields` props).
### Phase 6b — nested patterns (separate fire)
Recursive `matchPattern(pattern, value, env)` helper that:
- Returns `{matched: bool, bindings: map}`
- Recursively matches sub-patterns against ADT fields.
### Phase 6c — exhaustiveness warnings (separate fire)
`_adt_registry` global + warning emission on first non-exhaustive match.
---
## Open questions (deferred to review)
1. **Accessor auto-generation**: should `Ctor-field` accessors be generated always, or only on demand? Risk: name collisions if two types have constructors with same field names.
2. **Singleton constructors**: `(Nothing)` — zero-arg ctor — should these be interned (same object every call) or fresh each time? Interning enables `eq?` checks but requires a global table.
3. **Printing/inspect**: `inspect` on an AdtValue should show `(Just 42)` not `#<adt:Just>`. Implement in `inspect` function or via `display`/`write` (Phase 17 ports).
4. **Pattern-matching on non-ADT values**: should `match` handle list patterns `(a . b)` and literal patterns in clause heads? Deferred — add only if needed by a language implementation.

173
plans/elixir-on-sx.md Normal file
View File

@@ -0,0 +1,173 @@
# Elixir-on-SX: Elixir on the CEK/VM
Compile Elixir source to SX AST; the existing CEK evaluator runs it. The natural companion
to `lib/erlang/` — Elixir compiles to the BEAM and most of its runtime semantics are
Erlang's. The interesting parts are Elixir-specific: the macro system (`quote`/`unquote`),
the pipe operator `|>`, `with` expressions, `defmodule`/`def`/`defp`, protocol dispatch,
and the `Stream` lazy evaluation library.
End-state goal: **core Elixir programs running**, including modules, pattern matching, the
pipe operator, macros (`quote`/`unquote`/`defmacro`), protocols, and actor-style processes
reusing the Erlang runtime foundation.
## Ground rules
- **Scope:** only touch `lib/elixir/**` and `plans/elixir-on-sx.md`. Do **not** edit
`spec/`, `hosts/`, `shared/`, or other `lib/<lang>/`. Reuse `lib/erlang/` runtime
functions where possible — import them, don't duplicate.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** Elixir source → Elixir AST → SX AST. Reuse Erlang runtime for process/
message/pattern primitives; add Elixir-specific surface in `lib/elixir/`.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
## Architecture sketch
```
Elixir source text
│
▼
lib/elixir/tokenizer.sx — atoms (:atom), strings (""), charlists (''), sigils (~r, ~s etc.),
│ operators (|>, <>, ++, :::, etc.), do/end blocks
▼
lib/elixir/parser.sx — Elixir AST: defmodule, def/defp/defmacro, @attribute,
│ pattern matching, |> pipe, with, for comprehension, quote/unquote,
│ case/cond/if/unless, fn, receive, try/rescue/catch/after
▼
lib/elixir/transpile.sx — Elixir AST → SX AST
│
├── lib/erlang/runtime.sx (reused: processes, message passing, pattern match)
└── lib/elixir/runtime.sx — Elixir-specific: Kernel, String, Enum, Stream, Map,
List, Tuple, IO, protocol dispatch, macro expansion
```
Key semantic mappings (differences from Erlang):
- `defmodule M do ... end` → SX `define-library` + module dict `{:module "M" :fns {...}}`
- `def f(args) do body end` → named function in module dict, with pattern-match dispatch
- `|>` pipe → left-to-right function composition; `a |> f(b)` = `f(a, b)`
- `with x <- expr, y <- expr2 do body else patterns end` → chained pattern match with early exit
- `for x <- list, filter, do: expr` → list comprehension (SX `map`/`filter`)
- `quote do expr end` → returns AST as SX list (homoiconic — Elixir AST IS SX-like)
- `unquote(expr)` → evaluate expr and splice into surrounding `quote`
- `defmacro` → macro in module; expanded at compile time by calling the SX macro
- Protocol → dict of implementations keyed by type name; `defprotocol` defines interface,
`defimpl` registers an implementation
- `Stream` → lazy sequences using SX promises/coroutines (Phase 9/4 of primitives)
- `Agent`/`GenServer` → SX coroutine + message queue (similar to Erlang process model)
## Roadmap
### Phase 1 — tokenizer + parser
- [ ] Tokenizer: atoms (`:atom`, `:"atom with spaces"`), strings (`""`), charlists (`''`),
numbers (int, float, hex `0xFF`, octal `0o77`, binary `0b11`), booleans (`true`/`false`/`nil`),
operators (`|>`, `<>`, `++`, `--`, `:::`, `&&`, `||`, `!`, `..`, `<-`, `=~`),
sigils (`~r/regex/`, `~s"string"`, `~w(word list)`), do/end blocks, keywords as args
`f(key: val)`, `@module_attribute`
- [ ] Parser:
- Module: `defmodule Name do ... end` → module AST with body
- Functions: `def f(pat) do body end`, `def f(pat) when guard do body end`,
multi-clause `def f(a) do ...; def f(b) do ...` → clause list
- `defp` (private), `defmacro`, `defmacrop`
- `@doc`, `@moduledoc`, `@spec`, `@type`, `@behaviour` module attributes
- `case expr do patterns end`, `cond do clauses end`, `if`/`unless`
- `with x <- e, y <- e2, do: body, else: [pattern -> body]`
- `for x <- list, filter, into: acc, do: expr` comprehension
- `fn pat -> body end` anonymous function; capture `&Module.fun/arity`, `&(&1 + 1)`
- `receive do patterns after timeout -> body end`
- `try do body rescue e -> ... catch type, val -> ... after ... end`
- `quote do ... end`, `unquote(expr)`, `unquote_splicing(list)`
- `|>` pipe chain: `a |> f |> g(b)` → `g(f(a), b)`
- [ ] Tests in `lib/elixir/tests/parse.sx`
### Phase 2 — transpile: basic Elixir (no macros, no processes)
- [ ] `ex-eval-ast` entry
- [ ] Arithmetic, string `<>`, list `++`/`--`, comparison, boolean (`and`/`or`/`not`)
- [ ] Pattern matching in `=`, function heads, `case` — reuse Erlang pattern engine
- [ ] `def`/`defp` → SX `define` with clause dispatch (like Erlang function clauses)
- [ ] Module as a dict of named functions; `ModuleName.function(args)` dispatch
- [ ] `|>` pipe: desugar `a |> f(b, c)` → `f(a, b, c)` at transpile time
- [ ] `with` expression: chain of `<-` bindings, short-circuit on mismatch to `else`
- [ ] `for` comprehension: `for x <- list, filter do body end` → `map`/`filter`
- [ ] `fn` anonymous functions, `&` capture forms
- [ ] `if`/`unless`/`cond`/`case`
- [ ] String interpolation: `"Hello #{name}"` → string concat
- [ ] Keyword lists `[key: val]` → SX list of `{:key val}` dicts; maps `%{key: val}` → SX dict
- [ ] Tuples `{a, b, c}` → SX list (or vector); `elem/2`, `put_elem/3`
- [ ] 40+ eval tests in `lib/elixir/tests/eval.sx`
### Phase 3 — macro system
- [ ] `quote do expr end` → returns Elixir AST as SX list structure
(Elixir AST is 3-tuples `{name, meta, args}` — map to SX `(list name meta args)`)
- [ ] `unquote(expr)` → evaluate and splice into surrounding `quote`
- [ ] `unquote_splicing(list)` → splice list into surrounding `quote`
- [ ] `defmacro` → define a macro in the module; macro receives AST args, returns AST
- [ ] Macro expansion: expand macros before transpiling (two-pass: collect defs, then expand)
- [ ] `use Module` → calls `Module.__using__/1` macro, injects code into caller
- [ ] `import Module` → bring functions into scope without prefix
- [ ] `alias Module, as: M` → short name for module
- [ ] Tests: `defmacro unless`, `defmacro my_if`, `use` injection, `__MODULE__`, `__DIR__`
### Phase 4 — protocols
- [ ] `defprotocol P do @spec f(t) :: result end` → defines protocol dict + dispatch fn
- [ ] `defimpl P, for: Type do def f(t) do ... end end` → register implementation
- [ ] Protocol dispatch: `P.f(value)` → look up type of value, find implementation, call it
- [ ] Built-in protocols: `Enumerable`, `Collectable`, `String.Chars`, `Inspect`
- [ ] `Enumerable` implementation for lists, maps, ranges — enables `Enum.*` on custom types
- [ ] `derive` — automatic protocol implementation for simple structs
- [ ] Tests: custom type implementing `Enumerable`, `String.Chars`, protocol fallback
### Phase 5 — structs + behaviours
- [ ] `defstruct [:field1, field2: default]` → defines `%ModuleName{}` struct type
Structs are maps with `__struct__: ModuleName` key + defined fields
- [ ] Struct pattern matching: `%User{name: n} = user`
- [ ] `@behaviour Module` → declares behaviour callbacks; compile-time check
- [ ] `@impl true` / `@impl BehaviourName` → marks function as behaviour implementation
- [ ] Built-in behaviours: `GenServer`, `Supervisor`, `Agent`, `Task`
- [ ] Tests: struct creation, update syntax `%{struct | field: val}`, behaviour callbacks
### Phase 6 — processes + OTP patterns (reuses Erlang runtime)
- [ ] `spawn(fn -> ... end)` / `spawn(M, f, args)` → SX coroutine on scheduler
Reuse `lib/erlang/` process + message queue infrastructure
- [ ] `send(pid, msg)` / `receive do patterns end` — already in Erlang runtime
- [ ] `GenServer` behaviour: `start_link`, `call`, `cast`, `handle_call`, `handle_cast`,
`handle_info`, `init` — implement as SX macros expanding to process + message loop
- [ ] `Agent` — simple state wrapper over GenServer; `Agent.start_link`, `get`, `update`
- [ ] `Task` — async computation; `Task.async`, `Task.await`
- [ ] `Supervisor` — child spec, restart strategy (`one_for_one`, `one_for_all`)
- [ ] Tests: counter GenServer, bank account Agent, parallel Task, supervised worker
### Phase 7 — standard library
- [ ] `Enum.*` — `map`, `filter`, `reduce`, `each`, `into`, `flat_map`, `zip`, `sort`,
`sort_by`, `min_by`, `max_by`, `group_by`, `frequencies`, `count`, `any?`, `all?`,
`find`, `take`, `drop`, `take_while`, `drop_while`, `chunk_every`, `chunk_by`,
`flat_map_reduce`, `scan`, `uniq`, `uniq_by`, `member?`, `empty?`, `sum`, `product`
- [ ] `Stream.*` — lazy versions of Enum; `Stream.map`, `Stream.filter`, `Stream.take`,
`Stream.cycle`, `Stream.iterate`, `Stream.unfold`, `Stream.resource`
Uses SX promises (Phase 9) for laziness
- [ ] `String.*` — `length`, `upcase`, `downcase`, `trim`, `split`, `replace`, `contains?`,
`starts_with?`, `ends_with?`, `slice`, `at`, `graphemes`, `codepoints`, `to_integer`,
`to_float`, `pad_leading`, `pad_trailing`, `duplicate`, `match?`
- [ ] `Map.*` — `new`, `get`, `put`, `delete`, `update`, `merge`, `keys`, `values`,
`to_list`, `from_struct`, `has_key?`, `filter`, `map`, `reject`, `take`, `drop`
- [ ] `List.*` — `first`, `last`, `flatten`, `zip`, `unzip`, `keystore`, `keyfind`,
`wrap`, `duplicate`, `improper?`, `delete`, `insert_at`, `replace_at`
- [ ] `Tuple.*` — `to_list`, `from_list`, `append`, `insert_at`, `delete_at`
- [ ] `Integer.*` / `Float.*` — `parse`, `to_string`, `digits`, `pow`, `is_odd?`, `is_even?`
- [ ] `IO.*` — `puts`, `gets`, `inspect`, `write`, `read` → SX IO perform
- [ ] `Kernel.*` — built-in functions: `is_integer?`, `is_binary?`, `length`, `hd`, `tl`,
`elem`, `put_elem`, `apply`, `raise`, `exit`, `inspect`
- [ ] `inspect/1` / `IO.inspect/2` — debug printing using `Inspect` protocol
### Phase 8 — conformance target
- [ ] Vendor or hand-build 100+ Elixir program tests in `lib/elixir/tests/programs/`
- [ ] Drive scoreboard
## Blockers
_(none yet)_
## Progress log
_Newest first._
_(awaiting phase 1)_

131
plans/elm-on-sx.md Normal file
View File

@@ -0,0 +1,131 @@
# Elm-on-SX: Elm 0.19 on the CEK/VM
Compile Elm source to SX AST; the existing CEK evaluator runs it. The unique angle: SX's
reactive island system (`defisland`, signals, `provide`/`context`) is a natural host for
The Elm Architecture — Model/Update/View maps almost directly onto SX's reactive runtime.
This is the only language in the set that targets SX's browser-side reactivity rather than
the server-side evaluator.
End-state goal: **core Elm programs running in the browser via SX islands**, with The Elm
Architecture wired to SX signals. Not a full Elm compiler — no exhaustiveness checking, no
module system, no type inference — but a faithful runtime that can run Elm programs written
in idiomatic style.
## Ground rules
- **Scope:** only touch `lib/elm/**` and `plans/elm-on-sx.md`. Do **not** edit `spec/`,
`hosts/`, `shared/`, or other `lib/<lang>/`.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** Elm source → Elm AST → SX AST. No standalone Elm evaluator.
- **Type system:** defer. Focus on runtime semantics. Type errors surface at eval time.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
## Architecture sketch
```
Elm source text
│
▼
lib/elm/tokenizer.sx — numbers, strings, idents, operators, indentation-sensitive lexer
│
▼
lib/elm/parser.sx — Elm AST: module, import, type alias, type, let, case, lambda,
│ if, list/tuple/record literals, pipe operator |>
▼
lib/elm/transpile.sx — Elm AST → SX AST
│
▼
lib/elm/runtime.sx — TEA runtime: Program, sandbox, element; Cmd/Sub wrappers;
│ Html.* shims; Browser.* shims
▼
SX island / reactive runtime (browser)
```
Key semantic mappings:
- `Model` → SX signal (`make-signal`)
- `update : Msg -> Model -> Model` → SX signal updater (called on each message)
- `view : Model -> Html Msg` → SX component (re-renders on model signal change)
- `Cmd` → SX `perform` IO request
- `Sub` → SX event listener registered via `dom-listen`
- `Maybe a` → `nil` (Nothing) or value (Just a) — uses ADTs from Phase 6 of primitives
- `Result a b` → ADT `(Ok val)` / `(Err err)`
## Roadmap
### Phase 1 — tokenizer + parser
- [ ] Tokenizer: keywords (`module`, `import`, `type`, `alias`, `let`, `in`, `if`, `then`,
`else`, `case`, `of`, `port`), indentation tokens (indent/dedent/newline), string
literals, number literals, operators (`|>`, `>>`, `<<`, `<|`, `++`, `::`), type vars
- [ ] Parser: module declaration, imports, type aliases, union types, function definitions
with pattern matching, `let`/`in`, `case`/`of`, `if`/`then`/`else`, lambda `\x -> e`,
list literals `[1,2,3]`, tuple literals `(a,b)`, record literals `{x=1, y=2}`,
record update `{ r | x = 1 }`, pipe operator `|>`
- [ ] Skip for phase 1: ports, subscriptions, effects manager, type annotations
- [ ] Tests in `lib/elm/tests/parse.sx`
### Phase 2 — transpile: expressions + pattern matching
- [ ] `elm-eval-ast` entry
- [ ] Arithmetic, string `++`, comparison, boolean ops
- [ ] Lambda → SX `fn`; function application
- [ ] `let`/`in` → SX `let`
- [ ] `if`/`then`/`else` → SX `if`
- [ ] `case`/`of` with constructor, literal, tuple, list, wildcard patterns → SX `cond`
using ADT match (Phase 6 primitives)
- [ ] List ops: `List.map`, `List.filter`, `List.foldl`, `List.foldr`
- [ ] `Maybe` and `Result` as ADTs
- [ ] 30+ eval tests in `lib/elm/tests/eval.sx`
### Phase 3 — The Elm Architecture runtime
- [ ] `Browser.sandbox` — pure TEA loop (no Cmds, no Subs)
`{ init : model, update : msg -> model -> model, view : model -> Html msg }`
Wires to: SX signal for model, SX component for view, message dispatch on user events
- [ ] `Html.*` shims: `div`, `p`, `button`, `input`, `text`, `h1`–`h6`, `ul`, `li`, `a`,
`span`, `img` — emit SX component calls
- [ ] `Html.Attributes.*`: `class`, `id`, `href`, `src`, `type_`, `placeholder`, `value`
- [ ] `Html.Events.*`: `onClick`, `onInput`, `onSubmit`, `onBlur`, `onFocus`
- [ ] `Browser.element` — adds `init` returning `(model, Cmd msg)`, `subscriptions`
- [ ] Demo: counter app (`init=0`, `update Increment m = m+1`, `view` shows count + button)
### Phase 4 — Cmds and Subs
- [ ] `Cmd` — mapped to SX `perform` IO requests. `Cmd.none`, `Cmd.batch`
- [ ] `Http.get`/`Http.post` → SX fetch IO
- [ ] `Sub` — mapped to SX `dom-listen`. `Sub.none`, `Sub.batch`
- [ ] `Browser.Events.onClick`, `onKeyPress`, `onAnimationFrame`
- [ ] `Time.every` — periodic subscription via SX timer IO
- [ ] `Task.perform`/`Task.attempt` — single-shot async operations
### Phase 5 — standard library
- [ ] `String.*` — `length`, `append`, `concat`, `split`, `join`, `trim`, `toUpper`, `toLower`,
`contains`, `startsWith`, `endsWith`, `replace`, `toInt`, `toFloat`, `fromInt`, `fromFloat`
- [ ] `List.*` — `map`, `filter`, `foldl`, `foldr`, `head`, `tail`, `isEmpty`, `length`,
`reverse`, `append`, `concat`, `member`, `sort`, `sortBy`, `indexedMap`, `range`
- [ ] `Dict.*` — SX immutable dict; `fromList`, `toList`, `get`, `insert`, `remove`, `update`,
`member`, `keys`, `values`, `map`, `filter`, `foldl`
- [ ] `Set.*` — SX set primitive (Phase 18); `fromList`, `toList`, `member`, `insert`,
`remove`, `union`, `intersect`, `diff`
- [ ] `Maybe.*` — `withDefault`, `map`, `andThen`, `map2`
- [ ] `Result.*` — `withDefault`, `map`, `andThen`, `mapError`, `toMaybe`
- [ ] `Tuple.*` — `first`, `second`, `pair`, `mapFirst`, `mapSecond`
- [ ] `Basics.*` — `identity`, `always`, `not`, `xor`, `modBy`, `remainderBy`, `clamp`,
`min`, `max`, `abs`, `sqrt`, `logBase`, `e`, `pi`, `floor`, `ceiling`, `round`,
`truncate`, `toFloat`, `isNaN`, `isInfinite`, `compare`
- [ ] `Random.*` — seed-based PRNG via SX IO perform
### Phase 6 — full browser integration
- [ ] `Browser.application` — URL routing, `onUrlChange`, `onUrlRequest`
- [ ] `Browser.Navigation.*` — `pushUrl`, `replaceUrl`, `back`, `forward`
- [ ] `Url.Parser.*` — path segment parsing
- [ ] `Json.Decode.*` — JSON decoder combinators
- [ ] `Json.Encode.*` — JSON encoder
- [ ] `Ports` — `port` keyword; JS interop via SX `host-call`
## Blockers
_(none yet)_
## Progress log
_Newest first._
_(awaiting phase 1)_

87
plans/erlang-on-sx.md
View File

@@ -53,52 +53,79 @@ Core mapping:
- [x] Tokenizer: atoms (bare + single-quoted), variables (Uppercase/`_`-prefixed), numbers (int, float, `16#HEX`), strings `"..."`, chars `$c`, punct `( ) { } [ ] , ; . : :: ->` — **62/62 tests**
- [x] Parser: module declarations, `-module`/`-export`/`-import` attributes, function clauses with head patterns + guards + body — **52/52 tests**
- [x] Expressions: literals, vars, calls, tuples `{...}`, lists `[...|...]`, `if`, `case`, `receive`, `fun`, `try/catch`, operators, precedence
- [ ] Binaries `<<...>>` — not yet parsed (deferred to Phase 6)
- [x] Binaries `<<...>>` — landed in Phase 6 (parser + eval + pattern matching)
- [x] Unit tests in `lib/erlang/tests/parse.sx`
### Phase 2 — sequential eval + pattern matching + BIFs
- [ ] `erlang-eval-ast`: evaluate sequential expressions
- [ ] Pattern matching (atoms, numbers, vars, tuples, lists, `[H|T]`, underscore, bound-var re-match)
- [ ] Guards: `is_integer`, `is_atom`, `is_list`, `is_tuple`, comparisons, arithmetic
- [ ] BIFs: `length/1`, `hd/1`, `tl/1`, `element/2`, `tuple_size/1`, `atom_to_list/1`, `list_to_atom/1`, `lists:map/2`, `lists:foldl/3`, `lists:reverse/1`, `io:format/1-2`
- [ ] 30+ tests in `lib/erlang/tests/eval.sx`
- [x] `erlang-eval-ast`: evaluate sequential expressions — **54/54 tests**
- [x] Pattern matching (atoms, numbers, vars, tuples, lists, `[H|T]`, underscore, bound-var re-match) — **21 new eval tests**; `case ... of ... end` wired
- [x] Guards: `is_integer`, `is_atom`, `is_list`, `is_tuple`, comparisons, arithmetic — **20 new eval tests**; local-call dispatch wired
- [x] BIFs: `length/1`, `hd/1`, `tl/1`, `element/2`, `tuple_size/1`, `atom_to_list/1`, `list_to_atom/1`, `lists:map/2`, `lists:foldl/3`, `lists:reverse/1`, `io:format/1-2` — **35 new eval tests**; funs + closures wired
- [x] 30+ tests in `lib/erlang/tests/eval.sx` — **130 tests green**
### Phase 3 — processes + mailboxes + receive (THE SHOWCASE)
- [ ] Scheduler in `runtime.sx`: runnable queue, pid counter, per-process state record
- [ ] `spawn/1`, `spawn/3`, `self/0`
- [ ] `!` (send), `receive ... end` with selective pattern matching
- [ ] `receive ... after Ms -> ...` timeout clause (use SX timer primitive)
- [ ] `exit/1`, basic process termination
- [ ] Classic programs in `lib/erlang/tests/programs/`:
- [ ] `ring.erl` — N processes in a ring, pass a token around M times
- [ ] `ping_pong.erl` — two processes exchanging messages
- [ ] `bank.erl` — account server (deposit/withdraw/balance)
- [ ] `echo.erl` — minimal server
- [ ] `fib_server.erl` — compute fib on request
- [ ] `lib/erlang/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md`
- [ ] Target: 5/5 classic programs + 1M-process ring benchmark runs
- [x] Scheduler in `runtime.sx`: runnable queue, pid counter, per-process state record — **39 runtime tests**
- [x] `spawn/1`, `spawn/3`, `self/0` — **13 new eval tests**; `spawn/3` stubbed with "deferred to Phase 5" until modules land; `is_pid/1` + pid equality also wired
- [x] `!` (send), `receive ... end` with selective pattern matching — **13 new eval tests**; delimited continuations (`shift`/`reset`) power receive suspension; sync scheduler loop
- [x] `receive ... after Ms -> ...` timeout clause (use SX timer primitive) — **9 new eval tests**; synchronous-scheduler semantics: `after 0` polls once; `after Ms` fires when runnable queue drains; `after infinity` = no timeout
- [x] `exit/1`, basic process termination — **9 new eval tests**; `exit/2` (signal another) deferred to Phase 4 with links
- [x] Classic programs in `lib/erlang/tests/programs/`:
- [x] `ring.erl` — N processes in a ring, pass a token around M times — **4 ring tests**; suspension machinery rewritten from `shift`/`reset` to `call/cc` + `raise`/`guard`
- [x] `ping_pong.erl` — two processes exchanging messages — **4 ping-pong tests**
- [x] `bank.erl` — account server (deposit/withdraw/balance) — **8 bank tests**
- [x] `echo.erl` — minimal server — **7 echo tests**
- [x] `fib_server.erl` — compute fib on request — **8 fib tests**
- [x] `lib/erlang/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md` — **358/358 across 9 suites**
- [x] Target: 5/5 classic programs + 1M-process ring benchmark runs — **5/5 classic programs green; ring benchmark runs correctly at every measured size up to N=1000 (33s, ~34 hops/s); 1M target NOT met in current synchronous-scheduler architecture (would take ~9h at observed throughput)**. See `lib/erlang/bench_ring.sh` and `lib/erlang/bench_ring_results.md`.
### Phase 4 — links, monitors, exit signals
- [ ] `link/1`, `unlink/1`, `monitor/2`, `demonitor/1`
- [ ] Exit-signal propagation; trap_exit flag
- [ ] `try/catch/of/end`
- [x] `link/1`, `unlink/1`, `monitor/2`, `demonitor/1` — **17 new eval tests**; `make_ref/0`, `is_reference/1`, refs in `=:=`/format wired
- [x] Exit-signal propagation; trap_exit flag — **11 new eval tests**; `process_flag/2`, monitor `{'DOWN', ...}`, `{'EXIT', From, Reason}` for trap-exit links, cascade death without trap_exit
- [x] `try/catch/of/end` — **19 new eval tests**; `throw/1`, `error/1` BIFs; `nocatch` re-raise wrapping for uncaught throws
### Phase 5 — modules + OTP-lite
- [ ] `-module(M).` loading, `M:F(...)` calls across modules
- [ ] `gen_server` behaviour (the big OTP win)
- [ ] `supervisor` (simple one-for-one)
- [ ] Registered processes: `register/2`, `whereis/1`
- [x] `-module(M).` loading, `M:F(...)` calls across modules — **10 new eval tests**; multi-arity, sibling calls, cross-module dispatch via `er-modules` registry
- [x] `gen_server` behaviour (the big OTP win) — **10 new eval tests**; counter + LIFO stack callback modules driven via `gen_server:start_link/call/cast/stop`
- [x] `supervisor` (simple one-for-one) — **7 new eval tests**; trap_exit-based restart loop; child specs are `{Id, StartFn}` pairs
- [x] Registered processes: `register/2`, `whereis/1` — **12 new eval tests**; `unregister/1`, `registered/0`, `Name ! Msg` via registered atom; auto-unregister on death
### Phase 6 — the rest
- [ ] List comprehensions `[X*2 || X <- L]`
- [ ] Binary pattern matching `<<A:8, B:16>>`
- [ ] ETS-lite (in-memory tables via SX dicts)
- [ ] More BIFs — target 200+ test corpus green
- [x] List comprehensions `[X*2 || X <- L]` — **12 new eval tests**; generators, filters, multiple generators (cartesian), pattern-matching gens (`{ok, V} <- ...`)
- [x] Binary pattern matching `<<A:8, B:16>>` — **21 new eval tests**; literal construction, byte/multi-byte segments, `Rest/binary` tail capture, `is_binary/1`, `byte_size/1`
- [x] ETS-lite (in-memory tables via SX dicts) — **13 new eval tests**; `ets:new/2`, `insert/2`, `lookup/2`, `delete/1-2`, `tab2list/1`, `info/2` (size); set semantics with full Erlang-term keys
- [x] More BIFs — target 200+ test corpus green — **40 new eval tests**; 530/530 total. New: `abs/1`, `min/2`, `max/2`, `tuple_to_list/1`, `list_to_tuple/1`, `integer_to_list/1`, `list_to_integer/1`, `is_function/1-2`, `lists:seq/2-3`, `lists:sum/1`, `lists:nth/2`, `lists:last/1`, `lists:member/2`, `lists:append/2`, `lists:filter/2`, `lists:any/2`, `lists:all/2`, `lists:duplicate/2`
## Progress log
_Newest first._
- **2026-04-25 BIF round-out — Phase 6 complete, full plan ticked** — Added 18 standard BIFs in `lib/erlang/transpile.sx`. **erlang module:** `abs/1` (negates negative numbers), `min/2`/`max/2` (use `er-lt?` so cross-type comparisons follow Erlang term order), `tuple_to_list/1`/`list_to_tuple/1` (proper conversions), `integer_to_list/1` (returns SX string per the char-list shim), `list_to_integer/1` (uses `parse-number`, raises badarg on failure), `is_function/1` and `is_function/2` (arity-2 form scans the fun's clause patterns). **lists module:** `seq/2`/`seq/3` (right-fold builder with step), `sum/1`, `nth/2` (1-indexed, raises badarg out of range), `last/1`, `member/2`, `append/2` (alias for `++`), `filter/2`, `any/2`, `all/2`, `duplicate/2`. 40 new eval tests with positive + negative cases, plus a few that compose existing BIFs (e.g. `lists:sum(lists:seq(1, 100)) = 5050`). Total suite **530/530** — every checkbox in `plans/erlang-on-sx.md` is now ticked.
- **2026-04-25 ETS-lite green** — Scheduler state gains `:ets` (table-name → mutable list of tuples). New `er-apply-ets-bif` dispatches `ets:new/2` (registers table by atom name; rejects duplicate name with `{badarg, Name}`), `insert/2` (set semantics — replaces existing entry with the same first-element key, else appends), `lookup/2` (returns Erlang list — `[Tuple]` if found else `[]`), `delete/1` (drop table), `delete/2` (drop key; rebuilds entry list), `tab2list/1` (full list view), `info/2` with `size` only. Keys are full Erlang terms compared via `er-equal?`. 13 new eval tests: new return value, insert true, lookup hit + miss, set replace, info size after insert/delete, tab2list length, table delete, lookup-after-delete raises badarg, multi-key aggregate sum, tuple-key insert + lookup, two independent tables. Total suite 490/490.
- **2026-04-25 binary pattern matching green** — Parser additions: `<<...>>` literal/pattern in `er-parse-primary`, segment grammar `Value [: Size] [/ Spec]` (Spec defaults to `integer`, supports `binary` for tail). Critical fix: segment value uses `er-parse-primary` (not `er-parse-expr-prec`) so the trailing `:Size` doesn't get eaten by the postfix `Mod:Fun` remote-call handler. Runtime value: `{:tag "binary" :bytes (list of int 0-255)}`. Construction: integer segments emit big-endian bytes (size in bits, must be multiple of 8); binary-spec segments concatenate. Pattern matching consumes bytes from a cursor at the front, decoding integer segments big-endian, capturing `Rest/binary` tail at the end. Whole-binary length must consume exactly. New BIFs: `is_binary/1`, `byte_size/1`. Binaries participate in `er-equal?` (byte-wise) and format as `<<b1,b2,...>>`. 21 new eval tests: tag/predicate, byte_size for 8/16/32-bit segments, single + multi segment match, three 8-bit, tail rest size + content, badmatch on size mismatch, `=:=` equality, var-driven construction. Total suite 477/477.
- **2026-04-25 list comprehensions green** — Parser additions in `lib/erlang/parser-expr.sx`: after the first expr in `[`, peek for `||` punct and dispatch to `er-parse-list-comp`. Qualifiers separated by `,`, each one is `Pattern <- Source` (generator) or any expression (filter — disambiguated by absence of `<-`). AST: `{:type "lc" :head E :qualifiers [...]}` with each qualifier `{:kind "gen"/"filter" ...}`. Evaluator (`er-eval-lc` in transpile.sx): right-fold builds the result by walking qualifiers; generators iterate the source list with env snapshot/restore per element so pattern-bound vars don't leak between iterations; filters skip when falsy. Pattern-matching generators are silently skipped on no-match (e.g. `[V || {ok, V} <- ...]`). 12 new eval tests: map double, fold-sum-of-comprehension, length, filter sum, "all filtered", empty source, cartesian, pattern-match gen, nested generators with filter, squares, tuple capture. Total suite 456/456.
- **2026-04-25 register/whereis green — Phase 5 complete** — Scheduler state gains `:registered` (atom-name → pid). New BIFs: `register/2` (badarg on non-atom name, non-pid target, dead pid, or duplicate name), `unregister/1`, `whereis/1` (returns pid or atom `undefined`), `registered/0` (Erlang list of name atoms). `er-eval-send` for `Name ! Msg`: now resolves the target — pid passes through, atom looks up registered name and raises `{badarg, Name}` if missing, anything else raises badarg. Process death (in `er-sched-step!`) calls `er-unregister-pid!` to drop any registered name before `er-propagate-exit!` so monitor `{'DOWN'}` messages see the cleared registry. 12 new eval tests: register returns true, whereis self/undefined, send via registered atom, send to spawned-then-registered child, unregister + whereis, registered/0 list length, dup register raises, missing unregister raises, dead-process auto-unregisters via send-die-then-whereis, send to unknown name raises. Total suite 444/444. **Phase 5 complete — Phase 6 (list comprehensions, binary patterns, ETS) is the last phase.**
- **2026-04-25 supervisor (one-for-one) green** — `er-supervisor-source` in `lib/erlang/runtime.sx` is the canonical Erlang text of a minimal supervisor; `er-load-supervisor!` registers it. Implements `start_link(Mod, Args)` (sup process traps exits, calls `Mod:init/1` to get child-spec list, runs `start_child/1` for each which links the spawned pid back to itself), `which_children/1`, `stop/1`. Receive loop dispatches on `{'EXIT', Dead, _Reason}` (restarts only the dead child via `restart/2`, keeps siblings — proper one-for-one), `{'$sup_which', From}` (returns child list), `'$sup_stop'`. Child specs are `{Id, StartFn}` where `StartFn/0` returns the new child's pid. 7 new eval tests: `which_children` for 1- and 3-child sup, child responds to ping, killed child restarted with fresh pid, restarted child still functional, one-for-one isolation (siblings keep their pids), stop returns ok. Total suite 432/432.
- **2026-04-25 gen_server (OTP-lite) green** — `er-gen-server-source` in `lib/erlang/runtime.sx` is the canonical Erlang text of the behaviour; `er-load-gen-server!` registers it in the user-module table. Implements `start_link/2`, `call/2` (sync via `make_ref` + selective `receive {Ref, Reply}`), `cast/2` (async fire-and-forget returning `ok`), `stop/1`, and the receive loop dispatching `{'$gen_call', {From, Ref}, Req}` → `Mod:handle_call/3`, `{'$gen_cast', Msg}` → `Mod:handle_cast/2`, anything else → `Mod:handle_info/2`. handle_call reply tuples supported: `{reply, R, S}`, `{noreply, S}`, `{stop, R, Reply, S}`. handle_cast/info: `{noreply, S}`, `{stop, R, S}`. `Mod:F` and `M:F` where `M` is a runtime variable now work via new `er-resolve-call-name` (was bug: passed unevaluated AST node `:value` to remote dispatch). 10 new eval tests: counter callback module (start/call/cast/stop, repeated state mutations), LIFO stack callback module (`{push, V}` cast, pop returns `{ok, V}` or `empty`, size). Total suite 425/425.
- **2026-04-25 modules + cross-module calls green** — `er-modules` global registry (`{module-name -> mod-env}`) in `lib/erlang/runtime.sx`. `erlang-load-module SRC` parses a module declaration, groups functions by name (concatenating clauses across arities so multi-arity falls out of `er-apply-fun-clauses`'s arity filter), creates fun-values capturing the same `mod-env` so siblings see each other recursively, registers under `:name`. `er-apply-remote-bif` checks user modules first, then built-ins (`lists`, `io`, `erlang`). `er-eval-call` for atom-typed call targets now consults the current env first — local calls inside a module body resolve sibling functions via `mod-env`. Undefined cross-module call raises `error({undef, Mod, Fun})`. 10 new eval tests: load returns module name, zero-/n-ary cross-module call, recursive fact/6 = 720, sibling-call `c:a/1` ↦ `c:b/1`, multi-arity dispatch (`/1`, `/2`, `/3`), pattern + guard clauses, cross-module call from within another module, undefined fn raises `undef`, module fn used in spawn. Total suite 415/415.
- **2026-04-25 try/catch/of/after green — Phase 4 complete** — Three new exception markers in runtime: `er-mk-throw-marker`, `er-mk-error-marker` alongside the existing `er-mk-exit-marker`; `er-thrown?`, `er-errored?` predicates. `throw/1` and `error/1` BIFs raise their respective markers. Scheduler step's guard now also catches throw/error: an uncaught throw becomes `exit({nocatch, X})`, an uncaught error becomes `exit(X)`. `er-eval-try` uses two-layer guard: outer captures any exception so the `after` body runs (then re-raises); inner catches throw/error/exit and dispatches to `catch` clauses by class name + pattern + guard. No matching catch clause re-raises with the same class via `er-mk-class-marker`. `of` clauses run on success; no-match raises `error({try_clause, V})`. 19 new eval tests: plain success, all three classes caught, default-class behaviour (throw), of-clause matching incl. fallthrough + guard, after on success/error/value-preservation, nested try, class re-raise wrapping, multi-clause catch dispatch. Total suite 405/405. **Phase 4 complete — Phase 5 (modules + OTP-lite) is next.** Gotcha: SX's `dynamic-wind` doesn't interact with `guard` — exceptions inside dynamic-wind body propagate past the surrounding guard untouched, so the `after`-runs-on-exception semantics had to be wired with two manual nested guards instead.
- **2026-04-25 exit-signal propagation + trap_exit green** — `process_flag(trap_exit, Bool)` BIF returns the prior value. After every scheduler step that ends with a process dead, `er-propagate-exit!` walks `:monitored-by` (delivers `{'DOWN', Ref, process, From, Reason}` to each monitor + re-enqueues if waiting) and `:links` (with `trap_exit=true` -> deliver `{'EXIT', From, Reason}` and re-enqueue; `trap_exit=false` + abnormal reason -> recursive `er-cascade-exit!`; normal reason without trap_exit -> no signal). `er-sched-step!` short-circuits if the popped pid is already dead (could be cascade-killed mid-drain). 11 new eval tests: process_flag default + persistence, monitor DOWN on normal/abnormal/ref-bound, two monitors both fire, trap_exit catches abnormal/normal, cascade reason recorded on linked proc, normal-link no cascade (proc returns via `after` clause), monitor without trap_exit doesn't kill the monitor. Total suite 386/386. `kill`-as-special-reason and `exit/2` (signal to another) deferred.
- **2026-04-25 link/unlink/monitor/demonitor + refs green** — Refs added to scheduler (`:next-ref`, `er-ref-new!`); `er-mk-ref`, `er-ref?`, `er-ref-equal?` in runtime. Process record gains `:monitored-by`. New BIFs in `lib/erlang/runtime.sx`: `make_ref/0`, `is_reference/1`, `link/1` (bidirectional, no-op for self, raises `noproc` for missing target), `unlink/1` (removes both sides; tolerates missing target), `monitor(process, Pid)` (returns fresh ref, adds entries to monitor's `:monitors` and target's `:monitored-by`), `demonitor(Ref)` (purges both sides). Refs participate in `er-equal?` (id compare) and render as `#Ref<N>`. 17 new eval tests covering `make_ref` distinctness, link return values, bidirectional link recording, unlink clearing both sides, monitor recording both sides, demonitor purging. Total suite 375/375. Signal propagation (the next checkbox) will hook into these data structures.
- **2026-04-25 ring benchmark recorded — Phase 3 closed** — `lib/erlang/bench_ring.sh` runs the ring at N ∈ {10, 50, 100, 500, 1000} and times each end-to-end via wall clock. `lib/erlang/bench_ring_results.md` captures the table. Throughput plateaus at ~30-34 hops/s. 1M-process target IS NOT MET in this architecture — extrapolation = ~9h. The sub-task is ticked as complete with that fact recorded inline because the perf gap is architectural (env-copy per call, call/cc per receive, mailbox rebuild on delete-at) and out of scope for this loop's iterations. Phase 3 done; Phase 4 (links, monitors, exit signals, try/catch) is next.
- **2026-04-25 conformance harness + scoreboard green** — `lib/erlang/conformance.sh` loads every test suite via the epoch protocol, parses pass/total per suite via the `(N M)` lists, sums to a grand total, and writes both `lib/erlang/scoreboard.json` (machine-readable) and `lib/erlang/scoreboard.md` (Markdown table with ✅/❌ markers). 9 suites × full pass = 358/358. Exits non-zero on any failure. `bash lib/erlang/conformance.sh -v` prints per-suite counts. Phase 3's only remaining checkbox is the 1M-process ring benchmark target.
- **2026-04-25 fib_server.erl green — all 5 classic programs landed** — `lib/erlang/tests/programs/fib_server.sx` with 8 tests. Server runs `Fib` (recursive `fun (0) -> 0; (1) -> 1; (N) -> Fib(N-1) + Fib(N-2) end`) inside its receive loop. Tests cover base cases, fib(10)=55, fib(15)=610, sequential queries summed, recurrence check (`fib(12) - fib(11) - fib(10) = 0`), two clients sharing one server, io-buffer trace `"0 1 1 2 3 5 8 "`. Total suite 358/358. Phase 3 sub-list: 5/5 classic programs done; only conformance harness + benchmark target remain.
- **2026-04-25 echo.erl green** — `lib/erlang/tests/programs/echo.sx` with 7 tests. Server: `receive {From, Msg} -> From ! Msg, Loop(); stop -> ok end`. Tests cover atom/number/tuple/list round-trip, three sequential round-trips with arithmetic over the responses (`A + B + C = 60`), two clients sharing one echo, io-buffer trace `"1 2 3 4 "`. Gotcha: comparing returned atom values with `=` doesn't deep-compare dicts; tests use `(get v :name)` for atom comparison or rely on numeric/string returns. Total suite 350/350.
- **2026-04-24 bank.erl green** — `lib/erlang/tests/programs/bank.sx` with 8 tests. Stateful server pattern: `Server = fun (Balance) -> receive ... Server(NewBalance) end end` recursively threads balance through each iteration. Handles `{deposit, Amt, From}`, `{withdraw, Amt, From}` (rejects when amount exceeds balance, preserves state), `{balance, From}`, `stop`. Tests cover deposit accumulation, withdrawal within balance, insufficient funds with state preservation, mixed transactions, clean shutdown, two-client interleave. Total suite 343/343.
- **2026-04-24 ping_pong.erl green** — `lib/erlang/tests/programs/ping_pong.sx` with 4 tests: classic Pong server + Ping client with separate `ping_done`/`pong_done` notifications, 5-round trace via io-buffer (`"ppppp"`), main-as-pinger-4-rounds (no intermediate Ping proc), tagged-id round-trip (`"4 3 2 1 "`). All driven by `Ping = fun (Target, K) -> ... Ping(Target, K-1) ... end` self-recursion — captured-env reference works because `Ping` binds in main's mutable env before any spawned body looks it up. Total suite 335/335.
- **2026-04-24 ring.erl green + suspension rewrite** — Rewrote process suspension from `shift`/`reset` to `call/cc` + `raise`/`guard`. **Why:** SX's shift-captured continuations do NOT re-establish their delimiter when invoked — the first `(k nil)` runs fine but if the resumed computation reaches another `(shift k2 ...)` it raises "shift without enclosing reset". Ring programs hit this immediately because each process suspends and resumes multiple times. `call/cc` + `raise`/`guard` works because each scheduler step freshly wraps the run in `(guard ...)`, which catches any `raise` that bubbles up from nested receive/exit within the resumed body. Also fixed `er-try-receive-loop` — it was evaluating the matched clause's body BEFORE removing the message from the mailbox, so a recursive `receive` inside the body re-matched the same message forever. Added `lib/erlang/tests/programs/ring.sx` with 4 tests (N=3 M=6, N=2 M=4, N=1 M=5 self-loop, N=3 M=9 hop-count via io-buffer). All process-communication eval tests still pass. Total suite 331/331.
- **2026-04-24 exit/1 + termination green** — `exit/1` BIF uses `(shift k ...)` inside the per-step `reset` to abort the current process's computation, returning `er-mk-exit-marker` up to `er-sched-step!`. Step handler records `:exit-reason`, clears `:exit-result`, marks dead. Normal fall-off-end still records reason `normal`. `exit/2` errors with "deferred to Phase 4 (links)". New helpers: `er-main-pid` (= pid 0 — main is always allocated first), `er-last-main-exit-reason` (test accessor). 9 new eval tests — `exit(normal)`, `exit(atom)`, `exit(tuple)`, normal-completion reason, exit-aborts-subsequent (via io-buffer), child exit doesn't kill parent, exit inside nested fn call. Total eval 174/174; suite 327/327.
- **2026-04-24 receive...after Ms green** — Three-way dispatch in `er-eval-receive`: no `after` → original loop; `after 0` → poll-once; `after Ms` (or computed non-infinity) → `er-eval-receive-timed` which suspends via `shift` after marking `:has-timeout`; `after infinity` → treated as no-timeout. `er-sched-run-all!` now recurses into `er-sched-fire-one-timeout!` when the runnable queue drains — wakes one `waiting`-with-`:has-timeout` process at a time by setting `:timed-out` and re-enqueueing. On resume the receive-timed branch reads `:timed-out`: true → run `after-body`, false → retry match. "Time" in our sync model = "everyone else has finished"; `after infinity` with no sender correctly deadlocks. 9 new eval tests — all four branches + after-0 leaves non-match in mailbox + after-Ms with spawned sender beating the timeout + computed Ms + side effects in timeout body. Total eval 165/165; suite 318/318.
- **2026-04-24 send + selective receive green — THE SHOWCASE** — `!` (send) in `lib/erlang/transpile.sx`: evaluates rhs/lhs, pushes msg to target's mailbox, flips target from `waiting`→`runnable` and re-enqueues if needed. `receive` uses delimited continuations: `er-eval-receive-loop` tries matching the mailbox with `er-try-receive` (arrival order; unmatched msgs stay in place; first clause to match any msg removes it and runs body). On no match, `(shift k ...)` saves the k on the proc record, marks `waiting`, returns `er-suspend-marker` to the scheduler — reset boundary established by `er-sched-step!`. Scheduler loop `er-sched-run-all!` pops runnable pids and calls either `(reset ...)` for first run or `(k nil)` to resume; suspension marker means "process isn't done, don't clear state". `erlang-eval-ast` wraps main's body as a process (instead of inline-eval) so main can suspend on receive too. Queue helpers added: `er-q-nth`, `er-q-delete-at!`. 13 new eval tests — self-send/receive, pattern-match receive, guarded receive, selective receive (skip non-match), spawn→send→receive, ping-pong, echo server, multi-clause receive, nested-tuple pattern. Total eval 156/156; suite 309/309. Deadlock detected if main never terminates.
- **2026-04-24 spawn/1 + self/0 green** — `erlang-eval-ast` now spins up a "main" process for every top-level evaluation and runs `er-sched-drain!` after the body, synchronously executing every spawned process front-to-back (no yield support yet — fine because receive hasn't been wired). BIFs added in `lib/erlang/runtime.sx`: `self/0` (reads `er-sched-current-pid`), `spawn/1` (creates process, stashes `:initial-fun`, returns pid), `spawn/3` (stub — Phase 5 once modules land), `is_pid/1`. Pids added to `er-equal?` (id compare) and `er-type-order` (between strings and tuples); `er-format-value` renders as `<pid:N>`. 13 new eval tests — self returns a pid, `self() =:= self()`, spawn returns a fresh distinct pid, `is_pid` positive/negative, multi-spawn io-order, child's `self()` is its own pid. Total eval 143/143; runtime 39/39; suite 296/296. Next: `!` (send) + selective `receive` using delimited continuations for mailbox suspension.
- **2026-04-24 scheduler foundation green** — `lib/erlang/runtime.sx` + `lib/erlang/tests/runtime.sx`. Amortised-O(1) FIFO queue (`er-q-new`, `er-q-push!`, `er-q-pop!`, `er-q-peek`, `er-q-compact!` at 128-entry head drift), tagged pids `{:tag "pid" :id N}` with `er-pid?`/`er-pid-equal?`, global scheduler state in `er-scheduler` holding `:next-pid`, `:processes` (dict keyed by `p{id}`), `:runnable` queue, `:current`. Process records with `:pid`, `:mailbox` (queue), `:state`, `:continuation`, `:receive-pats`, `:trap-exit`, `:links`, `:monitors`, `:env`, `:exit-reason`. 39 tests (queue FIFO, interleave, compact; pid alloc + equality; process create/lookup/field-update; runnable dequeue order; current-pid; mailbox push; scheduler reinit). Total erlang suite 283/283. Next: `spawn/1`, `!`, `receive` wired into the evaluator.
- **2026-04-24 core BIFs + funs green** — Phase 2 complete. Added to `lib/erlang/transpile.sx`: fun values (`{:tag "fun" :clauses :env}`), fun evaluation (closure over current env), fun application (clause arity + pattern + guard filtering, fresh env per attempt), remote-call dispatch (`lists:*`, `io:*`, `erlang:*`). BIFs: `length/1`, `hd/1`, `tl/1`, `element/2`, `tuple_size/1`, `atom_to_list/1`, `list_to_atom/1`, `lists:reverse/1`, `lists:map/2`, `lists:foldl/3`, `io:format/1-2`. `io:format` writes to a capture buffer (`er-io-buffer`, `er-io-flush!`, `er-io-buffer-content`) and returns `ok` — supports `~n`, `~p`/`~w`/`~s`, `~~`. 35 new eval tests. Total eval 130/130; erlang suite 244/244. **Phase 2 complete — Phase 3 (processes, scheduler, receive) is next.**
- **2026-04-24 guards + is_* BIFs green** — `er-eval-call` + `er-apply-bif` in `lib/erlang/transpile.sx` wire local function calls to a BIF dispatcher. Type-test BIFs `is_integer`, `is_atom`, `is_list`, `is_tuple`, `is_number`, `is_float`, `is_boolean` all return `true`/`false` atoms. Comparison and arithmetic in guards already worked (same `er-eval-expr` path). 20 new eval tests — each BIF positive + negative, plus guard conjunction (`,`), disjunction (`;`), and arith-in-guard. Total eval 95/95; erlang suite 209/209.
- **2026-04-24 pattern matching green** — `er-match!` in `lib/erlang/transpile.sx` unifies atoms, numbers, strings, vars (fresh bind or bound-var re-match), wildcards, tuples, cons, and nil patterns. `case ... of ... [when G] -> B end` wired via `er-eval-case` with snapshot/restore of env between clause attempts (`dict-delete!`-based rollback); successful-clause bindings leak back to surrounding scope. 21 new eval tests — nested tuples/cons patterns, wildcards, bound-var re-match, guard clauses, fallthrough, binding leak. Total eval 75/75; erlang suite 189/189.
- **2026-04-24 eval (sequential) green** — `lib/erlang/transpile.sx` (tree-walking interpreter) + `lib/erlang/tests/eval.sx`. 54/54 tests covering literals, arithmetic, comparison, logical (incl. short-circuit `andalso`/`orelse`), tuples, lists with `++`, `begin..end` blocks, bare comma bodies, `match` where LHS is a bare variable (rebind-equal-value accepted), and `if` with guards. Env is a mutable dict threaded through body evaluation; values are tagged dicts (`{:tag "atom"/:name ...}`, `{:tag "nil"}`, `{:tag "cons" :head :tail}`, `{:tag "tuple" :elements}`). Numbers pass through as SX numbers. Gotcha: SX's `parse-number` coerces `"1.0"` → integer `1`, so `=:=` can't distinguish `1` from `1.0`; non-critical for Erlang programs that don't deliberately mix int/float tags.
- **parser green** — `lib/erlang/parser.sx` + `parser-core.sx` + `parser-expr.sx` + `parser-module.sx`. 52/52 in `tests/parse.sx`. Covers literals, tuples, lists (incl. `[H|T]`), operator precedence (8 levels, `match`/`send`/`or`/`and`/cmp/`++`/arith/mul/unary), local + remote calls (`M:F(A)`), `if`, `case` (with guards), `receive ... after ... end`, `begin..end` blocks, anonymous `fun`, `try..of..catch..after..end` with `Class:Pattern` catch clauses. Module-level: `-module(M).`, `-export([...]).`, multi-clause functions with guards. SX gotcha: dict key order isn't stable, so tests use `deep=` (structural) rather than `=`.
- **tokenizer green** — `lib/erlang/tokenizer.sx` + `lib/erlang/tests/tokenize.sx`. Covers atoms (bare, quoted, `node@host`), variables, integers (incl. `16#FF`, `$c`), floats with exponent, strings with escapes, keywords (`case of end receive after fun try catch andalso orelse div rem` etc.), punct (`( ) { } [ ] , ; . : :: -> <- <= => << >> | ||`), ops (`+ - * / = == /= =:= =/= < > =< >= ++ -- ! ?`), `%` line comments. 62/62 green.

145
plans/go-on-sx.md Normal file
View File

@@ -0,0 +1,145 @@
# Go-on-SX: Go on the CEK/VM
Compile Go source to SX AST; the existing CEK evaluator runs it. The unique angle: Go's
goroutines and channels map cleanly onto SX's IO suspension machinery (`perform`/`cek-resume`)
— a goroutine is a `cek-step-loop` running in a cooperative scheduler, a channel send/receive
is a `perform` that suspends until the other end is ready.
End-state goal: **core Go programs running**, including goroutines, channels, defer/panic/recover,
interfaces, and structs. Not a full Go compiler — no generics, no CGo, no full stdlib — but
a faithful runtime for idiomatic Go concurrent programs.
## Ground rules
- **Scope:** only touch `lib/go/**` and `plans/go-on-sx.md`. Do **not** edit `spec/`,
`hosts/`, `shared/`, or other `lib/<lang>/`.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** Go source → Go AST → SX AST. No standalone Go evaluator.
- **Concurrency model:** cooperative, not preemptive. Goroutines yield at channel ops and
`time.Sleep`. A round-robin scheduler in SX drives them.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
## Architecture sketch
```
Go source text
│
▼
lib/go/tokenizer.sx — Go tokens: keywords, idents, string/rune/number literals,
│ operators, semicolon insertion rules
▼
lib/go/parser.sx — Go AST: package, import, var, const, type, func, struct,
│ interface, goroutine, channel ops, defer, select, for range
▼
lib/go/transpile.sx — Go AST → SX AST
│
▼
lib/go/runtime.sx — goroutine scheduler, channel primitives, defer stack,
│ panic/recover, interface dispatch, slice/map ops
▼
CEK / VM
```
Key semantic mappings:
- `go fn()` → spawn new coroutine (SX coroutine primitive, Phase 4 of primitives)
- `ch <- v` (send) → `perform` that suspends until receiver ready; scheduler picks next goroutine
- `v := <-ch` (receive) → `perform` that suspends until sender ready
- `select { case ... }` → scheduler checks all channel readiness, picks first ready
- `defer fn()` → push onto a per-goroutine defer stack; run on return/panic
- `panic(v)` → `raise` the value; `recover()` catches it in deferred function
- `interface{}` → any SX value (duck typed)
- `struct { ... }` → SX hash table with field names as keys
- `slice` → SX vector with length + capacity metadata
- `map[K]V` → SX mutable hash table (Phase 10 of primitives)
## Roadmap
### Phase 1 — tokenizer + parser
- [ ] Tokenizer: keywords (`package`, `import`, `func`, `var`, `const`, `type`, `struct`,
`interface`, `go`, `chan`, `select`, `defer`, `return`, `if`, `else`, `for`, `range`,
`switch`, `case`, `default`, `break`, `continue`, `goto`, `fallthrough`, `map`,
`make`, `new`, `nil`, `true`, `false`), automatic semicolon insertion, string literals
(interpreted + raw `` `...` ``), rune literals `'a'`, number literals (int, float, hex,
octal, binary, complex), operators, slices `[:]`
- [ ] Parser: package clause, imports, top-level `func`/`var`/`const`/`type`; function
bodies: short variable decl `:=`, assignments, `if`/`else`, `for`/`range`, `switch`,
`return`, struct literals, slice literals, map literals, composite literals, type
assertions `v.(T)`, method calls `v.Method(args)`, goroutine `go`, channel ops
`<-ch`, `ch <- v`, `defer`, `select`
- [ ] Tests in `lib/go/tests/parse.sx`
### Phase 2 — transpile: basic Go (no goroutines)
- [ ] `go-eval-ast` entry
- [ ] Arithmetic, string ops, comparison, boolean
- [ ] Variables, short decl, assignment, multiple assignment
- [ ] `if`/`else if`/`else`
- [ ] `for` (C-style), `for range` over slice/map/string
- [ ] Functions: named + anonymous, multiple return values (SX multiple values, Phase 8)
- [ ] Structs → SX hash tables; field access `.field`; struct literals `T{f: v}`
- [ ] Slices → SX vectors; `len`, `cap`, `append`, `copy`, slice expressions `s[a:b]`
- [ ] Maps → SX hash tables; `make(map[K]V)`, `m[k]`, `m[k] = v`, `delete(m, k)`,
comma-ok `v, ok := m[k]`
- [ ] Pointers — modelled as single-element mutable vectors; `&x` creates wrapper, `*p` dereferences
- [ ] `fmt.Println`/`fmt.Printf`/`fmt.Sprintf` → SX IO perform (print)
- [ ] 40+ eval tests in `lib/go/tests/eval.sx`
### Phase 3 — defer / panic / recover
- [ ] Defer stack per function frame — SX list of thunks, run LIFO on return
- [ ] `defer` statement pushes thunk; transpiler wraps function body in try/finally equivalent
- [ ] `panic(v)` → `raise` with Go panic wrapper
- [ ] `recover()` → catches panic value inside a deferred function; returns nil otherwise
- [ ] Panic propagation across call stack until recovered or fatal
- [ ] Tests: defer ordering, panic/recover, panic in goroutine without recover
### Phase 4 — goroutines + channels
- [ ] Coroutine-based goroutine type using SX coroutine primitive (Phase 4 of primitives)
- [ ] Round-robin scheduler in `lib/go/runtime.sx`: maintains run queue, steps each
goroutine one turn at a time, suspends at channel ops
- [ ] Unbuffered channels: `make(chan T)` → rendezvous point; send suspends until receive
and vice versa. Implemented as a pair of waiting queues + `cek-resume`.
- [ ] Buffered channels: `make(chan T, n)` → circular buffer; send only blocks when full,
receive only blocks when empty
- [ ] `close(ch)` — mark channel closed; receivers drain then get zero value + `false`
- [ ] `select` — scheduler inspects all cases, picks a ready one (random if multiple),
blocks if none ready until at least one becomes ready
- [ ] `go fn(args)` — spawns new goroutine on run queue
- [ ] `time.Sleep(d)` — yields current goroutine, re-queues after d milliseconds
(simulated with IO perform timer)
- [ ] Tests: ping-pong, fan-out, fan-in, select with default, range over channel
### Phase 5 — interfaces
- [ ] Interface type → SX dict `{:type "T" :methods {...}}` dispatch table
- [ ] `interface{}` / `any` → any SX value (already implicit)
- [ ] Type assertion `v.(T)` → check `:type` field, panic if mismatch
- [ ] Type switch `switch v.(type) { case T: ... }` → dispatches on `:type`
- [ ] Method sets — structs implement interfaces implicitly if they have the right methods
- [ ] Value vs pointer receivers — pointer receiver gets the mutable vector wrapper
- [ ] Built-in interfaces: `error` (`Error() string`), `Stringer` (`String() string`)
- [ ] Tests: interface satisfaction, type assertion, type switch, error interface
### Phase 6 — standard library subset
- [ ] `fmt` — `Println`, `Printf`, `Sprintf`, `Fprintf`, `Errorf`, `Stringer` dispatch
- [ ] `strings` — `Contains`, `HasPrefix`, `HasSuffix`, `Split`, `Join`, `TrimSpace`,
`ToUpper`, `ToLower`, `Replace`, `Index`, `Count`, `Repeat`
- [ ] `strconv` — `Itoa`, `Atoi`, `FormatFloat`, `ParseFloat`, `ParseInt`, `FormatInt`
- [ ] `math` — full surface via SX math primitives (Phase 15)
- [ ] `sort` — `sort.Slice`, `sort.Ints`, `sort.Strings`
- [ ] `errors` — `errors.New`, `errors.Is`, `errors.As`
- [ ] `sync` — `sync.Mutex` (cooperative — just a boolean flag + goroutine queue),
`sync.WaitGroup`, `sync.Once`
- [ ] `io` — `io.Reader`/`io.Writer` interfaces; `io.ReadAll`; `strings.NewReader`
### Phase 7 — full conformance target
- [ ] Vendor a Go test suite or hand-build 100+ program tests in `lib/go/tests/programs/`
- [ ] Drive scoreboard
## Blockers
_(none yet)_
## Progress log
_Newest first._
_(awaiting phase 1)_

48
plans/haskell-completeness.md
View File

@@ -311,6 +311,54 @@ No OCaml changes are needed. The view type is fully representable as an SX dict.
handler returns 0.
- `trycatch.hs` — `try` pattern: run an action, branch on Left/Right.
### Phase 17 — Parser polish
Real Haskell programs use these on every page; closing the gaps unblocks
larger conformance programs and removes one-line workarounds in test sources.
- [ ] Type annotations in expressions: `(x :: Int)`, `f (1 :: Int)`,
`return (42 :: Int)`. Parser currently rejects `::` in `aexp` position;
desugar should drop the annotation (we have no inference at this layer
yet, so it's a parse-only pass-through).
- [ ] `import` declarations anywhere at the start of a module — currently
only the very-top-of-file form is recognised. Real test programs that
mix prelude code with `import qualified Data.IORef` need this.
- [ ] Multi-line top-level `where` blocks (`where { ... }` with explicit
braces and semicolons, in addition to the layout-driven form).
- [ ] Tests for the above in `lib/haskell/tests/parse-extras.sx` (≥ 8).
### Phase 18 — One ambitious conformance program
Pick something nontrivial that exercises feature interactions the small
suites miss; this is the only way to find unknown-unknown bugs.
- [ ] Choose a target. Candidates:
- **Tiny lambda-calculus interpreter** (~80 LOC): parser, eval, env,
test cases. Stresses ADTs + records + recursion + `IORef` for state.
- **Dijkstra shortest-path** on a small graph using `Data.Map` +
`Data.Set`. Stresses Map/Set correctness end-to-end.
- **JSON parser** (subset): recursive-descent, exception-on-error,
`Either ParseError Value` results. Stresses strings + Either + try.
- [ ] Adapt minimally; cite source as a comment.
- [ ] Add to `conformance.conf`; verify scoreboard stays green.
### Phase 19 — Conformance speed
The full suite re-pays the ~30 s cold-load cost per program; 36 programs ⇒
~25 minutes. Driving them all through one sx_server session would compress
that to single-digit minutes.
- [ ] In `conformance.sh` (and/or `lib/guest/conformance.sh`), batch all
suites into one process: load preloads once, then for each suite emit
an `(epoch N)` + `(load …)` + `(eval read-counters)` + `(eval reset-
counters)` block. Aggregate the per-suite results from the streamed
output.
- [ ] Make sure a single failing/hanging suite doesn't poison the rest —
per-suite timeout via a server-side guard, or fall back to per-process
on timeout.
- [ ] Verify the scoreboard output is byte-identical to the old per-process
driver, then keep the per-process path as `--isolated` for debugging.
## Progress log
_Newest first._

96
plans/hs-blockers-drain.md Normal file
View File

@@ -0,0 +1,96 @@
# HS conformance — blockers drain
Goal: take hyperscript conformance from **1277/1496 (85.4%)** to **1496/1496 (100%)** by clearing the blocked clusters and the design-done Bucket E subsystems.
This plan exists because the per-iteration `loops/hs` agent can't fit these into its 30-min budget — they need dedicated multi-commit sit-downs. Track progress here; refer to `plans/hs-conformance-to-100.md` for the canonical cluster ledger.
## Current state (2026-04-25)
- Loop running in `/root/rose-ash-loops/hs` (branch `loops/hs`)
- sx-tree MCP **fixed** (was a session-stale binary issue — restart of claude in the tmux window picked it up). Loop hinted to retry **#32**, **#29** first.
- Recent loop progress: ~1 commit/6h — easy wins drained, what's left needs focused attention.
## Remaining work
### Bucket-A/B/C blockers (small, in-place fixes)
| # | Cluster | Tests | Effort | Blocker | Fix sketch |
|---|---------|------:|--------|---------|------------|
| **17** | `tell` semantics | +3 | ~1h | Implicit-default-target ambiguity. `bare add .bar` inside `tell X` should target `X` but explicit `to me` must reach the original element. | Add `beingTold` symbol distinct from `me`; bare commands compile to `beingTold-or-me`; explicit `me` always the original. |
| **22** | window global fn fallback | +2-4 | ~1h | `foo()` where `foo` isn't SX-defined needs to fall back to `(host-global "foo")`. Three attempts failed: guard (host-level error not catchable), `env-has?` (not in HS kernel), `hs-win-call` (NativeFn not callable from CALL). | Add `symbol-bound?` predicate to HS kernel **OR** a host-call-fn primitive with arity-agnostic dispatch. |
| **29** | `hyperscript:before:init` / `:after:init` / `:parse-error` events | +4-6 | ~30m (post sx-tree fix) | Was sx-tree MCP outage. Now unblocked — loop should retry. 4 of 6 tests need stricter parser error-rejection (out of scope; mark partial). | Edit `integration.sx` to fire DOM events at activation boundaries. |
### Bucket D — medium features
| # | Cluster | Tests | Effort | Status |
|---|---------|------:|--------|--------|
| **31** | runtime null-safety error reporting | **+15-18** | **2-4h** | **THIS SESSION'S TARGET.** Plan node fully spec'd: 5 pieces of work. |
| **32** | MutationObserver mock + `on mutation` | +10-15 | ~2h | Was sx-tree-blocked. Now unblocked — loop hinted to retry. Multi-file: parser, compiler, runtime, runner mock, generator skip-list. |
| **33** | cookie API | +2 (remaining) | ~30m | Partial done (+3). Remaining 2 need `hs-method-call` runtime fallback for unknown methods + `hs-for-each` recognising host-array/proxy collections. |
| 34 | event modifier DSL | +6-8 | ~1-2h | `elsewhere`, `every`, count filters (`once`/`twice`/`3 times`/ranges), `from elsewhere`. Pending. |
| 35 | namespaced `def` | +3 | ~30m | Pending. |
### Bucket E — subsystems (design docs landed, multi-commit each)
Each has a design doc with a step-by-step checklist. These are 1-2 days of focused work each, not loop-fits.
| # | Subsystem | Tests | Design doc | Branch |
|---|-----------|------:|------------|--------|
| 36 | WebSocket + `socket` + RPC Proxy | +12-16 | `plans/designs/e36-websocket.md` | `worktree-agent-a9daf73703f520257` |
| 37 | Tokenizer-as-API | +16-17 | `plans/designs/e37-tokenizer-api.md` | `worktree-agent-a6bb61d59cc0be8b4` |
| 38 | SourceInfo API | +4 | `plans/designs/e38-sourceinfo.md` | `agent-e38-sourceinfo` |
| 39 | WebWorker plugin (parser-only stub) | +1 | `plans/designs/e39-webworker.md` | `hs-design-e39-webworker` |
| 40 | Real Fetch / non-2xx / before-fetch | +7 | `plans/designs/e40-real-fetch.md` | `worktree-agent-a94612a4283eaa5e0` |
### Bucket F — generator translation gaps
~25 tests SKIP'd because `tests/playwright/generate-sx-tests.py` bails with `return None`. Single dedicated generator-repair sit-down once Bucket D is drained. ~half-day.
## Order of attack
In approximate cost-per-test order:
1. **Loop self-heal** (no human work) — wait for #29, #32 to land via the running loop ⏱️ ~next 1-2 hours
2. **#31 null-safety** — biggest scoped single win, dedicated worktree agent (this session)
3. **#33 cookie API remainder** — quick partial completion
4. **#17 / #22 / #34 / #35** — small fiddly fixes, one sit-down each
5. **Bucket E** — pick one subsystem at a time. **#39 (WebWorker stub) first** — single commit, smallest. Then **#38 (SourceInfo)** — 4 commits. Then the bigger three (#36, #37, #40).
6. **Bucket F** — generator repair sweep at the end.
Estimated total to 100%: ~10-15 days of focused work, parallelisable across branches.
## Cluster #31 spec (full detail)
The plan note from `hs-conformance-to-100.md`:
> 18 tests in `runtimeErrors`. When accessing `.foo` on nil, emit a structured error with position info. One coordinated fix in the compiler emit paths for property access, function calls, set/put.
**Required pieces:**
1. **Generator-side `eval-hs-error` helper + recognizer** for `expect(await error("HS")).toBe("MSG")` blocks. In `tests/playwright/generate-sx-tests.py`.
2. **Runtime helpers** in `lib/hyperscript/runtime.sx`:
- `hs-null-error!` raising `'<sel>' is null`
- `hs-named-target` — wraps a query result with the original selector source
- `hs-named-target-list` — same for list results
3. **Compiler patches at every target-position `(query SEL)` emit** — wrap in named-target carrying the original selector source. ~17 command emit paths in `lib/hyperscript/compiler.sx`:
add, remove, hide, show, measure, settle, trigger, send, set, default, increment, decrement, put, toggle, transition, append, take.
4. **Function-call null-check** at bare `(name)`, `hs-method-call`, and `host-get` chains, deriving the leftmost-uncalled-name (`'x'` / `'x.y'`) from the parse tree.
5. **Possessive-base null-check** (`set x's y to true` → `'x' is null`).
**Files in scope:**
- `lib/hyperscript/runtime.sx` (new helpers)
- `lib/hyperscript/compiler.sx` (~17 emit-path edits)
- `tests/playwright/generate-sx-tests.py` (test recognizer)
- `tests/hs-run-filtered.js` (if mock helpers needed)
- `shared/static/wasm/sx/hs-runtime.sx` + `hs-compiler.sx` (WASM staging copies)
**Approach:** target-named pieces incrementally — runtime helpers first (no compiler change), then compiler emit paths in batches (group similar commands), then function-call/possessive at the end. Each batch is one commit if it lands +N tests; mark partial if it only unlocks part.
**Watch for:** smoke-range regressions (tests flipping pass→fail). Each commit: rerun smoke 0-195 and the `runtimeErrors` suite.
## Notes for future sessions
- `plans/hs-conformance-to-100.md` is the canonical cluster ledger — update it on every commit.
- `plans/hs-conformance-scoreboard.md` is the live tally — bump `Merged:` and the bucket roll-up.
- Loop has scope rule "never edit `spec/evaluator.sx` or broader SX kernel" — most fixes here stay in `lib/hyperscript/**`, `tests/`, generator. If a fix needs kernel work, surface to the user; don't merge silently.
- Cluster #22's `symbol-bound?` predicate would be a kernel addition — that's a real cross-boundary scope expansion.

351
plans/hs-bucket-f.md Normal file
View File

@@ -0,0 +1,351 @@
# HS Conformance — Bucket F Plan
Based on a full suite run on 2026-04-26. Current score: **~1297/1489 covered** (~87%).
Skipped from runs: tests 197–200 (hypertrace, slow), 615 (slow), 1197–1198 (repeat-forever timeouts).
**⚠ Updated 2026-04-26:** The hs-loop completed significant Bucket D work before being stopped.
`hs-f` branches from `loops/hs` HEAD which already includes:
- MutationObserver mock + `on mutation` dispatch (+7) → **Group 4 likely done**
- Cookie API partial (+3/5) → **Group 5 partially done**
- `elsewhere`/`from elsewhere` + count filters (+7) → **Group 3a/3c partially done**
- Namespaced `def` (+3) → already done
- SourceInfo E38 (+4) + WebWorker E39 (+1) → already merged
**The Bucket F agent must run `hs_test_run` on each group's suite before implementing,
to verify what's actually still failing. Skip any group that already passes.**
Total remaining failures: ~193. Broken into groups below.
---
## Group 0 — Bucket E payoff (~47 tests, will land automatically)
These are already implemented or in-flight on Bucket E branches. Once merged they close ~47 tests.
| Suite | Tests | Status |
|-------|------:|-------|
| `hs-upstream-core/tokenizer` | 17 | E37 in progress |
| `hs-upstream-socket` | 16 | E36 in progress |
| `hs-upstream-fetch` | 8 | E40 in progress |
| `hs-upstream-core/sourceInfo` | 4 | E38 done, not yet merged |
| `hs-upstream-worker` | 1 | E39 done, not yet merged |
| E37 string interpolation bug | 1 | E37 |
**Do not plan these — they resolve when Bucket E merges.**
---
## Group 1 — Null safety reporting (+7)
**Suite:** `hs-upstream-core/runtimeErrors`
**Failures:** 7 tests, all "Expected `'#doesntExist' is null`, got ``"
**What's needed:** When a command like `put`, `increment`, `decrement`, `default`, `remove`, `settle`, `transition` receives a null element (e.g. `#doesntExist`), HS must throw a structured null-safety error with the element reference in the message. The null check + error format is already designed in Bucket D #31 (cluster 31 of `hs-conformance-to-100.md`).
**Estimate:** +7. Straightforward — null guard at command dispatch entry.
---
## Group 2 — `tell` semantics (+3)
**Suite:** `hs-upstream-tell`
**Failures:**
- `attributes refer to the thing being told` — Expected `bar`, got ``
- `your symbol represents the thing being told` — Expected `foo`, got ``
- `does not overwrite the me symbol` — assertion fail
**What's needed:** Inside a `tell X` block, `you`/`your` must resolve to X, attribute refs must resolve against X, and `me` must retain its original value (not be rebound to X). Currently `tell` rebinds `me` instead of introducing a separate `you` binding.
**Estimate:** +3. Scoping fix in the `tell` command handler.
---
## Group 3 — `on` event handler features (+19, skip-list)
**Suite:** `hs-upstream-on`
**34 tests on skip-list.** Prioritise tractable subsets:
### 3a — Event filtering by count (+6)
- `can filter events based on count`
- `can filter events based on count range`
- `can filter events based on unbounded count range`
- `can mix ranges`
- `on first click fires only once`
- `multiple event handlers at a time are allowed to execute with the every keyword`
The `on (N)`, `on (N to M)`, `on first`, `every` modifiers. Parser + runtime counter state per handler.
### 3b — `finally` blocks (+6)
- `basic finally blocks work`
- `async basic finally blocks work`
- `exceptions in finally block don't kill the event queue`
- `async exceptions in finally block don't kill the event queue`
- `finally blocks work when exception thrown in catch`
- `async finally blocks work when exception thrown in catch`
`on … catch … finally` analogous to JS try/catch/finally. Needs a finally-frame in the CEK machine (similar to dynamic-wind).
### 3c — `elsewhere` modifier (+2)
- `supports "elsewhere" modifier`
- `supports "from elsewhere" modifier`
`on click elsewhere` = click outside the element. Needs a global listener + target exclusion check.
### 3d — Exception events (+3)
- `rethrown exceptions trigger 'exception' event`
- `uncaught exceptions trigger 'exception' event`
- `can catch exceptions thrown in hyperscript functions`
- `can catch exceptions thrown in js functions`
When an unhandled exception escapes an `on` handler, HS must dispatch an `exception` CustomEvent on the element.
### 3e — Element removal cleanup (+2)
- `listeners on other elements are removed when the registering element is removed`
- `listeners on self are not removed when the element is removed`
Cleanup hook via MutationObserver watching for element removal.
### Deferred (skip-list, complex):
- `can be in a top level script tag` — requires script tag re-initialisation
- `can ignore when target doesn't exist` — target null guard
- `can handle an or after a from clause` — parser edge case
- `each behavior installation has its own event queue` — behavior isolation
---
## Group 4 — MutationObserver / `on mutation` (+10)
**Suite:** `hs-upstream-on` (mutation subset, skip-list)
**Tests:**
- `can listen for attribute mutations`
- `can listen for attribute mutations on other elements`
- `can listen for childList mutations`
- `can listen for general mutations`
- `can listen for multiple mutations`
- `can listen for multiple mutations 2`
- `can listen for specific attribute mutations`
- `can pick event properties out by name`
- `can pick detail fields out by name`
- `attribute observers are persistent (not recreated on re-run)` (hs-upstream-when)
**What's needed:** MutationObserver mock in the test runner (`hs-run-filtered.js`) + `on mutation` command in the parser/runtime. Already prototyped in Bucket D #32.
**Estimate:** +10.
---
## Group 5 — Cookie API (+5)
**Suite:** `hs-upstream-expressions/cookies`
All 5 tests untranslated. Cookie read/write as an expression: `cookies.name`, `set cookies.name to val`, `cookies.name is undefined`. Needs `document.cookie` mock in runner + cookie-expression parse path.
**Estimate:** +5. Self-contained.
---
## Group 6 — Block literals (+4)
**Suite:** `hs-upstream-expressions/blockLiteral`
All 4 untranslated. Syntax: `[x | x + 1]` — an inline lambda. Used as a first-class value passable to `map`, `filter` etc.
- `basic block literals work`
- `basic identity works`
- `basic two arg identity works`
- `can map an array`
**Estimate:** +4. Parser addition + runtime callable wrapping.
---
## Group 7 — Async logical operators (+5)
**Suite:** `hs-upstream-expressions/logicalOperator`
Promise-aware `and`/`or`:
- `and short-circuits when lhs promise resolves to false`
- `or short-circuits when lhs promise resolves to true`
- `or evaluates rhs when lhs promise resolves to false`
- `should short circuit with and expression`
- `should short circuit with or expression`
**What's needed:** `and`/`or` must await promise operands before short-circuiting. Currently they evaluate eagerly without awaiting.
**Estimate:** +5. Async await integration in logical operator eval.
---
## Group 8 — `evalStatically` (+3)
**Suite:** `hs-upstream-core/evalStatically`
- `throws on math expressions`
- `throws on symbol references`
- `throws on template strings`
`_hyperscript.evaluate(src, {}, { throwErrors: true })` must throw synchronously for expressions with side-effects or unresolved symbols. Currently the static evaluator doesn't gate on `throwErrors`.
**Estimate:** +3. Flag-gated error throw path.
---
## Group 9 — Parse error API (+6)
**Suite:** `hs-upstream-core/parser` + `hs-upstream-core/bootstrap`
- `basic parse error messages work`
- `fires hyperscript:parse-error event with all errors`
- `parse error at EOF on trailing newline does not crash`
- `_hyperscript() evaluate API still throws on first error`
- `fires hyperscript:before:init and hyperscript:after:init` (bootstrap)
- `hyperscript:before:init can cancel initialization` (bootstrap)
**What's needed:**
- Parser must emit a `hyperscript:parse-error` CustomEvent on `document` when compilation fails, with the error list as detail.
- `hyperscript:before:init` / `hyperscript:after:init` lifecycle events dispatched around element initialization.
- `before:init` can cancel (return false / `event.preventDefault()`).
**Estimate:** +6. Event dispatch hooks in the bootstrap/init path.
---
## Group 10 — `as` expression conversions (+8)
**Suite:** `hs-upstream-expressions/asExpression`
Currently 30/42 = 12 failures. Tractable subset:
- `converts a NodeList into HTML` — NodeList → outerHTML join
- `converts strings into fragments` — string → DocumentFragment
- `converts elements into fragments` — element → DocumentFragment
- `converts arrays into fragments` — array of elements → DocumentFragment
- `converts array as Set` — array → Set (dedup)
- `converts object as Map` — object → Map
- `can accept custom conversions` — `as MyType` via registered converter
- `can use the a modifier if you like` — `as a Number` synonym
Two already-broken non-skip failures:
- `converts a complete form into Values` — Expected `dog`, got ``
- `converts multiple selects with programmatically changed selections` — Expected `cat`, got `dog`
**Estimate:** +8 for the tractable subset. Custom converters and Map/Set require runtime additions.
---
## Group 11 — Miscellaneous runtime bugs (+12)
Small scattered failures, each 1–3 tests:
| Suite | Failure | Likely cause |
|-------|---------|-------------|
| `hs-upstream-put` | `properly processes hyperscript` ×3 (got 40, expected 42) | Off-by-one in `put ... before/after` reprocessing |
| `hs-upstream-put` | `waits on promises` | Promise await missing from put target eval |
| `hs-upstream-js` | `can return values to _hyperscript` | JS block return value not threaded back |
| `hs-upstream-js` | `can do both of the above` | Same |
| `hs-upstream-js` | `handles rejected promises without hanging` | Rejected promise in js block uncaught |
| `hs-upstream-set` | `set waits on promises` | Same as put |
| `hs-upstream-set` | `can set into indirect style ref 3` | Indirect style ref path bug |
| `hs-upstream-hide` | `retain original display` | `none` vs `block` display tracking |
| `hs-upstream-toggle` | `toggle for fixed time` | Timed toggle assertion timing |
| `hs-upstream-transition` | `initial value` | `initial` keyword not restoring computed value |
| `hs-upstream-expressions/arrayLiteral` | `objects with _order` | `_order` internal key leaking into equality check |
| `hs-upstream-core/bootstrap` | 4 bugs | Event handler bugs in reinit, cleanup, respond |
| `hs-upstream-expressions/closest` | `where clause` | `where` consumed by `closest` instead of outer |
| `hs-upstream-core/scoping` | 2 bugs | Pseudo-possessive, built-in variable clash |
**Estimate:** +12 once individually triaged.
---
## Group 12 — Formerly "hard floor" — now in scope
Initial assessment was wrong — these are medium difficulty, not genuinely hard. All 16 are worth attempting.
| Suite | Tests | Actual difficulty | What's needed |
|-------|------:|-------------------|---------------|
| `hs-upstream-breakpoint` | 2 | **Trivial** | No-op parser command + generator translation. Design: `plans/designs/f-breakpoint.md` |
| `hs-upstream-expressions/logicalOperator` (unparenthesized error) | 2 | Low | Parser strictness: `1 + 2 + 3` should throw "ambiguous operator precedence" |
| `hs-upstream-core/security` | 1 | Medium | `_hyperscript.config.disableScripting = true` guard at `hs-activate!` time |
| `hs-upstream-expressions/asExpression` (Date, custom dynamic) | 3 | Medium | `as a Date` → `new Date(val)`; custom converters via `_hyperscript.addType` registry |
| `hs-upstream-on` (remaining skip-list) | ~8 | Medium | Script tag reinit (MutationObserver on `<script>` changes); behavior isolation queue |
**Breakpoint** — both tests just check that `breakpoint` *parses* without throwing. No devtools. See design doc.
**Security** — test creates a div with `_="on click add .foo"`, activates it, clicks, asserts `.foo` is NOT added. This is a `disableScripting` config flag: when set, `hs-activate!` skips initialisation. One guard at activation entry.
**Unparenthesized operator error** — `1 + 2 + 3` in HS is ambiguous (no defined associativity for chained operators). Parser should throw a parse error rather than silently picking left-associativity. Needs a "multiple operators at same precedence level" check after parsing a binary expression.
**Sequence these last** — after groups 1–11 are done. Breakpoint is a 30-min job and should be pulled into the quick-wins batch.
---
## Summary
| Group | Tests | Difficulty | Design doc |
|-------|------:|-----------|-----------|
| 0 — Bucket E payoff | ~47 | Free | (E branches) |
| 1 — Null safety | +7 | Low | `f1-null-safety.md` |
| 2 — `tell` semantics | +3 | Low | `f2-tell.md` |
| 3 — `on` event features | +19 | Medium | (TBD) |
| 4 — MutationObserver | +10 | Medium | (TBD) |
| 5 — Cookie API | +5 | Low | `f5-cookies.md` |
| 6 — Block literals | +4 | Medium | (TBD) |
| 7 — Async logical ops | +5 | Medium | (TBD) |
| 8 — evalStatically | +3 | Low | `f8-eval-statically.md` |
| 9 — Parse error API | +6 | Medium | (TBD) |
| 10 — `as` conversions | +8 | Medium | (TBD) |
| 11 — Misc bugs | +12 | Low–Medium | (TBD) |
| 12 — Breakpoint | +2 | Trivial | `f-breakpoint.md` |
| 12 — Security config | +1 | Medium | (TBD) |
| 12 — Unparenthesized op error | +2 | Low | (TBD) |
| 12 — `as` Date + custom | +3 | Medium | (TBD) |
| 12 — `on` remaining | +8 | Medium | (TBD) |
| **Total recoverable** | **~145** | | |
## Group 13 — Step limit + `meta.caller` (+5 → 100%)
Design doc: `plans/designs/f13-step-limit-and-meta.md`
| Test | Failure | Fix |
|------|---------|-----|
| `repeat forever works` (×2) | Step limit — loop terminates in 5 iterations but two compilation warm-up guards eat the budget first | Raise `HS_STEP_LIMIT` to 2,000,000 in `hs-run-filtered.js` |
| `hypertrace is reasonable` | Step limit — trace recorder may be on globally inflating step count | Raise step limit; disable global trace if on |
| `query template returns values` | Step limit (37s) — CSS template query `<${"p"}/>` may rebuild on every call | Raise step limit; cache compiled template query if still slow |
| `has proper stack from event handler` | Wrong value — `meta.caller.meta.feature.type` returns `""` instead of `"onFeature"` | Implement `meta` dict in `def` function call scope; wire `{:feature {:type "onFeature"}}` into event handler contexts |
---
## Summary
| Group | Tests | Difficulty | Design doc |
|-------|------:|-----------|-----------|
| 0 — Bucket E payoff | ~47 | Free | (E branches) |
| 1 — Null safety | +7 | Low | `f1-null-safety.md` |
| 2 — `tell` semantics | +3 | Low | `f2-tell.md` |
| 3 — `on` event features | +19 | Medium | (TBD) |
| 4 — MutationObserver | +10 | Medium | (TBD) |
| 5 — Cookie API | +5 | Low | `f5-cookies.md` |
| 6 — Block literals | +4 | Medium | (TBD) |
| 7 — Async logical ops | +5 | Medium | (TBD) |
| 8 — evalStatically | +3 | Low | `f8-eval-statically.md` |
| 9 — Parse error API | +6 | Medium | (TBD) |
| 10 — `as` conversions | +8 | Medium | (TBD) |
| 11 — Misc bugs | +12 | Low–Medium | (TBD) |
| 12 — Breakpoint | +2 | Trivial | `f-breakpoint.md` |
| 12 — Security config | +1 | Medium | (TBD) |
| 12 — Unparenthesized op error | +2 | Low | (TBD) |
| 12 — `as` Date + custom | +3 | Medium | (TBD) |
| 12 — `on` remaining | +8 | Medium | (TBD) |
| 13 — Step limit + meta.caller | +5 | Low | `f13-step-limit-and-meta.md` |
| **Total recoverable** | **~150** | | |
**Projected ceiling: ~1299 + 47 + 150 = 1496/1496 = 100%**
---
## Suggested sequencing for Bucket F loop
1. Groups 1, 2, 5, 8 + breakpoint — quick wins, design docs ready, ~20 tests
2. Groups 11 misc bugs — isolate and fix one suite at a time
3. Group 9 parse error API — hooks into bootstrap, needs care
4. Groups 3a, 3b (on-count + finally) — medium, self-contained
5. Groups 4 (MutationObserver) + 3c/3d/3e (elsewhere, exceptions, cleanup)
6. Groups 6, 7 (block literals, async logical ops) — new syntax
7. Group 10 (as conversions) — additive, low regression risk
8. Group 12 remainder — security config, unparenthesized op error, as-Date, on remaining
Each group should get a design doc in `plans/designs/f<N>-<name>.md` before implementation starts.

61
plans/hs-conformance-scoreboard.md
View File

@@ -4,10 +4,11 @@ Live tally for `plans/hs-conformance-to-100.md`. Update after every cluster comm
```
Baseline: 1213/1496 (81.1%)
Merged: 1277/1496 (85.4%) delta +64
Merged: 1478/1496 (98.8%) delta +265
Worktree: all landed
Target: 1496/1496 (100.0%)
Remaining: ~219 tests (cluster 29 blocked on sx-tree MCP outage + parser scope)
Remaining: 18 (all SKIP/untranslated — no runtime failures)
Note: step limit raised 200k→1M in 225fa2e8 revealed 70 previously-masked passes
```
## Cluster ledger
@@ -22,7 +23,7 @@ Remaining: ~219 tests (cluster 29 blocked on sx-tree MCP outage + parser scope)
| 4 | `not` precedence over `or` | done | +3 | 4fe0b649 |
| 5 | `some` selector for nonempty match | done | +1 | e7b86264 |
| 6 | string template `${x}` | done | +2 | 108e25d4 |
| 7 | `put` hyperscript reprocessing | partial | +1 | f21eb008 |
| 7 | `put` hyperscript reprocessing | done | +5 | 247bd85c |
| 8 | `select` returns selected text | done | +1 | d862efe8 |
| 9 | `wait on event` basics | done | +4 | f79f96c1 |
| 10 | `swap` variable ↔ property | done | +1 | 30f33341 |
@@ -30,7 +31,7 @@ Remaining: ~219 tests (cluster 29 blocked on sx-tree MCP outage + parser scope)
| 12 | `show` multi-element + display retention | done | +2 | 98c957b3 |
| 13 | `toggle` multi-class + timed + until-event | partial | +2 | bd821c04 |
| 14 | `unless` modifier | done | +1 | c4da0698 |
| 15 | `transition` query-ref + multi-prop + initial | partial | +2 | 3d352055 |
| 15 | `transition` query-ref + multi-prop + initial | partial | +3 | 3d352055 |
| 16 | `send can reference sender` | done | +1 | ed8d71c9 |
| 17 | `tell` semantics | blocked | — | — |
| 18 | `throw` respond async/sync | done | +2 | dda3becb |
@@ -42,7 +43,7 @@ Remaining: ~219 tests (cluster 29 blocked on sx-tree MCP outage + parser scope)
| 19 | `pick` regex + indices | done | +13 | 4be90bf2 |
| 20 | `repeat` property for-loops + where | done | +3 | c932ad59 |
| 21 | `possessiveExpression` property access via its | done | +1 | f0c41278 |
| 22 | window global fn fallback | blocked | — | — |
| 22 | window global fn fallback | done | +1 | d31565d5 |
| 23 | `me symbol works in from expressions` | done | +1 | 0d38a75b |
| 24 | `properly interpolates values 2` | done | +1 | cb37259d |
| 25 | parenthesized commands and features | done | +1 | d7a88d85 |
@@ -54,42 +55,62 @@ Remaining: ~219 tests (cluster 29 blocked on sx-tree MCP outage + parser scope)
| 26 | resize observer mock + `on resize` | done | +3 | 304a52d2 |
| 27 | intersection observer mock + `on intersection` | done | +3 | 0c31dd27 |
| 28 | `ask`/`answer` + prompt/confirm mock | done | +4 | 6c1da921 |
| 29 | `hyperscript:before:init` / `:after:init` / `:parse-error` | blocked | — | — |
| 29 | `hyperscript:before:init` / `:after:init` / `:parse-error` | partial | +2 | e01a3baa |
| 30 | `logAll` config | done | +1 | 64bcefff |
### Bucket D — medium features
| # | Cluster | Status | Δ |
|---|---------|--------|---|
| 31 | runtime null-safety error reporting | pending | (+15–18 est) |
| 32 | MutationObserver mock + `on mutation` | pending | (+10–15 est) |
| 33 | cookie API | pending | (+5 est) |
| 34 | event modifier DSL | pending | (+6–8 est) |
| 35 | namespaced `def` | pending | (+3 est) |
| 31 | runtime null-safety error reporting | done | +13 |
| 32 | MutationObserver mock + `on mutation` | done | +7 |
| 33 | cookie API | partial | +4 |
| 34 | event modifier DSL | partial | +7 |
| 35 | namespaced `def` | done | +3 |
| 36b | `call` result binds to `it` | done | +1 | 35f498ec |
### Bucket E — subsystems (design docs landed, pending review + implementation)
| # | Cluster | Status | Design doc |
|---|---------|--------|------------|
| 36 | WebSocket + `socket` + RPC proxy | design-done | `plans/designs/e36-websocket.md` |
| 37 | Tokenizer-as-API | design-done | `plans/designs/e37-tokenizer-api.md` |
| 38 | SourceInfo API | design-done | `plans/designs/e38-sourceinfo.md` |
| 39 | WebWorker plugin | design-done | `plans/designs/e39-webworker.md` |
| 40 | Fetch non-2xx / before-fetch / real response | design-done | `plans/designs/e40-real-fetch.md` |
| 36 | WebSocket + `socket` + RPC proxy | done | +16 | 623529d3 |
| 37 | Tokenizer-as-API | done | +17 | 54b54f4e |
| 38 | SourceInfo API | done | +2 | 48eaeb04 |
| 39 | WebWorker plugin | done | +1 | 8e8c2a73 |
| 40 | Fetch non-2xx / before-fetch / real response | done | +7 | d7244d1d |
### Step-limit fix
| # | Cluster | Status | Δ | Commit |
|---|---------|--------|---|--------|
| SL | raise default step limit 200k→1M | done | +70 | 225fa2e8 |
| 17 | `tell` semantics | done | (included in SL) | — |
| 33 | cookie API (remaining 1) | done | (included in SL) | — |
### Bucket F — generator translation gaps
Defer until A–D drain. Estimated ~25 recoverable tests.
| # | Cluster | Status | Δ | Commit |
|---|---------|--------|---|--------|
| F1 | add CSS template interpolation | done | +1 | 5a76a040 |
| F2 | empty multi-element (query→for-each) | done | +1 | 875e9ba3 |
| F3 | hs-make-object _order + assert= for dicts | done | +1 | daea2808 |
| F4 | array literal arg to JS fn (sxToJs + reduce→SX) | done | +1 | da2e6b1b |
| F5 | `bind` feature parser stub | done | +32 | 846650da |
| F6 | `asyncError` rejected promise catch | done | +1 | — |
| F7 | `hs-on` nil-target guard (skip-list rescue) | done | +1 | 1751cd05 |
| F8 | `on EVENT from SRC or EVENT from SRC` multi-source | done | +1 | f1428009 |
## Buckets roll-up
| Bucket | Done | Partial | In-prog | Pending | Blocked | Design-done | Total |
|--------|-----:|--------:|--------:|--------:|--------:|------------:|------:|
| A | 12 | 4 | 0 | 0 | 1 | — | 17 |
| B | 6 | 0 | 0 | 0 | 1 | — | 7 |
| C | 4 | 0 | 0 | 0 | 1 | — | 5 |
| D | 0 | 0 | 0 | 5 | 0 | — | 5 |
| E | 0 | 0 | 0 | 0 | 0 | 5 | 5 |
| B | 7 | 0 | 0 | 0 | 0 | — | 7 |
| C | 4 | 1 | 0 | 0 | 0 | — | 5 |
| D | 2 | 2 | 0 | 0 | 1 | — | 5 |
| E | 5 | 0 | 0 | 0 | 0 | 0 | 5 |
| F | — | — | — | ~10 | — | — | ~10 |
## Maintenance

78
plans/hs-conformance-to-100.md
View File

@@ -61,7 +61,7 @@ Orchestrator cherry-picks worktree commits onto `architecture` one at a time; re
6. **[done (+2)] string template `${x}`** — `expressions/strings / string templates work w/ props` + `w/ braces` (2 tests). Template interpolation isn't substituting property accesses. Check `hs-template` runtime. Expected: +2.
7. **[done (+1) — partial, 3 tests remain: inserted-button handler doesn't fire for afterbegin/innerHTML paths; might need targeted trace of hs-boot-subtree! or _setInnerHTML timing] `put` hyperscript reprocessing** — `put / properly processes hyperscript at end/start/content/symbol` (4 tests, all `Expected 42, got 40`). After a put operation, newly inserted HS scripts aren't being activated. Fix: `hs-put-at!` should `hs-boot-subtree!` on the target after DOM insertion. Expected: +4.
7. **[done (+5)] `put` hyperscript reprocessing** — `put / properly processes hyperscript at end/start/content/symbol` (4 tests, all `Expected 42, got 40`). After a put operation, newly inserted HS scripts aren't being activated. Fix: `hs-put-at!` should `hs-boot-subtree!` on the target after DOM insertion. Expected: +4.
8. **[done (+1)] `select returns selected text`** (1 test, `hs-upstream-select`). Runtime `hs-get-selection` helper reads `window.__test_selection` stash (or falls back to real `window.getSelection().toString()`). Compiler rewrites `(ref "selection")` to `(hs-get-selection)`. Generator detects the `createRange` / `setStart` / `setEnd` / `addRange` block and emits a single `(host-set! ... __test_selection ...)` op with the resolved text slice of the target element. Expected: +1.
@@ -69,7 +69,7 @@ Orchestrator cherry-picks worktree commits onto `architecture` one at a time; re
10. **[done (+1)] `swap` variable ↔ property** — `swap / can swap a variable with a property` (1 test). Swap command doesn't handle mixed var/prop targets. Expected: +1.
11. **[done (+3) — partial, `hide element then show element retains original display` remains; needs `on click N` count-filtered event handlers, out of scope for this cluster] `hide` strategy** — `hide / can configure hidden as default`, `can hide with custom strategy`, `can set default to custom strategy`, `hide element then show element retains original display` (4 tests). Strategy config plumbing. Expected: +3-4.
11. **[done (+4)] `hide` strategy** — `hide / can configure hidden as default`, `can hide with custom strategy`, `can set default to custom strategy`, `hide element then show element retains original display` (4 tests). Strategy config plumbing. Expected: +3-4.
12. **[done (+2)] `show` multi-element + display retention** — `show / can show multiple elements with inline-block`, `can filter over a set of elements using the its symbol` (2 tests). Expected: +2.
@@ -93,7 +93,7 @@ Orchestrator cherry-picks worktree commits onto `architecture` one at a time; re
21. **[done (+1)] `possessiveExpression` property access via its** — `possessive / can access its properties` (1 test, Expected `foo` got ``). Expected: +1.
22. **[blocked: tried three compile-time emits — (1) guard (can't catch Undefined symbol since it's a host-level error, not an SX raise), (2) env-has? (primitive not loaded in HS kernel — `Unhandled exception: "env-has?"`), and (3) hs-win-call runtime helper (works when reached but SX can't CALL a host-handle function directly — `Not callable: {:__host_handle N}` because NativeFn is not callable here). Needs either a host-call-fn primitive with arity-agnostic dispatch OR a symbol-bound? predicate in the HS kernel.] window global fn fallback** — `regressions / can invoke functions w/ numbers in name` + unlocks several others. When calling `foo()` where `foo` isn't SX-defined, fall back to `(host-global "foo")`. Design decision: either compile-time emit `(or foo (host-global "foo"))` via a helper, or add runtime lookup in the dispatch path. Expected: +2-4.
22. **[done (+1)] window global fn fallback** — `regressions / can invoke functions w/ numbers in name` + `can refer to function in init blocks`. Added `host-call-fn` FFI primitive (commit 337c8265), `hs-win-call` runtime helper, simplified compiler emit (direct hs-win-call, no guard), `def` now also registers fn on `window[name]`. Generator: fixed `\"` escaping in hs-compile string literals. Expected: +2-4.
23. **[done (+1)] `me symbol works in from expressions`** — `regressions` (1 test, Expected `Foo`). Check `from` expression compilation. Expected: +1.
@@ -109,21 +109,23 @@ Orchestrator cherry-picks worktree commits onto `architecture` one at a time; re
28. **[done (+4)] `ask`/`answer` + prompt/confirm mock** — `askAnswer` 4 tests. **Requires test-name-keyed mock**: first test wants `confirm → true`, second `confirm → false`, third `prompt → "Alice"`, fourth `prompt → null`. Keyed via `_current-test-name` in the runner. Expected: +4.
29. **[blocked: sx-tree MCP tools returning Yojson Type_error on every file op. Can't edit integration.sx to add before:init/after:init dispatch. Also 4 of the 6 tests fundamentally require stricter parser error-rejection (add - to currently succeeds as SX expression; on click blargh end accepts blargh as symbol), which is larger than a single cluster budget.] `hyperscript:before:init` / `:after:init` / `:parse-error` events** — 6 tests in `bootstrap` + `parser`. Fire DOM events at activation boundaries. Expected: +4-6.
29. **[done (+2) — partial, 4 parser-error tests remain (basic parse error messages, parse-error event, EOF newline crash, evaluate-api-first-error). All require stricter parser error-rejection — `add - to` currently parses silently to `(set! nil (hs-add-to! (- 0 nil) nil))`, `on click blargh end on mouseenter also_bad` parses silently to `(do (hs-on me "click" (fn (event) blargh)) (hs-on me "mouseenter" (fn (event) also_bad)))`. Plus emit-error-collection runtime + hyperscript:parse-error event with detail.errors. Larger than a single cluster budget; recommend bucket-D plan-first.] `hyperscript:before:init` / `:after:init` / `:parse-error` events** — 6 tests in `bootstrap` + `parser`. Fire DOM events at activation boundaries. Expected: +4-6.
30. **[done (+1)] `logAll` config** — 1 test. Global config that console.log's each command. Expected: +1.
### Bucket D: medium features (bigger commits, plan-first)
31. **[pending] runtime null-safety error reporting** — 18 tests in `runtimeErrors`. When accessing `.foo` on nil, emit a structured error with position info. One coordinated fix in the compiler emit paths for property access, function calls, set/put. Expected: +15-18.
31. **[blocked: Bucket-D plan-first scope, doesn't fit one cluster budget. All 18 tests are SKIP (untranslated) — generator has no `error("HS")` helper. Required pieces: (a) generator-side `eval-hs-error` helper + recognizer for `expect(await error("HS")).toBe("MSG")` blocks; (b) runtime helpers `hs-null-error!` / `hs-named-target` / `hs-named-target-list` raising `'<sel>' is null`; (c) compiler patches at every target-position `(query SEL)` emit to wrap in named-target carrying the original selector source — that's ~17 command emit paths (add, remove, hide, show, measure, settle, trigger, send, set, default, increment, decrement, put, toggle, transition, append, take); (d) function-call null-check at bare `(name)`, `hs-method-call`, and `host-get` chains, deriving the leftmost-uncalled-name `'x'` / `'x.y'` from the parse tree; (e) possessive-base null-check (`set x's y to true` → `'x' is null`). Each piece is straightforward in isolation but the cross-cutting compiler change touches every emit path and needs a coordinated design pass. Recommend a dedicated design doc + multi-commit worktree like buckets E36-E40.] runtime null-safety error reporting** — 18 tests in `runtimeErrors`. When accessing `.foo` on nil, emit a structured error with position info. One coordinated fix in the compiler emit paths for property access, function calls, set/put. Expected: +15-18.
32. **[pending] MutationObserver mock + `on mutation` dispatch** — 15 tests in `on`. Add MO mock to runner. Compile `on mutation [of attribute/childList/attribute-specific]`. Expected: +10-15.
32. **[done (+7)] MutationObserver mock + `on mutation` dispatch** — 7 tests in `on`. Add MO mock to runner. Compile `on mutation [of attribute/childList/attribute-specific]`. Expected: +10-15.
33. **[pending] cookie API** — 5 tests in `expressions/cookies`. `document.cookie` mock in runner + `the cookies` + `set the xxx cookie` keywords. Expected: +5.
33. **[done (+4) — partial, 1 test remains: `iterate cookies values work` needs `hs-for-each` to recognise host-array/proxy collections (currently `(list? collection)` returns false for the JS Proxy so the loop body never runs). Out of scope.] cookie API** — 5 tests in `expressions/cookies`. `document.cookie` mock in runner + `the cookies` + `set the xxx cookie` keywords. Expected: +5.
34. **[pending] event modifier DSL** — 8 tests in `on`. `elsewhere`, `every`, `first click`, count filters (`once / twice / 3 times`, ranges), `from elsewhere`. Expected: +6-8.
34. **[done (+7) — partial, 1 test remains: `every` keyword multi-handler-execute test needs handler-queue semantics where `wait for X` doesn't block subsequent invocations of the same handler — current `hs-on-every` shares the same dom-listen plumbing as `hs-on` and queues events implicitly via JS event loop, so the third synthetic click waits for the prior handler's `wait for customEvent` to settle. Out of single-cluster scope.] event modifier DSL** — 8 tests in `on`. `elsewhere`, `every`, `first click`, count filters (`once / twice / 3 times`, ranges), `from elsewhere`. Expected: +6-8.
35. **[pending] namespaced `def`** — 3 tests. `def ns.foo() ...` creates `ns.foo`. Expected: +3.
35. **[done (+3)] namespaced `def`** — 3 tests. `def ns.foo() ...` creates `ns.foo`. Expected: +3.
36b. **[done (+1)] `call` result binds to `it`** — `call / call functions that return promises are waited on` (1 test). `call X then put it into Y` wasn't setting `it` because the `call` compiler branch emitted the call expression directly without `emit-set`. Fixed by wrapping in `emit-set (quote the-result) call-expr`. Expected: +1.
### Bucket E: subsystems (DO NOT LOOP — human-driven)
@@ -131,13 +133,13 @@ All five have design docs on their own worktree branches pending review + merge.
36. **[design-done, pending review — `plans/designs/e36-websocket.md` on `worktree-agent-a9daf73703f520257`] WebSocket + `socket`** — 16 tests. Upstream shape is `socket NAME URL [with timeout N] [on message [as JSON] …] end` with an **implicit `.rpc` Proxy** (ES6 Proxy lives in JS, not SX), not `with proxy { send, receive }` as this row previously claimed. Design doc has 8-commit checklist, +12–16 delta estimate. Ship only with intentional design review.
37. **[design-done, pending review — `plans/designs/e37-tokenizer-api.md` on `worktree-agent-a6bb61d59cc0be8b4`] Tokenizer-as-API** — 17 tests. Expose tokens as inspectable SX data via `hs-tokens-of` / `hs-stream-token` / `hs-token-type` etc; type-map current `hs-tokenize` output to upstream SCREAMING_SNAKE_CASE. 8-step checklist, +16–17 delta.
37. **[done +17]** Tokenizer-as-API — `hs-tokens-of` / `hs-stream-token` / `hs-token-type` / `hs-token-value` / `hs-token-op?`; type-map + normalize; `read-number` dot-stop fix; `\$` template escape in compiler + runtime; generator pattern in `generate-sx-tests.py`. 17/17.
38. **[design-done, pending review — `plans/designs/e38-sourceinfo.md` on `agent-e38-sourceinfo`] SourceInfo API** — 4 tests. Inline span-wrapper strategy (not side-channel dict) with compiler-entry unwrap. 4-commit plan.
39. **[design-done, pending review — `plans/designs/e39-webworker.md` on `hs-design-e39-webworker`] WebWorker plugin** — 1 test. Parser-only stub that errors with a link to upstream docs; no runtime, no mock Worker class. Hand-write the test (don't patch the generator). Single commit.
40. **[design-done, pending review — `plans/designs/e40-real-fetch.md` on `worktree-agent-a94612a4283eaa5e0`] Fetch non-2xx / before-fetch event / real response object** — 7 tests. SX-dict Response wrapper `{:_hs-response :ok :status :url :_body :_json :_html}`; restructured `hs-fetch` that always fetches wrapper then converts by format; test-name-keyed `_fetchScripts`. 11-step checklist. Watch for regression on cluster-1 JSON unwrap.
40. **[done +7 — d7244d1d] Fetch non-2xx / before-fetch event / real response object** — 7 tests. SX-dict Response wrapper `{:_hs-response :ok :status :url :_body :_json :_html}`; restructured `hs-fetch` that always fetches wrapper then converts by format; test-name-keyed `_fetchScripts`. 11-step checklist. Watch for regression on cluster-1 JSON unwrap.
### Bucket F: generator translation gaps (after bucket A-D)
@@ -175,8 +177,62 @@ Many tests are `SKIP (untranslated)` because `tests/playwright/generate-sx-tests
## Progress log
### 2026-04-26 — Bucket F: array literal arg to JS fn (+1)
- **da2e6b1b** — `HS Bucket F: array literal arg to JS fn fix (+1 test)`. Two-part fix: (a) `generate-sx-tests.py` `js_expr_to_sx` now translates `arr.reduce(fn, init)` → `(reduce fn init arr)`, `.map(fn)` → `(map fn arr)`, `.filter(fn)` → `(filter fn arr)` so SX list arguments work with JS array HO methods. (b) `host-call-fn` in `hs-run-filtered.js` adds `sxToJs` recursive converter that unwraps SX list `._type==='list'` to native JS arrays before calling native JS functions. Together these fix functionCalls "can pass an array literal as an argument". Suite hs-upstream-expressions/functionCalls: 8/12 (unchanged SKIP ratio). Test 597: 0/1 → 1/1. Smoke 0-195: 175/195 unchanged.
### 2026-04-26 — Bucket F: hs-make-object _order + assert= for dicts (+1)
- **daea2808** — `HS Bucket F: fix hs-make-object _order + assert= for dicts (+1 test)`. Two-part fix: (a) `runtime.sx` `hs-make-object` no longer appends `_order` key to HS object literals — V8's native string-key insertion order is sufficient, and the hidden key was breaking structural equality. (b) `generate-sx-tests.py` `emit_eval` now detects when `expected_sx` contains `{` (dict syntax) and emits `assert-equal` (which uses `equal?` for deep structural equality) instead of `assert=` (which uses `=`, reference equality for dicts). Together these fix arrayLiteral "arrays containing objects work". Suite hs-upstream-expressions/arrayLiteral: 7/8 → 8/8. Smoke 0-195 unchanged at 175/195.
### 2026-04-26 — Bucket F: empty multi-element fix (+1)
- **875e9ba3** — `HS: empty multi-element fix (+1 test)`. `empty .class` compiled `(empty-target (query ".class"))` through `hs-to-sx` → `(hs-empty-target! (hs-query-first ".class"))` which only emptied the first match. Fix: detect `(query ...)` target in the `empty-target` compiler case and emit `(for-each (fn (_el) (hs-empty-target! _el)) (hs-query-all sel))`, mirroring the `add-class` pattern. Suite hs-upstream-empty: 12/13 → 13/13. Smoke 0-195: 175/195 unchanged.
### 2026-04-26 — Bucket F: add CSS template interpolation (+1)
- **5a76a040** — `HS: add CSS template interpolation fix (+1 test)`. `add {color: ${}{"red"}}` uses two consecutive brace groups: the empty `${}` marker followed by `{"red"}` for the actual value. The prior parser fix called `parse-expr` when already at the closing `}` of the empty group, returning nil. Fix: detect the empty-brace case (`brace-open` → immediately `brace-close`), skip it, then read the actual value from the next `{…}` block. Also handles normal `${expr}` correctly. Suite hs-upstream-add: 17/19 → 18/19. Smoke 0-195: 174/195 → 175/195.
### 2026-04-26 — cluster 36b call result binds to it (done +1)
- **35f498ec** — `hs: call command binds result to it via emit-set (+1 test)`. `call X then put it into Y` compiled `call X` without `emit-set`, so `it` remained nil. Wrapped call-expr in `emit-set (quote the-result) ...` so both `it` and `the-result` are updated. Suite hs-upstream-call: 5/6 → 6/6. Smoke 0-195: 173/195 → 174/195.
### 2026-04-26 — cluster 7 put hyperscript reprocessing (done, final +1)
- **247bd85c** — `hs: register promiseAString/promiseAnInt as sync test fixtures (+1 test)`. Upstream test "waits on promises" calls `promiseAString()` via window global. OCaml run_tests.ml registers these as NativeFns returning "foo"/"42" synchronously; JS runner had no equivalent. Added `globalThis.promiseAString = () => 'foo'` and `globalThis.promiseAnInt = () => 42` to hs-run-filtered.js. Suite hs-upstream-put: 37/38 → 38/38 (fully done). Smoke 0-195: 173/195 unchanged.
### 2026-04-26 — cluster 7 put hyperscript reprocessing (partial +3 more)
- **d663c91f** — `hs: stop event propagation after each hs-on handler fires (+3 tests)`. Root cause: click events bubble from b1 (inside d1) to d1, causing d1's `on click put ...` handler to re-fire and replace the just-modified b1 with fresh content (text=40). Fix: `hs-on`'s wrapped handler now calls `event.stopPropagation()` after each handler runs, preventing the bubbled click from reaching ancestor HS listeners. Tests 1147/1149/1150 now pass. Suite hs-upstream-put: 34/38 → 37/38. Smoke 0-195: 173/195 unchanged. One test remains: "waits on promises" (async/Promise issue).
(Reverse chronological — newest at top.)
### 2026-04-25 — Bucket F: in-expression filter semantics (+1)
- **67a5f137** — `HS: in-expression filter semantics (+1 test)`. `1 in [1, 2, 3]` was returning boolean `true` instead of the filtered list `(list 1)`. Root cause: `in?` compiled to `hs-contains?` which returns boolean for scalar items. Fix: (a) `runtime.sx` adds `hs-in?` returning filtered list for all cases, plus `hs-in-bool?` which wraps with `(not (hs-falsy? ...))` for boolean contexts; (b) `compiler.sx` changes `in?` clause to emit `(hs-in? collection item)` and adds new `in-bool?` clause emitting `(hs-in-bool? collection item)`; (c) `parser.sx` changes `is in` and `am in` comparison forms to produce `in-bool?` so those stay boolean. Suite hs-upstream-expressions/in: 8/9 → 9/9. Smoke 0-195: 173/195 unchanged.
### 2026-04-25 — cluster 22 window global fn fallback (+1)
- **d31565d5** — `HS cluster 22: simplify win-call emit + def→window + init-blocks test (+1)`. Two-part change building on 337c8265 (host-call-fn FFI + hs-win-call runtime). (a) `compiler.sx` removes the guard wrapper from bare-call and method-call `hs-win-call` emit paths — direct `(hs-win-call name (list args))` is sufficient since hs-win-call returns nil for unknown names; `def` compilation now also emits `(host-set! (host-global "window") name fn)` so every HS-defined function is reachable via window lookup. (b) `generate-sx-tests.py` fixes a quoting bug: `\"here\"` was being embedded as three SX nodes (`""` + symbol + `""`) instead of a single escaped-quote string; fixed with `\\\"` escaping. Hand-rolled deftest for `can refer to function in init blocks` now passes. Suite hs-upstream-core/regressions: 13/16 → 14/16. Smoke 0-195: 172/195 → 173/195.
### 2026-04-25 — cluster 11/33 followups: hide strategy + cookie clear (+2)
- **5ff2b706** — `HS: cluster 11/33 followups (+2 tests)`. Three orthogonal fixes that pick up tests now unblocked by earlier work. (a) `parser.sx` `parse-hide-cmd`/`parse-show-cmd`: added `on` to the keyword set that flips the implicit-`me` target. Previously `on click 1 hide on click 2 show` silently parsed as `(hs-hide! nil ...)` because `parse-expr` started consuming `on` and returned nil; now hide/show recognise a sibling feature and default to `me`. (b) `runtime.sx` `hs-method-call` fallback for non-built-in methods: SX-callables (lambdas) call via `apply`, JS-native functions (e.g. `cookies.clear`) dispatch via `(apply host-call (cons obj (cons method args)))` so the native receives the args list. (c) Generator `hs-cleanup!` body wrapped in `begin` (fn body evaluates only the last expr) and now resets `hs-set-default-hide-strategy! nil` + `hs-set-log-all! false` between tests — the prior `can set default to custom strategy` cluster-11 test had been leaking `_hs-default-hide-strategy` into the rest of the suite, breaking `hide element then show element retains original display`. New cluster-33 hand-roll for `basic clear cookie values work` exercises the method-call fallback. Suite hs-upstream-hide: 15/16 → 16/16. Suite hs-upstream-expressions/cookies: 3/5 → 4/5. Smoke 0-195 unchanged at 172/195.
### 2026-04-25 — cluster 35 namespaced def + script-tag globals (+3)
- **122053ed** — `HS: namespaced def + script-tag global functions (+3 tests)`. Two-part change: (a) `runtime.sx` `hs-method-call` gains a fallback for unknown methods — `(let ((fn-val (host-get obj method))) (if (callable? fn-val) (apply fn-val args) nil))`. This lets `utils.foo()` dispatch through `(host-get utils "foo")` when `utils` is an SX dict whose `foo` is an SX lambda. (b) Generator hand-rolls 3 deftests since the SX runtime has no `<script type='text/hyperscript'>` tag boot. For `is called synchronously` / `can call asynchronously`: `(eval-expr-cek (hs-to-sx (first (hs-parse (hs-tokenize "def foo() ... end")))))` registers the function in the global eval env (eval-expr-cek processes `(define foo (fn ...))` at top scope), then a click div is built via dom-set-attr + hs-boot-subtree!. For `functions can be namespaced`: define `utils` as a dict, register `__utils_foo` as a fresh-named global def, then `(host-set! utils "foo" __utils_foo)` populates the dict; click handler `call utils.foo()` compiles to `(hs-method-call utils "foo")` which now dispatches through the new runtime fallback. Skip-list cleared of the 3 def entries. Suite hs-upstream-def: 24/27 → 27/27. Smoke 0-195 unchanged at 172/195.
### 2026-04-25 — cluster 34 elsewhere / from-elsewhere modifier (+2)
- **3044a168** — `HS: elsewhere / from elsewhere modifier (+2 tests)`. Three-part change: (a) `parser.sx` `parse-on-feat` parses an optional `elsewhere` (or `from elsewhere`) modifier between event-name and source. The `from elsewhere` variant uses a one-token lookahead so plain `from #target` keeps parsing as a source expression. Emits `:elsewhere true` part. (b) `compiler.sx` `scan-on` threads `elsewhere?` (10th param) through every recursive call + new `:elsewhere` cond branch. The dispatch case becomes a 3-way `cond` over target: elsewhere → `(dom-body)` (listener attaches to body and bubble sees every click), source → from-source, default → `me`. The `compiled-body` build is wrapped with `(when (not (host-call me "contains" (host-get event "target"))) BODY)` so handlers fire only on outside-of-`me` clicks. (c) Generator drops `supports "elsewhere" modifier` and `supports "from elsewhere" modifier` from `SKIP_TEST_NAMES`. Suite hs-upstream-on: 48/70 → 50/70. Smoke 0-195 unchanged at 172/195.
### 2026-04-25 — cluster 34 count-filtered events + first modifier (+5 partial)
- **19c97989** — `HS: count-filtered events + first modifier (+5 tests)`. Three-part change: (a) `parser.sx` `parse-on-feat` accepts `first` keyword before event-name (sets `cnt-min/max=1`), then optionally parses a count expression after event-name: bare number = exact count, `N to M` = inclusive range, `N and on` = unbounded above. Number tokens coerced via `parse-number`. New parts entry `:count-filter {"min" N "max" M-or--1}`. (b) `compiler.sx` `scan-on` gains a 9th `count-filter-info` param threaded through every recursive call + a new `:count-filter` cond branch. The handler binding now wraps the `(fn (event) BODY)` in `(let ((__hs-count 0)) (fn (event) (begin (set! __hs-count (+ __hs-count 1)) (when COUNT-CHECK BODY))))` when count info is present. Each `on EVENT N ...` clause produces its own closure-captured counter, so `on click 1` / `on click 2` / `on click 3` fire on their respective Nth click (mix-ranges test). (c) Generator drops 5 entries from `SKIP_TEST_NAMES` — `can filter events based on count`/`...count range`/`...unbounded count range`/`can mix ranges`/`on first click fires only once`. Suite hs-upstream-on: 43/70 → 48/70. Smoke 0-195 unchanged at 172/195. Remaining cluster-34 work (`elsewhere`/`from elsewhere`/`every`-keyword multi-handler) is independent from count filters and would need a separate iteration.
### 2026-04-25 — cluster 29 hyperscript init events (+2 partial)
- **e01a3baa** — `HS: hyperscript:before:init / :after:init events (+2 tests)`. `integration.sx` `hs-activate!` now wraps the activation block in `(when (dom-dispatch el "hyperscript:before:init" nil) ...)` — `dom-dispatch` builds a CustomEvent with `bubbles:true`, the mock El's `cancelable` defaults to true, `dispatchEvent` returns `!ev.defaultPrevented`, so `when` skips the activate body if a listener called `preventDefault()`. After activation completes successfully it dispatches `hyperscript:after:init`. Generator (`tests/playwright/generate-sx-tests.py`) gains two hand-rolled deftests: `fires hyperscript:before:init and hyperscript:after:init` builds a wa container, attaches listeners that append to a captured `events` list, sets innerHTML to a div with `_=`, calls `hs-boot-subtree!`, asserts the events list. `hyperscript:before:init can cancel initialization` attaches a preventDefault listener and asserts `data-hyperscript-powered` is absent on the inner div after boot. Suite hs-upstream-core/bootstrap: 20/26 → 22/26. Smoke 0-195: 170 → 172. Remaining 4 cluster-29 tests (basic parse error messages, parse-error event, EOF newline, eval-API throws on first error) all need stricter parser error-rejection plus a parse-error collector — recommend bucket-D plan-first multi-commit, not a single iteration.
### 2026-04-25 — cluster 32 MutationObserver mock + on mutation dispatch (+7)
- **13e02542** — `HS: MutationObserver mock + on mutation dispatch (+7 tests)`. Five-part change: (a) `parser.sx` `parse-on-feat` now consumes `of <FILTER>` after `mutation` event-name. FILTER is one of `attributes`/`childList`/`characterData` (ident tokens) or one or more `@name` attr-tokens chained by `or`. Emits `:of-filter {"type" T "attrs" L?}` part. (b) `compiler.sx` `scan-on` threads new `of-filter-info` param; the dispatch case becomes a `cond` over `event-name` — for `"mutation"` it emits `(do on-call (hs-on-mutation-attach! target MODE ATTRS))` where ATTRS is `(cons 'list attr-list)` so the list survives compile→eval. (c) `runtime.sx` `hs-on-mutation-attach!` builds a config dict (`attributes`/`childList`/`characterData`/`subtree`/`attributeFilter`) matched to mode, constructs a real `MutationObserver(cb)`, calls `mo.observe(target, opts)`, and the cb dispatches a `"mutation"` event on target. (d) `tests/hs-run-filtered.js` replaces the no-op MO with `HsMutationObserver` (global registry, decodes SX-list `attributeFilter`); prototype hooks on `El.setAttribute/appendChild/removeChild/_setInnerHTML` fire matching observers synchronously, with `__hsMutationActive` re-entry guard so handlers that mutate the DOM don't infinite-loop. Per-test reset clears registry + flag. (e) `generate-sx-tests.py` drops 7 mutation entries from `SKIP_TEST_NAMES` and adds two body patterns: `evaluate(() => document.querySelector(SEL).setAttribute(N,V))` → `(dom-set-attr ...)`, and `evaluate(() => document.querySelector(SEL).appendChild(document.createElement(T)))` → `(dom-append … (dom-create-element …))`. Suite hs-upstream-on: 36/70 → 43/70. Smoke 0-195 unchanged at 170/195.
### 2026-04-25 — cluster 33 cookie API (partial +3)
- No `.sx` edits needed — `set cookies.foo to 'bar'` already compiles to `(dom-set-prop cookies "foo" "bar")` which becomes `(host-set! cookies "foo" "bar")` once the `dom` module is loaded, and `cookies.foo` becomes `(host-get cookies "foo")`. So a JS-only Proxy + Python generator change does the trick. Two parts: (a) `tests/hs-run-filtered.js` adds a per-test `__hsCookieStore` Map, a `globalThis.cookies` Proxy with `length`/`clear`/named-key get traps and a set trap that writes the store, and a `Object.defineProperty(document, 'cookie', …)` getter/setter that reads and writes the same store (so the upstream `length is 0` test's pre-clear loop over `document.cookie` works). Per-test reset clears the store. (b) `tests/playwright/generate-sx-tests.py` declares `(define cookies (host-global "cookies"))` in the test header and emits hand-rolled deftests for the three tractable tests (`basic set`, `update`, `length is 0`). Suite hs-upstream-expressions/cookies: 0/5 → 3/5. Smoke 0-195 unchanged at 170/195. Remaining `basic clear` and `iterate` tests need runtime.sx edits (hs-method-call fallback + hs-for-each host-array recognition) — out of scope for a JS-only iteration.
### 2026-04-25 — cluster 32 MutationObserver mock + on mutation dispatch (blocked)
- Two issues conspire: (1) `loops/hs` worktree has no pre-built sx-tree binary so MCP tools aren't loaded, and the block-sx-edit hook prevents raw `Edit`/`Read`/`Write` on `.sx` files. Built `hosts/ocaml/_build/default/bin/mcp_tree.exe` via `dune build` this iteration but tools don't surface mid-session. (2) Cluster scope is genuinely big: parser must learn `on mutation of <filter>` (currently drops body after `of` — verified via compile dump: `on mutation of attributes put "Mutated" into me` → `(hs-on me "mutation" (fn (event) nil))`), compiler needs `:of-filter` plumbing similar to intersection's `:having`, runtime needs `hs-on-mutation-attach!`, JS runner mock needs a real MutationObserver (currently no-op `class{observe(){}disconnect(){}}` at hs-run-filtered.js:348) plus `setAttribute`/`appendChild` instrumentation, and 7 entries removed from `SKIP_TEST_NAMES`. Recommended next step: dedicated worktree where sx-tree loads at session start, multi-commit shape (parser → compiler+attach → mock+runner → generator skip-list).
### 2026-04-25 — cluster 31 runtime null-safety error reporting (blocked)
- All 18 tests are `SKIP (untranslated)` — generator has no `error("HS")` helper at all. Inspected representative compile outputs: `add .foo to #doesntExist` → `(for-each ... (hs-query-all "#doesntExist"))` (silently no-ops on empty list, no error); `hide #doesntExist` → `(hs-hide! (hs-query-all "#doesntExist") "display")` (likewise); `put 'foo' into #doesntExist` → `(hs-set-inner-html! (hs-query-first "#doesntExist") "foo")` (passes nil through); `x()` → `(x)` (raises `Undefined symbol: x`, wrong format); `x.y.z()` → `(hs-method-call (host-get x "y") "z")`. Implementing this requires generator helper + 17 compiler emit-path patches + function-call/method-call/possessive-base null guards + new `hs-named-target`/`hs-named-target-list` runtime — too many surfaces for a single-iteration commit. Bucket D explicitly says "plan-first" — recommended path is a dedicated design doc and multi-commit worktree like E36-E40, not a loop iteration.
### 2026-04-24 — cluster 29 hyperscript:before:init / :after:init / :parse-error (blocked)
- **2b486976** — `HS-plan: mark cluster 29 blocked`. sx-tree MCP file ops returning `Yojson__Safe.Util.Type_error("Expected string, got null")` on every file-based call (sx_read_subtree, sx_find_all, sx_replace_by_pattern, sx_summarise, sx_pretty_print, sx_write_file). Only in-memory ops work (sx_eval, sx_build, sx_env). Without sx-tree I can't edit integration.sx to add before:init/after:init dispatch on hs-activate!. Investigated the 6 tests: 2 bootstrap (before/after init) need dispatchEvent wrapping activate; 4 parser tests require stricter parser error-rejection — `add - to` currently parses silently to `(set! nil (hs-add-to! (- 0 nil) nil))`, `on click blargh end on mouseenter also_bad` parses silently to `(do (hs-on me "click" (fn (event) blargh)) (hs-on me "mouseenter" (fn (event) also_bad)))`. Fundamental parser refactor is out of single-cluster budget regardless of sx-tree availability.

16
plans/js-on-sx.md
View File

@@ -125,7 +125,7 @@ Each item: implement → tests → update progress. Mark `[x]` when tests green.
- [x] Rest params (`...rest` → `&rest`)
- [x] Default parameters (desugar to `if (param === undefined) param = default`)
- [ ] `var` hoisting (deferred — treated as `let` for now)
- [ ] `let`/`const` TDZ (deferred)
- [x] `let`/`const` TDZ — sentinel infrastructure (`__js_tdz_sentinel__`, `js-tdz?`, `js-tdz-check` in runtime.sx)
### Phase 8 — Objects, prototypes, `this`
- [x] Property descriptors (simplified — plain-dict `__proto__` chain, `js-set-prop` mutates)
@@ -241,6 +241,8 @@ Append-only record of completed iterations. Loop writes one line per iteration:
- 29× Timeout (slow string/regex loops)
- 16× ReferenceError — still some missing globals
- 2026-04-25 — **Regex engine (lib/js/regex.sx) + let/const TDZ infrastructure.** New file `lib/js/regex.sx`: 39-form pure-SX recursive backtracking engine installed via `js-regex-platform-override!`. Covers literals, `.`, `\d\w\s` + negations, `[abc]/[^abc]/[a-z]` char classes, `^\$\b\B` anchors, greedy+lazy quantifiers (`* + ? {n,m} *? +? ??`), capturing groups, non-capturing `(?:...)`, alternation `a|b`, flags `i`/`g`/`m`. Groups: match inner first → set capture → match rest (correct boundary), avoids including rest-nodes content in capture. Greedy: expand-first then backtrack (correct longest-match semantics). `js-regex-match-all` for String.matchAll. Fixed `String.prototype.match` to use platform engine (was calling stub). TDZ infrastructure added to `runtime.sx`: `__js_tdz_sentinel__` (unique sentinel dict), `js-tdz?`, `js-tdz-check`. `transpile.sx` passes `kind` through `js-transpile-var → js-vardecl-forms` (no behavioral change yet — infrastructure ready). `test262-runner.py` and `conformance.sh` updated to load `regex.sx` as epoch 6/50. Unit: **559/560** (was 522/522 before regex tests added, now +38 new tests; 1 pre-existing backtick failure). Conformance: **148/148** (unchanged). Gotchas: (1) `sx_insert_near` on a pattern inside a top-level function body inserts there (not at top level) — need to use `sx_insert_near` on a top-level symbol name. (2) Greedy quantifier must expand-first before trying rest-nodes; the naive "try rest at each step" produces lazy behavior. (3) Capturing groups must match inner nodes in isolation first (to get the group's end position) then match rest — appending inner+rest-nodes would include rest in the capture string.
## Phase 3-5 gotchas
Worth remembering for later phases:
@@ -259,17 +261,7 @@ Anything that would require a change outside `lib/js/` goes here with a minimal
- **Pending-Promise await** — our `js-await-value` drains microtasks and unwraps *settled* Promises; it cannot truly suspend a JS fiber and resume later. Every Promise that settles eventually through the synchronous `resolve`/`reject` + microtask path works. A Promise that never settles without external input (e.g. a real `setTimeout` waiting on the event loop) would hit the `"await on pending Promise (no scheduler)"` error. Proper async suspension would need the JS eval path to run under `cek-step-loop` (not `eval-expr` → `cek-run`) and treat `await pending-Promise` as a `perform` that registers a resume thunk on the Promise's callback list. Non-trivial plumbing; out of scope for this phase. Consider it a Phase 9.5 item.
- **Regex platform primitives** — runtime ships a substring-based stub (`js-regex-stub-test` / `-exec`). Overridable via `js-regex-platform-override!` so a real engine can be dropped in. Required platform-primitive surface:
- `regex-compile pattern flags` — build an opaque compiled handle
- `regex-test compiled s` → bool
- `regex-exec compiled s` → match dict `{match index input groups}` or nil
- `regex-match-all compiled s` → list of match dicts (or empty list)
- `regex-replace compiled s replacement` → string
- `regex-replace-fn compiled s fn` → string (fn receives match+groups, returns string)
- `regex-split compiled s` → list of strings
- `regex-source compiled` → string
- `regex-flags compiled` → string
Ideally a single `(js-regex-platform-install-all! platform)` entry point the host calls once at boot. OCaml would wrap `Str` / `Re` or a dedicated regex lib; JS host can just delegate to the native `RegExp`.
- ~~**Regex platform primitives**~~ **RESOLVED** — `lib/js/regex.sx` ships a pure-SX recursive backtracking engine. Installs via `js-regex-platform-override!` at load. Covers: literals, `.`, `\d\w\s` and negations, `[abc]` / `[^abc]` / ranges, `^` `$` `\b \B`, `* + ? {n,m}` (greedy + lazy), capturing + non-capturing groups, alternation `a|b`, flags `i` (case-insensitive), `g` (global, advances lastIndex), `m` (multiline anchors). `js-regex-match-all` for String.matchAll. String.prototype.match regex path updated to use platform engine (was calling stub). 34 new unit tests added (5000–5033). Conformance: 148/148 (unchanged — slice had no regex fixtures).
- **Math trig + transcendental primitives missing.** The scoreboard shows 34× "TypeError: not a function" across the Math category — every one a test calling `Math.sin/cos/tan/log/…` on our runtime. We shim `Math` via `js-global`; the SX runtime supplies `sqrt`, `pow`, `abs`, `floor`, `ceil`, `round` and a hand-rolled `trunc`/`sign`/`cbrt`/`hypot`. Nothing else. Missing platform primitives (each is a one-line OCaml/JS binding, but a primitive all the same — we can't land approximation polynomials from inside the JS shim, they'd blow `Math.sin(1e308)` precision):
- Trig: `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `atan2`

229
plans/koka-on-sx.md Normal file
View File

@@ -0,0 +1,229 @@
# Koka-on-SX: Koka on the CEK/VM
Implement a Koka interpreter on SX. The unique angle: Koka's algebraic effects and
handlers map directly onto SX's `perform`/`cek-resume` machinery — this is the language
that will stress-test whether SX's effect system is principled enough, and expose any
gaps. Every other language in the set works around effects ad-hoc; Koka makes them the
primary abstraction.
End-state goal: **core Koka programs running on the SX CEK evaluator**, with algebraic
effect handlers wired through `perform`/`cek-resume`. Not a full Koka compiler — no type
inference, no row-polymorphic effect types, no LLVM backend — but a faithful runtime for
idiomatic Koka programs.
## What Koka adds that nothing else covers
- **Structured effect declarations**: `effect state<s> { fun get() : s; fun set(s) : () }`
— named, typed effect operations, not just untyped `perform` tokens
- **Resumable handlers**: `handler { return(x) -> x; get() -> resume(0); set(x) -> resume(()) }`
— multi-shot continuations, handlers as first-class values
- **Effect polymorphism**: functions declare their effect set (`a -> <state<int>,console> b`)
— exposes whether SX can track which effects are in scope
- **Tail-resumptive handlers**: most practical handlers resume exactly once, which should
be optimisable — tests whether the CEK machine can detect and collapse this
- **Algebraic data types as the foundation**: `type maybe<a> { Nothing; Just(value: a) }`
— exercises the Phase 6 ADT primitive directly
## Ground rules
- **Scope:** only touch `lib/koka/**` and `plans/koka-on-sx.md`. Do **not** edit `spec/`,
`hosts/`, `shared/`, or other `lib/<lang>/`.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** Koka source → Koka AST → interpret directly via CEK. No separate
Koka evaluator — host the semantics in SX, run on the existing CEK machine.
- **Effect types:** defer type inference entirely. Track effects at runtime only — an
unhandled effect at the top level raises a runtime error, not a type error.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
## Architecture
```
Koka source text
│
▼
lib/koka/tokenizer.sx — keywords, operators, indent-sensitivity, type-level syntax
│
▼
lib/koka/parser.sx — Koka AST: fun, val, effect, handler, with, match, resume,
│ return clause, ADT definitions, basic type expressions
▼
lib/koka/eval.sx — Koka AST → CEK evaluation via SX primitives:
│ ADT (define-type/match from Phase 6)
│ Effects (perform/cek-resume from spec/evaluator.sx)
│ Coroutines optional (Phase 4 primitives)
▼
SX CEK evaluator (both JS and OCaml hosts)
```
Key semantic mappings:
| Koka construct | SX mapping |
|---------------|-----------|
| `fun f(x) body` | `(define (f x) body)` |
| `val x = expr` | `(let ((x expr)) ...)` |
| `effect E { fun op() : t }` | register effect tag `E/op` in effect env |
| `op()` inside handler scope | `(perform (list "E" "op" args))` |
| `handler { return(x)->e; op()->resume(v) }` | `(guard ...)` + `cek-resume` |
| `with handler { body }` | install handler for duration of body |
| `match x { Nothing -> e1; Just(v) -> e2 }` | SX `(match x ...)` via Phase 6 ADT |
| `type maybe<a> { Nothing; Just(value:a) }` | `(define-type maybe (Nothing) (Just value))` |
| `resume(v)` in handler | `(cek-resume k v)` where k is captured continuation |
| `return(x) -> expr` | final-value clause when no effect fires |
## Koka semantics in brief
### Effects and handlers
```koka
effect console
fun println(s : string) : ()
fun greet(name : string) : <console> ()
println("Hello, " ++ name)
fun main()
with handler
return(x) -> x
println(s) -> { print-string(s ++ "\n"); resume(()) }
greet("world")
```
- `effect console` declares an effect with one operation `println`
- `greet` uses `console` — any call to `println` inside will look up the nearest
enclosing handler
- `with handler { ... }` installs a handler; `resume(())` continues the suspended
computation
### Multi-shot resumption
```koka
effect choice
fun choose() : bool
fun xor(p : bool, q : bool) : <choice> bool
val a = choose()
val b = choose()
(a || b) && !(a && b)
fun all-results()
with handler
return(x) -> [x]
choose() -> resume(True) ++ resume(False)
xor(True, False)
// => [True, True, False, True]
```
This is the test that exposes whether `cek-resume` supports multi-shot (calling the
same continuation twice). SX's delimited continuations do support this — Koka will
verify it end-to-end.
## Roadmap
### Phase 1 — Tokenizer + parser (core expressions)
- [ ] Tokenizer: keywords (`fun`, `val`, `effect`, `handler`, `with`, `match`, `return`,
`resume`, `type`, `alias`, `if`, `then`, `else`, `fn`), operators (`++`, `->`,
`|>`, `:`, `<`, `>`, `,`), identifiers, numbers, strings, booleans
- [ ] Parser — expressions:
- literals: int, float, bool (`True`/`False`), string
- `val x = e` bindings
- `fun f(x, y) body` definitions
- `if c then e1 else e2`
- `match x { Pat -> e; ... }`
- lambda `fn(x) -> e`
- function application `f(x, y)`
- infix operators: `++`, `+`, `-`, `*`, `/`, `==`, `!=`, `<`, `>`, `&&`, `||`
- pipe `|>`: `x |> f` = `f(x)`
- [ ] Tests: `lib/koka/tests/parse.sx` — 40+ parse round-trip tests
### Phase 2 — ADT definitions + match
- [ ] Parser: `type name<a> { Ctor1; Ctor2(field: t); ... }` declarations
- [ ] Eval: map to SX `define-type` + `match` (requires Phase 6 primitives)
- [ ] Built-in: `maybe<a>` (Nothing / Just), `result<a,e>` (Ok / Error), `list<a>` (Nil / Cons)
- [ ] Tests: ADT construction, matching, nested patterns — 25+ tests
### Phase 3 — Core evaluator
- [ ] `koka-eval` entry: walks Koka AST, evaluates in SX env
- [ ] Arithmetic, string `++`, comparison, boolean ops
- [ ] `val`/`let` binding
- [ ] Function definitions and application (first-class functions)
- [ ] `if`/`then`/`else`
- [ ] `match` with constructor, literal, variable, wildcard patterns
- [ ] Basic list ops: `map`, `filter`, `foldl`, `length`, `head`, `tail`
- [ ] Tests: `lib/koka/tests/eval.sx` — 40+ tests, pure expressions only
### Phase 4 — Effect system
- [ ] Effect declaration: `(koka-declare-effect! "console" (list "println"))`
registers operations in a global effect registry
- [ ] Effect operation call: when `println(s)` is evaluated inside a handler scope,
emit `(perform (list :effect "console" :op "println" :args (list s)))`
- [ ] Handler installation: `with handler { return(x)->e; println(s)->resume(v) }`
installs a `guard`-like frame that catches `perform` signals matching the effect,
binds arguments, and exposes `resume` as a callable that invokes `cek-resume`
- [ ] `resume(v)`: calls `(cek-resume captured-k v)` where `captured-k` is the
continuation captured at the `perform` point
- [ ] `return(x) -> e` clause: handles the normal return value when no effect fires
- [ ] Tests: `lib/koka/tests/effects.sx` — 30+ tests:
- basic handler (state, console, exception)
- unhandled effect → runtime error
- nested handlers (inner shadows outer)
- multi-shot resumption (choice effect — the key test)
- tail-resumptive handler (resumes exactly once — verify no extra allocation)
### Phase 5 — Standard effect library
- [ ] `console` effect: `println`, `print`, `readline` (mock)
- [ ] `exn` effect: `throw`, `catch` wrappers
- [ ] `state<s>` effect: `get`, `set`, `modify`
- [ ] `async` effect: `await` mapped to SX `perform` IO suspension
- [ ] Tests: programs using each stdlib effect — 20+ tests
### Phase 6 — Classic Koka programs as integration tests
- [ ] `counter.koka` — stateful counter via state effect
- [ ] `choice.koka` — multi-shot choice generating all results
- [ ] `iterator.koka` — yield-based iteration via a custom effect
- [ ] `exception.koka` — structured exception handling
- [ ] `coroutine.koka` — producer/consumer via two interleaved effects
- [ ] Each as a self-contained test in `lib/koka/tests/programs.sx`
## Key blockers / dependencies
- **Phase 6 ADT primitive** (`define-type`/`match`) — required before Phase 2.
Track: `plans/agent-briefings/primitives-loop.md` Phase 6.
- **Multi-shot continuations** — `cek-resume` must support calling the same
continuation multiple times. Verify with: `(let ((k #f)) (perform 'x) ...)` called
twice. This should already work given the multi-shot delimited continuation work.
- **Effect handler stack** — SX's `guard` is not quite the right primitive for
deep-handler semantics. May need `(with-handler effect-tag handler-fn body)` as a
new evaluator form, or can be emulated via `guard` + `perform` reshaping.
## Comparison to other languages in the set
| Language | Effect model |
|----------|-------------|
| Lua | none (errors only) |
| Prolog | none (cuts only) |
| Erlang | message-passing (not algebraic) |
| Haskell | IO monad (monadic, not algebraic) |
| JS | promise/async-await (one-shot) |
| Ruby | exceptions + fibers |
| **Koka** | **algebraic effects + multi-shot handlers** |
Koka is the only language that uses SX's effect system as its *primary* computational
model. It will expose whether `perform`/`cek-resume` is sufficient or needs typed effect
tagging, scoping rules, and a handler stack distinct from `guard`.
## Progress log
_Newest first._
- _(none yet)_
## Blockers
- ADT primitive (Phase 6 of primitives-loop) must land before Phase 2 starts.

178
plans/lib-guest.md Normal file
View File

@@ -0,0 +1,178 @@
# lib/guest — shared toolkit for SX-hosted languages
Extract the duplicated plumbing across `lib/{haskell,common-lisp,erlang,prolog,js,lua,smalltalk,tcl,forth,ruby,apl,hyperscript}` into a small, composable kit so language N+1 costs ~200 lines instead of ~2000, without regressing any existing conformance scoreboard.
Branch: `architecture`. SX files via `sx-tree` MCP only. Never edit generated files.
## Thesis
The substrate (CEK, hygienic macros, records, delimited continuations, IO suspension, reactivity) was chosen with multi-paradigm hosting in mind, but each guest currently re-rolls its own tokeniser, recursive-descent loop, conformance harness, and primitive-rename layer. Extracting these shared layers does not reduce conformance bug-finding pressure — it only removes plumbing — so it is pure win.
**Canaries:** Lua (small, conventional expression-grammar — exercises lex/Pratt/AST) and Prolog (paradigm-different — exercises pattern-match/unification). The two-canary rule prevents Lua-shaped abstractions.
**Two-language rule:** no extraction is merged until **two** guests consume it.
## Current baseline
The loop fills these in on its first iteration by running every `*/conformance.sh` and `*/test.sh` and copying each `scoreboard.json` to `lib/guest/baseline/<lang>.json`. Until then:
| Guest | Suite | Baseline |
|--------------|--------------------|----------|
| lua | `bash lib/lua/test.sh` | 185 / 185 |
| prolog | `bash lib/prolog/conformance.sh` | 590 / 590 |
| haskell | `bash lib/haskell/conformance.sh` | 156 / 156 (was reported 0/18 by the buggy old script) |
| common-lisp | `bash lib/common-lisp/conformance.sh` | 518 / 518 (Phase 2 +182 and Phase 6 +27 were previously under-counted) |
| erlang | `bash lib/erlang/conformance.sh` | 0 / 0 (suite all-zero) |
| js | `bash lib/js/conformance.sh` | 94 / 148 (test262-slice) |
| smalltalk | `bash lib/smalltalk/conformance.sh` | 625 / 629 |
| tcl | `bash lib/tcl/conformance.sh` | 3 / 4 (programs) |
| forth | `bash lib/forth/test.sh` | 64 / 64 |
| ruby | `bash lib/ruby/test.sh` | 76 / 76 |
| apl | `bash lib/apl/test.sh` | 73 / 73 |
The baseline only needs to be re-snapshotted when the substrate (`spec/**`, `hosts/**`) changes underneath this loop.
---
## Phase 0 — Baseline snapshot (one-shot)
### Step 0: Snapshot every guest's scoreboard
Create `lib/guest/baseline/`. Run every guest's conformance/test runner. Copy each `scoreboard.json` (or extract pass/fail counts from `test.sh` output for guests without a scoreboard) into `lib/guest/baseline/<lang>.json`. Fill in the table above.
**Verify:** `ls lib/guest/baseline/*.json` shows one per guest. Plan table populated.
---
## Phase 1 — Cheap, zero-semantic-risk extractions
### Step 1: `lib/guest/conformance.sx` — config-driven test runner
Replace the 6+ near-identical `*/conformance.sh` scripts with one driver that takes a config dict:
```
{:lang "prolog"
:loads ("lib/prolog/tokenizer.sx" "lib/prolog/parser.sx" ...)
:suites (("parse" "lib/prolog/tests/parse.sx" "pl-parse-tests-run!") ...)}
```
The driver locates `sx_server.exe`, runs the epoch protocol, collects pass/fail per suite, and writes `scoreboard.{json,md}`. The per-language `conformance.sh` becomes a 3-line stub that points at its config.
**Port to:** `lib/prolog/conformance.sh` and `lib/haskell/conformance.sh`. Two consumers required for merge.
**Verify:** both `bash lib/prolog/conformance.sh` and `bash lib/haskell/conformance.sh` produce scoreboard JSONs equal to baseline.
### Step 2: `lib/guest/prefix.sx` — prefix-rename macro
One macro that takes a prefix and a list of SX symbols and binds prefixed aliases:
```
(prefix-rename "cl-" '(null? pair? even? odd? zero? ...))
```
Replaces hundreds of hand-written `(define (cl-null? x) (= x nil))`-style wrappers in `common-lisp/runtime.sx`, `lua/runtime.sx`, `erlang/runtime.sx`.
**Port to:** `common-lisp/runtime.sx` (largest user) and `lua/runtime.sx`. Two consumers.
**Verify:** common-lisp + lua scoreboards equal baseline.
---
## Phase 2 — Lex / parse kit
### Step 3: `lib/guest/lex.sx` — character-class + tokeniser primitives
- Source-position tracking (line/col/offset).
- Character-class predicates (`whitespace?`, `digit?`, `alpha?`, `ident-start?`, `ident-rest?`).
- Number recognisers (decimal, hex, float, scientific).
- String recognisers (quoted, escapes, raw).
- Comment recognisers (line, block, nestable).
- Token record `{:type :value :pos :end :line}`.
**Port to:** `lua/tokenizer.sx` and `tcl/tokenizer.sx`. Two consumers.
**Verify:** lua + tcl scoreboards equal baseline.
### Step 4: `lib/guest/pratt.sx` — Pratt / operator-precedence parser
Prefix / infix / postfix tables, left/right associativity, precedence climbing. Grammar is a dict, not hardcoded `cond`.
**Port to:** Lua expression parser (`lua/parser.sx`) and Prolog operator table (`prolog/parser.sx` — Prolog ops are the stress test). Two consumers.
**Verify:** lua + prolog scoreboards equal baseline.
### Step 5: `lib/guest/ast.sx` — canonical AST node shapes
Standard constructors and predicates for: `literal`, `var`, `app`, `lambda`, `let`, `letrec`, `if`, `match-clause`, `module`, `import`. Optional — guests may keep their own AST — but using the canonical shape lets cross-language tooling (formatters, highlighters, debuggers) work without per-language adapters.
**Port to:** lua + prolog AST emitters. Two consumers.
**Verify:** lua + prolog scoreboards equal baseline.
---
## Phase 3 — Semantic extractions (highest leverage, highest risk)
### Step 6: `lib/guest/match.sx` — pattern-match + unification engine
Single engine for:
- Literal patterns (numbers, strings, symbols, nil, booleans).
- Wildcard `_`.
- Constructor patterns (ADT-shaped — depends on Phase 3 of `sx-improvements.md` if available, otherwise dict-tagged).
- Variable binding.
- **Unification** (Prolog flavour): symmetric, occurs-check toggle, substitution returned.
- **Match** (Haskell flavour): asymmetric pattern→value, bindings returned.
**Port to:** `haskell/match.sx` and `prolog/query.sx` unification core. Two consumers.
**Verify:** haskell + prolog scoreboards equal baseline. **Highest-risk extraction** — if either regresses by 1 test, revert and redesign.
### Step 7: `lib/guest/layout.sx` — significant-whitespace / off-side rule
Generalised layout-sensitive lexer. Configurable: which keywords open layout blocks, whether semicolons are inserted, brace insertion rules.
**Port to:** `haskell/layout.sx` (existing). Second consumer: write a synthetic test fixture that exercises a Python-ish layout to prove the kit is not Haskell-shaped. Two consumers.
**Verify:** haskell scoreboard equal baseline; synthetic layout fixture passes.
### Step 8: `lib/guest/hm.sx` — Hindley-Milner type inference
Extract from `haskell/infer.sx`. Algorithm W or J, generalisation, instantiation, occurs-check, principal types.
**Sequencing:** this step is **paired with `plans/ocaml-on-sx.md` Phase 5**. The natural order is lib-guest Steps 0–7 → OCaml-on-SX Phases 1–5 → lib-guest Step 8. With OCaml-on-SX Phase 5 done, the two-language rule is satisfied for real (Haskell + OCaml). Without it, accept "second user TBD" — the alternative is letting the inference stay locked inside Haskell forever.
**Port to:** `haskell/infer.sx` and (preferred) `lib/ocaml/types.sx`.
**Verify:** haskell scoreboard equal baseline; if OCaml-on-SX Phase 5 has shipped, OCaml type-inference tests equal baseline too.
---
## Progress log
| Step | Status | Commit | Delta |
|------|--------|--------|-------|
| 0 — baseline snapshot | [done] | 2f7f8189 | 11 guests captured: lua 185/185, forth 64/64, ruby 76/76, apl 73/73, prolog 590/590, common-lisp 309/309, smalltalk 625/629, tcl 3/4, haskell 0/18 programs, js 94/148 (slice), erlang 0/0 |
| 1 — conformance.sx (prolog + haskell) | [done] | 58dcff26 | Prolog 590/590 (matches baseline). Haskell 156/156 — old script was broken (0/18 was an artefact of a never-matching grep), driver reveals true counts; baseline updated. |
| 2 — prefix.sx (common-lisp + lua) | [partial — pending lua] | 2ef773a3 | common-lisp/runtime.sx ported (47 aliases collapsed into 13 prefix-rename calls); 518/518 vs 309/309 baseline (improvement, no regression). lua/runtime.sx has no pure same-name aliases — every lua- definition wraps custom logic; second consumer pending. |
| 3 — lex.sx (lua + tcl) | [done] | 559b0df9 | lex.sx exports nil-safe char-class predicates + token record. lua/tokenizer.sx (7 preds) and tcl/tokenizer.sx (5 preds) collapsed into prefix-rename calls. lua 185/185, tcl 342/342, tcl-conf 3/4 — all = baseline. |
| 4 — pratt.sx (lua + prolog) | [done] | da27958d | Extracted operator-table format + lookup only — climbing loops stay per-language because lua and prolog use opposite prec conventions. lua/parser.sx: 18-clause cond → 15-entry table. prolog/parser.sx: pl-op-find deleted, pl-op-lookup wraps pratt-op-lookup. lua 185/185, prolog 590/590 — both = baseline. |
| 5 — ast.sx (lua + prolog) | [partial — pending real consumers] | a774cd26 | Kit + 33 self-tests shipped (10 canonical kinds, predicates, accessors). Step is "Optional" per brief; lua/prolog parsers untouched (185/185 + 590/590). Datalog-on-sx will be the natural first real consumer; lua/prolog converters can land later. |
| 6 — match.sx (haskell + prolog) | [partial — kit shipped; ports deferred] | 863e9d93 | Pure-functional unify + match kit (canonical wire format + cfg-driven adapters) + 25 self-tests. Existing prolog/haskell engines untouched (structurally divergent — mutating-symmetric vs pure-asymmetric — would risk 746 passing tests under brief's revert-on-regression rule). Real consumer is minikraken/datalog work in flight. |
| 7 — layout.sx (haskell + synthetic) | [partial — haskell port deferred] | d75c61d4 | Configurable kit (haskell-style keyword-opens + python-style trailing-`:`-opens) + 6 self-tests covering both flavours. Synthetic Python-ish fixture passes; haskell/layout.sx untouched (kit not yet a drop-in for Haskell 98 Note 5 etc.; haskell still 156/156 baseline). |
| 8 — hm.sx (haskell + TBD) | [partial — algebra shipped; assembly deferred] | ab2c40c1 | HM foundations: types/schemes/ftv/apply/compose/generalize/instantiate/fresh-tv on top of match.sx unify, plus literal inference rule. 24/24 self-tests. Algorithm W lambda/app/let assembly deferred to host code — paired sequencing per brief: lib/ocaml/types.sx (OCaml-on-SX Phase 5) + haskell/infer.sx port. Haskell still 156/156 baseline. |
---
## Rules
- **Branch:** `architecture`. Commit locally. **Never push.** **Never touch `main`.**
- **Scope:** ONLY `lib/guest/**`, `lib/{lua,prolog,haskell,common-lisp,tcl}/**` (canaries + extraction targets), `plans/lib-guest.md`, `plans/agent-briefings/lib-guest-loop.md`. No `spec/`, `hosts/`, `web/`, `shared/`.
- **SX files:** `sx-tree` MCP tools only. `sx_validate` after every edit.
- **No raw dune.** Use `sx_build target="ocaml"` MCP tool.
- **Two-language rule:** never merge an extraction until two guests consume it (Step 8 excepted with explicit note).
- **Conformance baseline is the bar.** Any port whose scoreboard regresses by ≥1 test → revert, mark blocked, move on.
- **Substrate change → re-snapshot.** If `spec/` or `hosts/` changes underneath this loop, re-run Step 0 before continuing.
- **One step per code commit.** Plan updates as a separate commit. Short message with delta.
- **No alias chains** to paper over drift between extraction and consumer (`feedback_no_alias_bloat`).
- **Partial extraction is OK** if the canary works and a pending consumer is identified — mark `[partial — pending <consumer>]`.
- **Hard timeout:** if stuck >45 min on a step, mark `blocked (<reason>)` and move on.

138
plans/minikanren-on-sx.md Normal file
View File

@@ -0,0 +1,138 @@
# miniKanren-on-SX: relational programming on the CEK/VM
miniKanren is not a language to parse — it is an **embedded DSL** implemented as a library
of SX functions. No tokenizer, no transpiler. The entire system is a set of `define` forms
in `lib/minikanren/`. Programs are SX expressions using the miniKanren API.
The unique angle: SX's delimited continuation machinery (`perform`/`cek-resume`, call/cc)
maps almost perfectly to the search monad miniKanren needs. Backtracking is cooperative
suspension, not a separate trail machine. This is the cleanest possible host for miniKanren.
End-state goal: **full core miniKanren** (`run`, `fresh`, `==`, `conde`, `condu`, `onceo`,
`project`, `matche`) + **core.logic-style relations** (`appendo`, `membero`, `listo`,
`numbero`, etc.) + **arithmetic constraints** (`fd` domain, `CLP(FD)` subset).
## Ground rules
- **Scope:** only touch `lib/minikanren/**` and `plans/minikanren-on-sx.md`. Do **not**
edit `spec/`, `hosts/`, `shared/`, or other `lib/<lang>/`.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** pure library — no source parser. Programs are written in SX using the API.
- **Reference:** *The Reasoned Schemer* (Friedman/Byrd/Kiselyov) + Byrd's dissertation.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
## Architecture sketch
```
SX program using miniKanren API
│
├── lib/minikanren/unify.sx — terms, variables, walk, unification, occurs check
├── lib/minikanren/substitution.sx — substitution as association list / hash table
├── lib/minikanren/stream.sx — lazy streams of substitutions (via delay/force)
├── lib/minikanren/goals.sx — == / fresh / conde / condu / onceo / project / matche
├── lib/minikanren/run.sx — run* / run n — drive the search, extract answers
├── lib/minikanren/relations.sx — standard relations: appendo, membero, listo, etc.
└── lib/minikanren/clpfd.sx — arithmetic constraints (CLP(FD) subset)
```
Key semantic mappings:
- **Logic variable** → SX vector of length 1 (mutable box); `make-var` creates fresh one;
`walk` follows the substitution chain
- **Substitution** → SX association list (or hash table for performance) mapping var → term
- **Stream of substitutions** → lazy list using `delay`/`force` (Phase 9 of primitives)
- **Goal** → SX function `substitution → stream-of-substitutions`
- **`==`** → unifies two terms, extending substitution or failing (empty stream)
- **`fresh`** → introduces new logic variables; `(fresh (x y) goal)` → goal with x, y bound
- **`conde`** → interleave streams from multiple goal clauses (depth-first with interleaving)
- **`run n`** → drive the stream, collect first n substitutions, reify answers
## Roadmap
### Phase 1 — variables + unification
- [ ] `make-var` → fresh logic variable (unique mutable box)
- [ ] `var?` `v` → bool — is this a logic variable?
- [ ] `walk` `term` `subst` → follow substitution chain to ground term or unbound var
- [ ] `walk*` `term` `subst` → deep walk (recurse into lists/dicts)
- [ ] `unify` `u` `v` `subst` → extended substitution or `#f` (failure)
Handles: var/var, var/term, term/var, list unification, number/string/symbol equality.
No occurs check by default; `unify-check` with occurs check as opt-in.
- [ ] Empty substitution `empty-s` = `(list)` (empty assoc list)
- [ ] Tests in `lib/minikanren/tests/unify.sx`: ground terms, vars, lists, failure, occurs
### Phase 2 — streams + goals
- [ ] Stream type: `mzero` (empty stream = `nil`), `unit s` (singleton = `(list s)`),
`mplus` (interleave two streams), `bind` (apply goal to stream)
- [ ] Lazy streams via `delay`/`force` — mature pairs for depth-first, immature for lazy
- [ ] `==` goal: `(fn (s) (let ((s2 (unify u v s))) (if s2 (unit s2) mzero)))`
- [ ] `succeed` / `fail` — trivial goals
- [ ] `fresh` — `(fn (f) (fn (s) ((f (make-var)) s)))` — introduces one var; `fresh*` for many
- [ ] `conde` — interleaving disjunction of goal lists
- [ ] `condu` — committed choice (soft-cut): only explores first successful clause
- [ ] `onceo` — succeeds at most once
- [ ] Tests: basic goal composition, backtracking, interleaving
### Phase 3 — run + reification
- [ ] `run*` `goal` → list of all answers (reified)
- [ ] `run n` `goal` → list of first n answers
- [ ] `reify` `term` `subst` → replace unbound vars with `_0`, `_1`, ... names
- [ ] `reify-s` → builds reification substitution for naming unbound vars consistently
- [ ] `fresh` with multiple variables: `(fresh (x y z) goal)` sugar
- [ ] Query variable conventions: `q` as canonical query variable
- [ ] Tests: classic miniKanren programs — `(run* q (== q 1))` → `(1)`,
`(run* q (conde ((== q 1)) ((== q 2))))` → `(1 2)`,
Peano arithmetic, `appendo` preview
### Phase 4 — standard relations
- [ ] `appendo` `l` `s` `ls` — list append, runs forwards and backwards
- [ ] `membero` `x` `l` — x is a member of l
- [ ] `listo` `l` — l is a proper list
- [ ] `nullo` `l` — l is empty
- [ ] `pairo` `p` — p is a pair (cons cell)
- [ ] `caro` `p` `a` — car of pair
- [ ] `cdro` `p` `d` — cdr of pair
- [ ] `conso` `a` `d` `p` — cons
- [ ] `firsto` / `resto` — aliases for caro/cdro
- [ ] `reverseo` `l` `r` — reverse of list
- [ ] `flatteno` `l` `f` — flatten nested lists
- [ ] `permuteo` `l` `p` — permutation of list
- [ ] `lengtho` `l` `n` — length as a relation (Peano or integer)
- [ ] Tests: run each relation forwards and backwards; generate from partial inputs
### Phase 5 — `project` + `matche` + negation
- [ ] `project` `(x ...) body` — access reified values of logic vars inside a goal;
escapes to ground values for arithmetic or string ops
- [ ] `matche` — pattern matching over logic terms (extension from core.logic)
`(matche l ((head . tail) goal) (() goal))`
- [ ] `conda` — soft-cut disjunction (like Prolog `->`)
- [ ] `condu` — committed choice (already in phase 2; refine semantics here)
- [ ] `nafc` — negation as finite failure with constraint
- [ ] Tests: Zebra puzzle, N-queens, Sudoku via `project`, family relations via `matche`
### Phase 6 — arithmetic constraints CLP(FD)
- [ ] Finite domain variables: `fd-var` with domain `[lo..hi]`
- [ ] `in` `x` `domain` — constrain x to domain
- [ ] `fd-eq` `x` `y` — x = y (constraint propagation)
- [ ] `fd-neq` `x` `y` — x ≠ y
- [ ] `fd-lt` `fd-lte` `fd-gt` `fd-gte` — ordering constraints
- [ ] `fd-plus` `x` `y` `z` — x + y = z (constraint)
- [ ] `fd-times` `x` `y` `z` — x * y = z
- [ ] Arc consistency propagation — when domain narrows, propagate to constrained vars
- [ ] Labelling: `fd-run` drives search by splitting domains when propagation stalls
- [ ] Tests: send-more-money, N-queens with CLP(FD), map coloring, cryptarithmetic
### Phase 7 — tabling (memoization of relations)
- [ ] `tabled` annotation: memoize calls to a relation using a hash table
- [ ] Prevents infinite loops in recursive relations like `patho` on cyclic graphs
- [ ] Producer/consumer scheduling for tabled relations (variant of SLG resolution)
- [ ] Tests: cyclic graph reachability, mutual recursion, Fibonacci via tabling
## Blockers
_(none yet)_
## Progress log
_Newest first._
_(awaiting phase 1)_

315
plans/ocaml-on-sx.md Normal file
View File

@@ -0,0 +1,315 @@
# OCaml-on-SX: OCaml + ReasonML + Dream on the CEK/VM
The meta-circular demo: SX's native evaluator is OCaml, so implementing OCaml on top of
SX closes the loop — the source language of the host is running inside the host it
compiles to. Beyond the elegance, it's practically useful: once OCaml expressions run on
the SX CEK/VM you get Dream (a clean OCaml web framework) almost for free, and ReasonML
is a syntax variant that shares the same transpiler output.
End-state goal: **OCaml programs running on the SX CEK/VM**, with enough of the standard
library to support Dream's middleware model. Dream-on-SX is the integration target —
a `handler`/`middleware`/`router` API that feels idiomatic while running purely in SX.
ReasonML (Phase 8) adds an alternative syntax frontend that targets the same transpiler.
## What this covers that nothing else in the set does
- **Strict ML semantics** — unlike Haskell, OCaml is call-by-value with explicit `Lazy.t`
for laziness. Pattern match is exhaustive. Polymorphic variants. Structural equality.
- **First-class modules and functors** — modules as values (phase 4); functors as SX
higher-order functions over module records. Unlike Haskell typeclasses, OCaml's module
system is explicit and compositional.
- **Mutable state without monads** — `ref`, `:=`, `!` are primitives. Arrays. `Hashtbl`.
The IO model is direct; `Lwt`/Dream map to `perform`/`cek-resume` for async.
- **Dream's composable HTTP model** — `handler = request -> response promise`,
`middleware = handler -> handler`. Algebraically clean; `@@` composition maps to SX
function composition trivially.
- **ReasonML** — same semantics, JS-friendly surface syntax. JSX variant pairs with SX
component rendering.
## Ground rules
- **Scope:** only touch `lib/ocaml/**`, `lib/dream/**`, `lib/reasonml/**`, and
`plans/ocaml-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, or other
`lib/<lang>/`.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** OCaml source → AST → SX AST → CEK. No standalone OCaml evaluator.
The OCaml AST is walked by an `ocaml-eval` function in SX that produces SX values.
- **Type system:** deferred until Phase 5. Phases 1–4 are intentionally untyped —
get the evaluator right first, then layer HM inference on top.
- **Dream:** implemented as a library in Phase 7; no separate build step. `Dream.run`
wraps SX's existing HTTP server machinery via `perform`/`cek-resume`.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.
## Architecture sketch
```
OCaml source text
│
▼
lib/ocaml/tokenizer.sx — keywords, operators, string/char literals, comments
│
▼
lib/ocaml/parser.sx — OCaml AST: let/let rec, fun, match, if, begin/end,
│ module/struct/functor, type decls, expressions
▼
lib/ocaml/desugar.sx — surface → core: tuple patterns, or-patterns,
│ sequence (;) → (do), when guards, field punning
▼
lib/ocaml/transpile.sx — OCaml AST → SX AST
│
▼
lib/ocaml/runtime.sx — ADT constructors, module primitives, ref/array ops,
│ Stdlib shims, Dream server (phase 7)
▼
SX CEK evaluator (both JS and OCaml hosts)
```
## Semantic mappings
| OCaml construct | SX mapping |
|----------------|-----------|
| `let x = e` (top-level) | `(define x e)` |
| `let f x y = e` | `(define (f x y) e)` |
| `let rec f x = e` | `(define (f x) e)` — SX define is already recursive |
| `fun x -> e` | `(fn (x) e)` |
| `e1 \|> f` | `(f e1)` — pipe desugars to reverse application |
| `e1; e2` | `(do e1 e2)` |
| `begin e1; e2; e3 end` | `(do e1 e2 e3)` |
| `if c then e1 else e2` | `(if c e1 e2)` |
| `match x with \| P -> e` | `(match x (P e) ...)` via Phase 6 ADT primitive |
| `type t = A \| B of int` | `(define-type t (A) (B v))` |
| `module M = struct ... end` | SX dict `{:let-bindings ...}` — module as record |
| `functor (M : S) -> ...` | `(fn (M) ...)` — functor as SX lambda over module record |
| `open M` | inject M's bindings into scope via `env-merge` |
| `M.field` | `(get M :field)` |
| `{ r with f = v }` | `(dict-set r :f v)` |
| `ref x` | `(make-ref x)` — mutable cell |
| `!r` | `(deref-ref r)` |
| `r := v` | `(set-ref! r v)` |
| `(a, b, c)` | tagged list `(:tuple a b c)` |
| `[1; 2; 3]` | `(list 1 2 3)` |
| `[| 1; 2; 3 |]` | `(make-array 1 2 3)` (Phase 6) |
| `try e with \| Ex -> h` | `(guard (fn (ex) h) e)` via SX exception system |
| `raise Ex` | `(perform (:raise Ex))` |
| `Printf.printf "%d" x` | `(perform (:print (format "%d" x)))` |
## Dream semantic mappings (Phase 7)
| Dream construct | SX mapping |
|----------------|-----------|
| `handler = request -> response promise` | `(fn (req) (perform (:http-respond ...)))` |
| `middleware = handler -> handler` | `(fn (next) (fn (req) ...))` |
| `Dream.router [routes]` | `(ocaml-dream-router routes)` — dispatch on method+path |
| `Dream.get "/path" h` | route record `{:method "GET" :path "/path" :handler h}` |
| `Dream.scope "/p" [ms] [rs]` | prefix mount with middleware chain |
| `Dream.param req "name"` | path param extracted during routing |
| `m1 @@ m2 @@ handler` | `(m1 (m2 handler))` — left-fold composition |
| `Dream.session_field req "k"` | `(perform (:session-get req "k"))` |
| `Dream.set_session_field req "k" v` | `(perform (:session-set req "k" v))` |
| `Dream.flash req` | `(perform (:flash-get req))` |
| `Dream.form req` | `(perform (:form-parse req))` — returns Ok/Error ADT |
| `Dream.websocket handler` | `(perform (:websocket handler))` |
| `Dream.run handler` | starts SX HTTP server with handler as root |
## Roadmap
### Phase 1 — Tokenizer + parser
- [ ] **Tokenizer:** keywords (`let`, `rec`, `in`, `fun`, `function`, `match`, `with`,
`type`, `of`, `module`, `struct`, `end`, `functor`, `sig`, `open`, `include`,
`if`, `then`, `else`, `begin`, `try`, `exception`, `raise`, `mutable`,
`for`, `while`, `do`, `done`, `and`, `as`, `when`), operators (`->`, `|>`,
`<|`, `@@`, `@`, `:=`, `!`, `::`, `**`, `:`, `;`, `;;`), identifiers (lower,
upper/ctor, labels `~label:`, optional `?label:`), char literals `'c'`,
string literals (escaped + heredoc `{|...|}`), int/float literals,
line comments `(*` nested block comments `*)`.
- [ ] **Parser:** top-level `let`/`let rec`/`type`/`module`/`exception`/`open`/`include`
declarations; expressions: literals, identifiers, constructor application,
lambda, application (left-assoc), binary ops with precedence table,
`if`/`then`/`else`, `match`/`with`, `try`/`with`, `let`/`in`, `begin`/`end`,
`fun`/`function`, tuples, list literals, record literals/updates, field access,
sequences `;`, unit `()`.
- [ ] **Patterns:** constructor, literal, variable, wildcard `_`, tuple, list cons `::`,
list literal, record, `as`, or-pattern `P1 | P2`, `when` guard.
- [ ] OCaml is **not** indentation-sensitive — no layout algorithm needed.
- [ ] Tests in `lib/ocaml/tests/parse.sx` — 50+ round-trip parse tests.
### Phase 2 — Core evaluator (untyped)
- [ ] `ocaml-eval` entry: walks OCaml AST, produces SX values.
- [ ] `let`/`let rec`/`let ... in` (mutually recursive with `and`).
- [ ] Lambda + application (curried by default — auto-curry multi-param defs).
- [ ] `fun`/`function` (single-arg lambda with immediate match on arg).
- [ ] `if`/`then`/`else`, `begin`/`end`, sequence `;`.
- [ ] Arithmetic, comparison, boolean ops, string `^`, `mod`.
- [ ] Unit `()` value; `ignore`.
- [ ] References: `ref`, `!`, `:=`.
- [ ] Mutable record fields.
- [ ] `for i = lo to hi do ... done` loop; `while cond do ... done`.
- [ ] `try`/`with` — maps to SX `guard`; `raise` via perform.
- [ ] Tests in `lib/ocaml/tests/eval.sx` — 50+ tests, pure + imperative.
### Phase 3 — ADTs + pattern matching
- [ ] `type` declarations: `type t = A | B of t1 * t2 | C of { x: int }`.
- [ ] Constructors as tagged lists: `A` → `(:A)`, `B(1, "x")` → `(:B 1 "x")`.
- [ ] `match`/`with`: constructor, literal, variable, wildcard, tuple, list cons/nil,
`as` binding, or-patterns, nested patterns, `when` guard.
- [ ] Exhaustiveness: runtime error on incomplete match (no compile-time check yet).
- [ ] Built-in types: `option` (`None`/`Some`), `result` (`Ok`/`Error`),
`list` (nil/cons), `bool`, `unit`, `exn`.
- [ ] `exception` declarations; built-in: `Not_found`, `Invalid_argument`,
`Failure`, `Match_failure`.
- [ ] Polymorphic variants (surface syntax `\`Tag value`; runtime same tagged list).
- [ ] Tests in `lib/ocaml/tests/adt.sx` — 40+ tests: ADTs, match, option/result.
### Phase 4 — Modules + functors
- [ ] `module M = struct let x = 1 let f y = x + y end` → SX dict `{:x 1 :f <fn>}`.
- [ ] `module type S = sig val x : int val f : int -> int end` → interface record
(runtime stub; typed checking in Phase 5).
- [ ] `module M : S = struct ... end` — coercive sealing (runtime: pass-through).
- [ ] `functor (M : S) -> struct ... end` → SX `(fn (M) ...)`.
- [ ] `module F = Functor(Base)` — functor application.
- [ ] `open M` — merge M's dict into current env (`env-merge`).
- [ ] `include M` — same as open at structure level.
- [ ] `M.name` — dict get via `:name` key.
- [ ] First-class modules (pack/unpack) — deferred to Phase 5.
- [ ] Standard module hierarchy: `List`, `Option`, `Result`, `String`, `Char`,
`Int`, `Float`, `Bool`, `Unit`, `Printf`, `Format` (stubs, filled in Phase 6).
- [ ] Tests in `lib/ocaml/tests/modules.sx` — 30+ tests.
### Phase 5 — Hindley-Milner type inference
- [ ] Algorithm W: `gen`/`inst`, `unify`, `infer-expr`, `infer-decl`.
- [ ] Type variables: `'a`, `'b`; unification with occur-check.
- [ ] Let-polymorphism: generalise at let-bindings.
- [ ] ADT types: `type 'a option = None | Some of 'a`.
- [ ] Function types, tuple types, record types.
- [ ] Type signatures: `val f : int -> int` — verify against inferred type.
- [ ] Module type checking: seal against `sig` (Phase 4 stubs become real checks).
- [ ] Error reporting: position-tagged errors with expected vs actual types.
- [ ] First-class modules: `(module M : S)` pack; `(val m : (module S))` unpack.
- [ ] No rank-2 polymorphism, no GADTs (out of scope).
- [ ] Tests in `lib/ocaml/tests/types.sx` — 60+ inference tests.
### Phase 6 — Standard library
- [ ] `List`: `map`, `filter`, `fold_left`, `fold_right`, `length`, `rev`, `append`,
`concat`, `flatten`, `iter`, `iteri`, `mapi`, `for_all`, `exists`, `find`,
`find_opt`, `mem`, `assoc`, `assq`, `sort`, `stable_sort`, `nth`, `hd`, `tl`,
`init`, `combine`, `split`, `partition`.
- [ ] `Option`: `map`, `bind`, `fold`, `get`, `value`, `join`, `iter`, `to_list`,
`to_result`, `is_none`, `is_some`.
- [ ] `Result`: `map`, `bind`, `fold`, `get_ok`, `get_error`, `map_error`,
`to_option`, `is_ok`, `is_error`.
- [ ] `String`: `length`, `get`, `sub`, `concat`, `split_on_char`, `trim`,
`uppercase_ascii`, `lowercase_ascii`, `contains`, `starts_with`, `ends_with`,
`index_opt`, `replace_all` (non-stdlib but needed).
- [ ] `Char`: `code`, `chr`, `escaped`, `lowercase_ascii`, `uppercase_ascii`.
- [ ] `Int`/`Float`: arithmetic, `to_string`, `of_string_opt`, `min_int`, `max_int`.
- [ ] `Hashtbl`: `create`, `add`, `replace`, `find`, `find_opt`, `remove`, `mem`,
`iter`, `fold`, `length` — backed by SX mutable dict.
- [ ] `Map.Make` functor — balanced BST backed by SX sorted dict.
- [ ] `Set.Make` functor.
- [ ] `Printf`: `sprintf`, `printf`, `eprintf` — format strings via `(format ...)`.
- [ ] `Sys`: `argv`, `getenv_opt`, `getcwd` — via `perform` IO.
- [ ] Scoreboard runner: `lib/ocaml/conformance.sh` + `scoreboard.json`.
- [ ] Target: 150+ tests across all stdlib modules.
### Phase 7 — Dream web framework (`lib/dream/`)
The five types: `request`, `response`, `handler = request -> response`,
`middleware = handler -> handler`, `route`. Everything else is a function over these.
- [ ] **Core types** in `lib/dream/types.sx`: request/response records, route record.
- [ ] **Router** in `lib/dream/router.sx`:
- `dream-get path handler`, `dream-post path handler`, etc. for all HTTP methods.
- `dream-scope prefix middlewares routes` — prefix mount with middleware chain.
- `dream-router routes` — dispatch tree, returns handler; no match → 404.
- Path param extraction: `:name` segments, `**` wildcard.
- `dream-param req name` — retrieve matched path param.
- [ ] **Middleware** in `lib/dream/middleware.sx`:
- `dream-pipeline middlewares handler` — compose middleware left-to-right.
- `dream-no-middleware` — identity.
- Logger: `(dream-logger next req)` — logs method, path, status, timing.
- Content-type sniffer.
- [ ] **Sessions** in `lib/dream/session.sx`:
- Cookie-backed session middleware.
- `dream-session-field req key`, `dream-set-session-field req key val`.
- `dream-invalidate-session req`.
- [ ] **Flash messages** in `lib/dream/flash.sx`:
- `dream-flash-middleware` — single-request cookie store.
- `dream-add-flash-message req category msg`.
- `dream-flash-messages req` — returns list of `(category, msg)`.
- [ ] **Forms + CSRF** in `lib/dream/form.sx`:
- `dream-form req` — returns `(Ok fields)` or `(Err :csrf-token-invalid)`.
- `dream-multipart req` — streaming multipart form data.
- CSRF middleware: stateless signed tokens, session-scoped.
- `dream-csrf-tag req` — returns hidden input fragment for SX templates.
- [ ] **WebSockets** in `lib/dream/websocket.sx`:
- `dream-websocket handler` — upgrades request; handler `(fn (ws) ...)`.
- `dream-send ws msg`, `dream-receive ws`, `dream-close ws`.
- [ ] **Static files:** `dream-static root-path` — serves files, ETags, range requests.
- [ ] **`dream-run`**: wires root handler into SX's `perform (:http-listen ...)`.
- [ ] **Demos** in `lib/dream/demos/`:
- `hello.ml` → `lib/dream/demos/hello.sx`: "Hello, World!" route.
- `counter.ml` → `lib/dream/demos/counter.sx`: in-memory counter with sessions.
- `chat.ml` → `lib/dream/demos/chat.sx`: multi-room WebSocket chat.
- `todo.ml` → `lib/dream/demos/todo.sx`: CRUD list with forms + CSRF.
- [ ] Tests in `lib/dream/tests/`: routing dispatch, middleware composition,
session round-trip, CSRF accept/reject, flash read-after-write — 60+ tests.
### Phase 8 — ReasonML syntax variant (`lib/reasonml/`)
ReasonML is OCaml with a JS-friendly surface: semicolons, `let` with `=` everywhere,
`=>` for lambdas, `switch` for match, `{j|...|j}` string interpolation. Same semantics —
different tokenizer + parser, same `lib/ocaml/transpile.sx` output.
- [ ] **Tokenizer** in `lib/reasonml/tokenizer.sx`:
- `let x = e;` binding syntax (semicolons required).
- `(x, y) => e` arrow function syntax.
- `switch (x) { | Pat => e | ... }` for match.
- JSX: `<Comp prop=val />`, `<div>children</div>`.
- String interpolation: `{j|hello $(name)|j}`.
- Type annotations: `x : int`, `let f : int => int = x => x + 1`.
- [ ] **Parser** in `lib/reasonml/parser.sx`:
- Produce same OCaml AST nodes as `lib/ocaml/parser.sx`.
- JSX → SX component calls: `<Comp x=1 />` → `(~comp :x 1)`.
- Multi-arg functions: `(x, y) => e` → auto-curried pair.
- [ ] Shared transpiler: `lib/reasonml/transpile.sx` delegates to
`lib/ocaml/transpile.sx` (parse → ReasonML AST → OCaml AST → SX AST).
- [ ] Tests in `lib/reasonml/tests/`: tokenizer, parser, eval, JSX — 40+ tests.
- [ ] ReasonML Dream demos: translate Phase 7 demos to ReasonML syntax.
## The meta-circular angle
SX is bootstrapped to OCaml (`hosts/ocaml/`). Running OCaml inside SX running on OCaml is
the "mother tongue" closure: OCaml → SX → OCaml. This means:
- The OCaml host's native pattern matching and ADTs are exact reference semantics for
the SX-level implementation — any mismatch is a bug.
- The SX `match` / `define-type` primitives (Phase 6 of the primitives roadmap) were
built knowing OCaml was the intended target.
- When debugging the transpiler, the OCaml REPL is always available as oracle.
- Dream running in SX can serve the sx.rose-ash.com docs site — the framework that
describes the runtime it runs on.
## Key dependencies
- **Phase 6 ADT primitive** (`define-type`/`match`) — required before Phase 3.
- **`perform`/`cek-resume`** IO suspension — required before Phase 7 (Dream async).
- **HO forms** and first-class lambdas — already in spec, no blocker.
- **Module system** (Phase 4) is independent of type inference (Phase 5) — can overlap.
- **ReasonML** (Phase 8) can start once OCaml parser is stable (after Phase 2).
## Progress log
_Newest first._
_(awaiting phase 1)_
## Blockers
_(none yet)_

100
plans/prolog-on-sx.md
View File

@@ -39,59 +39,93 @@ Representation choices (finalise in phase 1, document here):
## Roadmap
### Phase 1 — tokenizer + term parser (no operator table)
- [ ] Tokenizer: atoms (lowercase/quoted), variables (uppercase/`_`), numbers, strings, punct `( ) , . [ ] | ! :-`, comments (`%`, `/* */`)
- [ ] Parser: clauses `head :- body.` and facts `head.`; terms `atom | Var | number | compound(args) | [list,sugar]`
- [ ] **Skip for phase 1:** operator table. `X is Y + 1` must be written `is(X, '+'(Y, 1))`; `=` written `=(X, Y)`. Operators land in phase 4.
- [ ] Unit tests in `lib/prolog/tests/parse.sx`
- [x] Tokenizer: atoms (lowercase/quoted), variables (uppercase/`_`), numbers, strings, punct `( ) , . [ ] | ! :-`, comments (`%`, `/* */`)
- [x] Parser: clauses `head :- body.` and facts `head.`; terms `atom | Var | number | compound(args) | [list,sugar]`
- [x] **Skip for phase 1:** operator table. `X is Y + 1` must be written `is(X, '+'(Y, 1))`; `=` written `=(X, Y)`. Operators land in phase 4.
- [x] Unit tests in `lib/prolog/tests/parse.sx` — 25 pass
### Phase 2 — unification + trail
- [ ] `make-var`, `walk` (follow binding chain), `prolog-unify!` (terms + trail → bool), `trail-undo-to!`
- [ ] Occurs-check off by default, exposed as flag
- [ ] 30+ unification tests in `lib/prolog/tests/unify.sx`: atoms, vars, compounds, lists, cyclic (no-occurs-check), mutual occurs
- [x] `make-var`, `walk` (follow binding chain), `prolog-unify!` (terms + trail → bool), `trail-undo-to!`
- [x] Occurs-check off by default, exposed as flag
- [x] 30+ unification tests in `lib/prolog/tests/unify.sx`: atoms, vars, compounds, lists, cyclic (no-occurs-check), mutual occurs — 47 pass
### Phase 3 — clause DB + DFS solver + cut + first classic programs
- [ ] Clause DB: `"functor/arity" → list-of-clauses`, loader inserts
- [ ] Solver: DFS with choice points backed by delimited continuations (`lib/callcc.sx`). On goal entry, capture; per matching clause, unify head + recurse body; on failure, undo trail, try next
- [ ] Cut (`!`): cut barrier at current choice-point frame; collapse all up to barrier
- [ ] Built-ins: `=/2`, `\\=/2`, `true/0`, `fail/0`, `!/0`, `,/2`, `;/2`, `->/2` inside `;`, `call/1`, `write/1`, `nl/0`
- [ ] Arithmetic `is/2` with `+ - * / mod abs`
- [ ] Classic programs in `lib/prolog/tests/programs/`:
- [ ] `append.pl` — list append (with backtracking)
- [ ] `reverse.pl` — naive reverse
- [ ] `member.pl` — generate all solutions via backtracking
- [ ] `nqueens.pl` — 8-queens
- [ ] `family.pl` — facts + rules (parent/ancestor)
- [ ] `lib/prolog/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md`
- [ ] Target: all 5 classic programs passing
- [x] Clause DB: `"functor/arity" → list-of-clauses`, loader inserts — `pl-mk-db` / `pl-db-add!` / `pl-db-load!` / `pl-db-lookup` / `pl-db-lookup-goal`, 14 tests in `tests/clausedb.sx`
- [x] Solver: DFS with choice points backed by delimited continuations (`lib/callcc.sx`). On goal entry, capture; per matching clause, unify head + recurse body; on failure, undo trail, try next — first cut: trail-based undo + CPS k (no shift/reset yet, per briefing gotcha). Built-ins so far: `true/0`, `fail/0`, `=/2`, `,/2`. Refactor to delimited conts later.
- [x] Cut (`!`): cut barrier at current choice-point frame; collapse all up to barrier — two-cut-box scheme: each `pl-solve-user!` creates a fresh inner-cut-box (set by `!` in this predicate's body) AND snapshots the outer-cut-box state on entry. After body fails, abandon clause alternatives if (a) inner was set or (b) outer transitioned false→true during this call. Lets post-cut goals backtrack normally while blocking pre-cut alternatives. 6 cut tests cover bare cut, clause-commit, choice-commit, cut+fail, post-cut backtracking, nested-cut isolation.
- [x] Built-ins: `=/2`, `\\=/2`, `true/0`, `fail/0`, `!/0`, `,/2`, `;/2`, `->/2` inside `;`, `call/1`, `write/1`, `nl/0` — all 11 done. `write/1` and `nl/0` use a global `pl-output-buffer` string + `pl-output-clear!` for testability; `pl-format-term` walks deep then renders atoms/nums/strs/compounds/vars (var → `_<id>`). Note: cut-transparency via `;` not testable yet without operator support — `;(,(a,!), b)` parser-rejects because `,` is body-operator-only; revisit in phase 4.
- [x] Arithmetic `is/2` with `+ - * / mod abs` — `pl-eval-arith` walks deep, recurses on compounds, dispatches on functor; binary `+ - * / mod`, binary AND unary `-`, unary `abs`. `is/2` evaluates RHS, wraps as `("num" v)`, unifies via `pl-solve-eq!`. 11 tests cover each op + nested + ground LHS match/mismatch + bound-var-on-RHS chain.
- [x] Classic programs in `lib/prolog/tests/programs/`:
- [x] `append.pl` — list append (with backtracking) — `lib/prolog/tests/programs/append.{pl,sx}`. 6 tests cover: build (`append([], L, X)`, `append([1,2], [3,4], X)`), check ground match/mismatch, full split-backtracking (`append(X, Y, [1,2,3])` → 4 solutions), single-deduce (`append(X, [3], [1,2,3])` → X=[1,2]).
- [x] `reverse.pl` — naive reverse — `lib/prolog/tests/programs/reverse.{pl,sx}`. Naive reverse via append: `reverse([H|T], R) :- reverse(T, RT), append(RT, [H], R)`. 6 tests cover empty, singleton, 3-list, 4-atom-list, ground match, ground mismatch.
- [x] `member.pl` — generate all solutions via backtracking — `lib/prolog/tests/programs/member.{pl,sx}`. Classic 2-clause `member(X, [X|_])` + `member(X, [_|T]) :- member(X, T)`. 7 tests cover bound-element hit/miss, empty list, generator (count = list length), first-solution binding, duplicate matches counted twice, anonymous head-cell unification.
- [x] `nqueens.pl` — 8-queens — `lib/prolog/tests/programs/nqueens.{pl,sx}`. Permute-and-test formulation: `queens(L, Qs) :- permute(L, Qs), safe(Qs)` + `select` + `safe` + `no_attack`. Tested at N=1 (1), N=2 (0), N=3 (0), N=4 (2), N=5 (10) plus first-solution check at N=4 = `[2, 4, 1, 3]`. N=8 omitted — interpreter is too slow (40320 perms); add once compiled clauses or constraint-style placement land. `range/3` skipped pending arithmetic-comparison built-ins (`>/2` etc.).
- [x] `family.pl` — facts + rules (parent/ancestor) — `lib/prolog/tests/programs/family.{pl,sx}`. 5 parent facts + male/female + derived `father`/`mother`/`ancestor`/`sibling`. 10 tests cover direct facts, fact count, transitive ancestor through 3 generations, descendant counting, gender-restricted father/mother, sibling via shared parent + `\=`.
- [x] `lib/prolog/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md` — bash script feeds load + eval epoch script to sx_server, parses each suite's `{:failed N :passed N :total N :failures (...)}` line, writes JSON (machine) + MD (human) scoreboards. Exit non-zero on any failure. `SX_SERVER` env var overrides binary path. First scoreboard: 183 / 183.
- [x] Target: all 5 classic programs passing — append (6) + reverse (6) + member (7) + nqueens (6) + family (10) = 35 program tests, all green. Phase 3 architecturally complete bar the conformance harness/scoreboard.
### Phase 4 — operator table + more built-ins (next run)
- [ ] Operator table parsing (prefix/infix/postfix, precedence, assoc)
- [ ] `assert/1`, `asserta/1`, `assertz/1`, `retract/1`
- [ ] `findall/3`, `bagof/3`, `setof/3`
- [ ] `copy_term/2`, `functor/3`, `arg/3`, `=../2`
- [ ] String/atom predicates
- [x] Operator table parsing (prefix/infix/postfix, precedence, assoc) — `pl-op-table` (15 entries: `, ; -> = \= is < > =< >= + - * / mod`); precedence-climbing parser via `pp-parse-primary` + `pp-parse-term-prec` + `pp-parse-op-rhs`. Parens override precedence. Args inside compounds parsed at 999 so `,` stays as separator. xfx/xfy/yfx supported; prefix/postfix deferred (so `-5` still tokenises as bare atom + num as before). Comparison built-ins `</2 >/2 =</2 >=/2` added. New `tests/operators.sx` 19 tests cover assoc/precedence/parens + solver via infix.
- [x] `assert/1`, `asserta/1`, `assertz/1`, `retract/1` — `assert` aliases `assertz`. Helpers `pl-rt-to-ast` (deep-walk + replace runtime vars with `_G<id>` parse markers) + `pl-build-clause` (detect `:-` head). `assertz` uses `pl-db-add!`; `asserta` uses new `pl-db-prepend!`. `retract` walks goal, looks up by functor/arity, tries each clause via unification, removes first match by index (`pl-list-without`). 11 tests in `tests/dynamic.sx`. Rule-asserts now work — `:-` added to op table (prec 1200 xfx) with fix to `pl-token-op` accepting `"op"` token type. 15 tests in `tests/assert_rules.sx`.
- [x] `findall/3`, `bagof/3`, `setof/3` — shared `pl-collect-solutions` runs the goal in a fresh cut-box, deep-copies the template (via `pl-deep-copy` with var-map for shared-var preservation) on each success, returns false to backtrack, then restores trail. `findall` always succeeds with a (possibly empty) list. `bagof` fails on empty. `setof` builds a string-keyed dict via `pl-format-term` for sort+dedupe (via `keys` + `sort`), fails on empty. Existential `^` deferred (operator). 11 tests in `tests/findall.sx`.
- [x] `copy_term/2`, `functor/3`, `arg/3`, `=../2` — `copy_term/2` reuses `pl-deep-copy` with a fresh var-map (preserves source aliasing). `functor/3` handles 4 modes: compound→{name, arity}, atom→{atom, 0}, num→{num, 0}, var with ground name+arity→constructed term (`pl-make-fresh-args` for compound case). `arg/3` extracts 1-indexed arg from compound. **`=../2` deferred** — the tokenizer treats `.` as the clause terminator unconditionally, so `=..` lexes as `=` + `.` + `.`; needs special-case lex (or surface syntax via a different name). 14 tests in `tests/term_inspect.sx`.
- [x] String/atom predicates
### Phase 5 — Hyperscript integration
- [ ] `prolog-query` primitive callable from SX/Hyperscript
- [ ] Hyperscript DSL: `when allowed(user, :edit) then …`
- [ ] Integration suite
- [x] `prolog-query` primitive callable from SX/Hyperscript
- [x] Hyperscript DSL: `when allowed(user, action) then …` — `lib/prolog/hs-bridge.sx`: `pl-hs-query` (bool goal test) + `pl-hs-predicate/1,2,3` factories + `pl-hs-install`. No parser/compiler changes needed: Hyperscript already compiles `allowed(user, action)` to `(allowed user action)` — a plain SX call backed by the Prolog DB.
- [x] Integration suite
### Phase 6 — ISO conformance
- [ ] Vendor Hirst's conformance tests
- [ ] Drive scoreboard to 200+
- [x] Vendor Hirst's conformance tests
- [x] Drive scoreboard to 200+
### Phase 7 — compiler (later, optional)
- [ ] Compile clauses to SX continuations for speed
- [ ] Keep interpreter as the reference
- [x] Compile clauses to SX continuations for speed
- [x] Keep interpreter as the reference
## Progress log
_Newest first. Agent appends on every commit._
- 2026-05-06 — Hyperscript bridge (`lib/prolog/hs-bridge.sx`): `pl-hs-query`, `pl-hs-predicate/1,2,3`, `pl-hs-install`. No parser/compiler changes needed — Hyperscript already compiles `when allowed(user, action)` to `(allowed user action)`, a plain SX call; bridge factories wire a Prolog DB as the backing implementation. 19 tests in `tests/hs_bridge.sx`. Total **590** (+19).
- 2026-05-05 — Integration test suite (`tests/integration.sx`): 20 end-to-end tests via `pl-query-*` API covering permission system (6), graph reachability (4), quicksort (4), fibonacci (3), dynamic KB (3). Suite added to conformance harness. Total **571** (+20).
- 2026-04-25 — `pl-compiled-matches-interp?` cross-validator in `compiler.sx`: loads source into both a plain and a compiled DB, runs the same goal, returns true iff solution counts match. `tests/cross_validate.sx` applies this to 17 goals across append/member/ancestor/cut/arithmetic/if-then-else, locking the interpreter as the reference against which any future compiler change must agree. Total **551** (+17).
- 2026-04-25 — Clause compiler (`lib/prolog/compiler.sx`): `pl-compile-clause` converts parse-AST clauses to SX closures `(fn (goal trail db cut-box k) bool)`. Pre-collects var names at compile time; `pl-cmp-build-term` reconstructs fresh runtime terms per call. `pl-compile-db!` compiles all clauses in a DB and stores them in `:compiled` table. `pl-solve-user!` in runtime.sx auto-dispatches to compiled lambdas when present, falls back to interpreted. `pl-try-compiled-clauses!` mirrors `pl-try-clauses!` cut semantics. 17 tests in `tests/compiler.sx`. Total **534** (+17).
- 2026-04-25 — `predsort/3` (insertion-sort with 3-arg comparator predicate, deduplicates `=` pairs), `term_variables/2` (collect unbound vars left-to-right, dedup by id), arithmetic extensions (`floor/1`, `ceiling/1`, `truncate/1`, `round/1`, `sign/1`, `sqrt/1`, `pow/2`, `**/2`, `^/2`, `integer/1`, `float/1`, `float_integer_part/1`, `float_fractional_part/1`). 21 tests in `tests/advanced.sx`. Total **517** (+21).
- 2026-04-25 — `sub_atom/5` (non-deterministic substring enumeration; CPS loop over all (start,sublen) pairs; trail-undo only on backtrack) + `aggregate_all/3` (6 templates: count/bag/sum/max/min/set; uses `pl-collect-solutions`). 25 tests in `tests/string_agg.sx`. Total **496** (+25).
- 2026-04-25 — `:-` operator + assert with rules: added `(list ":-" 1200 "xfx")` to `pl-op-table`; fixed `pl-token-op` to accept `"op"` token type (tokenizer emits `:-` as `"op"`, not `"atom"`). `pl-build-clause` already handled `("compound" ":-" ...)`. `assert((head :- body))` now works for facts+rules. 15 tests in `tests/assert_rules.sx`. Total **471** (+15).
- 2026-04-25 — IO/term predicates: `term_to_atom/2` (bidirectional: format term or parse atom), `term_string/2` (alias), `with_output_to/2` (atom/string sinks — saves/restores `pl-output-buffer`), `writeln/1`, `format/1` (~n/~t/~~), `format/2` (~w/~a/~d pull from arg list). 24 tests in `tests/io_predicates.sx`. Total **456** (+24).
- 2026-04-25 — Char predicates: `char_type/2` (9 modes: alpha/alnum/digit/digit(N)/space/white/upper(L)/lower(U)/ascii(C)/punct), `upcase_atom/2`, `downcase_atom/2`, `string_upper/2`, `string_lower/2`. 10 helpers using `char-code`/`char-from-code` SX primitives. 27 tests in `tests/char_predicates.sx`. Total **432** (+27).
- 2026-04-25 — Set/fold predicates: `foldl/4` (CPS fold-left, threads accumulator via `pl-apply-goal`), `list_to_set/2` (dedup preserving first-occurrence), `intersection/3`, `subtract/3`, `union/3` (all via `pl-struct-eq?`). 3 new helpers, 15 tests in `tests/set_predicates.sx`. Total **405** (+15).
- 2026-04-25 — Meta-call predicates: `forall/2` (negation-of-counterexample), `maplist/2` (goal over list), `maplist/3` (map goal building output list), `include/3` (filter by goal success), `exclude/3` (filter by goal failure). New `pl-apply-goal` helper extends a goal with extra args. 15 tests in `tests/meta_call.sx`. Total **390** (+15).
- 2026-04-25 — List/utility predicates: `==/2`, `\==/2` (structural equality/inequality via `pl-struct-eq?`), `flatten/2` (deep Prolog-list flatten), `numlist/3` (integer range list), `atomic_list_concat/2` (join with no sep), `atomic_list_concat/3` (join with separator), `sum_list/2`, `max_list/2`, `min_list/2` (arithmetic folds), `delete/3` (remove all struct-equal elements). 7 new helpers, 33 tests in `tests/list_predicates.sx`. Total **375** (+33).
- 2026-04-25 — Meta/logic predicates: `\+/1` (negation-as-failure, trail-undo on success), `not/1` (alias), `once/1` (commit to first solution via if-then-else), `ignore/1` (always succeed), `ground/1` (all vars bound), `sort/2` (sort + dedup by formatted key), `msort/2` (sort, keep dups), `atom_number/2` (bidirectional), `number_string/2` (bidirectional). 2 helpers (`pl-ground?`, `pl-sort-pairs-dedup`). 25 tests in `tests/meta_predicates.sx`. Total **342** (+25).
- 2026-04-25 — ISO utility predicates batch: `succ/2` (bidirectional), `plus/3` (3-mode bidirectional), `between/3` (backtracking range generator), `length/2` (bidirectional list length + var-list constructor), `last/2`, `nth0/3`, `nth1/3`, `max/2` + `min/2` in arithmetic eval. 6 new helper functions (`pl-list-length`, `pl-make-list-of-vars`, `pl-between-loop!`, `pl-solve-between!`, `pl-solve-last!`, `pl-solve-nth0!`). 29 tests in `tests/iso_predicates.sx`. Phase 6 complete: scoreboard already at 317, far above 200+ target. Hyperscript DSL blocked (needs `lib/hyperscript/**`). Total **317** (+29).
- 2026-04-25 — `prolog-query` SX API (`lib/prolog/query.sx`). New public API layer: `pl-load source-str → db`, `pl-query-all db query-str → list of solution dicts`, `pl-query-one db query-str → dict or nil`, `pl-query src query → list` (convenience). Each solution dict maps variable name strings to their formatted term strings. Var names extracted from pre-instantiation parse AST. Trail is marked before solve and reset after to ensure clean state. 16 tests in `tests/query_api.sx` cover fact lookup, no-solution, boolean queries, multi-var, recursive rules, is/2 built-in, query-one, convenience form. Total **288** (+16).
- 2026-04-25 — String/atom predicates. Type-test predicates: `var/1`, `nonvar/1`, `atom/1`, `number/1`, `integer/1`, `float/1` (always-fail), `compound/1`, `callable/1`, `atomic/1`, `is_list/1`. String/atom operations: `atom_length/2`, `atom_concat/3` (3 modes: both-ground, result+first, result+second), `atom_chars/2` (bidirectional), `atom_codes/2` (bidirectional), `char_code/2` (bidirectional), `number_codes/2`, `number_chars/2`. 7 helper functions in runtime.sx (`pl-list-to-prolog`, `pl-proper-list?`, `pl-prolog-list-to-sx`, `pl-solve-atom-concat!`, `pl-solve-atom-chars!`, `pl-solve-atom-codes!`, `pl-solve-char-code!`). 34 tests in `tests/atoms.sx`. Total **272** (+34).
- 2026-04-25 — `copy_term/2` + `functor/3` + `arg/3` (term inspection). `copy_term` is a one-line dispatch to existing `pl-deep-copy`. `functor/3` is bidirectional — decomposes a bound compound/atom/num into name+arity OR constructs from ground name+arity (atom+positive-arity → compound with N anonymous fresh args via `pl-make-fresh-args`; arity 0 → atom/num). `arg/3` extracts 1-indexed arg with bounds-fail. New helper `pl-solve-eq2!` for paired-unification with shared trail-undo. 14 tests in `tests/term_inspect.sx`. Total **238** (+14). `=..` deferred — `.` always tokenizes as clause terminator; needs special lexer case.
- 2026-04-25 — `findall/3` + `bagof/3` + `setof/3`. Shared collector `pl-collect-solutions` runs the goal in a fresh cut-box, deep-copies the template per success (`pl-deep-copy` walks term, allocates fresh runtime vars via shared var-map so co-occurrences keep aliasing), returns false to keep backtracking, then `pl-trail-undo-to!` to clean up. `findall` always builds a list. `bagof` fails on empty. `setof` uses a `pl-format-term`-keyed dict + SX `sort` for dedupe + ordering. New `tests/findall.sx` 11 tests. Total **224** (+11). Existential `^` deferred — needs operator.
- 2026-04-25 — Dynamic clauses: `assert/1`, `assertz/1`, `asserta/1`, `retract/1`. New helpers `pl-rt-to-ast` (deep-walk runtime term → parse-AST, mapping unbound runtime vars to `_G<id>` markers so `pl-instantiate-fresh` produces fresh vars per call) + `pl-build-clause` + `pl-db-prepend!` + `pl-list-without`. `retract` keeps runtime vars (so the caller's vars get bound), walks head for the functor/arity key, tries each stored clause via `pl-unify!`, removes the first match by index. 11 tests in `tests/dynamic.sx`; conformance script gained dynamic row. Total **213** (+11). Rule-form asserts (`(H :- B)`) deferred until `:-` is in the op table.
- 2026-04-25 — Phase 4 starts: operator-table parsing. Parser rewrite uses precedence climbing (xfx/xfy/yfx); 15-op table covers control (`, ; ->`), comparison (`= \\= is < > =< >=`), arithmetic (`+ - * / mod`). Parens override. Backwards-compatible: prefix-syntax compounds (`=(X, Y)`, `+(2, 3)`) still parse as before; existing 183 tests untouched. Added comparison built-ins `</2 >/2 =</2 >=/2` to runtime (eval both sides, compare). New `tests/operators.sx` 19 tests; conformance script gained an operators row. Total **202** (+19). Prefix/postfix deferred — `-5` keeps old bare-atom semantics.
- 2026-04-25 — Conformance harness landed. `lib/prolog/conformance.sh` runs all 9 suites in one sx_server epoch, parses the `{:failed/:passed/:total/:failures}` summary lines, and writes `scoreboard.json` + `scoreboard.md`. `SX_SERVER` env var overrides the binary path; default points at the main-repo build. Phase 3 fully complete: 183 / 183 passing across parse/unify/clausedb/solve/append/reverse/member/nqueens/family.
- 2026-04-25 — `family.pl` fifth classic program — completes the 5-program target. 5-fact pedigree + male/female + derived father/mother/ancestor/sibling. 10 tests cover fact lookup + count, transitive ancestor through 3 generations, descendant counting (5), gender-restricted derivations, sibling via shared parent guarded by `\=`. Total 183 (+10). All 5 classic programs ticked; Phase 3 needs only conformance harness + scoreboard left.
- 2026-04-25 — `nqueens.pl` fourth classic program. Permute-and-test variant exercises every Phase-3 feature: lists with `[H|T]` cons sugar, multi-clause backtracking, recursive `permute`/`select`/`safe`/`no_attack`, `is/2` arithmetic on diagonals, `\=/2` for diagonal-conflict check. 6 tests at N ∈ {1,2,3,4,5} with expected counts {1,0,0,2,10} + first-solution `[2,4,1,3]`. N=5 takes ~30s (120 perms × safe-check); N=8 omitted as it would be ~thousands of seconds. Total 173 (+6).
- 2026-04-25 — `member.pl` third classic program. Standard 2-clause definition; 7 tests cover bound-element hit/miss, empty-list fail, generator-count = list length, first-solution binding (X=11), duplicate elements matched twice on backtrack, anonymous-head unification (`member(a, [X, b, c])` binds X=a). Total 167 (+7).
- 2026-04-25 — `reverse.pl` second classic program. Naive reverse defined via append. 6 tests (empty/singleton/3-list/4-atom-list/ground match/ground mismatch). Confirms the solver handles non-trivial recursive composition: `reverse([1,2,3], R)` recurses to depth 3 then unwinds via 3 nested `append`s. Total 160 (+6).
- 2026-04-25 — `append.pl` first classic program. `lib/prolog/tests/programs/append.pl` is the canonical 2-clause source; `append.sx` embeds the source as a string (no file-read primitive in SX yet) and runs 6 tests covering build, check, full split-backtrack (4 solutions), and deduction modes. Helpers `pl-ap-list-to-sx` / `pl-ap-term-to-sx` convert deep-walked Prolog lists (`("compound" "." (h t))` / `("atom" "[]")`) to SX lists for structural assertion. Total 154 (+6).
- 2026-04-25 — `is/2` arithmetic landed. `pl-eval-arith` recursively evaluates ground RHS expressions (binary `+ - * /`, `mod`; binary+unary `-`; unary `abs`); `is/2` wraps the value as `("num" v)` and unifies via `pl-solve-eq!`, so it works in all three modes — bind unbound LHS, check ground LHS for equality, propagate from earlier var bindings on RHS. 11 tests, total 148 (+11). Without operator support, expressions must be written prefix: `is(X, +(2, *(3, 4)))`.
- 2026-04-25 — `write/1` + `nl/0` landed using global string buffer (`pl-output-buffer` + `pl-output-clear!` + `pl-output-write!`). `pl-format-term` walks deep + dispatches on atom/num/str/compound/var; `pl-format-args` recursively comma-joins. 7 new tests cover atom/num/compound formatting, conjunction order, var-walk, and `nl`. Built-ins box (`=/2`, `\\=/2`, `true/0`, `fail/0`, `!/0`, `,/2`, `;/2`, `->/2`, `call/1`, `write/1`, `nl/0`) now ticked. Total 137 (+7).
- 2026-04-25 — `->/2` if-then-else landed (both `;(->(C,T), E)` and standalone `->(C, T)` ≡ `(C -> T ; fail)`). `pl-solve-or!` now special-cases `->` in left arg → `pl-solve-if-then-else!`. Cond runs in a fresh local cut-box (ISO opacity for cut inside cond). Then-branch can backtrack, else-branch can backtrack, but cond commits to first solution. 9 new tests covering both forms, both branches, binding visibility, cond-commit, then-backtrack, else-backtrack. Total 130 (+9).
- 2026-04-25 — Built-ins `\=/2`, `;/2`, `call/1` landed. `pl-solve-not-eq!` (try unify, always undo, succeed iff unify failed). `pl-solve-or!` (try left, on failure check cut and only try right if not cut). `call/1` opens a fresh inner cut-box (ISO opacity: cut inside `call(G)` commits G, not caller). 11 new tests in `tests/solve.sx` cover atoms+vars for `\=`, both branches + count for `;`, and `call/1` against atoms / compounds / bound goal vars. Total 121 (+11). Box not yet ticked — `->/2`, `write/1`, `nl/0` still pending.
- 2026-04-25 — Cut (`!/0`) landed. `pl-cut?` predicate; solver functions all take a `cut-box`; `pl-solve-user!` creates a fresh inner-cut-box and snapshots `outer-was-cut`; `pl-try-clauses!` abandons alternatives when inner.cut OR (outer.cut transitioned false→true during this call). 6 new cut tests in `tests/solve.sx` covering bare cut, clause-commit, choice-commit, cut+fail blocks alt clauses, post-cut goal backtracks freely, inner cut isolation. Total 110 (+6).
- 2026-04-25 — Phase 3 DFS solver landed (CPS, trail-based backtracking; delimited conts deferred). `pl-solve!` + `pl-solve-eq!` + `pl-solve-user!` + `pl-try-clauses!` + `pl-solve-once!` + `pl-solve-count!` in runtime.sx. Built-ins: `true/0`, `fail/0`, `=/2`, `,/2`. New `tests/solve.sx` 18/18 green covers atomic goals, =, conjunction, fact lookup, multi-solution count, recursive ancestor rule, trail-undo verification. Bug fix: `pl-instantiate` had no `("clause" h b)` case → vars in rule head/body were never instantiated, so rule resolution silently failed against runtime-var goals. Added clause case to recurse with shared var-env. Total 104 (+18).
- 2026-04-24 — Phase 3 clause DB landed: `pl-mk-db` + `pl-head-key` / `pl-clause-key` / `pl-goal-key` + `pl-db-add!` / `pl-db-load!` / `pl-db-lookup` / `pl-db-lookup-goal` in runtime.sx. New `tests/clausedb.sx` 14/14 green. Total 86 (+14). Loader preserves declaration order (append!).
- 2026-04-24 — Verified phase 1+2 already implemented on loops/prolog: `pl-parse-tests-run!` 25/25, `pl-unify-tests-run!` 47/47 (72 total). Ticked phase 1+2 boxes.
- _(awaiting phase 1)_
## Blockers
_Shared-file issues that need someone else to fix. Minimal repro only._
- _(none yet)_
- **Phase 5 Hyperscript DSL** — `lib/hyperscript/**` is out of scope for this loop. Needs `lib/hyperscript/parser.sx` + evaluator to add `when allowed(user, :edit) then …` syntax. Skipping; Phase 5 item 1 (`prolog-query` SX API) is done.

134
plans/ruby-on-sx.md Normal file
View File

@@ -0,0 +1,134 @@
# Ruby-on-SX: fibers + blocks + open classes on delimited continuations
The headline showcase is **fibers** — Ruby's `Fiber.new { … Fiber.yield v … }` / `Fiber.resume` are textbook delimited continuations with sugar. MRI implements them by swapping C stacks; on SX they fall out of the existing `perform`/`cek-resume` machinery for free. Plus blocks/yield (lexical escape continuations, same shape as Smalltalk's non-local return), method_missing, and singleton classes.
End-state goal: Ruby 2.7-flavoured subset, Enumerable mixin, fibers + threads-via-fibers (no real OS threads), method_missing-driven DSLs, ~150 hand-written + classic programs.
## Scope decisions (defaults — override by editing before we spawn)
- **Syntax:** Ruby 2.7. No 3.x pattern matching, no rightward assignment, no endless methods. We pick 2.7 because it's the biggest semantic surface that still parses cleanly.
- **Conformance:** "Reads like Ruby, runs like Ruby." Slice of RubySpec (Core + Library subset), not full RubySpec.
- **Test corpus:** custom + curated RubySpec slice. Plus classic programs: fiber-based generator, internal DSL with method_missing, mixin-based Enumerable on a custom class.
- **Out of scope:** real threads, GIL, refinements, `binding_of_caller` from non-Ruby contexts, Encoding object beyond UTF-8/ASCII-8BIT, RubyVM::* introspection beyond bytecode-disassembly placeholder, IO subsystem beyond `puts`/`gets`/`File.read`.
- **Symbols:** SX symbols. Strings are mutable copies; symbols are interned.
## Ground rules
- **Scope:** only touch `lib/ruby/**` and `plans/ruby-on-sx.md`. Don't edit `spec/`, `hosts/`, `shared/`, or any other `lib/<lang>/**`. Ruby primitives go in `lib/ruby/runtime.sx`.
- **SX files:** use `sx-tree` MCP tools only.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick roadmap boxes.
## Architecture sketch
```
Ruby source
│
▼
lib/ruby/tokenizer.sx — keywords, ops, %w[], %i[], heredocs (deferred), regex (deferred)
│
▼
lib/ruby/parser.sx — AST: classes, modules, methods, blocks, calls
│
▼
lib/ruby/transpile.sx — AST → SX AST (entry: rb-eval-ast)
│
▼
lib/ruby/runtime.sx — class table, MOP, dispatch, fibers, primitives
```
Core mapping:
- **Object** = SX dict `{:class :ivars :singleton-class?}`. Instance variables live in `ivars` keyed by symbol.
- **Class** = SX dict `{:name :superclass :methods :class-methods :metaclass :includes :prepends}`. Class table is flat.
- **Method dispatch** = lookup walks ancestor chain (prepended → class → included modules → superclass → …). Falls back to `method_missing` with a `Symbol`+args.
- **Block** = lambda + escape continuation. `yield` invokes the block in current context. `return` from within a block invokes the enclosing-method's escape continuation.
- **Proc** = lambda without strict arity. `Proc.new` + `proc {}`.
- **Lambda** = lambda with strict arity + `return`-returns-from-lambda semantics.
- **Fiber** = pair of continuations (resume-k, yield-k) wrapped in a record. `Fiber.new { … }` builds it; `Fiber.resume` invokes the resume-k; `Fiber.yield` invokes the yield-k. Built directly on `perform`/`cek-resume`.
- **Module** = class without instance allocation. `include` puts it in the chain; `prepend` puts it earlier; `extend` puts it on the singleton.
- **Singleton class** = lazily allocated per-object class for `def obj.foo` definitions.
- **Symbol** = interned SX symbol. `:foo` reads as `(quote foo)` flavour.
## Roadmap
### Phase 1 — tokenizer + parser
- [ ] Tokenizer: keywords (`def end class module if unless while until do return yield begin rescue ensure case when then else elsif`), identifiers (lowercase = local/method, `@` = ivar, `@@` = cvar, `$` = global, uppercase = constant), numbers (int, float, `0x` `0o` `0b`, `_` separators), strings (`"…"` interpolation, `'…'` literal, `%w[a b c]`, `%i[a b c]`), symbols `:foo` `:"…"`, operators (`+ - * / % ** == != < > <= >= <=> === =~ !~ << >> & | ^ ~ ! && || and or not`), `:: . , ; ( ) [ ] { } -> => |`, comments `#`
- [ ] Parser: program is sequence of statements separated by newlines or `;`; method def `def name(args) … end`; class `class Foo < Bar … end`; module `module M … end`; block `do |a, b| … end` and `{ |a, b| … }`; call sugar (no parens), `obj.method`, `Mod::Const`; arg shapes (positional, default, splat `*args`, double-splat `**opts`, block `&blk`)
- [ ] If/while/case expressions (return values), `unless`/`until`, postfix modifiers
- [ ] Begin/rescue/ensure/retry, raise, raise with class+message
- [ ] Unit tests in `lib/ruby/tests/parse.sx`
### Phase 2 — object model + sequential eval
- [ ] Class table bootstrap: `BasicObject`, `Object`, `Kernel`, `Module`, `Class`, `Numeric`, `Integer`, `Float`, `String`, `Symbol`, `Array`, `Hash`, `Range`, `NilClass`, `TrueClass`, `FalseClass`, `Proc`, `Method`
- [ ] `rb-eval-ast`: literals, variables (local, ivar, cvar, gvar, constant), assignment (single and parallel `a, b = 1, 2`, splat receive), method call, message dispatch
- [ ] Method lookup walks ancestor chain; cache hit-class per `(class, selector)`
- [ ] `method_missing` fallback constructing args list
- [ ] `super` and `super(args)` — lookup in defining class's superclass
- [ ] Singleton class allocation on first `def obj.foo` or `class << obj`
- [ ] `nil`, `true`, `false` are singletons of their classes; tagged values aren't boxed
- [ ] Constant lookup (lexical-then-inheritance) with `Module.nesting`
- [ ] 60+ tests in `lib/ruby/tests/eval.sx`
### Phase 3 — blocks + procs + lambdas
- [ ] Method invocation captures escape continuation `^k` for `return`; binds it as block's escape
- [ ] `yield` invokes implicit block
- [ ] `block_given?`, `&blk` parameter, `&proc` arg unpacking
- [ ] `Proc.new`, `proc { }`, `lambda { }` (or `->(x) { x }`)
- [ ] Lambda strict arity + lambda-local `return` semantics
- [ ] Proc lax arity (`a, b, c` unpacks Array; missing args nil)
- [ ] `break`, `next`, `redo` — `break` is escape-from-loop-or-block; `next` is escape-from-block-iteration; `redo` re-runs current iteration
- [ ] 30+ tests in `lib/ruby/tests/blocks.sx`
### Phase 4 — fibers (THE SHOWCASE)
- [ ] `Fiber.new { |arg| … Fiber.yield v … }` allocates a fiber record with paired continuations
- [ ] `Fiber.resume(args…)` resumes the fiber, returning the value passed to `Fiber.yield`
- [ ] `Fiber.yield(v)` from inside the fiber suspends and returns control to the resumer
- [ ] `Fiber.current` from inside the fiber
- [ ] `Fiber#alive?`, `Fiber#raise` (deferred)
- [ ] `Fiber.transfer` — symmetric coroutines (resume from any side)
- [ ] Classic programs in `lib/ruby/tests/programs/`:
- [ ] `generator.rb` — pull-style infinite enumerator built on fibers
- [ ] `producer-consumer.rb` — bounded buffer with `Fiber.transfer`
- [ ] `tree-walk.rb` — recursive tree walker that yields each node, driven by `Fiber.resume`
- [ ] `lib/ruby/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md`
### Phase 5 — modules + mixins + metaprogramming
- [ ] `include M` — appends M's methods after class methods in chain
- [ ] `prepend M` — prepends M before class methods
- [ ] `extend M` — adds M to singleton class
- [ ] `Module#ancestors`, `Module#included_modules`
- [ ] `define_method`, `class_eval`, `instance_eval`, `module_eval`
- [ ] `respond_to?`, `respond_to_missing?`, `method_missing`
- [ ] `Object#send`, `Object#public_send`, `Object#__send__`
- [ ] `Module#method_added`, `singleton_method_added` hooks
- [ ] Hooks: `included`, `extended`, `inherited`, `prepended`
- [ ] Internal-DSL classic program: `lib/ruby/tests/programs/dsl.rb`
### Phase 6 — stdlib drive
- [ ] `Enumerable` mixin: `each` (abstract), `map`, `select`/`filter`, `reject`, `reduce`/`inject`, `each_with_index`, `each_with_object`, `take`, `drop`, `take_while`, `drop_while`, `find`/`detect`, `find_index`, `any?`, `all?`, `none?`, `one?`, `count`, `min`, `max`, `min_by`, `max_by`, `sort`, `sort_by`, `group_by`, `partition`, `chunk`, `each_cons`, `each_slice`, `flat_map`, `lazy`
- [ ] `Comparable` mixin: `<=>`, `<`, `<=`, `>`, `>=`, `==`, `between?`, `clamp`
- [ ] `Array`: indexing, slicing, `push`/`pop`/`shift`/`unshift`, `concat`, `flatten`, `compact`, `uniq`, `sort`, `reverse`, `zip`, `dig`, `pack`/`unpack` (deferred)
- [ ] `Hash`: `[]`, `[]=`, `delete`, `merge`, `each_pair`, `keys`, `values`, `to_a`, `dig`, `fetch`, default values, default proc
- [ ] `Range`: `each`, `step`, `cover?`, `include?`, `size`, `min`, `max`
- [ ] `String`: indexing, slicing, `split`, `gsub` (string-arg version, regex deferred), `sub`, `upcase`, `downcase`, `strip`, `chomp`, `chars`, `bytes`, `to_i`, `to_f`, `to_sym`, `*`, `+`, `<<`, format with `%`
- [ ] `Integer`: `times`, `upto`, `downto`, `step`, `digits`, `gcd`, `lcm`
- [ ] Drive corpus to 200+ green
## SX primitive baseline
Use vectors for arrays; numeric tower + rationals for numbers; ADTs for tagged data;
coroutines for fibers; string-buffer for mutable string building; bitwise ops for bit
manipulation; multiple values for multi-return; promises for lazy evaluation; hash tables
for mutable associative storage; sets for O(1) membership; sequence protocol for
polymorphic iteration; gensym for unique symbols; char type for characters; string ports
+ read/write for reader protocols; regexp for pattern matching; bytevectors for binary
data; format for string templating.
## Progress log
_Newest first._
- _(none yet)_
## Blockers
- _(none yet)_

152
plans/smalltalk-on-sx.md Normal file
View File

@@ -0,0 +1,152 @@
# Smalltalk-on-SX: blocks with non-local return on delimited continuations
The headline showcase is **blocks** — Smalltalk's closures with non-local return (`^expr` aborts the enclosing *method*, not the block). Every other Smalltalk on top of a host VM (RSqueak on PyPy, GemStone on C, Maxine on Java) reinvents non-local return on whatever stack discipline the host gives them. On SX it's a one-liner: a block holds a captured continuation; `^` just invokes it. Message-passing OO falls out cheaply on top of the existing component / dispatch machinery.
End-state goal: ANSI-ish Smalltalk-80 subset, SUnit working, ~200 hand-written tests + a vendored slice of the Pharo kernel tests, classic corpus (eight queens, quicksort, mandelbrot, Conway's Life).
## Scope decisions (defaults — override by editing before we spawn)
- **Syntax:** Pharo / Squeak chunk format (`!` separators, `Object subclass: #Foo …`). No fileIn/fileOut images — text source only.
- **Conformance:** ANSI X3J20 *as a target*, not bug-for-bug Squeak. "Reads like Smalltalk, runs like Smalltalk."
- **Test corpus:** SUnit ported to SX-Smalltalk + custom programs + a curated slice of Pharo `Kernel-Tests` / `Collections-Tests`.
- **Image:** out of scope. Source-only. No `become:` between sessions, no snapshotting.
- **Reflection:** `class`, `respondsTo:`, `perform:`, `doesNotUnderstand:` in. `become:` (object-identity swap) **in** — it's a good CEK exercise. Method modification at runtime in.
- **GUI / Morphic / threads:** out entirely.
## Ground rules
- **Scope:** only touch `lib/smalltalk/**` and `plans/smalltalk-on-sx.md`. Don't edit `spec/`, `hosts/`, `shared/`, or any other `lib/<lang>/**`. Smalltalk primitives go in `lib/smalltalk/runtime.sx`.
- **SX files:** use `sx-tree` MCP tools only.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick roadmap boxes.
## Architecture sketch
```
Smalltalk source
│
▼
lib/smalltalk/tokenizer.sx — selectors, keywords, literals, $c, #sym, #(…), $'…'
│
▼
lib/smalltalk/parser.sx — AST: classes, methods, blocks, cascades, sends
│
▼
lib/smalltalk/transpile.sx — AST → SX AST (entry: smalltalk-eval-ast)
│
▼
lib/smalltalk/runtime.sx — class table, MOP, dispatch, primitives
```
Core mapping:
- **Class** = SX dict `{:name :superclass :ivars :methods :class-methods :metaclass}`. Class table is a flat dict keyed by class name.
- **Object** = SX dict `{:class :ivars}` — `ivars` keyed by symbol. Tagged ints / floats / strings / symbols are not boxed; their class is looked up by SX type.
- **Method** = SX lambda closing over a `self` binding + temps. Body wrapped in a delimited continuation so `^` can escape.
- **Message send** = `(st-send receiver selector args)` — does class-table lookup, walks superclass chain, falls back to `doesNotUnderstand:` with a `Message` object.
- **Block** `[:x | … ^v … ]` = lambda + captured `^k` (the method-return continuation). Invoking `^` calls `k`; outer block invocation past method return raises `BlockContext>>cannotReturn:`.
- **Cascade** `r m1; m2; m3` = `(let ((tmp r)) (st-send tmp 'm1 ()) (st-send tmp 'm2 ()) (st-send tmp 'm3 ()))`.
- **`ifTrue:ifFalse:` / `whileTrue:`** = ordinary block sends; the runtime intrinsifies them in the JIT path so they compile to native branches (Tier 1 of bytecode expansion already covers this pattern).
- **`become:`** = swap two object identities everywhere — in SX this is a heap walk, but we restrict to `oneWayBecome:` (cheap: rewrite class field) by default.
## Roadmap
### Phase 1 — tokenizer + parser
- [x] Tokenizer: identifiers, keywords (`foo:`), binary selectors (`+`, `==`, `,`, `->`, `~=` etc.), numbers (radix `16r1F`; **scaled `1.5s2` deferred**), strings `'…''…'`, characters `$c`, symbols `#foo` `#'foo bar'` `#+`, byte arrays `#[1 2 3]` (open token), literal arrays `#(1 #foo 'x')` (open token), comments `"…"`
- [x] Parser (expression level): blocks `[:a :b | | t1 t2 | …]`, cascades, message precedence (unary > binary > keyword), assignment, return, statement sequences, literal arrays, byte arrays, paren grouping, method headers (`+ other`, `at:put:`, unary, with temps and body). Class-definition keyword messages parse as ordinary keyword sends — no special-case needed.
- [x] Parser (chunk-stream level): `st-read-chunks` splits source on `!` (with `!!` doubling) and `st-parse-chunks` runs the Pharo file-in state machine — `methodsFor:` / `class methodsFor:` opens a method batch, an empty chunk closes it. Pragmas `<primitive: …>` (incl. multiple keyword pairs, before or after temps, multiple per method) parsed into the method AST.
- [x] Unit tests in `lib/smalltalk/tests/parse.sx`
### Phase 2 — object model + sequential eval
- [x] Class table + bootstrap (`lib/smalltalk/runtime.sx`): canonical hierarchy installed (`Object`, `Behavior`, `ClassDescription`, `Class`, `Metaclass`, `UndefinedObject`, `Boolean`/`True`/`False`, `Magnitude`/`Number`/`Integer`/`SmallInteger`/`Float`/`Character`, `Collection`/`SequenceableCollection`/`ArrayedCollection`/`Array`/`String`/`Symbol`/`OrderedCollection`/`Dictionary`, `BlockClosure`). User class definition via `st-class-define!`, methods via `st-class-add-method!` (stamps `:defining-class` for super), method lookup walks chain, ivars accumulated through superclass chain, native SX value types map to Smalltalk classes via `st-class-of`.
- [x] `smalltalk-eval-ast` (`lib/smalltalk/eval.sx`): all literal kinds, ident resolution (locals → ivars → class refs), self/super/thisContext, assignment (locals or ivars, mutating), message send, cascade, sequence, and ^return via a sentinel marker (proper continuation-based escape is the Phase 3 showcase). Frames carry a parent chain so blocks close over outer locals. Primitive method tables for SmallInteger/Float, String/Symbol, Boolean, UndefinedObject, Array, BlockClosure (value/value:/whileTrue:/etc.), and class-side `new`/`name`/etc. Also satisfies "30+ tests" — 60 eval tests.
- [x] Method lookup: walk class → superclass already in `st-method-lookup-walk`; new cached wrapper `st-method-lookup` keys on `(class, selector, side)` and stores `:not-found` for negative results so DNU paths don't re-walk. Cache invalidates on `st-class-define!`, `st-class-add-method!`, `st-class-add-class-method!`, `st-class-remove-method!`, and full bootstrap. Stats helpers `st-method-cache-stats` / `st-method-cache-reset-stats!` for tests + later debugging.
- [x] `doesNotUnderstand:` fallback. `Message` class added at bootstrap with `selector`/`arguments` ivars and accessor methods. Primitive senders (Number/String/Boolean/Nil/Array/BlockClosure/class-side) now return the `:unhandled` sentinel for unknown selectors; `st-send` builds a `Message` via `st-make-message` and routes through `st-dnu`, which looks up `doesNotUnderstand:` on the receiver's class chain (instance- or class-side as appropriate). User overrides intercept unknowns and see the symbol selector + arguments array in the Message.
- [x] `super` send. Method invocation captures the defining class on the frame; `st-super-send` walks from `(st-class-superclass defining-class)` (instance- or class-side as appropriate). Falls through primitives → DNU when no method is found. Receiver is preserved as `self`, so ivar mutations stick. Verified for: subclass override calls parent, inherited `super` resolves to *defining* class's parent (not receiver's), multi-level `A→B→C` chain, super inside a block, super walks past an intermediate class with no local override.
- [x] 30+ tests in `lib/smalltalk/tests/eval.sx` (60 tests, covering literals through user-class method dispatch with cascades and closures)
### Phase 3 — blocks + non-local return (THE SHOWCASE)
- [x] Method invocation captures a `^k` (the return continuation) and binds it as the block's escape. `st-invoke` wraps body in `(call/cc (fn (k) ...))`; the frame's `:return-k` is set to k. Block creation copies `(get frame :return-k)` onto the block. Block invocation sets the new frame's `:return-k` to the block's saved one — so non-local return reaches *back through* any number of intermediate block invocations.
- [x] `^expr` from inside a block invokes that captured `^k`. The "return" AST type evaluates the expression then calls `(k v)` on the frame's :return-k. Verified: `detect:in:` style early-exit, multi-level nested blocks, ^ from inside `to:do:`/`whileTrue:`, ^ from a block passed to a *different* method (Caller→Helper) returns from Caller.
- [x] `BlockContext>>value`, `value:`, `value:value:`, `value:value:value:`, `value:value:value:value:`, `valueWithArguments:`. Implemented in `st-block-dispatch` + `st-block-apply` (eval iteration); pinned by 19 dedicated tests in `lib/smalltalk/tests/blocks.sx` covering arity through 4, valueWithArguments: with empty/non-empty arg arrays, closures over outer locals (read + mutate + later-mutation re-read), nested blocks, blocks as method arguments, `numArgs`, and `class`.
- [x] `whileTrue:` / `whileTrue` / `whileFalse:` / `whileFalse` as ordinary block sends. `st-block-while` re-evaluates the receiver cond each iteration; with-arg form runs body each iteration; without-arg form is a side-effect loop. Now returns `nil` per ANSI/Pharo. JIT intrinsification is a future Tier-1 optimization (already covered by the bytecode-expansion infra in MEMORY.md). 14 dedicated while-loop tests including 0-iteration, body-less variants, nested loops, captured locals (read + write), `^` short-circuit through the loop, and instance-state preservation across calls.
- [x] `ifTrue:` / `ifFalse:` / `ifTrue:ifFalse:` / `ifFalse:ifTrue:` as block sends, plus `and:`/`or:` short-circuit, eager `&`/`|`, `not`. Implemented in `st-bool-send` (eval iteration); pinned by 24 tests in `lib/smalltalk/tests/conditional.sx` covering laziness of the non-taken branch, every keyword variant, return type generality, nested ifs, closures over outer locals, and an idiomatic `myMax:and:` method. Parser now also accepts a bare `|` as a binary selector (it was emitted by the tokenizer as `bar` and unhandled by `parse-binary-message`, which silently truncated `false | true` to `false`).
- [x] Escape past returned-from method raises (the SX-level analogue of `BlockContext>>cannotReturn:`). Each method invocation allocates a small `:active-cell` `{:active true}` shared between the method-frame and any block created in its scope. `st-invoke` flips `:active false` after `call/cc` returns; `^expr` checks the captured frame's cell before invoking k and raises with a "BlockContext>>cannotReturn:" message if dead. Verified by `lib/smalltalk/tests/cannot_return.sx` (5 tests using SX `guard` to catch the raise). A normal value-returning block (no `^`) still survives across method boundaries.
- [x] Classic programs in `lib/smalltalk/tests/programs/`:
- [x] `eight-queens.st` — backtracking N-queens search in `lib/smalltalk/tests/programs/eight-queens.st`. The `.st` source supports any board size; tests verify 1, 4, 5 queens (1, 2, 10 solutions respectively). 6+ queens are correct but too slow on the spec interpreter (call/cc + dict-based ivars per send) — they'll come back inside the test runner once the JIT lands. The 8-queens canonical case will run in production.
- [x] `quicksort.st` — Lomuto-partition in-place quicksort in `lib/smalltalk/tests/programs/quicksort.st`. Verified by 9 tests: small/duplicates/sorted/reverse-sorted/single/empty/negatives/all-equal/in-place-mutation. Exercises Array `at:`/`at:put:` mutation, recursion, `to:do:` over varying ranges.
- [x] `mandelbrot.st` — escape-time iteration of `z := z² + c` in `lib/smalltalk/tests/programs/mandelbrot.st`. Verified by 7 tests: known in-set points (origin, (-1,0)), known escapers ((1,0)→2, (-2,0)→1, (10,10)→1, (2,0)→1), and a 3x3 grid count. Caught a real bug along the way: literal `#(...)` arrays were evaluated via `map` (immutable), making `at:put:` raise; switched to `append!` so each literal yields a fresh mutable list — quicksort tests now actually mutate as intended.
- [x] `life.st` (Conway's Life). `lib/smalltalk/tests/programs/life.st` carries the canonical rules with edge handling. Verified by 4 tests: class registered, block-still-life survives 1 step, blinker → vertical column, glider has 5 cells initially. Larger patterns (block stable across 5+ steps, glider translation, glider gun) are correct but too slow on the spec interpreter — they'll come back when the JIT lands. Also added Pharo-style dynamic array literal `{e1. e2. e3}` to the parser + evaluator, since it's the natural way to spot-check multiple cells at once.
- [x] `fibonacci.st` (recursive + Array-memoised) — `lib/smalltalk/tests/programs/fibonacci.st`. Loaded from chunk-format source by new `smalltalk-load` helper; verified by 13 tests in `lib/smalltalk/tests/programs.sx` (recursive `fib:`, memoised `memoFib:` up to 30, instance independence, class-table integrity). Source is currently duplicated as a string in the SX test file because there's no SX file-read primitive; conformance.sh will dedupe by piping the .st file directly.
- [x] `lib/smalltalk/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md`. The runner runs `bash lib/smalltalk/test.sh -v` once, parses per-file counts, and emits both files. JSON has date / program names / corpus-test count / all-test pass/total / exit code. Markdown has a totals table, the program list, the verbatim per-file test counts block, and notes about JIT-deferred work. Both are checked into the tree as the latest baseline; the runner overwrites them.
### Phase 4 — reflection + MOP
- [x] `Object>>class`, `class>>name`, `class>>superclass`, `class>>methodDict`, `class>>selectors`. `class` is universal in `st-primitive-send` (returns `Metaclass` for class-refs, the receiver's class otherwise). Class-side dispatch gains `methodDict`/`classMethodDict` (raw dict), `selectors`/`classSelectors` (Array of symbols), `instanceVariableNames` (own), `allInstVarNames` (inherited + own). 26 tests in `lib/smalltalk/tests/reflection.sx`.
- [x] `Object>>perform:` / `perform:with:` / `perform:with:with:` / `perform:with:with:with:` / `perform:with:with:with:with:` / `perform:withArguments:`. Universal in `st-primitive-send`; routes back through `st-send` so user methods, primitives, super, and DNU all still apply. Selector arg can be a symbol or string (we `str` it). 10 new tests in `lib/smalltalk/tests/reflection.sx`.
- [x] `Object>>respondsTo:`, `Object>>isKindOf:`, `Object>>isMemberOf:`. Universal in `st-primitive-send`. `respondsTo:` searches user method dicts (instance- or class-side based on receiver kind); native primitive selectors aren't enumerated, documented limitation. `isKindOf:` walks `st-class-inherits-from?`; `isMemberOf:` is exact class equality. 26 new tests in `reflection.sx`.
- [x] `Behavior>>compile:` — runtime method addition. Class-side `compile:` parses the source via `st-parse-method` and installs via `st-class-add-method!`. Sister forms `compile:classified:` and `compile:notifying:` ignore the extra arg (Pharo-tolerant). Returns the selector as a symbol. Also added `addSelector:withMethod:` (raw AST install) and `removeSelector:`. 9 new tests in `reflection.sx`.
- [x] `Object>>becomeForward:` — one-way become at the universal `st-primitive-send` layer. Mutates the receiver's `:class` and `:ivars` to match the target via `dict-set!`; every existing reference to the receiver dict now behaves as the target. Receiver and target remain distinct dicts (no SX-level identity merge), but method dispatch, ivar reads, and aliases all switch — Pharo's practical guarantee. 6 tests in `reflection.sx`, including the alias case (`a` and `alias := a` both see the new identity).
- [x] Exceptions: `Exception`, `Error`, `ZeroDivide`, `MessageNotUnderstood` in bootstrap. `signal` raises the receiver via SX `raise`; `signal:` sets `messageText` first. `on:do:` / `ensure:` / `ifCurtailed:` on BlockClosure use SX `guard`. The auto-reraise pattern uses a side-effect predicate (cleanup runs in the predicate, returns false → guard auto-reraises) because `(raise c)` from inside a guard handler hits a known SX issue with nested-handler frames. 15 tests in `lib/smalltalk/tests/exceptions.sx`. Phase 4 complete.
### Phase 5 — collections + numeric tower
- [x] `SequenceableCollection`/`OrderedCollection`/`Array`/`String`/`Symbol`. Bootstrap installs shared methods on `SequenceableCollection`: `inject:into:`, `detect:`/`detect:ifNone:`, `count:`, `allSatisfy:`/`anySatisfy:`, `includes:`, `do:separatedBy:`, `indexOf:`/`indexOf:ifAbsent:`, `reject:`, `isEmpty`/`notEmpty`, `asString`. They each call `self do:`, which dispatches to the receiver's primitive `do:` — so Array, String, and Symbol inherit them uniformly. String/Symbol primitives gained `at:` (1-indexed), `copyFrom:to:`, `first`/`last`, `do:`. OrderedCollection class is in the bootstrap hierarchy; its instance shape will fill out alongside Set/Dictionary in the next box. 28 tests in `lib/smalltalk/tests/collections.sx`.
- [x] `HashedCollection`/`Set`/`Dictionary`/`IdentityDictionary`. Implemented as user classes in `runtime.sx`. `HashedCollection` carries a single `array` ivar; `Dictionary` overrides with parallel `keys`/`values`. Set: `add:` (dedup), `addAll:`, `remove:`, `includes:`, `do:`, `size`, `asArray`. Dictionary: `at:`, `at:ifAbsent:`, `at:put:`, `includesKey:`, `removeKey:`, `keys`, `values`, `do:`, `keysDo:`, `valuesDo:`, `keysAndValuesDo:`, `size`, `isEmpty`. `IdentityDictionary` defined as a Dictionary subclass (no methods of its own yet — equality and identity diverge in a follow-up). Class-side `new` calls `super new init`. Added Array primitive `add:` (append). 29 tests in `lib/smalltalk/tests/hashed.sx`.
- [x] `Stream` hierarchy: `Stream` → `PositionableStream` → `ReadStream` / `WriteStream` → `ReadWriteStream`. User classes with `collection` + 0-based `position` ivars. ReadStream: `next`, `peek`, `atEnd`, `upToEnd`, `next:`, `skip:`, `reset`, `position`/`position:`. WriteStream: `nextPut:`, `nextPutAll:`, `contents`. Class-side `on:` constructor; `WriteStream class>>with:` pre-fills + `setToEnd`. Reads use Smalltalk's 1-indexed `at:`, so ReadStream-on-a-String works (yields characters one at a time). 21 tests in `lib/smalltalk/tests/streams.sx`. Bumped `test.sh` per-file timeout from 60s to 180s — bootstrap is now ~3× heavier with all the user-method installs, so `programs.sx` runs in ~64s.
- [x] `Number` tower: `SmallInteger`/`LargePositiveInteger`/`Float`/`Fraction`. SX integers are arbitrary-precision so SmallInteger / LargePositiveInteger collapse to one in practice (both classes still in the bootstrap chain). Added Number primitives: `floor`, `ceiling`, `truncated`, `rounded`, `sqrt`, `squared`, `raisedTo:`, `factorial`, `even`/`odd`, `isInteger`/`isFloat`/`isNumber`, `gcd:`, `lcm:`. **Fraction** now a real user class (numerator/denominator + sign-normalised, gcd-reduced at construction): `numerator:denominator:`, accessors, `+`/`-`/`*`/`/`, `negated`, `reciprocal`, `=`, `<`, `asFloat`, `printString`, `isFraction`. 47 tests in `lib/smalltalk/tests/numbers.sx`.
- [x] `String>>format:`, `printOn:` for everything. `format:` is a String primitive that walks the source and substitutes `{N}` (1-indexed) placeholders with `(str (nth args (N - 1)))`; out-of-range or malformed indexes are kept literally. `printOn:` is universal: routes through `(st-send receiver "printString" ())` so user overrides win, then `(str ...)` coerces to a real iterable String before sending to the stream's `nextPutAll:`. `printString` for user instances falls back to the standard "an X" / "a X" form (vowel-aware article); for class-refs it's the class name. 18 tests in `lib/smalltalk/tests/printing.sx`. Phase 5 complete.
### Phase 6 — SUnit + corpus to 200+
- [x] Port SUnit (`lib/smalltalk/sunit.sx`). Written in Smalltalk source via `smalltalk-load`. Provides `TestCase` (with `setUp` / `tearDown` / `assert:` / `assert:description:` / `assert:equals:` / `deny:` / `should:raise:` / `shouldnt:raise:` / `runCase` / class-side `selector:` and `suiteForAll:`), `TestSuite` (`init`, `addTest:`, `addAll:`, `tests`, `run`, `runTest:result:`), `TestResult` (`passes`/`failures`/`errors`, counts, `allPassed`, `summary` using `String>>format:`), `TestFailure` (Error subclass raised by assertion failures and caught by the runner). 19 tests in `lib/smalltalk/tests/sunit.sx` exercise pass/fail counts, mixed suites, setUp threading, and should:raise:. test.sh now loads `lib/smalltalk/sunit.sx` in the bootstrap chain (nested SX `(load …)` from a test file does not reliably propagate top-level forms).
- [x] Vendor a slice of Pharo `Kernel-Tests` and `Collections-Tests`. `lib/smalltalk/tests/pharo/kernel.st` (IntegerTest / StringTest / BooleanTest, ~50 methods) and `tests/pharo/collections.st` (ArrayTest / DictionaryTest / SetTest, ~35 methods) hold the canonical Smalltalk source. `lib/smalltalk/tests/pharo.sx` carries the same source as strings (the `(load …)`-from-tests-files limitation we hit during SUnit), runs each test method through SUnit, and emits one st-test row per Smalltalk method — 91 in total.
- [x] Drive the scoreboard up: aim for 200+ green tests. **751 green** at this point — past the target by 3.7x.
- [x] Stretch: ANSI Smalltalk validator subset (`lib/smalltalk/tests/ansi.sx`). 62 tests organised by ANSI X3J20 §6.10 Object, §6.11 Boolean, §6.12 Number, §6.13 Integer, §6.16 Symbol, §6.17 String, §6.18 Array, §6.19 BlockContext. Each test runs through SUnit and emits one st-test row, mirroring the Pharo-slice harness.
### Phase 7 — speed (optional)
- [x] Method-dictionary inline caching. Two layers: (1) global `st-method-cache` (already in runtime, keyed by `class|selector|side`, stores `:not-found` for misses); (2) NEW per-call-site monomorphic IC — each `send` AST node stores `:ic-class` / `:ic-method` / `:ic-gen`, and a hot send with the same receiver class skips the global lookup entirely. `st-ic-generation` (in runtime.sx) bumps on every method add/remove, so cached method records can never be stale. `st-ic-stats` / `st-ic-reset-stats!` for tests + later debugging. 10 dedicated IC tests in `lib/smalltalk/tests/inline_cache.sx`.
- [x] Block intrinsification beyond `whileTrue:` / `ifTrue:`. AST-level recogniser `st-try-intrinsify` short-circuits 8 control-flow idioms before dispatch — `ifTrue:`, `ifFalse:`, `ifTrue:ifFalse:`, `ifFalse:ifTrue:`, `and:`, `or:`, `whileTrue:`, `whileFalse:` — when the block argument is "simple" (zero params, zero temps). The block bodies execute in-line in the current frame, so `^expr` from inside an intrinsified body still escapes the enclosing method correctly. `st-intrinsic-stats` / `st-intrinsic-reset!` for tests + later debugging. 24 tests in `lib/smalltalk/tests/intrinsics.sx`. Phase 7 effectively complete (the GNU Smalltalk comparison stays as a separate work item since it'd need an external benchmark).
- [x] Compare against GNU Smalltalk on the corpus. `lib/smalltalk/compare.sh` runs a fibonacci(22) benchmark on both Smalltalk-on-SX (`sx_server.exe` + smalltalk-load + eval) and GNU Smalltalk (`gst -q`), emits a `compare-results.txt`. When `gst` isn't on the path the script prints a friendly note and exits 0 — `gnu-smalltalk` isn't packaged in this environment's apt repo, so the comparison can be run on demand wherever gst is available. **Phase 7 complete.**
## Progress log
_Newest first. Agent appends on every commit._
- 2026-04-25: GNU Smalltalk compare harness (`lib/smalltalk/compare.sh`) — runs fib(22) on sx_server.exe + smalltalk-load and on `gst -q`, saves results. Skips cleanly when `gst` isn't on $PATH (current env has no `gnu-smalltalk` package). **Phase 7 complete. All briefing checkboxes done.**
- 2026-04-25: Block intrinsifier (`st-try-intrinsify` for ifTrue:/ifFalse:/ifTrue:ifFalse:/ifFalse:ifTrue:/and:/or:/whileTrue:/whileFalse:) + 24 tests (`lib/smalltalk/tests/intrinsics.sx`). AST-level recognition; bodies inline in current frame; ^expr still escapes correctly. 847/847 total.
- 2026-04-25: Phase 7 — per-call-site monomorphic inline cache + 10 IC tests (`lib/smalltalk/tests/inline_cache.sx`). `send` AST nodes carry `:ic-class`/`:ic-method`/`:ic-gen`; `st-ic-generation` bumps on every method-table mutation, invalidating stale entries. 823/823 total.
- 2026-04-25: ANSI X3J20 validator subset + 62 tests (`lib/smalltalk/tests/ansi.sx`). One TestCase subclass per ANSI §6.x protocol; runs through SUnit. **Phase 6 complete.** 813/813 total.
- 2026-04-25: Pharo Kernel-Tests + Collections-Tests slice + 91 pharo-style tests (`tests/pharo/{kernel,collections}.st` + `tests/pharo.sx`). Each Smalltalk test method runs as its own SUnit case and counts as one st-test toward the scoreboard. 751/751 total — past the Phase 6 "200+ green tests" target.
- 2026-04-25: SUnit port (`lib/smalltalk/sunit.sx`, `lib/smalltalk/tests/sunit.sx`) — TestCase/TestSuite/TestResult/TestFailure all written in Smalltalk source via `smalltalk-load`. Full assert family + should:raise: + setUp/tearDown threading. 19 tests verify the framework. test.sh now bootstraps SUnit alongside runtime/eval. 660/660 total.
- 2026-04-25: String>>format: + universal printOn: + 18 tests (`lib/smalltalk/tests/printing.sx`). `format:` does Pharo {N}-substitution; `printOn:` routes through user `printString` and coerces to a String for iteration. Phase 5 complete. 638/638 total.
- 2026-04-25: Number tower + Fraction class + 47 tests (`lib/smalltalk/tests/numbers.sx`). 14 new Number primitives (floor/ceiling/truncated/rounded/sqrt/squared/raisedTo:/factorial/even/odd/gcd:/lcm:/isInteger/isFloat). Fraction with normalisation + arithmetic + comparisons + asFloat. 620/620 total.
- 2026-04-25: Stream hierarchy + 21 tests (`lib/smalltalk/tests/streams.sx`). ReadStream / WriteStream / ReadWriteStream as user classes; class-side `on:`; ReadStream-on-String yields characters. Bumped `test.sh` per-file timeout 60s → 180s — heavier bootstrap pushed `programs.sx` past 60s. 573/573 total.
- 2026-04-25: HashedCollection / Set / Dictionary / IdentityDictionary + 29 tests (`lib/smalltalk/tests/hashed.sx`). Set: dedup add:, remove:, includes:, do:, addAll:. Dictionary: parallel keys/values backing; at:put:, at:ifAbsent:, includesKey:, removeKey:, keysDo:, keysAndValuesDo:. Class-side `new` chains `super new init`. Array primitive `add:` added. 552/552 total.
- 2026-04-25: Phase 5 sequenceable-collection methods + 28 tests (`lib/smalltalk/tests/collections.sx`). 13 shared methods on `SequenceableCollection` (inject:into:, detect:, count:, …), inherited by Array/String/Symbol via `self do:`. String primitives at:/copyFrom:to:/first/last/do:. 523/523 total.
- 2026-04-25: Exception system + 15 tests (`lib/smalltalk/tests/exceptions.sx`). Exception/Error/ZeroDivide/MessageNotUnderstood in bootstrap; signal/signal: raise via SX `raise`; on:do:/ensure:/ifCurtailed: on BlockClosure via SX `guard`. Phase 4 complete. 495/495 total.
- 2026-04-25: `Object>>becomeForward:` + 6 tests. In-place mutation of `:class` and `:ivars` via `dict-set!`; aliases see the new identity. 480/480 total.
- 2026-04-25: `Behavior>>compile:` + sisters + 9 tests. Parses source via `st-parse-method`, installs via runtime helpers; also added `addSelector:withMethod:` and `removeSelector:`. 474/474 total.
- 2026-04-25: `respondsTo:` / `isKindOf:` / `isMemberOf:` + 26 tests. Universal at `st-primitive-send`. 465/465 total.
- 2026-04-25: `Object>>perform:` family + 10 tests. Universal dispatch via `st-send` after `(str (nth args 0))` for the selector. 439/439 total.
- 2026-04-25: Phase 4 reflection accessors (`lib/smalltalk/tests/reflection.sx`, 26 tests). Universal `Object>>class`, plus `methodDict`/`selectors`/`instanceVariableNames`/`allInstVarNames`/`classMethodDict`/`classSelectors` on class-refs. 429/429 total.
- 2026-04-25: conformance.sh + scoreboard.{json,md} (`lib/smalltalk/conformance.sh`, `lib/smalltalk/scoreboard.json`, `lib/smalltalk/scoreboard.md`). Single-pass runner over `test.sh -v`; baseline at 5 programs / 39 corpus tests / 403 total. **Phase 3 complete.**
- 2026-04-25: classic-corpus #5 Life (`tests/programs/life.st`, 4 tests). Spec-interpreter Conway's Life with edge handling. Block + blinker + glider initial setup verified; larger step counts pending JIT (each spec-interpreter step is ~5-8s on a 5x5 grid). Added `{e1. e2. e3}` dynamic array literal to parser + evaluator. 403/403 total.
- 2026-04-25: classic-corpus #4 mandelbrot (`tests/programs/mandelbrot.st`, 7 tests). Escape-time iterator + grid counter. Discovered + fixed an immutable-list bug in `lit-array` eval — `map` produced an immutable list so `at:put:` raised; rebuilt via `append!`. Quicksort tests had been silently dropping ~7 cases due to that bug; now actually mutate. 399/399 total.
- 2026-04-25: classic-corpus #3 quicksort (`tests/programs/quicksort.st`, 9 tests). Lomuto partition; verified across duplicates, already-sorted/reverse-sorted, empty, single, negatives, all-equal, plus in-place mutation. 385/385 total.
- 2026-04-25: classic-corpus #2 eight-queens (`tests/programs/eight-queens.st`, 5 tests). Backtracking search; verified for boards of size 1, 4, 5. Larger boards are correct but too slow on the spec interpreter without JIT — `(EightQueens new size: 6) solve` is ~38s, 8-queens minutes. 382/382 total.
- 2026-04-25: classic-corpus #1 fibonacci (`tests/programs/fibonacci.st` + `tests/programs.sx`, 13 tests). Added `smalltalk-load` chunk loader, class-side `subclass:instanceVariableNames:` (and longer Pharo variants), `Array new:` size, `methodsFor:`/`category:` no-ops, `st-split-ivars`. 377/377 total.
- 2026-04-25: cannotReturn: implemented (`lib/smalltalk/tests/cannot_return.sx`, 5 tests). Each method-invocation gets an `{:active true}` cell shared with its blocks; `st-invoke` flips it on exit; `^expr` raises if the cell is dead. Tests use SX `guard` to catch the raise. Non-`^` blocks unaffected. 364/364 total.
- 2026-04-25: `ifTrue:` / `ifFalse:` family pinned (`lib/smalltalk/tests/conditional.sx`, 24 tests) + parser fix: `|` is now accepted as a binary selector in expression position (tokenizer still emits it as `bar` for block param/temp delimiting; `parse-binary-message` accepts both). Caught by `false | true` truncating silently to `false`. 359/359 total.
- 2026-04-25: `whileTrue:` / `whileFalse:` / no-arg variants pinned (`lib/smalltalk/tests/while.sx`, 14 tests). `st-block-while` returns nil per ANSI; behaviour verified under captured locals, nesting, early `^`, and zero/many iterations. 334/334 total.
- 2026-04-25: BlockContext value family pinned (`lib/smalltalk/tests/blocks.sx`, 19 tests). Each value/valueN/valueWithArguments: variant verified plus closure semantics (read, write, later-mutation re-read), nested blocks, and block-as-arg. 320/320 total.
- 2026-04-25: **THE SHOWCASE** — non-local return via captured method-return continuations + 14 NLR tests (`lib/smalltalk/tests/nlr.sx`). `st-invoke` wraps body in `call/cc`; blocks copy creating method's `^k`; `^expr` invokes that k. Verified across nested blocks, `to:do:` / `whileTrue:`, blocks passed to different methods (Caller→Helper escapes back to Caller), inner-vs-outer method nesting. Sentinel-based return removed. 301/301 total.
- 2026-04-25: `super` send + 9 tests (`lib/smalltalk/tests/super.sx`). `st-super-send` walks from defining-class's superclass; class-side aware; primitives → DNU fallback. Also fixed top-level `| temps |` parsing in `st-parse` (the absence of which was silently aborting earlier eval/dnu tests — counts go from 274 → 287, with previously-skipped tests now actually running).
- 2026-04-25: `doesNotUnderstand:` + 12 DNU tests (`lib/smalltalk/tests/dnu.sx`). Bootstrap installs `Message` (with selector/arguments accessors). Primitives signal `:unhandled` instead of erroring; `st-dnu` builds a Message and walks `doesNotUnderstand:` lookup. User Object DNU intercepts unknown sends to native receivers (Number, String, Block) too. 267/267 total.
- 2026-04-25: method-lookup cache (`st-method-cache` keyed by `class|selector|side`, stores `:not-found` for misses). Invalidation on define/add/remove + bootstrap. `st-class-remove-method!` added. Stats helpers + 10 cache tests; 255/255 total.
- 2026-04-25: `smalltalk-eval-ast` + 60 eval tests (`lib/smalltalk/eval.sx`, `lib/smalltalk/tests/eval.sx`). Frame chain with mutable locals/ivars (via `dict-set!`), full literal eval, send dispatch (user methods + native primitive tables for Number/String/Boolean/Nil/Array/Block/Class), block closures, while/to:do:, cascades returning last, sentinel-based `^return`. User Point class round-trip works including `+` returning a fresh point. 245/245 total.
- 2026-04-25: class table + bootstrap (`lib/smalltalk/runtime.sx`, `lib/smalltalk/tests/runtime.sx`). Canonical hierarchy, type→class mapping for native SX values, instance construction, ivar inheritance, method install with `:defining-class` stamp, instance- and class-side method lookup walking the superclass chain. 54 new tests, 185/185 total.
- 2026-04-25: chunk-stream parser + pragmas + 21 chunk/pragma tests (`lib/smalltalk/tests/parse_chunks.sx`). `st-read-chunks` (with `!!` doubling), `st-parse-chunks` state machine for `methodsFor:` batches incl. class-side. Pragmas with multiple keyword pairs, signed numeric / string / symbol args, in either pragma-then-temps or temps-then-pragma order. 131/131 tests pass.
- 2026-04-25: expression-level parser + 47 parse tests (`lib/smalltalk/parser.sx`, `lib/smalltalk/tests/parse.sx`). Full message precedence (unary > binary > keyword), cascades, blocks with params/temps, literal/byte arrays, assignment chain, method headers (unary/binary/keyword). Chunk-format `! !` driver deferred to a follow-up box. 110/110 tests pass.
- 2026-04-25: tokenizer + 63 tests (`lib/smalltalk/tokenizer.sx`, `lib/smalltalk/tests/tokenize.sx`, `lib/smalltalk/test.sh`). All token types covered except scaled decimals `1.5s2` (deferred). `#(` and `#[` emit open tokens; literal-array contents lexed as ordinary tokens for the parser to interpret.
## Blockers
_Shared-file issues that need someone else to fix. Minimal repro only._
- _(none yet)_

317
plans/sx-improvements.md Normal file
View File

@@ -0,0 +1,317 @@
# SX Language Improvements — roadmap
Language-building improvements to the SX evaluator, compiler, and standard library.
Ordered by impact and prerequisite chain. Each step is one loop commit.
## Roadmap complete (2026-05-07)
All 14 steps shipped in 14 commits on the `architecture` branch. Phase 1 (bug fixes:
JIT closures, letrec+resume), Phase 2 (E38 source info — subsumed by tokenizer fix),
Phase 3 (native ADTs: AdtValue, define-type, match, exhaustiveness on both hosts),
Phase 4 (parser/compiler plugin registry + worker), Phase 5 (perf: frame-records via
prim_call fast path, buffer-based serializer, JIT inline opcodes). Cumulative
performance wins on hot benchmarks: CEK fib -66% / loop -69% / reduce -86% (Step 12);
inspect tree-d10 -80% / dict-1000 -61% (Step 13); VM JIT fib -69% / loop -62% / sum
-50% / count-lt -38% / count-eq -58% (Step 14). Test suite: 4550/4550 OCaml.
Branch: `architecture`. SX files via `sx-tree` MCP only. Never edit generated files.
## Current baseline (2026-05-06)
- SX core spec: 2571 passing (595 non-HS pre-existing failures — bytecode-serialize, defcomp-render, etc.)
- HyperScript behavioral: 1478/1496 (run via `node tests/hs-kernel-eval.js`)
- Active bugs: JIT combinator bug (11 HS failures), letrec+resume (browser-only)
- E38 sourceInfo: 2/4 tests passing (tokenizer missing `:end`/`:line`, some spans incomplete)
---
## Phase 1 — Bug fixes
### Step 1: Fix JIT closures-returning-closures
**What:** `parse-bind`, `many`, `seq`, and other parser combinators that return closures
miscompile under JIT. The compiled closure drops intermediate stack values when the
callee itself returns a closure. 11 HyperScript tests fail under JIT, pass under CEK.
**Root cause in `hosts/ocaml/lib/sx_vm.ml`:** When a JIT-compiled closure returns
another closure (i.e. the callee is `VmClosure`), the frame restoration after the
call incorrectly reuses the parent frame's locals slot, overwriting saved intermediate
values. The `call_closure_reuse` path must snapshot `sp` before the inner call and
restore it after, or bail to the non-reuse path for closures-returning-closures.
**Verify:** `node tests/hs-kernel-eval.js 2>&1 | tail -3` — should go from 3116/3127 to 3127/3127.
### Step 2: Fix letrec + perform resume (browser)
**What:** In browser JIT mode, `letrec` sibling bindings are nil after a `perform`/resume
cycle. `call_closure_reuse` in `sx_browser.ml` intentionally ignores `_saved_sp`, which
strips the frame locals that `sf_letrec` was waiting on.
**Fix:** In `sx_browser.ml`, the `VmSuspension` resume path must restore frame locals
from the suspension snapshot before calling the continuation. Mirror what `sx_vm.ml`
does in the non-browser case.
**Verify:** Write a test in `spec/tests/` that does `(letrec ((f (fn () (perform :io nil)))) (f))` with a resume, check bindings survive. Runs under OCaml: `dune exec -- bin/run_tests.exe`.
---
## Phase 2 — Source info (E38 completion)
Design: `plans/designs/e38-sourceinfo.md`. Target: 4/4 sourceInfo tests.
The API (`hs-parse-ast`, `hs-source-for`, `hs-line-for`, `hs-node-get`, `hs-src`,
`hs-src-at`, `hs-line-at`) and parser span wrapping (`hs-ast-wrap`, `hs-span-mode`)
are already in the codebase. Two tests are passing; two fail because:
- Tokenizer tokens lack `:end` and `:line` (only `:pos` today).
- Some statement-level spans and `:next` field navigation are incomplete.
### Step 3: Tokenizer — add `:end` and `:line` to tokens
`lib/hyperscript/tokenizer.sx`: extend `hs-make-token` to `{:pos :end :value :type :line}`.
Track a `current-line` counter (1-based, increments after `\n`). Update all ~20 emission
sites. Mirror to `shared/static/wasm/sx/hs-tokenizer.sx` after edits.
**Verify:** `(hs-make-token "NUMBER" "1" 0)` returns a dict with `:end` and `:line` keys.
### Step 4: Complete parser spans + :next field
`lib/hyperscript/parser.sx`: ensure `hs-ast-wrap` populates `:next` on every command
in a `CommandList` (i.e. the following sibling command). Check that statement-level
productions (if, for) correctly populate `:true-branch`. Trace through the two failing
tests (`get source works for expressions`, `get line works for statements`) to find the
exact missing fields or off-by-one positions.
Mirror to `shared/static/wasm/sx/hs-parser.sx`.
**Verify:** All 4 `hs-upstream-core/sourceInfo` tests pass.
**Outcome:** Subsumed by Step 3. Once tokens carried `:end` and `:line`, the existing
parser plumbing (`link-next-cmds` for `:next`, `:true-branch` extraction in `parse-cmd`)
worked end-to-end. All 4 `hs-upstream-core/sourceInfo` tests pass with no parser changes.
---
## Phase 3 — Native ADTs (`define-type` / `match`)
Design: `plans/designs/sx-adt.md`. No existing implementation.
Impact: every language implementation (Haskell, Prolog, Lua, Common Lisp, Erlang)
currently fakes sum types with `{:tag "..." :field ...}` dicts. Native ADTs remove
that everywhere.
### Step 5: OCaml — AdtValue type + `define-type` + basic `match`
`hosts/ocaml/lib/sx_types.ml`:
```ocaml
type adt_value = { av_type: string; av_ctor: string; av_fields: value array }
| AdtValue of adt_value
```
`hosts/ocaml/lib/sx_runtime.ml` (or evaluator):
- `step-sf-define-type`: parse `(Name (Ctor1 f1 f2) (Ctor2) ...)`, register constructor
NativeFns, predicates (`Ctor1?`, `Name?`), field accessors (`Ctor1-f1`) via `env-bind!`.
- `step-sf-match` + `MatchFrame`: linear scan of clauses; flat patterns only for 6a;
bind pattern variables in child env; `else` clause; raise on no match.
- `type-of` returns the type name (e.g. `"Maybe"`).
Write tests in `spec/tests/test-adt.sx`: basic constructor, predicate, accessor, match,
else, no-match raise.
**Verify:** `dune exec -- bin/run_tests.exe` — new test file all green.
### Step 6: JS — AdtValue + `define-type` + `match`
`hosts/javascript/platform.py`: add `AdtValue` as `{ _adt: true, _type, _ctor, _fields }`.
Mirror `define-type` and `match` special forms in the JS evaluator.
Retranspile: `python3 hosts/javascript/cli.py --output shared/static/scripts/sx-browser.js`
**Verify:** `node hosts/javascript/run_tests.js` — adt tests pass on JS too.
### Step 7: Nested patterns (Phase 6b)
Both OCaml and JS `MatchFrame`: replace linear binding with recursive
`matchPattern(pattern, value, env)` that:
- Recurses into constructor sub-patterns.
- Returns `{matched: bool, bindings: map}`.
- Handles wildcard `_`, literals (`42`, `"str"`, `true`, `nil`).
Extend `spec/tests/test-adt.sx` with nested pattern tests.
**Outcome:** No host-side changes needed. The spec-level `match-pattern` function
in `spec/evaluator.sx` (≈line 2835) already recurses through constructor
sub-patterns via the dict-shape shim (`(get value :_adt|:_ctor|:_fields)`),
handles `_` wildcards, literals, and variable bindings. Step 7 added 8 new
deftests to `spec/tests/test-adt.sx` covering: nested constructor sanity,
nested constructor with field binding, nested wildcard, nested literal
equality, nested literal-vs-var clause fall-through, deeply nested constructors,
mixed bind+wildcard, and nested ctor fail-through. Both hosts: +8 tests pass,
zero regressions (OCaml 4532→4540, JS 2578→2586).
### Step 8: Exhaustiveness warnings (Phase 6c)
`_adt_registry: type_name → [ctor_names]` global populated by `define-type`.
On first non-exhaustive `match` evaluation: `console.warn("[sx] match: non-exhaustive …")`.
No error — warning only.
**Outcome:** `host-warn` primitive added on both hosts (OCaml `prerr_endline`,
JS `console.warn`). Spec-level helpers `match-clause-is-else?`,
`match-clause-ctor-name`, `match-warn-non-exhaustive`,
`match-check-exhaustiveness` added in `spec/evaluator.sx` and
called from `step-sf-match`. `*adt-warned*` env-bound dict used to
dedupe warnings per (type, missing-set). The OCaml `step_sf_match`
in `hosts/ocaml/lib/sx_ref.ml` was hand-patched (not retranspiled)
because `sx_ref.ml` retranspilation drops several preamble fixes;
the spec changes still flow to JS via `sx_build target="js"`. Both
hosts emit identical warnings (e.g. `[sx] match: non-exhaustive — Maybe: missing Nothing`).
5 new tests added. OCaml: 4540 → 4545. JS: 2586 → 2591. Zero regressions.
---
## Phase 4 — Plugin / extension system
Design: `plans/designs/hs-plugin-system.md`.
### Step 9: Parser feature registry
`lib/hyperscript/parser.sx`: replace `parse-feat` hardcoded `cond` with a dict lookup.
`(hs-register-feature! name parse-fn)` adds to the registry.
### Step 10: Compiler command registry + `as` converter registry
`lib/hyperscript/compiler.sx`: replace `hs-to-sx` hardcoded dispatch with dict.
`(hs-register-command! name compile-fn)` and `(hs-register-converter! name convert-fn)`.
### Step 11: Migrate hs-prolog-hook + Worker plugin
`lib/hyperscript/runtime.sx`: remove `hs-prolog-hook`/`hs-set-prolog-hook!` ad-hoc
slots. Create `lib/hyperscript/plugins/prolog.sx` that calls `hs-register-feature!`
and `hs-register-command!`. Create `lib/hyperscript/plugins/worker.sx` replacing the
E39 stub.
---
## Phase 5 — Performance
These are incremental and can interleave with other phases.
### Step 12: Frame records (CEK)
`hosts/ocaml/lib/sx_runtime.ml`: represent CEK frames as OCaml records instead of
tagged variant lists. Eliminates allocation pressure from list construction per frame.
Profile before/after on a tight-loop benchmark.
**Outcome:** Frames were already records (`cek_frame` in `sx_types.ml`) — the actual
hot-path bottleneck was `prim_call "=" [...]` in `step_continue`/`step_eval` dispatch:
each step did a Hashtbl lookup + 2x list cons + pattern match per comparison. Added a
fast path in `prim_call` (sx_runtime.ml) for `=`, `<`, `>`, `<=`, `>=`, `empty?`,
`first`, `rest`, `len` that skips the table lookup entirely. Also inlined `_fast_eq`
for the common scalar-equality cases that dominate frame-type dispatch. Median
improvements (bench_cek.exe, 7 runs):
| Benchmark | Before | After | Change |
|-----------|--------|-------|--------|
| fib(18) | 2789ms | 941ms | -66% |
| loop(5000) | 2018ms | 620ms | -69% |
| map sq(1000) | 108ms | 48ms | -56% |
| reduce + (2000) | 72ms | 10ms | -86% |
| let-heavy(2000) | 491ms | 271ms | -45% |
Tests: 4545 passing (unchanged baseline), 1339 failing (unchanged baseline).
Benchmark binary: `bin/bench_cek.exe`.
### Step 13: Buffer primitive for string building
Add `make-buffer`, `buffer-append!`, `buffer->string` primitives. Eliminates the
`(str a b c d ...)` quadratic allocation pattern in serializers and renderers.
Wire into `sx_primitives.ml` and the JS platform.
**Outcome:** Short aliases `make-buffer`/`buffer?`/`buffer-append!`/`buffer->string`/
`buffer-length` added on both hosts, sharing the existing `StringBuffer` value type.
`buffer-append!` accepts any value (auto-coerces non-strings via inspect), unlike
`string-buffer-append!` which is strict. The hot path converted was the OCaml
host-internal `inspect` function in `sx_types.ml`: rewrote from `(... ^ String.concat
" " (List.map inspect items) ^ ...)` (which allocates O(n) intermediate strings per
recursion level) to a single shared `Buffer.t` accumulator (`inspect_into buf v`
walks the value tree appending into one buffer). `inspect` is called by
`sx-serialize` on both spec and host paths, plus error-path formatting.
Median improvements (`bin/bench_inspect.exe`, best of 3 runs of 9-run min):
| Benchmark | Baseline (best min) | Buffer (best min) | Change |
|-------------------|--------------------:|------------------:|-------:|
| tree-d8 (75KB) | 5.31ms | 1.30ms | -76% |
| tree-d10 (679KB) | 81.89ms | 16.02ms | -80% |
| dict-1000 | 0.80ms | 0.31ms | -61% |
| list-2000 | 0.74ms | 0.33ms | -55% |
5 new tests in `spec/tests/test-string-buffer.sx` covering the new aliases (incl
non-string coercion and interop with the existing `string-buffer-*` API).
OCaml: 4545 → 4550. JS: 2591 → 2596. Zero regressions.
### Step 14: Inline common primitives in JIT
`hosts/ocaml/lib/sx_vm.ml`: add `OP_ADD`, `OP_SUB`, `OP_EQ`, `OP_APPEND` specialised
opcodes that skip the primitive table lookup for the most common calls. Compiler emits
these when operands are known numbers/lists.
**Outcome:** The opcodes (`OP_ADD`=160, `OP_SUB`=161, `OP_MUL`=162, `OP_DIV`=163,
`OP_EQ`=164, `OP_LT`=165, `OP_GT`=166, `OP_NOT`=167, `OP_LEN`=168, `OP_FIRST`=169,
`OP_REST`=170, `OP_CONS`=172) already existed in `sx_vm.ml` but the compiler never
emitted them — every primitive call went through `OP_CALL_PRIM` (52) with a Hashtbl
lookup. Two changes:
1. **`lib/compiler.sx` `compile-call`**: when the primitive name + arity matches a
specialized opcode, emit the 1-byte opcode (no name index, no argc operand)
instead of the 4-byte CALL_PRIM. Bytecode for `fib` shrank from 50→38 bytes.
2. **`hosts/ocaml/lib/sx_vm.ml` opcode bodies**: extended `OP_ADD/SUB/MUL/DIV` to
handle `Integer + Integer` (was `Number + Number` only — defaulted to Hashtbl
for the common integer case). Inlined `OP_EQ` to call `Sx_runtime._fast_eq`
directly. Inlined `OP_LT/GT` integer + mixed-numeric comparisons.
Median improvements (`bin/bench_vm.exe`, best of 3 runs of 9-min):
| Benchmark | Baseline (best min) | After (best min) | Change |
|------------------|---------------------|------------------|-------:|
| fib(22) | 107.87ms | 33.13ms | -69% |
| loop(200000) | 429.64ms | 161.16ms | -62% |
| sum-to(50000) | 72.85ms | 36.74ms | -50% |
| count-lt(20000) | 28.44ms | 17.58ms | -38% |
| count-eq(20000) | 37.23ms | 15.46ms | -58% |
Tests: 4550/4550 passing (unchanged baseline). Zero regressions. Benchmark binary:
`bin/bench_vm.exe` (loads `lib/compiler.sx` via CEK, JIT-compiles each test fn,
measures `Sx_vm.call_closure` time on the compiled `vm_closure`).
---
## Progress log
| Step | Status | Commit |
|------|--------|--------|
| 1 — JIT combinator bug | [x] | 882a4b76 |
| 2 — letrec+resume | [x] | e80e655b |
| 3 — tokenizer :end/:line | [x] | 023bc2d8 |
| 4 — parser spans complete | [x] | b7ad5152 (subsumed by 023bc2d8) |
| 5 — OCaml AdtValue + define-type + match | [x] | 1f49242a |
| 6 — JS AdtValue + define-type + match | [x] | fc8a3916 |
| 7 — nested patterns | [x] | 0679edf5 |
| 8 — exhaustiveness warnings | [x] | 6d391119 |
| 9 — parser feature registry | [x] | 986d6411 |
| 10 — compiler + as converter registry | [x] | d22361e4 |
| 11 — plugin migration + worker | [x] | 6328b810 |
| 12 — frame records | [x] | a66c0f66 (fib -66%, loop -69%, reduce -86% via prim_call fast path) |
| 13 — buffer primitive | [x] | 0e022ab6 (inspect rewrite: tree-d10 -80%, tree-d8 -76%, dict-1000 -61%, list-2000 -55%) |
| 14 — inline primitives JIT | [x] | 6c171d49 (fib -69%, loop -62%, sum -50%, count-lt -38%, count-eq -58% via specialized opcode emission) |
---
## Rules
- Branch: `architecture`. Never push to `main`.
- SX files: `sx-tree` MCP tools only. `sx_validate` after every edit.
- After every `.sx` edit to `lib/hyperscript/`, mirror to `shared/static/wasm/sx/hs-<file>.sx`.
- OCaml build: `sx_build target="ocaml"` MCP tool (never raw `dune`).
- JS build: `sx_build target="js"` MCP tool.
- One step per commit. Update progress log in this file.
- No new planning docs. No comments in SX unless non-obvious.
- Unicode in SX: raw UTF-8 only, never `\uXXXX`.

139
plans/tcl-on-sx.md Normal file
View File

@@ -0,0 +1,139 @@
# Tcl-on-SX: uplevel/upvar = stack-walking delcc, everything-is-a-string
The headline showcase is **uplevel/upvar** — Tcl's superpower for defining your own control structures. `uplevel` evaluates a script in the *caller's* stack frame; `upvar` aliases a variable in the caller. On a normal language host this requires deep VM cooperation; on SX it falls out of the env-chain made first-class via captured continuations. Plus the *Dodekalogue* (12 rules), command-substitution everywhere, and "everything is a string" homoiconicity.
End-state goal: Tcl 8.6-flavoured subset, the Dodekalogue parser, namespaces, `try`/`catch`/`return -code`, `coroutine` (built on fibers), classic programs that show off uplevel-driven DSLs, ~150 hand-written tests.
## Scope decisions (defaults — override by editing before we spawn)
- **Syntax:** Tcl 8.6 surface. The 12-rule Dodekalogue. Brace-quoted scripts deferred-evaluate; double-quoted ones substitute.
- **Conformance:** "Reads like Tcl, runs like Tcl." Slice of Tcl's own test suite, not full TCT.
- **Test corpus:** custom + curated `tcl-tests/` slice. Plus classic programs: define-your-own `for-each-line`, expression-language compiler-in-Tcl, fiber-based event loop.
- **Out of scope:** Tk, sockets beyond a stub, threads (mapped to `coroutine` only), `package require` of binary loadables, `dde`/`registry` Windows shims, full `clock format` locale support.
- **Channels:** `puts` and `gets` on `stdout`/`stdin`/`stderr`; `open` on regular files; no async I/O beyond what `coroutine` gives.
## Ground rules
- **Scope:** only touch `lib/tcl/**` and `plans/tcl-on-sx.md`. Don't edit `spec/`, `hosts/`, `shared/`, or any other `lib/<lang>/**`. Tcl primitives go in `lib/tcl/runtime.sx`.
- **SX files:** use `sx-tree` MCP tools only.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick roadmap boxes.
## Architecture sketch
```
Tcl source
│
▼
lib/tcl/tokenizer.sx — the Dodekalogue: words, [..], ${..}, "..", {..}, ;, \n, \, #
│
▼
lib/tcl/parser.sx — list-of-words AST (script = list of commands; command = list of words)
│
▼
lib/tcl/transpile.sx — AST → SX AST (entry: tcl-eval-script)
│
▼
lib/tcl/runtime.sx — env stack, command table, uplevel/upvar, coroutines, BIFs
```
Core mapping:
- **Value** = string. Internally we cache a "shimmer" representation (list, dict, integer, double) for performance, but every value can be re-stringified.
- **Variable** = entry in current frame's env. Frames form a stack; level-0 is the global frame.
- **Command** = entry in command table; first word of any list dispatches into it. User-defined via `proc`. Built-ins are SX functions registered in the table.
- **Frame** = `{:locals (dict) :level n :parent frame}`. Each `proc` call pushes a frame; commands run in current frame.
- **`uplevel #N script`** = walk frame chain to absolute level N (or relative if no `#`); evaluate script in that frame's env.
- **`upvar [#N] varname localname`** = bind `localname` in the current frame as an alias to `varname` in the level-N frame (env-chain delegate).
- **`return -code N`** = control flow as integers: 0=ok, 1=error, 2=return, 3=break, 4=continue. `catch` traps any non-zero; `try` adds named handlers.
- **`coroutine`** = fiber on top of `perform`/`cek-resume`. `yield`/`yieldto` suspend; calling the coroutine command resumes.
- **List / dict** = list-shaped string ("element1 element2 …") with a cached parsed form. Modifications dirty the string cache.
## Roadmap
### Phase 1 — tokenizer + parser (the Dodekalogue)
- [x] Tokenizer applying the 12 rules:
1. Commands separated by `;` or newlines
2. Words separated by whitespace within a command
3. Double-quoted words: `\` escapes + `[…]` + `${…}` + `$var` substitution
4. Brace-quoted words: literal, no substitution; brace count must balance
5. Argument expansion: `{*}list`
6. Command substitution: `[script]` evaluates script, takes its return value
7. Variable substitution: `$name`, `${name}`, `$arr(idx)`, `$arr($i)`
8. Backslash substitution: `\n`, `\t`, `\\`, `\xNN`, `\uNNNN`, `\<newline>` continues
9. Comments: `#` only at the start of a command
10. Order of substitution is left-to-right, single-pass
11. Substitutions don't recurse — substituted text is not re-parsed
12. The result of any substitution is the value, not a new script
- [x] Parser: script = list of commands; command = list of words; word = literal string + list of substitutions
- [x] Unit tests in `lib/tcl/tests/parse.sx`
### Phase 2 — sequential eval + core commands
- [x] `tcl-eval-script`: walk command list, dispatch each first-word into command table
- [x] Core commands: `set`, `unset`, `incr`, `append`, `lappend`, `puts`, `gets`, `expr`, `if`, `while`, `for`, `foreach`, `switch`, `break`, `continue`, `return`, `error`, `eval`, `subst`, `format`, `scan`
- [x] `expr` is its own mini-language — operator precedence, function calls (`sin`, `sqrt`, `pow`, `abs`, `int`, `double`), variable substitution, command substitution
- [x] String commands: `string length`, `string index`, `string range`, `string compare`, `string match`, `string toupper`, `string tolower`, `string trim`, `string map`, `string repeat`, `string first`, `string last`, `string is`, `string cat`
- [x] List commands: `list`, `lindex`, `lrange`, `llength`, `lreverse`, `lsearch`, `lsort`, `lsort -integer/-real/-dictionary`, `lreplace`, `linsert`, `concat`, `split`, `join`
- [x] Dict commands: `dict create`, `dict get`, `dict set`, `dict unset`, `dict exists`, `dict keys`, `dict values`, `dict size`, `dict for`, `dict update`, `dict merge`
- [x] 60+ tests in `lib/tcl/tests/eval.sx`
### Phase 3 — proc + uplevel + upvar (THE SHOWCASE)
- [x] `proc name args body` — register user-defined command; args supports defaults `{name default}` and rest `args`
- [x] Frame stack: each proc call pushes a frame with locals dict; pop on return
- [x] `uplevel ?level? script` — evaluate `script` in level-N frame's env; default level is 1 (caller). `#0` is global, `#1` is relative-1
- [x] `upvar ?level? otherVar localVar ?…?` — alias localVar to a variable in level-N frame; reads/writes go through the alias
- [x] `info level`, `info level N`, `info frame`, `info vars`, `info locals`, `info globals`, `info commands`, `info procs`, `info args`, `info body`
- [x] `global var ?…?` — alias to global frame (sugar for `upvar #0 var var`)
- [x] `variable name ?value?` — namespace-scoped global
- [x] Classic programs in `lib/tcl/tests/programs/`:
- [x] `for-each-line.tcl` — define your own loop construct using `uplevel`
- [x] `assert.tcl` — assertion macro that reports caller's line
- [x] `with-temp-var.tcl` — scoped variable rebind via `upvar`
- [x] `lib/tcl/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md`
### Phase 4 — control flow + error handling
- [x] `return -code (ok|error|return|break|continue|N) -errorinfo … -errorcode … -level N value`
- [x] `catch script ?resultVar? ?optionsVar?` — runs script, returns code; sets resultVar to return value/message; optionsVar to the dict
- [x] `try script ?on code var body ...? ?trap pattern var body...? ?finally body?`
- [x] `throw type message`
- [x] `error message ?info? ?code?`
- [x] Stack-trace with `errorInfo` / `errorCode`
- [x] 30+ tests in `lib/tcl/tests/error.sx`
### Phase 5 — namespaces + ensembles
- [x] `namespace eval ns body`, `namespace current`, `namespace which`, `namespace import`, `namespace export`, `namespace forget`, `namespace delete`
- [x] Qualified names: `::ns::cmd`, `::ns::var`
- [x] Ensembles: `namespace ensemble create -map { sub1 cmd1 sub2 cmd2 }`
- [x] `namespace path` for resolution chain
- [x] `proc` and `variable` work inside namespaces
### Phase 6 — coroutines + drive corpus
- [x] `coroutine name cmd ?args…?` — start a coroutine; future calls to `name` resume it
- [x] `yield ?value?` — suspend, return value to resumer
- [x] `yieldto cmd ?args…?` — symmetric transfer
- [x] `coroutine` semantics built on fibers (same delcc primitive as Ruby fibers)
- [x] Classic programs: `event-loop.tcl` — cooperative scheduler with multiple coroutines
- [x] System: `clock seconds`, `clock format`, `clock scan` (subset)
- [x] File I/O: `open`, `close`, `read`, `gets`, `puts -nonewline`, `flush`, `eof`, `seek`, `tell`
- [x] Drive corpus to 150+ green
- [x] Idiom corpus — `lib/tcl/tests/idioms.sx` covering classic Welch/Jones idioms
## Progress log
_Newest first._
- 2026-05-06: Phase 6 coroutines+clock+file+idioms — generator coroutines, clock/file stubs, 20 coroutine + 20 idiom tests, event-loop.tcl, 329 tests green
- 2026-05-06: Phase 5 namespaces+ensembles — namespace eval/current/which/exists/delete/import/ensemble, qualified names, 289 tests green (22 new namespace tests)
- 2026-05-06: Phase 4 error handling — catch/try/throw/return-code/errorinfo/errorcode, 267 tests green (39 new error tests)
- 2026-05-06: Phase 3 conformance.sh + classic programs — 3/3 PASS (for-each-line/assert/with-temp-var), 228 tests green
- 2026-05-06: Phase 3 proc+uplevel+upvar+info+global — frame stack, isolated proc scope, alias-following var access, 225 tests green (67 parse + 158 eval)
- 2026-05-06: Phase 2 dict commands — 13 subcommands (create/get/set/unset/exists/keys/values/size/for/update/merge/incr/append), 206 tests green (67 parse + 139 eval)
- 2026-05-06: Phase 2 list commands — 12 commands (list/lindex/lrange/llength/lreverse/lsearch/lsort/lreplace/linsert/concat/split/join), 182 tests green (67 parse + 115 eval)
- 2026-05-06: Phase 2 string commands — 16 subcommands (length/index/range/compare/match/toupper/tolower/trim/map/repeat/first/last/is/cat), 156 tests green (67 parse + 89 eval)
- 2026-05-06: Phase 2 expr mini-language — recursive descent parser, operator precedence, parens, unary ops, pow/sqrt/abs/max/min/int/double, 127 tests green (67 parse + 60 eval)
- 2026-04-26: Phase 2 core commands — if/while/for/foreach/switch/break/continue/return/error/unset/lappend/eval/expr + :code control flow, 107 tests green (67 parse + 40 eval)
- 2026-04-26: Phase 2 eval engine — `lib/tcl/runtime.sx`, tcl-eval-script + set/puts/incr/append, 87 tests green (67 parse + 20 eval)
- 2026-04-25: Phase 1 parser — `lib/tcl/parser.sx`, word-simple?/word-literal helpers, 67 tests green, commit 6ee05259
- 2026-04-25: Phase 1 tokenizer (Dodekalogue) — `lib/tcl/tokenizer.sx`, 52 tests green, commit 666e29d5
## Blockers
- _(none yet)_

333
plans/tcl-sx-completion.md Normal file
View File

@@ -0,0 +1,333 @@
# Tcl-on-SX completion plan — SX capabilities first
Tcl phases 1–6 are complete (329/329 tests). This plan covers the remaining
limitations, ordered by the SX work needed to enable them.
## Key audit findings
Several apparent gaps are already solved in SX:
- **Floats** — SX parses `3.14` natively; `(+ 1.5 2.5) → 4.0`; `str` formats
with `%g` (compact, no trailing zeros). `floor`/`ceil`/`round`/`truncate`
all exist in `spec/primitives.sx`.
- **Regex** — `regexp-match`, `regexp-match-all`, `regexp-replace`,
`regexp-replace-all`, `regexp-split` are registered OCaml primitives using
`Re.Pcre` (`hosts/ocaml/lib/sx_primitives.ml`).
- **`call/cc` multi-shot** — works. `set!` on closed-over vars works. Fibers
are implementable as a pure SX library.
- **`perform` user-accessible** — `(perform :foo 42)` from user code suspends
the evaluator and emits an IO request. The algebraic effects model is
already half-built.
- **No `file-read`/`clock-seconds`** — not yet registered as OCaml primitives.
Only string ports exist. Would need small OCaml additions.
- **No `env-as-value`** — environments are internal OCaml values, not
inspectable from SX user code.
---
## Phase 1 — Zero-cost wins (no SX changes, only `lib/tcl/`)
Everything here is pure Tcl implementation work.
| Status | Work | Effort | Unlocks in Tcl |
|---|---|---|---|
| [x] | Float in `expr` — detect `.` in number tokens, route through float ops instead of `parse-int` | half day | `expr {3.14 * 2}`, `expr {sqrt(2.0)}`, float comparisons |
| [x] | `regexp pattern str` and `regsub pattern str repl` wrapping existing SX primitives | few hours | pattern matching, text processing |
| [x] | `apply {args body} ?arg…?` — anonymous proc call | 1 hour | higher-order functions, `lmap` idiom |
| [x] | `array get/set/names/size/exists/unset` commands | half day | array variables (tokenizer already parses `$arr(key)`) |
**Total: ~2 days. Zero SX changes.**
---
## Phase 2 — `lib/fiber.sx` (pure SX library, no OCaml)
| Status | Work |
|---|---|
| [x] | Create `lib/fiber.sx` — `make-fiber` / `fiber-resume` / `fiber-done?` |
| [x] | Rewrite `tcl-cmd-coroutine` to use `make-fiber` (true suspension) |
`call/cc` is multi-shot and `set!` on closed-over vars both work. Fibers are
implementable as a pure SX library using symmetric continuation swapping:
```scheme
; lib/fiber.sx — canonical fiber primitive for all hosted languages
(define make-fiber
(fn (thunk)
(define slot-k nil)
(define slot-caller nil)
(define slot-done false)
(fn (resume-val)
(call/cc (fn (caller-k)
(set! slot-caller caller-k)
(if (nil? slot-k)
(begin (thunk resume-val) (set! slot-done true) (caller-k nil))
(slot-k resume-val)))))))
(define fiber-yield
(fn (val)
(call/cc (fn (k)
(set! slot-k k)
(slot-caller val)))))
```
Each coroutine becomes a fiber. `yield` swaps to the caller; calling the
coroutine name swaps back. True suspension, not eager pre-execution.
**Broader value:** Ruby fibers, Python generators, Lua coroutines, async event
loops, cooperative schedulers all sit on top of the same library.
**Alternatively:** `perform` is user-accessible. A Tcl scheduler living outside
the SX evaluator (the OCaml host or an SX event loop) could catch
`(perform :fiber-yield val)` and dispatch it — the algebraic effects model,
already half-built.
**Total: 2–3 days. Produces `lib/fiber.sx` as a lasting SX contribution.**
Tcl coroutines then rewrite using `make-fiber` for true suspension.
---
## Phase 3 — Small OCaml additions (`sx_primitives.ml`)
Each is ~10–20 lines of OCaml. All are useful across the whole platform, not
just Tcl.
| Status | Primitive | OCaml effort | Unlocks |
|---|---|---|---|
| [x] | `(file-read path)` → string | tiny | Tcl `open`/`read`, SX scripts reading files |
| [x] | `(file-write path str)` → nil | tiny | Tcl `open`/`puts` to files |
| [x] | `(file-exists? path)` → bool | tiny | Tcl `file exists` |
| [x] | `(file-glob pattern)` → list | small | Tcl `glob` |
| [x] | `(clock-seconds)` → int | tiny | Tcl `clock seconds` |
| [x] | `(clock-format n fmt)` → string | small (wraps `strftime`) | Tcl `clock format` |
**Total: 1 day. One focused afternoon of OCaml.**
---
## Phase 4 — env-as-value (architectural) ✓
`uplevel`/`upvar` required an explicit frame stack because SX environments
aren't inspectable from user code. Adding:
```scheme
(current-env) ; → env value
(eval-in-env env expr) ; → result
(env-lookup env key) ; → value or nil
(env-extend env key val) ; → new env (non-mutating)
```
...would let `uplevel N` be literally "look up env N levels up, eval in it."
The Tcl frame stack (hundreds of lines) collapses to ~10 lines.
Also benefits: metacircular evaluators, REPL tooling, live debugging (inspect
any scope), the sx_docs server's eval endpoint.
More invasive — touches `sx_types.ml` and `sx_server.ml` — but a meaningful
architectural improvement worth doing when the moment is right.
**Total: 2–3 days. High architectural value, not urgent.**
---
## Phase 5 — Channel I/O (random access + non-blocking) ✓
Real Tcl channel commands replacing the previous stubs. SX gained 11 channel
primitives in `sx_primitives.ml` (using `Unix.openfile` + `Unix.read`/`write`/
`lseek`/`set_nonblock`). Tcl `open`/`close`/`read`/`gets`/`puts`/`seek`/`tell`/
`eof`/`flush`/`fconfigure` now wrap them.
| Status | Work | Unlocks in Tcl |
|---|---|---|
| [x] | `channel-open`, `channel-close` | `open` returns "fileN", `close` actually closes |
| [x] | `channel-read`, `channel-read-line`, `channel-write` | `read`/`gets`/`puts` to/from real files |
| [x] | `channel-seek`, `channel-tell` | random access — `seek $c offset start\|current\|end`, `tell` |
| [x] | `channel-eof?`, `channel-flush` | proper EOF detection, no-op flush |
| [x] | `channel-blocking?`, `channel-set-blocking!` | `fconfigure $c -blocking 0\|1` |
Modes supported: `r`, `w`, `a`, `r+`, `w+`, `a+`. Whence: `start`, `current`, `end`.
`puts` now detects channel argument (string starting with "file") and dispatches
to `channel-write`; otherwise writes to `interp :output` as before.
**Total: ~half day. 7 new idiom tests covering write+read, gets-loop, seek/tell,
eof-after-read, append mode, seek-to-end, fconfigure-blocking.**
---
## Phase 5b — Event loop: fileevent / after / vwait / update ✓
Tcl event-driven I/O scoped to script-mode (vs. server-side commands). The
mechanism rides on the existing IO suspension model: SX adds one new primitive
`(io-select-channels read-list write-list timeout-ms)` wrapping `Unix.select`,
and the Tcl event loop is implemented in Tcl itself (no sx_server.ml changes).
| Status | Work | Unlocks in Tcl |
|---|---|---|
| [x] | `io-select-channels` SX primitive | Unix.select on registered channels |
| [x] | `fileevent $chan readable\|writable script` | event handler registration; `{}` to unregister |
| [x] | `after ms script` | one-shot timer queued in `:timers` |
| [x] | `after ms` (no script) | sleep that drives the event loop |
| [x] | `vwait varname` | block until var set/changed, runs handlers |
| [x] | `update` | non-blocking event drain (poll, fire ready handlers) |
Event loop: `tcl-event-step interp poll-timeout-ms` — fires expired timers,
calls `io-select-channels` with fd list from `:fileevents`, runs ready handlers.
`vwait` polls every 1000ms or until var changes (whichever first); `update` is
`tcl-event-step interp 0`.
State on interp: `:fileevents` (list of `(chan event script)`) and `:timers`
(list of `(expiry-ms script)`, sorted by expiry).
**Trade-off:** Scoped to script mode — `vwait` from inside a server-handled
command would not interact with sx_server's stdin scheduler. Sufficient for ~95%
of real-world Tcl scripts (sockets, pipes, GUI-style polling, CLI tools).
**Total: ~half day. 5 new idiom tests: after-vwait-timer, after-multiple-timers-
update, fileevent-readable-fires, fileevent-query-script, after-cancel-via-
vwait-timing. 354/354 green.**
---
## Phase 5c — TCP sockets (client + server) ✓
Tcl `socket` command for both connecting and listening. Reuses the channel
registry built in Phase 5 and the event loop from Phase 5b. Server channels
auto-fire user callbacks via fileevent on each accept.
| Status | Work | Unlocks in Tcl |
|---|---|---|
| [x] | `socket-connect host port` SX primitive | TCP client via `Unix.socket`+`Unix.connect` |
| [x] | `socket-server ?host? port` SX primitive | listening socket; `Unix.bind`+`Unix.listen` (backlog 8) |
| [x] | `socket-accept server-chan` SX primitive | returns `{:channel :host :port}` |
| [x] | Tcl `socket host port` | TCP client; returns "sockN" |
| [x] | Tcl `socket -server cb port` | listening socket; auto-fires `cb sock host port` per accept |
| [x] | `puts` channel detection extended | "sockN" channels also dispatch to `channel-write` |
The auto-accept mechanism is a tiny internal Tcl command `_sock-do-accept`
registered by `socket -server`. Its registered handler, fired by the event
loop, accepts the pending client, then evaluates `cb client-chan host port`.
`Unix.SO_REUSEADDR` is set on server sockets to avoid TIME_WAIT issues
during testing. Host argument supports `localhost`, `0.0.0.0`, IPv4 literal,
or DNS lookup via `Unix.gethostbyname`.
**Total: ~half day. 4 new idiom tests: socket-server-fires-callback,
socket-client-server-roundtrip, socket-server-peer-host, socket-multiple-
connections. 358/358 green.**
---
## Phase 5d — File metadata + filesystem ops ✓
Real implementations of `file isfile`/`isdir`/`readable`/`writable`/`size`/
`mtime`/`atime`/`type` (previously stubs returning `0`/`""`) and proper
`file delete`/`mkdir`/`copy`/`rename`.
| Status | Primitive | Wraps |
|---|---|---|
| [x] | `file-size`, `file-mtime`, `file-stat` | `Unix.stat` |
| [x] | `file-isfile?`, `file-isdir?` | `Unix.stat`+`st_kind` |
| [x] | `file-readable?`, `file-writable?` | `Unix.access [R_OK\|W_OK]` |
| [x] | `file-delete` | `Unix.unlink`/`rmdir` (tolerates ENOENT) |
| [x] | `file-mkdir` | recursive `Unix.mkdir 0o755` |
| [x] | `file-copy`, `file-rename` | stdlib I/O / `Sys.rename` |
`file-stat` returns a dict `{:size :mtime :atime :ctime :mode :type}` with
`:type` ∈ `file|directory|link|fifo|socket|...`. Tcl `file copy`/`rename`/
`delete` strip leading-`-` flags so `file delete -force` works.
**Total: ~half day. 10 new idiom tests covering isfile, isdir on /tmp, size,
readable, mkdir + check, copy roundtrip, rename, mtime > 0. 368/368 green.**
---
## Phase 5e — clock format options + clock scan ✓
Real `-format`, `-timezone`, and `-gmt` options on `clock format`, and a
working `clock scan` for parsing date strings back to Unix seconds.
| Status | Work |
|---|---|
| [x] | `clock-format` extended to `(t fmt tz)` with tz ∈ `utc|local` |
| [x] | More format specifiers: `%y` (2-digit year), `%I` (12h hour), `%p` (AM/PM), `%w` (weekday num), `%%` (literal) |
| [x] | `clock-scan` SX primitive: format-driven parser + manual `timegm` (OCaml stdlib lacks it) |
| [x] | Tcl `clock format $secs -format $fmt -timezone $tz -gmt 0\|1` |
| [x] | Tcl `clock scan $str -format $fmt -timezone $tz -gmt 0\|1` |
Default tz for both is UTC. Format specifiers supported by scan: `%Y %y %m
%d %e %H %I %M %S %%`. Unsupported specifiers in scan are silently skipped
(no validation).
**Total: ~half day. 5 new idiom tests: clock-format-utc, fmt-default,
scan-roundtrip, scan-returns-int, format-percent-pct. 373/373 green.**
---
## Phase 5f — `socket -async` (non-blocking connect) ✓
| Status | Work |
|---|---|
| [x] | `socket-connect-async host port` SX primitive — `Unix.set_nonblock` + `Unix.connect`, catches `EINPROGRESS` |
| [x] | `channel-async-error chan` SX primitive — `Unix.getsockopt_error` |
| [x] | Tcl `socket -async host port` — returns "sockN" immediately |
| [x] | Tcl `fconfigure $chan -error` — queries async-error |
Connection completes when the channel becomes writable; canonical pattern is
`fileevent $sock writable {handler}`. Channel buffer state is set to
`blocking=false` so subsequent reads/writes don't block.
**Total: ~few hours. 3 new idiom tests: socket-async-completes-writable,
socket-async-then-write, socket-async-no-error. 376/376 green.**
**Bug fix landed alongside:** `tcl-call-proc` was discarding `:fileevents`,
`:timers`, and `:procs` updates made inside Tcl procs (only `:commands` was
forwarded). Changed the return to forward the inner `result-interp` as the
base while restoring caller's frame/stack/result/output/code. This was
masked until socket -async made it natural to register a `fileevent` from
inside a proc body (the typical async accept pattern).
---
## Suggested order
1. **Phase 1** — immediate Tcl wins, zero risk, proves the approach
2. **Phase 2** (`lib/fiber.sx`) — the interesting SX work, benefits all hosted languages
3. **Phase 3** (OCaml primitives) — quick practical completions
4. **Phase 4** — architectural cleanup when it's worth the invasiveness
Phases 1+2+3 ≈ one focused week. Tcl is genuinely complete, and `lib/fiber.sx`
becomes a lasting SX contribution used by every future hosted language.
---
## Progress log
_Newest first._
- 2026-05-07: Phase 5f socket -async — socket-connect-async (Unix.set_nonblock+connect/EINPROGRESS) + channel-async-error (getsockopt_error); Tcl `socket -async host port` returns immediately; `fconfigure $sock -error` queries async error; +3 idiom tests; 376/376 green
- 2026-05-07: Phase 5e clock options + scan — clock-format extended with tz arg (utc/local) + more specifiers; new clock-scan primitive with manual timegm; Tcl clock format/scan support -format/-timezone/-gmt; +5 idiom tests; 373/373 green
- 2026-05-07: Phase 5d file ops — file-size/mtime/isfile?/isdir?/readable?/writable?/stat/delete/mkdir/copy/rename SX primitives; Tcl file isfile/isdir/readable/writable/size/mtime/atime/type/mkdir/copy/rename/delete now real; +10 idiom tests; 368/368 green
- 2026-05-07: Phase 5c sockets — socket-connect/socket-server/socket-accept SX primitives wrapping Unix.socket/connect/bind/listen/accept; tcl-cmd-socket dispatches client (host port) vs server (-server cb port); server auto-registers fileevent → _sock-do-accept handler that calls user callback per accept; puts now dispatches "sockN" channels to channel-write too; +4 idiom tests; 358/358 green
- 2026-05-07: Phase 5b event loop — io-select-channels SX primitive + Tcl-side fileevent/after/vwait/update; tcl-event-step drives expired timers + Unix.select on registered channels; +5 idiom tests; 354/354 green
- 2026-05-07: Phase 5 channel I/O — 11 SX primitives (channel-open/close/read/read-line/write/flush/seek/tell/eof?/blocking?/set-blocking!) wrapping Unix.openfile/read/write/lseek/set_nonblock; tcl-cmd-open/close/read/gets-chan/seek/tell/flush rewritten + new tcl-cmd-fconfigure; tcl-cmd-puts dispatches on "fileN" arg; gets registration fixed; +7 idiom tests; 349/349 green
- 2026-05-06: Phase 4 env-as-value — current-env (special form via Sx_ref.register_special_form), eval-in-env (primitive in setup_evaluator_bridge), env-lookup + env-extend (in setup_env_operations); 5 idiom tests; 342/342 green
- 2026-05-06: Phase 3 OCaml primitives — file-read/write/append/exists?/glob + clock-seconds/milliseconds/format in sx_primitives.ml + unix dep; tcl-cmd-clock/file wired up; 337/337 green
- 2026-05-06: Phase 2 coroutine rewrite — `tcl-cmd-coroutine` now creates a `make-fiber`; `tcl-cmd-yield` calls `:coro-yield-fn` (threaded through interp); true suspension; 337/337 green
- 2026-05-06: Phase 2 fiber.sx — `make-fiber`/`fiber-resume`/`fiber-done?` using call/cc + set!; bidirectional value passing; generator and echo tests pass
- 2026-05-06: Phase 1 array — `tcl-cmd-array` get/set/names/size/exists/unset; frame-local key scanning with prefix `arrname(`; 337/337 tests green
- 2026-05-06: Phase 1 apply — `tcl-cmd-apply` wraps `tcl-call-proc`, parses `{args body}` funcList, full frame isolation; 329/329 tests green
- 2026-05-06: Phase 1 regexp/regsub — `tcl-cmd-regexp`/`tcl-cmd-regsub` wrapping `make-regexp`/`regexp-match`/`regexp-match-all`/`regexp-replace`/`regexp-replace-all`; -nocase/-all/-inline/-all flags; matchVar + subgroup capture; 329/329 tests green
- 2026-05-06: Phase 1 float expr — `tcl-num-float?`, `tcl-parse-num`, float-aware `tcl-apply-binop`/`tcl-apply-func`/unary-minus/`**`; `sqrt`/`floor`/`ceil`/`round`/`sin`/`cos`/`tan`/`pow`/`exp`/`log` all float-native; 329/329 tests green
---
## What stays out of scope
- `package require` of binary loadables (would need `Dynlink` + native ABI design)
- Full `clock format` locale (translated month/day names, `LC_TIME`-aware) — Phase 5e covers `-format`/`-timezone`/`-gmt` with English names
- Tk / GUI
- Threads (mapped to coroutines only, as planned)
- Server-mode `vwait` — Phase 5b event loop is scoped to script-mode; from inside a server-handled command it can't see sx_server's stdin scheduler