plans: sx-improvements roadmap + loop briefing (14 steps)
Phases: bug fixes (JIT combinator, letrec+resume), E38 source info completion, native ADTs (define-type/match), plugin system, performance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
86
plans/agent-briefings/sx-improvements-loop.md
Normal file
86
plans/agent-briefings/sx-improvements-loop.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# sx-improvements loop agent
|
||||
|
||||
Iterates `plans/sx-improvements.md` forever. One step per commit.
|
||||
|
||||
```
|
||||
description: sx-improvements loop
|
||||
subagent_type: general-purpose
|
||||
run_in_background: true
|
||||
isolation: worktree
|
||||
```
|
||||
|
||||
## Prompt
|
||||
|
||||
You are the sole background agent iterating `plans/sx-improvements.md` on the `architecture` branch of `/root/rose-ash`. One step per commit, forever. Never push.
|
||||
|
||||
## Restart baseline — check before each iteration
|
||||
|
||||
1. Read `plans/sx-improvements.md` — find the first unchecked `[ ]` step in the progress log.
|
||||
2. Read the step's section in the plan for exact implementation details.
|
||||
3. Run the verification command for that step to confirm it currently fails.
|
||||
4. Implement. Verify. Commit. Tick the `[ ]` → `[x]` in the progress log. Next.
|
||||
|
||||
## Test commands
|
||||
|
||||
- **OCaml spec:** `sx_build target="ocaml"` then check `bin/run_tests.exe` output.
|
||||
- **JS spec (no DOM):** `node hosts/javascript/run_tests.js 2>&1 | tail -3`
|
||||
- **HyperScript kernel:** `node tests/hs-kernel-eval.js 2>&1 | tail -3`
|
||||
- **Baseline SX tests (non-HS):** `node hosts/javascript/run_tests.js 2>&1 | grep -v "hs-upstream\|hs-compat\|hs-dev" | grep "Results:"`
|
||||
|
||||
Do NOT regress the pre-merge passing tests. After each step, confirm the count did not drop.
|
||||
|
||||
## Ground rules (hard)
|
||||
|
||||
- **Branch:** `architecture`. Never push. Never touch `main`.
|
||||
- **SX files:** `sx-tree` MCP tools ONLY (`sx_summarise`, `sx_read_subtree`, `sx_replace_node`, `sx_insert_child`, `sx_validate`). Read before edit. Validate after edit.
|
||||
- **Generated files:** NEVER edit `shared/static/wasm/sx/` or `shared/static/scripts/sx-*.js` directly. Rebuild via `sx_build`.
|
||||
- **HS mirror rule:** after editing any `lib/hyperscript/<f>.sx`, copy to `shared/static/wasm/sx/hs-<f>.sx` using `sx_write_file` with the same content.
|
||||
- **OCaml build:** `sx_build target="ocaml"` — never raw `dune exec`.
|
||||
- **JS build:** `sx_build target="js"`.
|
||||
- **One step per commit.** Tick the plan. Factual commit message.
|
||||
- **No new planning docs.** No comments in SX unless non-obvious.
|
||||
- **Unicode in SX:** raw UTF-8 only, never `\uXXXX` escapes.
|
||||
|
||||
## Step-specific notes
|
||||
|
||||
### Step 1 (JIT combinator bug)
|
||||
The bug is in `hosts/ocaml/lib/sx_vm.ml` — `call_closure_reuse` path strips locals when
|
||||
callee returns a closure. Look for the path where `call_closure_reuse` is invoked for a
|
||||
`VmClosure` return value. The fix is to not reuse frames when the call might return a
|
||||
closure, or to properly snapshot/restore `sp`. Check `spec/tests/test-parser-combinators.sx`
|
||||
for existing combinator tests; run `node tests/hs-kernel-eval.js` for the 11 failing HS tests.
|
||||
|
||||
### Step 2 (letrec+resume)
|
||||
The bug is browser-only (`hosts/ocaml/browser/sx_browser.ml`). Write a minimal
|
||||
`spec/tests/test-letrec-resume.sx` that exercises `letrec` + `perform` + resume and
|
||||
verify it passes under `run_tests.exe` (OCaml server mode). Then check what
|
||||
`sx_browser.ml` does differently in the VmSuspension resume path.
|
||||
|
||||
### Steps 3-4 (E38 source info)
|
||||
The API is already in `lib/hyperscript/runtime.sx`. The gap is in the tokenizer (no `:end`/`:line`)
|
||||
and some parser span completeness. Run the 4 sourceInfo tests to see exact failures:
|
||||
`node tests/hs-kernel-eval.js --suite sourceInfo` or grep results for `sourceInfo`.
|
||||
|
||||
### Steps 5-8 (ADTs)
|
||||
Full spec in `plans/designs/sx-adt.md`. Implement in OCaml first (Step 5), then mirror
|
||||
to JS (Step 6). Steps 7-8 build on top. Write `spec/tests/test-adt.sx` from scratch —
|
||||
start with a `(define-type Maybe (Just value) (Nothing))` suite covering constructor,
|
||||
predicate, accessor, basic match, else clause.
|
||||
|
||||
### Steps 9-11 (plugin system)
|
||||
Full spec in `plans/designs/hs-plugin-system.md`. The prolog hook migration (Step 11) is
|
||||
the most important for language-building — it's the pattern for all future embeds.
|
||||
|
||||
### Steps 12-14 (performance)
|
||||
Profile first. Use `sx_harness_eval` to measure throughput on a tight loop before and
|
||||
after each change. Only commit if there's a measurable win (>10%).
|
||||
|
||||
## General gotchas (all loops)
|
||||
|
||||
- SX `do` is R7RS iteration. Use `begin` for multi-expr sequences.
|
||||
- `cond`/`when`/`let` bodies evaluate only the last expression.
|
||||
- `type-of` on a user-defined function returns `"lambda"`.
|
||||
- Shell heredoc `||` gets eaten — escape or use `case`.
|
||||
- `env-bind!` creates new bindings; `env-set!` mutates existing (walks scope chain).
|
||||
- After OCaml edits: the build takes ~2 min. Run `sx_build target="ocaml"` and wait.
|
||||
- After JS edits: retranspile with `sx_build target="js"` then re-run tests.
|
||||
210
plans/sx-improvements.md
Normal file
210
plans/sx-improvements.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# SX Language Improvements — roadmap
|
||||
|
||||
Language-building improvements to the SX evaluator, compiler, and standard library.
|
||||
Ordered by impact and prerequisite chain. Each step is one loop commit.
|
||||
|
||||
Branch: `architecture`. SX files via `sx-tree` MCP only. Never edit generated files.
|
||||
|
||||
## Current baseline (2026-05-06)
|
||||
|
||||
- SX core spec: 2571 passing (595 non-HS pre-existing failures — bytecode-serialize, defcomp-render, etc.)
|
||||
- HyperScript behavioral: 1478/1496 (run via `node tests/hs-kernel-eval.js`)
|
||||
- Active bugs: JIT combinator bug (11 HS failures), letrec+resume (browser-only)
|
||||
- E38 sourceInfo: 2/4 tests passing (tokenizer missing `:end`/`:line`, some spans incomplete)
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Bug fixes
|
||||
|
||||
### Step 1: Fix JIT closures-returning-closures
|
||||
|
||||
**What:** `parse-bind`, `many`, `seq`, and other parser combinators that return closures
|
||||
miscompile under JIT. The compiled closure drops intermediate stack values when the
|
||||
callee itself returns a closure. 11 HyperScript tests fail under JIT, pass under CEK.
|
||||
|
||||
**Root cause in `hosts/ocaml/lib/sx_vm.ml`:** When a JIT-compiled closure returns
|
||||
another closure (i.e. the callee is `VmClosure`), the frame restoration after the
|
||||
call incorrectly reuses the parent frame's locals slot, overwriting saved intermediate
|
||||
values. The `call_closure_reuse` path must snapshot `sp` before the inner call and
|
||||
restore it after, or bail to the non-reuse path for closures-returning-closures.
|
||||
|
||||
**Verify:** `node tests/hs-kernel-eval.js 2>&1 | tail -3` — should go from 3116/3127 to 3127/3127.
|
||||
|
||||
### Step 2: Fix letrec + perform resume (browser)
|
||||
|
||||
**What:** In browser JIT mode, `letrec` sibling bindings are nil after a `perform`/resume
|
||||
cycle. `call_closure_reuse` in `sx_browser.ml` intentionally ignores `_saved_sp`, which
|
||||
strips the frame locals that `sf_letrec` was waiting on.
|
||||
|
||||
**Fix:** In `sx_browser.ml`, the `VmSuspension` resume path must restore frame locals
|
||||
from the suspension snapshot before calling the continuation. Mirror what `sx_vm.ml`
|
||||
does in the non-browser case.
|
||||
|
||||
**Verify:** Write a test in `spec/tests/` that does `(letrec ((f (fn () (perform :io nil)))) (f))` with a resume, check bindings survive. Runs under OCaml: `dune exec -- bin/run_tests.exe`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Source info (E38 completion)
|
||||
|
||||
Design: `plans/designs/e38-sourceinfo.md`. Target: 4/4 sourceInfo tests.
|
||||
|
||||
The API (`hs-parse-ast`, `hs-source-for`, `hs-line-for`, `hs-node-get`, `hs-src`,
|
||||
`hs-src-at`, `hs-line-at`) and parser span wrapping (`hs-ast-wrap`, `hs-span-mode`)
|
||||
are already in the codebase. Two tests are passing; two fail because:
|
||||
- Tokenizer tokens lack `:end` and `:line` (only `:pos` today).
|
||||
- Some statement-level spans and `:next` field navigation are incomplete.
|
||||
|
||||
### Step 3: Tokenizer — add `:end` and `:line` to tokens
|
||||
|
||||
`lib/hyperscript/tokenizer.sx`: extend `hs-make-token` to `{:pos :end :value :type :line}`.
|
||||
Track a `current-line` counter (1-based, increments after `\n`). Update all ~20 emission
|
||||
sites. Mirror to `shared/static/wasm/sx/hs-tokenizer.sx` after edits.
|
||||
|
||||
**Verify:** `(hs-make-token "NUMBER" "1" 0)` returns a dict with `:end` and `:line` keys.
|
||||
|
||||
### Step 4: Complete parser spans + :next field
|
||||
|
||||
`lib/hyperscript/parser.sx`: ensure `hs-ast-wrap` populates `:next` on every command
|
||||
in a `CommandList` (i.e. the following sibling command). Check that statement-level
|
||||
productions (if, for) correctly populate `:true-branch`. Trace through the two failing
|
||||
tests (`get source works for expressions`, `get line works for statements`) to find the
|
||||
exact missing fields or off-by-one positions.
|
||||
|
||||
Mirror to `shared/static/wasm/sx/hs-parser.sx`.
|
||||
|
||||
**Verify:** All 4 `hs-upstream-core/sourceInfo` tests pass.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Native ADTs (`define-type` / `match`)
|
||||
|
||||
Design: `plans/designs/sx-adt.md`. No existing implementation.
|
||||
|
||||
Impact: every language implementation (Haskell, Prolog, Lua, Common Lisp, Erlang)
|
||||
currently fakes sum types with `{:tag "..." :field ...}` dicts. Native ADTs remove
|
||||
that everywhere.
|
||||
|
||||
### Step 5: OCaml — AdtValue type + `define-type` + basic `match`
|
||||
|
||||
`hosts/ocaml/lib/sx_types.ml`:
|
||||
```ocaml
|
||||
type adt_value = { av_type: string; av_ctor: string; av_fields: value array }
|
||||
| AdtValue of adt_value
|
||||
```
|
||||
|
||||
`hosts/ocaml/lib/sx_runtime.ml` (or evaluator):
|
||||
- `step-sf-define-type`: parse `(Name (Ctor1 f1 f2) (Ctor2) ...)`, register constructor
|
||||
NativeFns, predicates (`Ctor1?`, `Name?`), field accessors (`Ctor1-f1`) via `env-bind!`.
|
||||
- `step-sf-match` + `MatchFrame`: linear scan of clauses; flat patterns only for 6a;
|
||||
bind pattern variables in child env; `else` clause; raise on no match.
|
||||
- `type-of` returns the type name (e.g. `"Maybe"`).
|
||||
|
||||
Write tests in `spec/tests/test-adt.sx`: basic constructor, predicate, accessor, match,
|
||||
else, no-match raise.
|
||||
|
||||
**Verify:** `dune exec -- bin/run_tests.exe` — new test file all green.
|
||||
|
||||
### Step 6: JS — AdtValue + `define-type` + `match`
|
||||
|
||||
`hosts/javascript/platform.py`: add `AdtValue` as `{ _adt: true, _type, _ctor, _fields }`.
|
||||
Mirror `define-type` and `match` special forms in the JS evaluator.
|
||||
Retranspile: `python3 hosts/javascript/cli.py --output shared/static/scripts/sx-browser.js`
|
||||
|
||||
**Verify:** `node hosts/javascript/run_tests.js` — adt tests pass on JS too.
|
||||
|
||||
### Step 7: Nested patterns (Phase 6b)
|
||||
|
||||
Both OCaml and JS `MatchFrame`: replace linear binding with recursive
|
||||
`matchPattern(pattern, value, env)` that:
|
||||
- Recurses into constructor sub-patterns.
|
||||
- Returns `{matched: bool, bindings: map}`.
|
||||
- Handles wildcard `_`, literals (`42`, `"str"`, `true`, `nil`).
|
||||
|
||||
Extend `spec/tests/test-adt.sx` with nested pattern tests.
|
||||
|
||||
### Step 8: Exhaustiveness warnings (Phase 6c)
|
||||
|
||||
`_adt_registry: type_name → [ctor_names]` global populated by `define-type`.
|
||||
On first non-exhaustive `match` evaluation: `console.warn("[sx] match: non-exhaustive …")`.
|
||||
No error — warning only.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Plugin / extension system
|
||||
|
||||
Design: `plans/designs/hs-plugin-system.md`.
|
||||
|
||||
### Step 9: Parser feature registry
|
||||
|
||||
`lib/hyperscript/parser.sx`: replace `parse-feat` hardcoded `cond` with a dict lookup.
|
||||
`(hs-register-feature! name parse-fn)` adds to the registry.
|
||||
|
||||
### Step 10: Compiler command registry + `as` converter registry
|
||||
|
||||
`lib/hyperscript/compiler.sx`: replace `hs-to-sx` hardcoded dispatch with dict.
|
||||
`(hs-register-command! name compile-fn)` and `(hs-register-converter! name convert-fn)`.
|
||||
|
||||
### Step 11: Migrate hs-prolog-hook + Worker plugin
|
||||
|
||||
`lib/hyperscript/runtime.sx`: remove `hs-prolog-hook`/`hs-set-prolog-hook!` ad-hoc
|
||||
slots. Create `lib/hyperscript/plugins/prolog.sx` that calls `hs-register-feature!`
|
||||
and `hs-register-command!`. Create `lib/hyperscript/plugins/worker.sx` replacing the
|
||||
E39 stub.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Performance
|
||||
|
||||
These are incremental and can interleave with other phases.
|
||||
|
||||
### Step 12: Frame records (CEK)
|
||||
|
||||
`hosts/ocaml/lib/sx_runtime.ml`: represent CEK frames as OCaml records instead of
|
||||
tagged variant lists. Eliminates allocation pressure from list construction per frame.
|
||||
Profile before/after on a tight-loop benchmark.
|
||||
|
||||
### Step 13: Buffer primitive for string building
|
||||
|
||||
Add `make-buffer`, `buffer-append!`, `buffer->string` primitives. Eliminates the
|
||||
`(str a b c d ...)` quadratic allocation pattern in serializers and renderers.
|
||||
Wire into `sx_primitives.ml` and the JS platform.
|
||||
|
||||
### Step 14: Inline common primitives in JIT
|
||||
|
||||
`hosts/ocaml/lib/sx_vm.ml`: add `OP_ADD`, `OP_SUB`, `OP_EQ`, `OP_APPEND` specialised
|
||||
opcodes that skip the primitive table lookup for the most common calls. Compiler emits
|
||||
these when operands are known numbers/lists.
|
||||
|
||||
---
|
||||
|
||||
## Progress log
|
||||
|
||||
| Step | Status | Commit |
|
||||
|------|--------|--------|
|
||||
| 1 — JIT combinator bug | [ ] | — |
|
||||
| 2 — letrec+resume | [ ] | — |
|
||||
| 3 — tokenizer :end/:line | [ ] | — |
|
||||
| 4 — parser spans complete | [ ] | — |
|
||||
| 5 — OCaml AdtValue + define-type + match | [ ] | — |
|
||||
| 6 — JS AdtValue + define-type + match | [ ] | — |
|
||||
| 7 — nested patterns | [ ] | — |
|
||||
| 8 — exhaustiveness warnings | [ ] | — |
|
||||
| 9 — parser feature registry | [ ] | — |
|
||||
| 10 — compiler + as converter registry | [ ] | — |
|
||||
| 11 — plugin migration + worker | [ ] | — |
|
||||
| 12 — frame records | [ ] | — |
|
||||
| 13 — buffer primitive | [ ] | — |
|
||||
| 14 — inline primitives JIT | [ ] | — |
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- Branch: `architecture`. Never push to `main`.
|
||||
- SX files: `sx-tree` MCP tools only. `sx_validate` after every edit.
|
||||
- After every `.sx` edit to `lib/hyperscript/`, mirror to `shared/static/wasm/sx/hs-<file>.sx`.
|
||||
- OCaml build: `sx_build target="ocaml"` MCP tool (never raw `dune`).
|
||||
- JS build: `sx_build target="js"` MCP tool.
|
||||
- One step per commit. Update progress log in this file.
|
||||
- No new planning docs. No comments in SX unless non-obvious.
|
||||
- Unicode in SX: raw UTF-8 only, never `\uXXXX`.
|
||||
Reference in New Issue
Block a user