GUEST-plan: claim step 0 — baseline snapshot

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

plans/agent-briefings/lib-guest-loop.md

@@ -0,0 +1,118 @@
+# lib/guest extraction loop (single agent, queue-driven)
+
+Role: iterates `plans/lib-guest.md` forever. Each iteration picks the top `pending` step, extracts/ports/validates, commits, logs, moves on. North star: every guest's `scoreboard.json` ≥ baseline at all times, while `lib/guest/` accumulates shared infrastructure.
+
+```
+description: lib/guest extraction loop
+subagent_type: general-purpose
+run_in_background: true
+```
+
+## Prompt
+
+You are the sole background agent working `/root/rose-ash/plans/lib-guest.md`. You work a prioritised queue, one step per code commit, indefinitely. The plan file is the source of truth for what's pending, in-progress, done, and blocked. Update it after every iteration.
+
+## Iteration protocol (follow exactly)
+
+### 1. Read state
+
+- Read `plans/lib-guest.md` in full.
+- Pick the first step with status `[ ]`. If all remaining are `[blocked]` or `[done]`, stop and report loop complete.
+- Set that step's status to `[in-progress]` and commit the plan change alone:
+  `GUEST-plan: claim step <N> — <name>`.
+
+### 2. Baseline (every iteration that touches a guest)
+
+Before any code edit, snapshot the **current** scoreboard for every guest this step will touch (extraction consumers + canaries):
+
+```
+bash lib/<guest>/conformance.sh    # or test.sh
+cp lib/<guest>/scoreboard.json /tmp/baseline-<guest>-step<N>.json
+```
+
+If the step is Step 0, the snapshot itself is the work — copy each guest's `scoreboard.json` (or harvest pass/fail counts from `test.sh` for guests without a scoreboard) into `lib/guest/baseline/<lang>.json`, populate the table in `plans/lib-guest.md`, commit, done.
+
+### 3. Do the work
+
+For each step the protocol is:
+1. Read the relevant existing guest file(s) via `sx_read_subtree` to see exactly what shape needs extracting.
+2. Draft `lib/guest/<file>.sx` via `sx_write_file` (validates by parsing).
+3. Port the **first** consumer to use it. Run that guest's conformance. Must equal baseline.
+4. Port the **second** consumer (the two-language rule). Run that guest's conformance. Must equal baseline.
+5. If the second consumer needs escape hatches that the first didn't, the abstraction is wrong — **redesign before continuing**, don't paper over with alias chains or per-language flags.
+
+For Step 0 only: just snapshot, no extraction.
+
+### 4. Verify
+
+For every guest the step touched:
+
+```
+bash lib/<guest>/conformance.sh   # or test.sh
+diff lib/<guest>/scoreboard.json /tmp/baseline-<guest>-step<N>.json
+```
+
+**Abort rule:** if any touched guest's scoreboard regresses by ≥1 test, do NOT commit code. Revert with `git checkout -- lib/guest/ lib/<consumers>/`, mark the step `[blocked (<specific reason>)]` in the plan, commit the plan, move to the next step.
+
+### 5. Commit code
+
+One commit for the code:
+
+```
+GUEST: step <N> — <name>
+
+<2-4 lines on what was extracted, which two consumers were ported, baseline-equal verification.>
+
+Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
+```
+
+### 6. Update plan + commit
+
+In `plans/lib-guest.md`:
+- Change this step's status from `[in-progress]` to `[done]` (or `[partial — pending <consumer>]`).
+- Fill in the Commit and Delta columns of the progress log.
+- If you re-snapshotted any baseline, update the Baseline column.
+
+Commit: `GUEST-plan: log step <N> done`.
+
+### 7. Move on
+
+Go back to step 1. Continue until:
+- All steps are `[done]` or `[blocked]`, OR
+- You hit your iteration budget, OR
+- You encounter a substrate-level failure (build broken, sx_server.exe missing) — stop and report.
+
+## Ground rules
+
+- **Branch:** `architecture`. Commit locally. **Never push.** **Never touch `main`.**
+- **Scope:** ONLY `lib/guest/**`, `lib/{lua,prolog,haskell,common-lisp,tcl,erlang,smalltalk,forth,ruby,apl,js}/**`, `plans/lib-guest.md`, `plans/agent-briefings/lib-guest-loop.md`. NO `spec/`, `hosts/`, `web/`, `shared/`.
+- **SX files:** `sx-tree` MCP tools ONLY. Never `Edit`/`Read`/`Write` on `.sx`. `sx_validate` after every edit.
+- **OCaml build:** `sx_build target="ocaml"` MCP tool. Never raw `dune`.
+- **Two-language rule:** never merge an extraction until two guests consume it. Step 8 (HM) is the only exception, marked explicitly.
+- **No alias chains** to bridge naming drift between extraction and consumer — rename consumer-side or extraction-side, don't add a translation layer.
+- **No new planning docs** beyond updating the plan file.
+- **No comments in SX** unless non-obvious.
+- **Unicode in SX:** raw UTF-8, never `\uXXXX`.
+- **Hard timeout:** >45 min on a step → mark `blocked`, move on.
+- **Partial fixes are OK.** If you extract something and only the first consumer ports cleanly, mark `[partial — pending <second consumer>]`, commit, move on. The next iteration that lands the second consumer flips it to `[done]`.
+
+## Gotchas from past sessions
+
+- `env-bind!` creates a binding; `env-set!` mutates an existing one (walks scope chain). Macros that want to introduce names use `env-bind!`.
+- SX `do` is R7RS iteration, not a sequence form. Use `begin` for multi-expr bodies.
+- `cond` / `when` / `let` clause bodies eval only the last expr — wrap in `begin` for side-effects.
+- `list?` returns false on raw JS Arrays — host-side data must be SX-converted.
+- `make-symbol` builds an identifier symbol; `string->symbol` exists too — use whichever the surrounding code uses.
+- `sx_validate` after every edit. The hook will block raw `Edit`/`Write` on `.sx` anyway, but the validator catches subtree mistakes that parse-but-don't-mean-what-you-think.
+- Guest `conformance.sh` scripts use the epoch protocol against `sx_server.exe`. If the server isn't built, run `sx_build target="ocaml"` first.
+- Each guest's `scoreboard.json` schema differs slightly — normalise to `{:totals {:pass N :fail M} :suites [...]}` when writing `lib/guest/baseline/<lang>.json`.
+- `lib/parser-combinators.sx` exists and is unused by any guest. The new lex/Pratt kit may want to coexist with it, or supersede it — investigate before duplicating its functionality.
+- Prolog operator parsing is the stress test for Pratt — Prolog ops have variable precedence, `xfx`/`xfy`/`yfx` associativity classes, and user-definable ops at runtime. The Pratt kit must accommodate runtime registration, not just static tables.
+- Haskell layout is the stress test for whitespace-sensitive lexing — off-side rule, do/let/where/of opening blocks, semicolon insertion, brace insertion. Don't ship `lib/guest/layout.sx` unless the haskell scoreboard equals baseline.
+
+## Starting state
+
+- Branch: `architecture`. HEAD at or near `40f0e733`.
+- Canaries: **Lua** + **Prolog**.
+- Plan file at `plans/lib-guest.md`. Step 0 (baseline snapshot) is the first iteration.
+- `lib/guest/` does not yet exist — create it on the Step 0 commit.

plans/lib-guest.md

@@ -0,0 +1,176 @@
+# lib/guest — shared toolkit for SX-hosted languages
+
+Extract the duplicated plumbing across `lib/{haskell,common-lisp,erlang,prolog,js,lua,smalltalk,tcl,forth,ruby,apl,hyperscript}` into a small, composable kit so language N+1 costs ~200 lines instead of ~2000, without regressing any existing conformance scoreboard.
+
+Branch: `architecture`. SX files via `sx-tree` MCP only. Never edit generated files.
+
+## Thesis
+
+The substrate (CEK, hygienic macros, records, delimited continuations, IO suspension, reactivity) was chosen with multi-paradigm hosting in mind, but each guest currently re-rolls its own tokeniser, recursive-descent loop, conformance harness, and primitive-rename layer. Extracting these shared layers does not reduce conformance bug-finding pressure — it only removes plumbing — so it is pure win.
+
+**Canaries:** Lua (small, conventional expression-grammar — exercises lex/Pratt/AST) and Prolog (paradigm-different — exercises pattern-match/unification). The two-canary rule prevents Lua-shaped abstractions.
+
+**Two-language rule:** no extraction is merged until **two** guests consume it.
+
+## Current baseline
+
+The loop fills these in on its first iteration by running every `*/conformance.sh` and `*/test.sh` and copying each `scoreboard.json` to `lib/guest/baseline/<lang>.json`. Until then:
+
+| Guest        | Suite              | Baseline |
+|--------------|--------------------|----------|
+| lua          | `bash lib/lua/test.sh`              | TBD      |
+| prolog       | `bash lib/prolog/conformance.sh`    | TBD      |
+| haskell      | `bash lib/haskell/conformance.sh`   | TBD      |
+| common-lisp  | `bash lib/common-lisp/conformance.sh` | TBD    |
+| erlang       | `bash lib/erlang/conformance.sh`    | TBD      |
+| js           | `bash lib/js/conformance.sh`        | TBD      |
+| smalltalk    | `bash lib/smalltalk/conformance.sh` | TBD      |
+| tcl          | `bash lib/tcl/conformance.sh`       | TBD      |
+| forth        | `bash lib/forth/test.sh`            | TBD      |
+| ruby         | `bash lib/ruby/test.sh`             | TBD      |
+| apl          | `bash lib/apl/test.sh`              | TBD      |
+
+The baseline only needs to be re-snapshotted when the substrate (`spec/**`, `hosts/**`) changes underneath this loop.
+
+---
+
+## Phase 0 — Baseline snapshot (one-shot)
+
+### Step 0: Snapshot every guest's scoreboard
+
+Create `lib/guest/baseline/`. Run every guest's conformance/test runner. Copy each `scoreboard.json` (or extract pass/fail counts from `test.sh` output for guests without a scoreboard) into `lib/guest/baseline/<lang>.json`. Fill in the table above.
+
+**Verify:** `ls lib/guest/baseline/*.json` shows one per guest. Plan table populated.
+
+---
+
+## Phase 1 — Cheap, zero-semantic-risk extractions
+
+### Step 1: `lib/guest/conformance.sx` — config-driven test runner
+
+Replace the 6+ near-identical `*/conformance.sh` scripts with one driver that takes a config dict:
+
+```
+{:lang "prolog"
+ :loads ("lib/prolog/tokenizer.sx" "lib/prolog/parser.sx" ...)
+ :suites (("parse" "lib/prolog/tests/parse.sx" "pl-parse-tests-run!") ...)}
+```
+
+The driver locates `sx_server.exe`, runs the epoch protocol, collects pass/fail per suite, and writes `scoreboard.{json,md}`. The per-language `conformance.sh` becomes a 3-line stub that points at its config.
+
+**Port to:** `lib/prolog/conformance.sh` and `lib/haskell/conformance.sh`. Two consumers required for merge.
+
+**Verify:** both `bash lib/prolog/conformance.sh` and `bash lib/haskell/conformance.sh` produce scoreboard JSONs equal to baseline.
+
+### Step 2: `lib/guest/prefix.sx` — prefix-rename macro
+
+One macro that takes a prefix and a list of SX symbols and binds prefixed aliases:
+
+```
+(prefix-rename "cl-" '(null? pair? even? odd? zero? ...))
+```
+
+Replaces hundreds of hand-written `(define (cl-null? x) (= x nil))`-style wrappers in `common-lisp/runtime.sx`, `lua/runtime.sx`, `erlang/runtime.sx`.
+
+**Port to:** `common-lisp/runtime.sx` (largest user) and `lua/runtime.sx`. Two consumers.
+
+**Verify:** common-lisp + lua scoreboards equal baseline.
+
+---
+
+## Phase 2 — Lex / parse kit
+
+### Step 3: `lib/guest/lex.sx` — character-class + tokeniser primitives
+
+- Source-position tracking (line/col/offset).
+- Character-class predicates (`whitespace?`, `digit?`, `alpha?`, `ident-start?`, `ident-rest?`).
+- Number recognisers (decimal, hex, float, scientific).
+- String recognisers (quoted, escapes, raw).
+- Comment recognisers (line, block, nestable).
+- Token record `{:type :value :pos :end :line}`.
+
+**Port to:** `lua/tokenizer.sx` and `tcl/tokenizer.sx`. Two consumers.
+
+**Verify:** lua + tcl scoreboards equal baseline.
+
+### Step 4: `lib/guest/pratt.sx` — Pratt / operator-precedence parser
+
+Prefix / infix / postfix tables, left/right associativity, precedence climbing. Grammar is a dict, not hardcoded `cond`.
+
+**Port to:** Lua expression parser (`lua/parser.sx`) and Prolog operator table (`prolog/parser.sx` — Prolog ops are the stress test). Two consumers.
+
+**Verify:** lua + prolog scoreboards equal baseline.
+
+### Step 5: `lib/guest/ast.sx` — canonical AST node shapes
+
+Standard constructors and predicates for: `literal`, `var`, `app`, `lambda`, `let`, `letrec`, `if`, `match-clause`, `module`, `import`. Optional — guests may keep their own AST — but using the canonical shape lets cross-language tooling (formatters, highlighters, debuggers) work without per-language adapters.
+
+**Port to:** lua + prolog AST emitters. Two consumers.
+
+**Verify:** lua + prolog scoreboards equal baseline.
+
+---
+
+## Phase 3 — Semantic extractions (highest leverage, highest risk)
+
+### Step 6: `lib/guest/match.sx` — pattern-match + unification engine
+
+Single engine for:
+- Literal patterns (numbers, strings, symbols, nil, booleans).
+- Wildcard `_`.
+- Constructor patterns (ADT-shaped — depends on Phase 3 of `sx-improvements.md` if available, otherwise dict-tagged).
+- Variable binding.
+- **Unification** (Prolog flavour): symmetric, occurs-check toggle, substitution returned.
+- **Match** (Haskell flavour): asymmetric pattern→value, bindings returned.
+
+**Port to:** `haskell/match.sx` and `prolog/query.sx` unification core. Two consumers.
+
+**Verify:** haskell + prolog scoreboards equal baseline. **Highest-risk extraction** — if either regresses by 1 test, revert and redesign.
+
+### Step 7: `lib/guest/layout.sx` — significant-whitespace / off-side rule
+
+Generalised layout-sensitive lexer. Configurable: which keywords open layout blocks, whether semicolons are inserted, brace insertion rules.
+
+**Port to:** `haskell/layout.sx` (existing). Second consumer: write a synthetic test fixture that exercises a Python-ish layout to prove the kit is not Haskell-shaped. Two consumers.
+
+**Verify:** haskell scoreboard equal baseline; synthetic layout fixture passes.
+
+### Step 8: `lib/guest/hm.sx` — Hindley-Milner type inference
+
+Extract from `haskell/infer.sx`. Algorithm W or J, generalisation, instantiation, occurs-check, principal types.
+
+**Port to:** `haskell/infer.sx`. Second consumer is speculative (no existing ML/Elm guest); accept "second user TBD" for this step only, since the alternative is letting the inference stay locked inside Haskell forever.
+
+**Verify:** haskell scoreboard equal baseline.
+
+---
+
+## Progress log
+
+| Step | Status | Commit | Delta |
+|------|--------|--------|-------|
+| 0 — baseline snapshot | [in-progress] | — | — |
+| 1 — conformance.sx (prolog + haskell) | [ ] | — | — |
+| 2 — prefix.sx (common-lisp + lua) | [ ] | — | — |
+| 3 — lex.sx (lua + tcl) | [ ] | — | — |
+| 4 — pratt.sx (lua + prolog) | [ ] | — | — |
+| 5 — ast.sx (lua + prolog) | [ ] | — | — |
+| 6 — match.sx (haskell + prolog) | [ ] | — | — |
+| 7 — layout.sx (haskell + synthetic) | [ ] | — | — |
+| 8 — hm.sx (haskell + TBD) | [ ] | — | — |
+
+---
+
+## Rules
+
+- **Branch:** `architecture`. Commit locally. **Never push.** **Never touch `main`.**
+- **Scope:** ONLY `lib/guest/**`, `lib/{lua,prolog,haskell,common-lisp,tcl}/**` (canaries + extraction targets), `plans/lib-guest.md`, `plans/agent-briefings/lib-guest-loop.md`. No `spec/`, `hosts/`, `web/`, `shared/`.
+- **SX files:** `sx-tree` MCP tools only. `sx_validate` after every edit.
+- **No raw dune.** Use `sx_build target="ocaml"` MCP tool.
+- **Two-language rule:** never merge an extraction until two guests consume it (Step 8 excepted with explicit note).
+- **Conformance baseline is the bar.** Any port whose scoreboard regresses by ≥1 test → revert, mark blocked, move on.
+- **Substrate change → re-snapshot.** If `spec/` or `hosts/` changes underneath this loop, re-run Step 0 before continuing.
+- **One step per code commit.** Plan updates as a separate commit. Short message with delta.
+- **No alias chains** to paper over drift between extraction and consumer (`feedback_no_alias_bloat`).
+- **Partial extraction is OK** if the canary works and a pending consumer is identified — mark `[partial — pending <consumer>]`.
+- **Hard timeout:** if stuck >45 min on a step, mark `blocked (<reason>)` and move on.