rose-ash/plans/forth-on-sx.md

# Forth-on-SX: stack language on the VM

The smallest serious second language — Forth's stack-based semantics map directly onto the SX bytecode VM (OP_DUP, OP_SWAP, OP_DROP already exist as arithmetic primitives or can be added trivially). Compile-mode / interpret-mode is the one genuinely novel piece, but it's a classic technique and small.

End-state goal: **passes John Hayes' ANS-Forth test suite** (the canonical Forth conformance harness — small, well-documented, targets the Core word set).

## Scope decisions (defaults — override)

- **Standard:** ANS-Forth 1994 Core word set + Core Extension. No ANS-Forth Optional word sets (File Access, Floating Point, Search Order, etc.) in the first run.
- **Test suite:** John Hayes' "Test Suite for ANS Forth" (~250 tests, public domain, widely used).
- **Case-sensitivity:** case-insensitive (ANS default).
- **Number base:** support `BASE` variable, defaults to 10. Hex and binary literals (`$FF`, `%1010`) per standard.

## Ground rules

- **Scope:** only touch `lib/forth/**` and `plans/forth-on-sx.md`. No edits to `spec/`, `hosts/`, `shared/`, or other language dirs.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** reader (not tokenizer — Forth is whitespace-delimited) → interpreter → dictionary-backed compiler. The compiler emits SX AST (not bytecode directly) so we inherit the VM.
- **Commits:** one feature per commit. Keep `## Progress log` updated.

## Architecture sketch

```
Forth source text
    │
    ▼
lib/forth/reader.sx      — whitespace-split words (that's it — no real tokenizer)
    │
    ▼
lib/forth/interpreter.sx — interpret mode: look up word in dict, execute
    │
    ▼
lib/forth/compiler.sx    — compile mode (`:` opens, `;` closes): emit SX AST
    │
    ▼
lib/forth/runtime.sx     — stack ops, dictionary, BASE, I/O
    │
    ▼
existing CEK / VM        — runs compiled definitions natively
```

Representation:
- **Stack** = SX list, push = cons, pop = uncons
- **Dictionary** = dict `word-name → {:kind :immediate? :body}` where kind is `:primitive` or `:colon-def`
- **A colon definition** compiles to a thunk `(lambda () <body-as-sx-sequence>)`
- **Compile-mode** is a flag on the interpreter state; `:` sets it, `;` clears and installs the new word
- **IMMEDIATE** words run at compile time

## Roadmap

### Phase 1 — reader + interpret mode
- [x] `lib/forth/reader.sx`: whitespace-split, number parsing (base-aware)
- [x] `lib/forth/runtime.sx`: stack as SX list, push/pop/peek helpers
- [x] Core stack words: `DUP`, `DROP`, `SWAP`, `OVER`, `ROT`, `-ROT`, `NIP`, `TUCK`, `PICK`, `ROLL`, `?DUP`, `DEPTH`, `2DUP`, `2DROP`, `2SWAP`, `2OVER`
- [x] Arithmetic: `+`, `-`, `*`, `/`, `MOD`, `/MOD`, `NEGATE`, `ABS`, `MIN`, `MAX`, `1+`, `1-`, `2+`, `2-`, `2*`, `2/`
- [x] Comparison: `=`, `<>`, `<`, `>`, `<=`, `>=`, `0=`, `0<>`, `0<`, `0>`
- [x] Logical: `AND`, `OR`, `XOR`, `INVERT` (32-bit two's-complement sim)
- [x] I/O: `.` (print), `.S` (show stack), `EMIT`, `CR`, `SPACE`, `SPACES`, `BL`
- [x] Interpreter loop: read word, look up, execute, repeat
- [x] Unit tests in `lib/forth/tests/test-phase1.sx` — 108/108 pass

### Phase 2 — colon definitions + compile mode
- [x] `:` opens compile mode and starts a definition
- [x] `;` closes it and installs into the dictionary
- [x] Compile mode: non-IMMEDIATE words are compiled as late-binding call thunks; numbers are compiled as pushers; IMMEDIATE words run immediately
- [x] `VARIABLE`, `CONSTANT`, `VALUE`, `TO`, `RECURSE`, `IMMEDIATE`
- [x] `@` (fetch), `!` (store), `+!`
- [x] Colon-def body is `(fn (s) (for-each op body))` — runs on CEK, inherits TCO
- [x] Tests in `lib/forth/tests/test-phase2.sx` — 26/26 pass

### Phase 3 — control flow + first Hayes tests green
- [x] `IF`, `ELSE`, `THEN` — compile to SX `if`
- [x] `BEGIN`, `UNTIL`, `WHILE`, `REPEAT`, `AGAIN` — compile to loops
- [x] `DO`, `LOOP`, `+LOOP`, `I`, `J`, `LEAVE` — counted loops (needs a return stack)
- [x] Return stack: `>R`, `R>`, `R@`, `2>R`, `2R>`, `2R@`
- [x] Vendor John Hayes' test suite to `lib/forth/ans-tests/`
- [x] `lib/forth/conformance.sh` + runner; `scoreboard.json` + `scoreboard.md`
- [x] Baseline: probably 30-50% Core passing after phase 3

### Phase 4 — strings + more Core
- [x] `S"`, `C"`, `."`, `TYPE`, `COUNT`, `CMOVE`, `FILL`, `BLANK`
- [x] `CHAR`, `[CHAR]`, `KEY`, `ACCEPT`
- [x] `BASE` manipulation: `DECIMAL`, `HEX`
- [x] `DEPTH`, `SP@`, `SP!`
- [ ] Drive Hayes Core pass-rate up

### Phase 5 — Core Extension + optional word sets
- [ ] Full Core + Core Extension
- [ ] File Access word set (via SX IO)
- [ ] String word set (`SLITERAL`, `COMPARE`, `SEARCH`)
- [ ] Target: 100% Hayes Core

### Phase 6 — speed
- [ ] Inline primitive calls during compile (skip dict lookup)
- [ ] Tail-call optimise colon-def endings
- [ ] JIT cooperation: mark compiled colon-defs as VM-eligible

## Progress log

_Newest first._

- **Phase 4 — `SP@`/`SP!` (+4; Hayes unchanged; `DEPTH` was already present).**
  `SP@` pushes the current data-stack depth (our closest analogue to a
  stack pointer — SX lists have no addressable backing). `SP!` pops a
  target depth and truncates the stack via `drop` on the dstack list.
  This preserves the save/restore idiom `SP@ … SP!` even though the
  returned "pointer" is really a count.

- **Phase 4 — `BASE`/`DECIMAL`/`HEX`/`BIN`/`OCTAL` (+9; Hayes unchanged).**
  Moved `base` from its top-level state slot into `state.vars["base"]`
  so the regular `@`/`!`/VARIABLE machinery works on it.
  `BASE` pushes the sentinel address `"base"`; `DECIMAL`/`HEX`/`BIN`/
  `OCTAL` are thin primitives that write into that slot. Parser
  reads through `vars` now. Hayes unchanged because the runner had
  already been stubbing `HEX`/`DECIMAL` — now real words, stubs
  removed from `hayes-runner.sx`.

- **Phase 4 — `CHAR`/`[CHAR]`/`KEY`/`ACCEPT` (+7 / Hayes 168→174).**
  `CHAR` parses the next token and pushes the first-char code. `[CHAR]`
  is IMMEDIATE: in compile mode it embeds the code as a compiled push
  op, in interpret mode it pushes inline. `KEY`/`ACCEPT` read from an
  optional `state.keybuf` string — empty buffer makes `KEY` raise
  `"no input available"` (matches ANS when stdin is closed) and
  `ACCEPT` returns `0`. Enough for Hayes to get past CHAR-gated
  clusters; real interactive IO lands later.

- **Phase 4 — strings: `S"`/`C"`/`."`/`TYPE`/`COUNT`/`CMOVE`/`CMOVE>`/`MOVE`/`FILL`/`BLANK`/`C@`/`C!`/`CHAR+`/`CHARS` (+16 / Hayes 165→168).**
  Added a byte-addressable memory model to state: `mem` (dict keyed by
  stringified address → integer byte) and `here` (next-free integer
  addr). Helpers `forth-alloc-bytes!` / `forth-mem-write-string!` /
  `forth-mem-read-string`. `S"`/`C"`/`."` are IMMEDIATE parsing words
  that consume tokens until one ends with `"`, then either copy content
  into memory at compile time (and emit a push of `addr`/`addr len` for
  the colon-def body) or do it inline in interpret mode. `TYPE` emits
  `u` bytes from `addr` via `char-from-code`. `COUNT` reads the length
  byte at a counted-string address and pushes (`addr+1`, `u`). `FILL`,
  `BLANK` (FILL with space), `CMOVE` (forward), `CMOVE>` (backward),
  and `MOVE` (auto-directional) mutate the byte dict. 193/193 internal
  tests, Hayes 168/590 (+3).

- **Phase 3 — Hayes conformance runner + baseline scoreboard (165/590, 28%).**
  `lib/forth/conformance.sh` preprocesses `ans-tests/core.fr` (strips `\`
  and `( ... )` comments + `TESTING` lines), splits the source on every
  `}T` so each Hayes test plus the small declaration blocks between
  them are one safe-resume chunk, and emits an SX driver that feeds
  the chunks through `lib/forth/hayes-runner.sx`. The runner registers
  `T{`/`->`/`}T` as Forth primitives that snapshot the dstack depth on
  `T{`, record actual on `->`, compare on `}T`, and install stub
  `HEX`/`DECIMAL`/`TESTING` so metadata doesn't halt the stream. Errors
  raised inside a chunk are caught by `guard` and the state is reset,
  so one bad test does not break the rest. Outputs
  `scoreboard.json` + `scoreboard.md`.

  First-run baseline: 165 pass / 425 error / 0 fail on the first 590
  chunks. The default cap sits at 590 because `core.fr` chunks beyond
  that rely on unsigned-integer wrap-around (e.g. `COUNT-BITS` with
  `BEGIN DUP WHILE … 2* REPEAT`) which never terminates on our
  bignum-based Forth; raise `MAX_CHUNKS` once those tests unblock.
  Majority of errors are missing Phase-4 words (`RSHIFT`, `LSHIFT`,
  `CELLS`, `S"`, `CHAR`, `SOURCE`, etc.) — each one implemented should
  convert a cluster of errors to passes.

- **Phase 3 — vendor Gerry Jackson's forth2012-test-suite.** Added
  `lib/forth/ans-tests/{tester.fr, core.fr, coreexttest.fth}` from
  https://github.com/gerryjackson/forth2012-test-suite (master, fetched
  2026-04-24). `tester.fr` is Hayes' `T{ ... -> ... }T` harness; `core.fr`
  is the ~1000-line Core word tests; `coreexttest.fth` is Core Ext
  (parked for later phases). Files are pristine — the conformance runner
  (next iteration) will consume them.

- **Phase 3 — `DO`/`LOOP`/`+LOOP`/`I`/`J`/`LEAVE` + return stack words (+16).**
  Counted loops compile onto the same PC-driven body runner. DO emits an
  enter-op (pops limit+start from data stack, pushes them to rstack) and
  pushes a `{:kind "do" :back PC :leaves ()}` marker onto cstack. LOOP/+LOOP
  emit a dict op (`:kind "loop"`/`"+loop"` with target=back-cell). The step
  handler pops index & reads limit, increments, and either restores the
  updated index + jumps back, or drops the frame and advances. LEAVE walks
  cstack for the innermost DO marker, emits a `:kind "leave"` dict op with
  a fresh target cell, and registers it on the marker's leaves list. LOOP
  patches all registered leave-targets to the exit PC and drops the marker.
  The leave op pops two from rstack (unloop) and branches. `I` peeks rtop;
  `J` reads rstack index 2 (below inner frame). Added non-immediate
  return-stack words `>R`, `R>`, `R@`, `2>R`, `2R>`, `2R@`. Nested
  DO/LOOP with J tested; LEAVE in nested loops exits only the inner.
  177/177 green.

- **Phase 3 — `BEGIN`/`UNTIL`/`WHILE`/`REPEAT`/`AGAIN` (+9).** Indefinite-loop
  constructs built on the same PC-driven body runner introduced for `IF`.
  BEGIN records the current body length on `state.cstack` (a plain numeric
  back-target). UNTIL/AGAIN pop that back-target and emit a `bif`/`branch`
  op whose target cell is set to the recorded PC. WHILE emits a forward
  `bif` with a fresh target cell and pushes it on the cstack *above* the
  BEGIN marker; REPEAT pops both (while-target first, then back-pc), emits
  an unconditional branch back to BEGIN, then patches the while-target to
  the current body length — so WHILE's false flag jumps past the REPEAT.
  Mixed compile-time layout (numeric back-targets + dict forward targets
  on the same cstack) is OK because the immediate words pop them in the
  order they expect. AGAIN works structurally but lacks a test without a
  usable mid-loop exit; revisit once `EXIT` lands. 161/161 green.

- **Phase 3 start — `IF`/`ELSE`/`THEN` (+18).** `lib/forth/compiler.sx`
  + `tests/test-phase3.sx`. Colon-def body switched from `for-each` to
  a PC-driven runner so branch ops can jump: ops now include dict tags
  `{"kind" "bif"|"branch" "target" cell}` alongside the existing
  `(fn (s) ...)` shape. IF compiles a `bif` with a fresh target cell
  pushed to `state.cstack`; ELSE emits an unconditional `branch`,
  patches the IF's target to the instruction after this branch, and
  pushes the new target; THEN patches the most recent target to the
  current body length. Nested IF/ELSE/THEN works via the cstack.
  Also fixed `EMIT`: `code-char` → `char-from-code` (spec-correct
  primitive name) so Phase 1/2 tests run green on sx_server.
  152/152 (Phase 1 + 2 + 3) green.

- **Phase 2 complete — colon defs, compile mode, VARIABLE/CONSTANT/VALUE/TO, @/!/+! (+26).**
  `lib/forth/compiler.sx` plus `tests/test-phase2.sx`.
  Colon-def body is a list of ops (one per source token) wrapped in a single
  lambda. References are late-binding thunks so forward/recursive references
  work via `RECURSE`. Redefinitions take effect immediately.
  VARIABLE creates a pusher for a symbolic address stored in `state.vars`.
  CONSTANT compiles to `(fn (s) (forth-push s v))`. VALUE/TO share the vars dict.
  Compiler rewrites `forth-interpret` to drive from a token list stored on
  state so parsing words (`:`, `VARIABLE`, `TO` etc.) can consume the next
  token with `forth-next-token!`. 134/134 (Phase 1 + 2) green.

- **Phase 1 complete — reader + interpret mode + core words (+108).**
  `lib/forth/{runtime,reader,interpreter}.sx` plus `tests/test-phase1.sx`.
  Stack as SX list (TOS = first). Dict is `{lowercased-name -> {:kind :body :immediate?}}`.
  Data + return stacks both mutable. Output buffered in state (no host IO yet).
  BASE-aware number parsing with `$`, `%`, `#` prefixes and `'c'` char literals.
  Bitwise AND/OR/XOR/INVERT simulated over 32-bit two's-complement.
  Integer `/` is truncated-toward-zero (ANS symmetric), MOD matches.
  Case-insensitive lookup. 108/108 tests green.

## Blockers

- _(none yet)_