rose-ash/plans/js-on-sx.md

# JS-on-SX: cherry-picked test262 conformance

Transpile a restricted JS subset to SX AST, run on the existing CEK/VM. Goal is runtime-hardening via conformance tests, plus a second hyperscript substrate (hyperscript.js running on SX).

## Ground rules for the loop

- **Scope:** only touch `lib/js/**` and `plans/js-on-sx.md`. Do **not** edit `spec/evaluator.sx`, `spec/primitives.sx`, `shared/sx/**`, or `lib/hyperscript/**` — the user is working on those elsewhere.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix them from this loop.
- **SX files:** use `sx-tree` MCP tools only (never `Edit`/`Read`/`Write` on `.sx` files). Use `sx_write_file` for new files, path-based or pattern-based edits for changes.
- **Architecture:** JS source → JS AST → SX AST → existing CEK. No standalone JS evaluator — reuse everything.
- **Tests:** mirror `lib/hyperscript/test.sh` pattern. Start with a tiny in-repo smoke suite; graduate to a cherry-picked test262 slice once the lexer+expression-parser cycle is green.

## The loop

A single long-running background agent works `plans/js-on-sx.md` forever, one feature per commit. It runs a prioritized queue driven by the real test262 scoreboard. The briefing lives at `plans/agent-briefings/loop.md`; the recovery helper is `plans/restore-loop.sh`.

The queue (condensed — full version in the briefing):

1. Baseline commit (stage what's on disk now).
2. Fix `lib/js/test262-runner.py` so it produces a real scoreboard.
3. Full scoreboard run across the whole `test/` tree.
4. Regex lexer/parser/runtime stub + Blockers entry listing platform primitives needed.
5. Scoreboard-driven: pick the worst-passing category each iteration; fix; re-score.
6. When the scoreboard plateaus, tackle deferred items (ASI, CEK-suspend await).

### Crash recovery

The agent process dies with the machine. **Work survives if it was committed** — that's why every iteration ends in a commit. After a crash:

```bash
bash plans/restore-loop.sh            # show state: tests, commits, scoreboard, regex hook
bash plans/restore-loop.sh --print    # also cat the briefing for easy copy-paste
```

Then respawn the agent by pasting `plans/agent-briefings/loop.md` into Claude Code via the `Agent` tool with `run_in_background=true`. The agent re-reads the plan, picks up wherever the queue has got to, and carries on.

## Architecture sketch

```
JS source text
    │
    ▼
lib/js/lexer.sx          — tokens: {:type :value :pos}
    │
    ▼
lib/js/parser.sx         — JS AST as SX trees, e.g. (js-binop "+" left right)
    │
    ▼
lib/js/transpile.sx      — JS AST → SX AST (reusable by CEK)
    │                       e.g. (js-binop "+" l r)  →  (+ (transpile l) (transpile r))
    ▼
existing CEK / VM        — evaluation, async/IO, exceptions already handled
```

Runtime shims in `lib/js/runtime.sx`: `js-global`, `js-console-log`, `js-typeof`, coercion helpers (`to-number`, `to-string`, `to-boolean`, abstract-equality), eventually the prototype chain.

## Roadmap

Each item: implement → tests → update progress. Mark `[x]` when tests green.

### Phase 1 — Lexer
- [x] Numeric literals (int, float, hex, exponent)
- [x] String literals (double, single, escape sequences, template strings later)
- [x] Identifiers + reserved words
- [x] Punctuation: `( ) { } [ ] , ; : . ...`
- [x] Operators: `+ - * / % ** = == === != !== < > <= >= && || ! ?? ?: & | ^ ~ << >> >>> += -= ...`
- [x] Comments (`//`, `/* */`)
- [ ] Automatic Semicolon Insertion (defer — initially require semicolons)

### Phase 2 — Expression parser (Pratt-style)
- [x] Literals → AST nodes
- [x] Binary operators with precedence
- [x] Unary operators (`- + ! ~ typeof void`)
- [x] Member access (`.`, `[]`)
- [x] Function calls
- [x] Array literals
- [x] Object literals (string/ident keys, shorthand later)
- [x] Conditional `a ? b : c`
- [x] Arrow functions (expression body only)

### Phase 3 — Transpile to SX AST
- [x] Numeric/string/bool/null literals → SX literals
- [x] Arithmetic: `+ - * / % **` with numeric coercion (`js-add` does string-concat dispatch)
- [x] Comparisons: `=== !== == != < > <= >=`
- [x] Logical: `&& ||` (short-circuit, value-returning via thunk trick); `??` nullish coalesce
- [x] Unary: `- + ! ~ typeof void`
- [x] Member access → `js-get-prop` (handles dicts, lists `.length`/index, strings)
- [x] Array literal → `(list ...)`
- [x] Object literal → `(dict)` + `dict-set!` sequence
- [x] Function call → SX call (callee can be ident, member, arrow)
- [x] Arrow fn → `(fn ...)` (params become SX symbols; closures inherited)
- [x] Assignment (`=` and compound `+= -= ...`) on ident/member/index targets
- [x] `js-eval` end-to-end: source → tokens → AST → SX → `eval-expr`

### Phase 4 — Runtime shims (`lib/js/runtime.sx`)
- [x] `js-typeof`, `js-to-number`, `js-to-string`, `js-to-boolean`
- [x] Abstract equality (`js-loose-eq`) vs strict (`js-strict-eq`)
- [x] Arithmetic (`js-add` `js-sub` `js-mul` `js-div` `js-mod` `js-pow` `js-neg` `js-pos`)
- [x] Logical (`js-and` `js-or` via thunks for lazy rhs) and `js-not` / `js-bitnot`
- [x] Relational (`js-lt` `js-gt` `js-le` `js-ge`) incl. lexicographic strings
- [x] `js-get-prop` / `js-set-prop` (dict/list/string; `.length`; numeric index)
- [x] `console.log` → `log-info` bridge (`console` dict wired)
- [x] `Math` object shim (`abs` `floor` `ceil` `round` `max` `min` `random` `PI` `E`)
- [x] `js-undefined` sentinel (keyword) distinct from `nil` (JS `null`)
- [x] Number parser `js-num-from-string` (handles int/float/±sign, no NaN yet)

### Phase 5 — Conformance harness
- [x] Cherry-picked slice vendored at `lib/js/test262-slice/` (69 fixtures across 7 categories)
- [x] Harness `lib/js/conformance.sh` — batch-load kernel, one epoch per fixture, substring-compare
- [x] Initial target: ≥50% pass. **Actual: 69/69 (100%).**

### Phase 6 — Statements
- [x] `var`/`let`/`const` declarations (all behave as `define` — block scope via SX lexical semantics)
- [x] `if`/`else`
- [x] `while`, `do..while`
- [x] `for (init; cond; step)`
- [x] `return`, `break`, `continue` (via `call/cc` continuation bindings `__return__` / `__break__` / `__continue__`)
- [x] Block scoping (via `begin` — lexical scope inherited, no TDZ)

### Phase 7 — Functions & scoping
- [x] `function` declarations (with call/cc-wrapped bodies for `return`)
- [x] Function expressions (named + anonymous)
- [x] Hoisting — function decls hoisted to enclosing scope (scan body first, emit defines ahead of statements)
- [x] Closures — work via SX `fn` env capture
- [x] Rest params (`...rest` → `&rest`)
- [x] Default parameters (desugar to `if (param === undefined) param = default`)
- [ ] `var` hoisting (deferred — treated as `let` for now)
- [ ] `let`/`const` TDZ (deferred)

### Phase 8 — Objects, prototypes, `this`
- [x] Property descriptors (simplified — plain-dict `__proto__` chain, `js-set-prop` mutates)
- [x] Prototype chain lookup (`js-dict-get-walk` walks `__proto__`)
- [x] `this` binding rules: method call, function call (undefined), arrow (lexical)
- [x] `new` + constructor semantics (fresh dict, proto linked, ctor called with `this`)
- [x] ES6 classes (sugar over prototypes, incl. `extends`)
- [x] Array mutation: `a[i] = v`, `a.push(...)` — via `set-nth!` / `append!`
- [x] Array builtins: push, pop, shift, slice, indexOf, join, concat, map, filter, forEach, reduce
- [x] String builtins: charAt, charCodeAt, indexOf, slice, substring, toUpperCase, toLowerCase, split, concat
- [x] `instanceof` and `in` operators

### Phase 9 — Async & Promises
- [x] Promise constructor + `.then` / `.catch` / `.finally`
- [x] `Promise.resolve` / `Promise.reject` / `Promise.all` / `Promise.race`
- [x] `async` functions (decl, expr, arrow) return Promises
- [x] `await` — synchronous-ish: drains microtasks, unwraps settled Promise
- [x] Microtask queue with FIFO drain (`__drain()` exposed to JS)
- [ ] True CEK suspension on `await` for pending Promises (deferred — needs cek-step-loop plumbing)

### Phase 10 — Error handling
- [x] `throw` statement → `(raise v)`
- [x] `try`/`catch`/`finally` (desugars to `guard` + optional finally wrapper)
- [x] Error hierarchy (`Error`, `TypeError`, `RangeError`, `SyntaxError`, `ReferenceError` as constructor shims)

### Phase 11 — Stretch / deferred
- ASI, regex literals, generators, iterators, destructuring, template strings with `${}`, tagged templates, Symbol, Proxy, typed arrays, ESM modules.

## Progress log

Append-only record of completed iterations. Loop writes one line per iteration: date, what was done, test count delta.

- 2026-04-23 — scaffold landed: lib/js/{lexer,parser,transpile,runtime}.sx stubs + test.sh. 7/7 smoke tests pass (js-tokenize/js-parse/js-transpile stubs + js-to-boolean coercion cases).
- 2026-04-23 — Phase 1 (Lexer) complete: numbers (int/float/hex/exp/leading-dot), strings (escapes), idents/keywords, punctuation, all operators (1-4 char, longest-match), // and /* */ comments. 38/38 tests pass. Gotchas found: `peek` and `emit!` are primitives (shadowed to `js-peek`, `js-emit!`); `cond` clauses take ONE body only, multi-expr needs `(do ...)` wrapper.
- 2026-04-23 — Phase 2 (Pratt expression parser) complete: literals, binary precedence (w/ `**` right-assoc), unary (`- + ! ~ typeof void`), member access (`.`/`[]`), call chains, array/object literals (ident+string+number keys), ternary, arrow fns (zero/one/many params; curried), assignment (right-assoc incl. compound `+=` etc.). AST node shapes all match the `js-*` names already wired. 47 new tests, 85/85 total. Most of the Phase 2 scaffolding was already written in an earlier session — this iteration verified every path, added the parser test suite, and greened everything on the first pass. No new gotchas beyond Phase 1.
- 2026-04-23 — Phase 3 (Transpile to SX AST) complete: dispatch on AST head using `js-tag?`/`symbol-name` inspection, emit SX trees built from `list`/`cons`/`make-symbol`. All binops, unaries, member/index, call (arbitrary callee), array (`list`), object (`let` + `dict` + `dict-set!`), ternary (`if` around `js-to-boolean`), arrow (`fn` with `make-symbol` params), assignment (ident → `set!`; member/index → `js-set-prop`; compound → combines). Short-circuit `&&`/`||` built via thunk passed to `js-and`/`js-or` — this preserves JS value-returning semantics and avoids re-evaluating lhs. `??` uses `let`+`if`. `js-eval src` pipelines `js-parse-expr` → `js-transpile` → `eval-expr`. 69 new assertions for end-to-end eval; 154/154 main suite.
- 2026-04-23 — Phase 4 (Runtime shims) complete: coercions (ToBoolean/ToNumber/ToString) including a tiny recursive string→number parser (handles ±int/float, no NaN yet); arithmetic with JS `+` dispatch (string-concat when either operand is string, else ToNumber); `js-strict-eq`/`js-loose-eq` with null↔undefined and boolean-coercion rules; relational with lexicographic string path via `char-code`; `js-get-prop`/`js-set-prop` covering dict/list/string with numeric index and `.length`; `Math` object, `console.log`, `js-undefined` sentinel. Several SX gotchas hit and noted below.
- 2026-04-23 — Phase 5 (Conformance harness) complete: 69 hand-picked fixtures under `lib/js/test262-slice/` covering arithmetic, comparison, loose+strict equality, logical, nullish, conditional, arrays, objects, strings, arrows, Math, typeof. Runner `lib/js/conformance.sh` builds one batch script (single kernel boot), one epoch per fixture, substring-matches the sibling `.expected` file. Scoring: **69/69 (100%)** — well past the 50% target. Real test262 integration deferred to later phase (needs network + Python harness; deferred to Phase 5.5 per plan).
- 2026-04-23 — Phases 6 + 7 (Statements + Functions) complete. Parser now accepts top-level programs — `js-parse` returns `(js-program (stmts...))`, with new node types `js-var / js-vardecl / js-block / js-if / js-while / js-do-while / js-for / js-return / js-break / js-continue / js-exprstmt / js-empty / js-funcdecl / js-funcexpr / js-param / js-rest`. Arrow bodies now accept `{...}` block form. Transpile dispatch extended with 14 new clauses. Loops built via `letrec` recursion; `break` / `continue` / `return` via lexical `call/cc` bindings (`__break__` / `__continue__` / `__return__`). Function declarations hoisted to the enclosing scope before other statements run (two-pass: `js-collect-funcdecls` scans, `js-transpile-stmt-list` replaces the hoisted entry with `nil`). Default params desugar to `(if (or (= x nil) (= x :js-undefined)) default x)` updates; rest params emit `&rest name`. Unit tests: **195/195** (154 → +41 for statements, functions, closures, loops). Conformance: **96/96** (69 → +27 new fixtures across `statements/`, `loops/`, `functions/`, `closures/`). Gotchas: (1) SX `do` is R7RS iteration, not sequence — must use `begin`. A `(do (x) ...)` where `x` is a list → "first: expected list, got N" because the do-form tries to parse its iteration bindings. (2) SX passes unsupplied fn params as `nil`, not an undefined sentinel — default-param init must test for both `nil` and `:js-undefined`. (3) `jp-collect-params` (used by arrow heads) doesn't understand rest/defaults; new `jp-parse-param-list` used for function declarations. Arrow rest/defaults deferred. (4) `...` lexes as `punct`, not `op`.

- 2026-04-23 — **Phase 9 (Async & Promises) complete.** New AST tags: `js-await`, `js-funcdecl-async`, `js-funcexpr-async`, `js-arrow-async`. Parser extended: `async` keyword consumed, dispatches by the next token (function/ident/paren). Primary parser grows a pre-function `async` case and a new `await` unary. Statement parser adds a two-token lookahead for `async function` decls. Runtime adds: microtask queue (`__js_microtask_queue__` dict cell + push/pop/empty/drain), `js-promise?` predicate, full `{:__js_promise__ true :state :value :callbacks}` object, `js-promise-resolve!`/`reject!`/`flush-callbacks!`, callback dispatch (`run-callback!` / `run-handler!` / `try-call` using `guard`), `.then` via `js-promise-then-internal!`, `.catch`/`.finally` derivative calls. `js-invoke-method` now routes Promise methods through `js-invoke-promise-method` (same single-dispatch no-closure pattern as Phase 8 list/string builtins). `Promise` constructor runs executor synchronously inside a guard so throws reject the Promise. Statics `resolve`/`reject`/`all`/`race` live in `__js_promise_statics__` dict; `js-get-prop` special-cases identity-equality against the `Promise` function. `js-async-wrap` wraps a thunk → Promise (fulfilled on return, rejected on throw, adopts returned Promises). `js-await-value` drains microtasks then unwraps a settled Promise or raises its reason; pending Promise = error (no scheduler — see Blockers). `js-eval` drains microtasks at end. `__drain()` exposed to JS so tests can force-run pending callbacks synchronously before reading a mutable result. Arity-tolerant call path `js-call-arity-tolerant` adapts 1-arg handler invocations to handlers declared with `()` (zero params) via `lambda-params` introspection. Unit tests: **254/254** (+31 parser + runtime). Conformance: **148/148** (+29: `test262-slice/promises/*` × 16, `test262-slice/async/*` × 13). Microtask ordering is FIFO (append on settle, drain one-at-a-time); spec-ish but not obsessive about thenable-adoption iteration count. Gotchas: (1) **`cond` needs `begin` for multi-body clauses** — same rule as Phase 1, bit me hard because the original draft had `(cond ((state) (side-effect) (side-effect2)))` which silently discarded the first expression as "predicate" and tried to call it as a function. (2) **`guard` with multi-body handler clauses** — same fix, `(guard (e (else (begin …))))`. (3) **`(= (type-of fn) "function")` is FALSE** — `type-of` returns `"lambda"` for user-defined fns; use `js-function?` which accepts lambda/function/component. (4) **Forward refs in SX work** because `define` is late-bound in the global env. (5) **Microtask semantics vs top-level last-expression** — `js-eval` evaluates all stmts THEN drains; if the last stmt reads `r` assigned in a `.then`, you'll see `nil` unless you insert `__drain()` between the setup and the read. (6) **`Promise.resolve(p)` returns p for existing Promises** — identity preserved via `(js-promise? v) → v` short-circuit. (7) **Strict arity in SX lambdas vs tolerant JS** — `() => side-effect()` in JS accepts extra args silently; SX `(fn () ...)` errors. Callback invocations go through `js-call-arity-tolerant` which introspects `lambda-params` and calls with no args if the handler has zero params.

- 2026-04-23 — **Queue item 1: baseline commit.** Staged `lib/js/` tree + `plans/` as committed by prior sessions. 278/280 unit (2 failing template-string edges: epoch 903 part-count off-by-one, 934 escaped-backtick ident-lookup), 148/148 slice. Runner stub at 0/8 with 7 timeouts. Commit `9e568ad8`. Out-of-scope changes in `lib/compiler.sx`, `lib/hyperscript/compiler.sx`, `shared/static/wasm/sx/hs-compiler.sx` intentionally left unstaged per briefing scope rules.
- 2026-04-23 — Phases 8 + 10 (Objects + Errors) complete in a single session. **Object model:** regular JS `function` bodies wrap with `(let ((this (js-this))) ...)` — a dynamic `this` via a global cell `__js_this_cell__`. Method calls `obj.m(args)` route through `js-invoke-method` which saves/restores the cell around the call, so `this` works without an explicit first-arg calling convention. Arrow functions don't wrap — they inherit the enclosing lexical `this`. **`new`:** creates a fresh dict with `__proto__` linked to the constructor's prototype dict, calls the constructor with `this` bound, returns the ctor's dict return (if any) else the new object. **Prototype chain:** lives in a side table `__js_proto_table__` keyed by `inspect(ctor)`. `ctor.prototype` access and assignment both go through this table. `js-dict-get-walk` walks the `__proto__` chain on dict property lookup. **Classes:** desugar to `(define Name ctor)` + `(js-reset-ctor-proto! Name)` (critical for redefinition) + `(dict-set! (js-get-ctor-proto Name) mname mfn)` for each method. `extends` chains by setting `(js-get-ctor-proto Child).__proto__ = (js-get-ctor-proto Parent)`. Default ctor with `extends` calls parent with same args. **Arrays:** `js-set-prop` on lists dispatches to `js-list-set!` which does in-bounds `set-nth!` or `append!` past end (pads with `js-undefined`). No shrinking (primitive gap — `pop-last!` is a no-op). **Array + String builtins** are routed through `js-invoke-method` directly via `js-invoke-list-method` / `js-invoke-string-method` to AVOID a VM JIT bug: returning a closure from a JIT-compiled function (which happened when `js-array-method` returned an inner `fn`) crashed with "VM undefined: else". Dispatching without closures works. **Throw/try/catch/finally:** `throw v` → `(raise v)`; try/catch → `(guard (e (else cbody)) body)`; finally wraps via `(let ((r try-tr)) finally-tr r)`. **Error hierarchy:** `Error`/`TypeError`/`RangeError`/`SyntaxError`/`ReferenceError` are constructor shims that set `this.message` + `this.name` on the new object. **`instanceof` + `in`:** parser precedence table extended to accept both as keywords at prec 10; binary-loop predicate extended to allow keyword-type tokens for these two. Unit tests: **223/223** (+28). Conformance: **119/119** (+23 new fixtures across `objects/` and `errors/`). Gotchas: (1) **Ctor-id collision on redefine** — `inspect` of a lambda is keyed by (name + arity), so redefining `class B` found the OLD proto-table entry. Fix: class decl always calls `js-reset-ctor-proto!`. (2) **VM closure bug** — functions returning inner closures from JIT-compiled bodies break: `(fn (arr) (fn (f) ...use arr...))` compiles to a VM closure for the outer that can't produce a working inner. Workaround: route all builtin method dispatch through a single (non-closure-returning) helper. (3) **`jp-parse-param-list` eats its own `(`** — don't prefix with `jp-expect! st "punct" "("`, the parser handles both. Class method parser hit this.

- 2026-04-23 — **Queue item 2: fixed test262 runner.** Root-cause of 7/8 timeouts: runner re-parsed the entire 197-line `assert.js` for every test in one big `js-eval` (8.3s/test) — and the real harness uses `i++` which our parser doesn't support yet, so every test immediately died with a parse error. New runner ships a minimal in-Python JS-stub harness (`Test262Error`, `assert.sameValue`/`notSameValue`/`throws`/`_isSameValue`/`_toString`, stub `verifyProperty`/`verifyPrimordialProperty`/`isConstructor`/`compareArray`) covering >99% of tests' actual surface, and replaces the per-batch subprocess with a long-lived `ServerSession` that loads the kernel + harness once and feeds each test as a separate `js-eval` over persistent stdin. Added skip rules for 80+ unsupported features (Atomics/BigInt/Proxy/Reflect/Symbol/Temporal/TypedArrays/generators/destructuring/etc.) and path prefixes (`intl402/`, `annexB/`, `built-ins/{Atomics,BigInt,Proxy,Reflect,Symbol,Temporal,*Array,*Buffer,…}/`) so the scoreboard reflects what's actually attempted. Scoreboard over 288 runnable Math tests: **56/288 (19.4%)** in 185s, rate ≈ 2.3 tests/s (prev: 0/8 with 7 timeouts). Top failure modes: 83× assertion-fail (real semantic gaps in Math.floor/ceil/trunc/etc. details), 62× ReferenceError (builtins we haven't shimmed, e.g. `isConstructor`), 46× TypeError "not a function", 35× parse errors (mostly `i++`, destructuring, tagged templates). 278/280 unit + 148/148 slice unchanged.

## Phase 3-5 gotchas

Worth remembering for later phases:

- **Varargs in SX** use `&rest args`, not bare `args`. `(fn args …)` errors with "Expected list, got symbol" — use `(fn (&rest args) …)`.
- **`make-symbol`** is the way to build an SX identifier-symbol at runtime for later `eval-expr`. Use it to turn JS idents into SX variable references.
- **`eval-expr`** is the primitive you want to evaluate an SX list you've just constructed. It's already in the kernel primitives.
- **Keywords print as quoted strings** via the epoch `eval` command — `:foo` comes back as `"foo"`. Affected the `js-undefined` test expectation.
- **`char-code` (not `code-char`)** for char→codepoint. No `bit-not` primitive — implement `~x` as `-(int(x)+1)`.
- **Epoch `eval` string must double-escape** to survive two nested SX string literals (`(eval "(js-eval \"…\")")`). The conformance harness uses a small Python helper to do that reliably.
- **Shell heredocs and `||`** — writing fixtures with `||` in a here-doc fed to a `while read` loop gets interpreted as a pipe mid-parse. Use `case|a\|\|b|…` style or hand-write the `||` cases.

## Blockers / shared-file issues

Anything that would require a change outside `lib/js/` goes here with a minimal repro. Don't fix from the loop.

- **Pending-Promise await** — our `js-await-value` drains microtasks and unwraps *settled* Promises; it cannot truly suspend a JS fiber and resume later. Every Promise that settles eventually through the synchronous `resolve`/`reject` + microtask path works. A Promise that never settles without external input (e.g. a real `setTimeout` waiting on the event loop) would hit the `"await on pending Promise (no scheduler)"` error. Proper async suspension would need the JS eval path to run under `cek-step-loop` (not `eval-expr` → `cek-run`) and treat `await pending-Promise` as a `perform` that registers a resume thunk on the Promise's callback list. Non-trivial plumbing; out of scope for this phase. Consider it a Phase 9.5 item.

## First-iteration checklist (scaffolding) — DONE

- [x] `lib/js/lexer.sx` — stub `js-tokenize`
- [x] `lib/js/parser.sx` — stub `js-parse`
- [x] `lib/js/transpile.sx` — stub `js-transpile`
- [x] `lib/js/runtime.sx` — stub `js-global`, `js-to-boolean`
- [x] `lib/js/test.sh` — epoch-protocol runner mirroring `lib/hyperscript/test.sh`
- [x] Smoke suite green (7/7)

Next iteration: Phase 1 — lexer. Start with numeric literals + identifiers + whitespace skipping; extend test.sh with tokenization assertions.