rose-ash/plans/scheme-on-sx.md

# Scheme-on-SX: the reflective-kit second-consumer port

The kernel-on-sx loop documented six reflective API candidates; two are now live (`env.sx`, `class-chain.sx`). Three more — `evaluator.sx`, `hygiene.sx`, `quoting.sx` — wait on a guest with operative-free lexical scope, hygienic syntax-transformer infrastructure, and quasiquote. **Scheme is exactly that guest.**

A correct R7RS-small implementation acts as second consumer for those three kits in one stroke. It also confirms a third independent consumer for `env.sx` (after Kernel + Tcl + Smalltalk), and a candidate fourth consumer for `class-chain.sx` (Scheme's record types have parent fields — though OO is non-core in Scheme so the fit is weaker).

## Strategic note on `combiner.sx`

Scheme has *no fexprs*. `combiner.sx`'s applicative/operative split is Kernel-specific machinery. **Scheme is not a second consumer for `combiner.sx`** — that file stays Kernel-only until a Maru, Klisp, or CL-fexpr port arrives. The current session's earlier claim that Scheme "unlocks four more reflective kits" was over-counted; the correct number is **three**.

## Scope decisions

- **Target dialect:** R7RS-small. Source-only — no images, no FFI, no C extensions, no JIT.
- **Numbers:** integers + floats. Rationals optional (defer to phase N+1). Complex out.
- **Tail-call optimisation:** required. Implemented via the existing SX CEK machinery — call recursion in the evaluator uses iterative `cek-call` rather than host recursion.
- **Continuations:** `call/cc` required for R7RS. Use SX's `call/cc` primitive directly.
- **Hygienic macros:** `syntax-rules` required. `syntax-case` deferred.
- **Char/string semantics:** Unicode codepoints; surface API matches R7RS section 6.
- **I/O:** minimal stub (`display`, `write`, `newline`, `read`) on SX's IO surface.
- **`define-library`:** required for module testing; implementation reuses SX's `define-library` if it's exposed, else hand-rolls a flat module registry.

## Architecture sketch

```
lib/scheme/parser.sx     — reader: numbers, strings, symbols, booleans,
                            chars #\c, vectors #(...), dotted-pairs (a . b),
                            quasi-quote sugar, datum comments #;, block
                            comments #| ... |#

lib/scheme/eval.sx       — eval-expr ENV: walks AST. Symbols → env-lookup.
                            Lists → look up head; if syntactic operator
                            (if/lambda/define/set!/quote/quasiquote/
                             let/let*/letrec/begin/cond/case/and/or/when/
                             unless/do), dispatch to native handler. Else
                            apply combiner (always applicative).

                          ENV is `lib/guest/reflective/env.sx` directly
                          — Scheme is the third consumer for env.sx with
                          NO adapter cfg (canonical wire shape).

lib/scheme/runtime.sx    — Standard environment, primitives, R7RS base.
                            Variadic arithmetic, list ops, string ops,
                            char ops, vector ops, define-record-type,
                            syntax-rules, etc.

lib/scheme/tests/        — Standard pattern: parse, eval, lambda+closure,
                            macros (syntax-rules), call/cc, define-library,
                            classic programs (factorial, Y, tree-walking,
                            named let, do-loop), R7RS conformance subset.
```

## Roadmap

### Phase 1 — Parser
- [ ] Reader for R7RS lexical syntax: integers, floats, strings (with escapes), symbols (extended-identifier-character set), booleans `#t`/`#f`/`#true`/`#false`, characters `#\c` `#\space` `#\newline`, vectors `#(...)`, dotted pairs `(a . b)`, quote/quasiquote/unquote/unquote-splicing sugar (same reader macros as Kernel).
- [ ] Datum comments `#;<datum>` (skip one whole expression).
- [ ] Block comments `#| ... |#` (nestable).
- [ ] Tests in `lib/scheme/tests/parse.sx`.

### Phase 2 — Evaluator + env
- [ ] `scheme-eval EXPR ENV` — primary entry, uses `lib/guest/reflective/env.sx` directly as the canonical scope chain. **Third consumer for env.sx.**
- [ ] Self-evaluating: numbers, booleans, strings, chars, vectors.
- [ ] Symbol lookup → `refl-env-lookup-with`.
- [ ] List → look up head; syntactic operators dispatch natively; otherwise applicative call with evaluated args.
- [ ] Tests in `lib/scheme/tests/eval.sx`.

### Phase 3 — Syntactic operators
- [ ] `if`, `quote`, `set!`, `define` (top-level + internal).
- [ ] `lambda` — fixed-arity, rest-arg via dot, multi-body via implicit `begin`.
- [ ] `let`, `let*`, `letrec`, `letrec*` — including named-let.
- [ ] `begin` — implicit + explicit.
- [ ] `cond`, `case`, `when`, `unless`, `and`, `or`, `do`.
- [ ] Tests for each.

### Phase 4 — Standard environment
- [ ] Variadic `+ - * /` and chained comparison.
- [ ] Type predicates (R7RS `number?`, `pair?`, `null?`, `symbol?`, `string?`, `procedure?`, `vector?`, `char?`, `boolean?`).
- [ ] List ops: `cons car cdr caar cadr ... cddddr` (or just a subset), `list length reverse append map filter fold-left fold-right for-each`.
- [ ] String ops: `string-length string-ref substring string-append string=? string<? char->integer integer->char`.
- [ ] Char ops: `char->integer integer->char char-alphabetic? char-numeric?` etc.
- [ ] Vector ops: `vector make-vector vector-length vector-ref vector-set! vector->list list->vector`.
- [ ] I/O: `display write newline read`.
- [ ] Numerical: `abs floor ceiling round truncate min max modulo quotient remainder gcd lcm expt`.
- [ ] Classic programs: factorial, fib, list reversal, tree map.

### Phase 5 — call/cc + dynamic-wind
- [ ] `call-with-current-continuation` / `call/cc`.
- [ ] `dynamic-wind`.
- [ ] `with-exception-handler`, `raise`, `error`.
- [ ] Tests: escape continuations, multi-shot via call/cc (chosen via host SX `call/cc`).

### Phase 6 — `syntax-rules` + hygiene
- [ ] `define-syntax`, `let-syntax`, `letrec-syntax`.
- [ ] `syntax-rules` pattern matching, ellipsis, template instantiation.
- [ ] Hygiene: scope-set / lifted-symbol implementation. **Second consumer for `lib/guest/reflective/hygiene.sx` extraction once that kit's API surface stabilises.**
- [ ] Tests: hygienic identifier capture, ellipsis patterns, recursive macros.

### Phase 7 — Reflection: `eval`, `interaction-environment`, etc.
- [ ] `eval EXPR ENV` — applicative form of the evaluator. **Second consumer for `lib/guest/reflective/evaluator.sx` extraction.**
- [ ] `interaction-environment`, `null-environment`, `scheme-report-environment`.
- [ ] `environment?` predicate.

### Phase 8 — `define-library` + module hygiene
- [ ] `define-library`, `import`, `export`.
- [ ] `cond-expand` for feature-flag conditionals.
- [ ] Tests: cross-library imports, identifier renaming.

### Phase 9 — Records
- [ ] `define-record-type` with constructor/predicate/accessors/mutators.
- [ ] Tests: typical record idioms.

### Phase 10 — Quasiquote runtime
- [ ] Backquote walker with depth tracking. **Second consumer for `lib/guest/reflective/quoting.sx` extraction.**
- [ ] Tests including nested quasiquote.

### Phase 11 — Conformance + scoreboard
- [ ] Curated R7RS test slice (Chibi, Larceny, or hand-picked).
- [ ] `lib/scheme/conformance.sh` + scoreboard.
- [ ] Drive conformance toward 100% on chosen slice.

## Reflective kit consumption — explicit mapping

| Kit | When it lands | How Scheme uses it |
|-----|--------------|-------------------|
| `lib/guest/reflective/env.sx` | Phase 2 | Direct — canonical wire shape, no cfg needed. Third consumer. |
| `lib/guest/reflective/evaluator.sx` | Phase 7 (will trigger the extraction) | Scheme's `eval`/`interaction-environment`/`null-environment` mirror the proposed `refl-eval`/`refl-make-environment`/`refl-current-env` triple. Second consumer → extraction unblocked. |
| `lib/guest/reflective/hygiene.sx` | Phase 6 | Scheme's hygienic `syntax-rules` is the canonical implementation of scope sets / lifted symbols. Second consumer for the deferred Shutt-style hygiene work — Scheme's hygiene goes BEYOND Kernel's by-default-static-env-extension into proper scope-set lifting. Drives the deferred research-grade kit. |
| `lib/guest/reflective/quoting.sx` | Phase 10 | Scheme's backquote walker is structurally identical to Kernel's `knl-quasi-walk`, with depth tracking added. Second consumer → extraction unblocked. |
| `lib/guest/reflective/combiner.sx` | NEVER (no fexprs) | Not applicable. Stays Kernel-only until a fexpr-having consumer arrives. |
| `lib/guest/reflective/short-circuit.sx` | Possibly Phase 3 | Scheme's `and`/`or` are syntactic, not operative; could be second consumer but adapter would need to bridge "macro that short-circuits" vs "operative that short-circuits". Marginal. |

## Ground rules

- **Scope:** only `lib/scheme/**` and `plans/scheme-on-sx.md` and `lib/guest/reflective/**` (for extraction work). Don't edit `spec/`, `hosts/`, `shared/`, or other `lib/<lang>/` directories.
- **Consume:** `lib/guest/lex.sx` (character predicates), `lib/guest/reflective/env.sx` (scope chain), eventually `evaluator.sx`/`hygiene.sx`/`quoting.sx` once extracted with Scheme as second consumer.
- **Commits:** one feature per commit. Short factual messages.
- **Tests:** every phase ends with a test file. Conformance scoreboard at the end.
- **Branch:** `loops/scheme`. Worktree pattern (already set up at `/root/rose-ash-loops/scheme`).
- **Substrate gaps:** filed to `sx-improvements.md`, not fixed in this loop.

## References

- R7RS-small: https://small.r7rs.org/attachment/r7rs.pdf
- Chibi Scheme — a small, readable R7RS implementation.
- Dybvig, "Three Implementation Models for Scheme" — for the hygiene story.
- Existing kernel-on-sx code in `lib/kernel/` — much of the parser, evaluator structure, and env handling carries over near-verbatim because Kernel and Scheme share lexical scope.

## Progress log

- 2026-05-14 — **Phases 1, 2, 3, 3.5, 4, 5abc, 6ab, 7, 8, 9, 10, 11 landed in one loop session.** 296 Scheme tests across 9 suites; ~1830 LoC of substrate. Test runner + scoreboard at `lib/scheme/test.sh` and `lib/scheme/scoreboard.md`. Three reflective kits unlocked: `env.sx` extracted directly as third consumer, `evaluator.sx` and `quoting.sx` second-consumer-ready for the kit-extraction commits (kit code is documented in `plans/kernel-on-sx.md`; Scheme consumer code is in place).

### Phase-by-phase outcomes

- Phase 1 (Parser, 62 tests): R7RS lexical syntax with reader macros, three comment flavours (`;`, `#;`, `#| |#`).
- Phase 2 (Eval + env third-consumer, 23 tests): `scheme-make-env` etc. are thin aliases for `refl-env-*` from `lib/guest/reflective/env.sx`. No adapter cfg needed — Scheme uses the canonical wire shape directly.
- Phase 3 (if/define/set!/begin/lambda + closures, 24 tests): factorial 10 → 3628800, counter via closed-over `set!`, curried lambda.
- Phase 3.5 (let/let*/cond/when/unless/and/or, 21 tests).
- Phase 4 (standard env + set! bugfix, 82 tests): variadic arithmetic, type predicates, list/string/char/vector ops, higher-order combinators. **Found and fixed an SX cond multi-expression branch bug** affecting set!. Bugfix unblocked 4 silently-failing tests in Phase 3.
- Phase 5a (call/cc, 8 tests): single-shot escape continuations.
- Phase 5b (raise/guard/with-exception-handler/error, 12 tests): catch-once-then-rehandle-outside pattern avoids handler-self-raise loops.
- Phase 5c (dynamic-wind, 5 tests): basic before-thunk-after with raise propagation. call/cc-escape tracking deferred.
- Phase 6a (define-syntax + syntax-rules, 12 tests): pattern matching with literals + pattern variables + list structure; template substitution.
- Phase 6b (syntax-rules ellipsis, 8 tests): tail-rest single-variable form. `(my-and 1 2 3)` etc. work.
- Phase 7 (eval / interaction-environment, 13 tests): **second consumer for evaluator.sx**. `interaction-environment` closes over the env being built, so user-side defines via `(eval ... ie)` persist across calls.
- Phase 8 (define-library + import, 7 tests): minimal module system. Private definitions stay in library env; only exports are visible after import.
- Phase 9 (define-record-type, 9 tests): tagged-dict records with optional mutators.
- Phase 10 (quasiquote runtime, 10 tests): **second consumer for quoting.sx**. Identical algorithm to Kernel's `knl-quasi-walk` — universal across reflective Lisps.
- Phase 11 (test.sh + scoreboard): single-process aggregating runner, scoreboard markdown.

### Deferred phases

- **Phase 6c — full hygiene**. Dybvig-style scope-sets / lifted-symbol algorithm. Would be the second consumer for the deferred `lib/guest/reflective/hygiene.sx`. Current macros work for common patterns but don't prevent introduced-binding capture. Research-grade work; warrants its own loop iteration.
- **Nested quasiquote depth tracking**.
- **R7RS module rich features** (`cond-expand`, `include`, import sets like `only`/`except`/`prefix`/`rename`).
- **Dotted-pair `(a b . rest)` parser syntax** + lambda rest-args.
- **Full call/cc + dynamic-wind interaction**: dynamic-extent re-entry/re-exit tracking.

### Chisel ledger update

This Scheme port satisfies the two-consumer rule for **three** reflective kits documented in the kernel-on-sx loop:

| Kit | Status |
|-----|--------|
| `env.sx` | Extracted — Scheme is the third consumer (after Kernel + Tcl/Smalltalk), uses the canonical shape directly with no cfg |
| `evaluator.sx` | Second consumer ready — Scheme `eval`/`interaction-environment`/`null-environment`/`scheme-report-environment` mirror the proposed `refl-eval`/`refl-current-env`/`refl-make-environment` triple |
| `quoting.sx` | Second consumer ready — Scheme `scm-quasi-walk` is structurally identical to Kernel's `knl-quasi-walk`; the only difference is the unquote keyword names (cfg parameterisation) |
| `hygiene.sx` | Still awaiting (needs Phase 6c) |
| `combiner.sx` | N/A — Scheme has no fexprs |
| `short-circuit.sx` | N/A — Scheme `and`/`or` are syntactic, not operative |

The kit-extraction commits themselves are follow-on work; this Scheme port is the consumer-side foundation.