rose-ash/plans/kernel-on-sx.md

# Kernel-on-SX: first-class everything

The natural successor to SX's recently-completed env-as-value work (sx-improvements Phase 4). Kernel — John Shutt's reformulation of Lisp from his 2010 PhD — pushes *first-class* all the way: environments, evaluators, special forms (operatives), lambda variants are all runtime values, manipulable by programs. SX already has env-as-value; Kernel is what env-as-value looks like *all the way*.

**The chisel:** *reflection*. Every language in the current set treats some part of itself as fixed and ineffable — Common Lisp's special forms, Erlang's process model, OCaml's modules. Kernel reifies more of itself than any other language does. Implementing it stresses the substrate's *self-knowledge*: which parts of evaluation does SX expose to user programs, and which stay opaque?

**What this exposes about the substrate:**
- Whether `eval-expr` can be called as a primitive on user-supplied environments without breaking invariants.
- Whether CEK frames can be reified as values (they currently aren't).
- Whether special-form dispatch can be table-driven and user-extensible at runtime.
- Whether the macro hygiene story extends to Shutt's "hygienic operatives" (operatives that don't capture).

**End-state goal:** Kernel's R-1RK core — `$vau`/`$lambda`/`wrap`/`unwrap`, first-class environments, the applicative–operative distinction, the standard environment, encapsulations.

## Ground rules
- Scope: `lib/kernel/**` and `plans/kernel-on-sx.md` only. Substrate work belongs to `sx-improvements.md` — if a feature is missing, file it there, don't fix from this plan.
- Consumes from `lib/guest/`: `core/lex.sx`, `core/pratt.sx` (s-expression-shaped, minimal demand), `core/ast.sx`, `core/match.sx`.
- **May propose** a new sub-layer `lib/guest/reflective/` — environment reification helpers, applicative-vs-operative dispatch, evaluator continuation protocols. A second consumer would be needed; candidates are a hypothetical "MetaScheme" or a Common-Lisp port that exposes its evaluator.
- Branch: `loops/kernel`. Standard worktree pattern.

## Architecture sketch

```
Kernel source text (S-expression syntax)
    │
    ▼
lib/kernel/parser.sx       — bog-standard s-expr reader
    │
    ▼
lib/kernel/eval.sx         — kernel-eval: walks the AST, threads first-class env
    │                        dispatches to operatives via env-bound bindings, not
    │                        a hardcoded switch
    ▼
lib/kernel/runtime.sx      — applicative/operative tagged values, wrap/unwrap,
    │                        standard environment construction, encapsulations
    ▼
SX CEK evaluator
```

## Semantic mappings

| Kernel construct | SX mapping |
|------------------|-----------|
| `($lambda (x) body)` | applicative: `(make-applicative (fn (x) body))` — args evaluated |
| `($vau (x) e body)` | operative: `(make-operative (fn (x e) body))` — args UN-evaluated, dynamic env passed as `e` |
| `(wrap op)` | applicative wrapping an operative: evaluate args, then call op |
| `(unwrap app)` | get the underlying operative of an applicative |
| `($define! x v)` | operative: bind `x` to `v` in dynamic env |
| `(eval expr env)` | call `kernel-eval` on `expr` in `env` — first-class |
| `(make-environment)` | fresh empty env |
| `(get-current-environment)` | reify the calling env (via SX env-as-value) |
| `($if c t e)` | operative: evaluate `c`, then `t` or `e` in dynamic env |

The whole interesting thing: there are no special forms hardcoded in the evaluator. `$if`, `$define!`, `$lambda` are all *operatives* bound in the standard environment. User code can rebind them. The evaluator is just `lookup-and-call`.

## Roadmap

### Phase 1 — Parser
- [x] S-expression reader with the standard atoms (number, string, symbol, boolean, nil) and lists.
- [x] Reader macros optional; defer to Phase 6.
- [x] Tests in `lib/kernel/tests/parse.sx`.

### Phase 2 — Core evaluator with first-class environments
- [x] `kernel-eval expr env` — primary entry, walks AST, threads env as a value.
- [x] Symbol lookup → environment value (using SX env-as-value primitives).
- [x] List → look up head, dispatch on tag (applicative vs operative).
- [x] No hardcoded special forms — even `if`/`define`/`lambda` are env-bound.
- [x] Tests in `lib/kernel/tests/eval.sx`.

### Phase 3 — `$vau` / `$lambda` / `wrap` / `unwrap`
- [x] Operative tagged value: `{:type :operative :params :env-param :body :static-env}`.
- [x] Applicative tagged value wraps an operative + the "evaluate args first" contract.
- [x] `$vau` builds operatives; `$lambda` is `wrap` ∘ `$vau`.
- [x] `wrap` / `unwrap` round-trip cleanly.
- [x] Tests: define a custom operative, define a custom applicative on top of it.

### Phase 4 — Standard environment
- [x] Standard env construction: bind `$if`, `$define!`, `$lambda`, `$vau`, `wrap`, `unwrap`, `eval`, `make-environment`, `get-current-environment`, plus arithmetic and list primitives.
- [x] Tests: classic Kernel programs (factorial, list operations, environment manipulation).

### Phase 5 — Encapsulations
- [x] `make-encapsulation-type` returns three operatives: encapsulator, predicate, decapsulator. Standard Kernel idiom for opaque types.
- [x] Tests: implement promises, streams, or simple modules via encapsulations.

### Phase 6 — Hygienic operatives (Shutt's later work)
- [x] Operatives that don't capture caller bindings — hygiene-by-default via static-env extension. Full scope-set / frame-stamp story is research-grade and documented but deferred.
- [x] Bridge to SX's hygienic macro story; extends proposed `lib/guest/reflective/` with `$let` and `$define-in!` hygiene primitives.
- [x] Tests: write an operative that introduces a binding and verify it doesn't shadow caller's same-named bindings.

### Phase 7 — Propose `lib/guest/reflective/`  *[partial — pending second consumer]*
- [x] Identified reusable env-reification + dispatch primitives across Phases 2–6. Consolidated API surface below as four candidate files: `env.sx`, `combiner.sx`, `evaluator.sx`, `hygiene.sx`.
- [ ] Find a second consumer (Common-Lisp's macro-expansion evaluator? a metacircular Scheme variant? a future plan). Until this lands, extraction is blocked by the two-consumer rule.
- [ ] Only extract once two consumers exist (per stratification rule). **Do not extract from this loop** — Kernel is one consumer; we need another before `lib/guest/reflective/` is real.

**Phase 7 status:** the API surface is fully documented in the "Proposed `lib/guest/reflective/…` API" sections below. Candidate second consumers in priority order:

1. **A metacircular Scheme** — Scheme can reuse `env.sx` directly (same scope semantics), borrow `evaluator.sx`'s eval/make-env/current-env triple, and pattern-match the `hygiene.sx` story (Scheme has identical lexical scope). Would NOT need `combiner.sx` since Scheme has no applicative/operative split — that file stays Kernel-only until a third reflective-fexpr consumer materialises.
2. **Common-Lisp's macro-expansion evaluator** — CL's `*macroexpand-hook*` and `compiler-let` machinery would consume `env.sx` (CL package envs map cleanly) and `evaluator.sx` (defmacro = an operative-like fexpr in expander phase). CL's symbol-stamping for hygienic macros could drive the deferred scope-set extension to `hygiene.sx`.
3. **A future Maru / Schemely port** — these languages have first-class fexprs and would use the whole kit verbatim.

When the second consumer arrives, the extraction work is: rename `kernel-*` → `refl-*` in the relevant files, move into `lib/guest/reflective/`, update both consumers' references. Estimated <500 lines moved, since the bulk is already cleanly separated by responsibility in this loop's commits.

## lib/guest feedback loop

**Consumes:** `core/lex`, `core/pratt`, `core/ast`, `core/match`.

**Stresses substrate:** env-as-value (Phase 4 of sx-improvements) under heavy use; `eval` as a primitive on user environments; potentially CEK frame reification.

**May propose:** `lib/guest/reflective/` sub-layer — environment manipulation, evaluator-as-value, applicative/operative dispatch protocols.

**Proposed `lib/guest/reflective/quoting.sx` API** (from quasiquote chiselling — pending second consumer):
- `(refl-quasi-walk FORM ENV)` — top-level entry. Recursively walks FORM; an `$unquote` sub-expression is evaluated in ENV and replaces itself in the result.
- `(refl-quasi-walk-list FORMS ENV)` — walks a list of forms, splicing `$unquote-splicing` results inline.
- `(refl-list-concat XS YS)` — pure-SX list concatenation (no host dependency on `append`).
- Driving insight: every reflective Lisp eventually adds quasiquote, and the recursion-with-splicing structure is identical across them. Nesting depth tracking (for `` ``e `` inside `` `e ``) is the only Kernel-specific complication; for the kit, a depth-tracking variant `refl-quasi-walk-depth FORM ENV DEPTH` would be the second-tier API.

**Proposed `lib/guest/reflective/hygiene.sx` API** (from Phase 6 chiselling — pending second consumer):
- The substrate decision: a user-defined combiner's body runs in `(extend STATIC-ENV)`, NOT in the dyn-env. Any `$define!` inside the body binds in this fresh child, so callers' envs stay untouched. This is the cheap, lexical-scope hygiene story that R-1RK has had since the start.
- `(refl-let BINDINGS BODY)` — bind names in a fresh child of dyn-env, evaluate body there. Values evaluated in OUTER env (parallel semantics).
- `(refl-define-in! ENV NAME EXPR)` — explicit-target bind. The operative that wants to mutate someone else's env says so explicitly.
- Full scope-set / frame-stamp hygiene (Shutt's later work, Racket-style) is research-grade and not implemented. The pieces would include: lifted symbols carrying a stamp set, `refl-introduce-symbol` to create a fresh-stamp name, `refl-symbol=?` that compares names *and* stamps. This belongs in a future Phase 7+ extraction once a second consumer wants it.

**Proposed `lib/guest/reflective/evaluator.sx` API** (from Phase 4 chiselling — pending second consumer):
- `(refl-eval EXPR ENV)` — the primary entry. Used to be implicit; exposing it as a function lets guests call into their own evaluator.
- `(refl-make-environment [PARENT])` — fresh evaluation context, optionally a child of an existing one.
- `(refl-current-env-operative)` — a Kernel-shaped operative that returns the dyn-env when called. Other reflective languages will need the same mechanism (an operative-equivalent that exposes "the env at this point").
- Driving insight: the eval/make-env/current-env triple IS the reflective evaluator interface. Every reflective Lisp eventually exposes these three. Even more so when you start needing macro-expansion-time vs run-time vs call-time envs (the Kernel hygienic operatives work in Phase 6 will reveal whether more `refl-env-at-foo-time` accessors should join the kit).

**Proposed `lib/guest/reflective/combiner.sx` API** (from Phase 3 chiselling — pending second consumer):
- `(refl-make-primitive-operative IMPL)` — IMPL receives `(args dyn-env)`, args unevaluated.
- `(refl-make-user-operative PARAMS EPARAM BODY STATIC-ENV)` — for $vau-like constructors. The EPARAM sentinel for "ignore dyn-env" is a fixed keyword (`:refl-ignore` in the proposal).
- `(refl-wrap OP)` / `(refl-unwrap APP)` — round-trip pair.
- `(refl-operative? V)` / `(refl-applicative? V)` / `(refl-combiner? V)`.
- `(refl-call-combiner COMBINER ARGS DYN-ENV)` — the dispatch fork. Pairs with `refl-eval` from the evaluator kit.
- Representation: `{:refl-tag :operative :impl FN}` or `{:refl-tag :operative :params P :env-param EP :body B :static-env SE}`; applicatives are `{:refl-tag :applicative :underlying OP}`. The dispatch decision lives in one fork: presence of `:impl` is primitive, presence of `:body` is user-defined.
- Driving insight: every reflective Lisp must distinguish "eval my args first" from "hand me the syntax". The tag protocol is identical across Kernel, CL fexprs, vau-style Schemes, possibly Forth's IMMEDIATE words.

**Proposed `lib/guest/reflective/env.sx` API** (from Phase 2 chiselling — pending second consumer per the two-consumer rule):
- `(refl-make-env)` / `(refl-extend-env PARENT)` — fresh / chained envs, plain SX dicts so they're easy to introspect.
- `(refl-env? V)` — predicate.
- `(refl-env-bind! ENV NAME VAL)` — local bind; parent is untouched.
- `(refl-env-has? ENV NAME)` — recursive presence check.
- `(refl-env-lookup ENV NAME)` — recursive lookup, raises on miss.
- Representation: `{:refl-tag :env :bindings DICT :parent ENV-OR-NIL}`. Pure-SX dicts so any guest can serialize, diff, snapshot, or rewind environments without help from the host.

The motivation is that SX's host `make-env` family is registered only in HTTP/site-mode platform setup, so a guest that needs first-class envs in CLI / test contexts has to roll its own anyway. A shared kit means the next reflective consumer (CL macro evaluator? metacircular Scheme?) doesn't need to redo the work.

**What it teaches:** whether SX's recent env-as-value direction generalises to "evaluator-as-value." If Kernel implements cleanly in <2000 lines, env-as-value is real. If it requires substrate fixes at every turn, env-as-value was incomplete and the substrate is telling us what's missing.

## References
- Shutt, "Fexprs as the basis of Lisp function application" (PhD thesis, 2010).
- Kernel Report (R-1RK): https://web.cs.wpi.edu/~jshutt/kernel.html
- Klisp implementation (Andres Navarro) — pragmatic reference.

## Progress log

- 2026-05-11 — `$quasiquote` runtime. The parser's reader macros (Phase 1.5) produced unevaluated `$quasiquote`/`$unquote`/`$unquote-splicing` forms; the runtime side now interprets them. `kernel-quasiquote-operative` walks the template via mutual recursion `knl-quasi-walk` ↔ `knl-quasi-walk-list`: atoms and empty lists pass through; an `($unquote X)` head form returns `(kernel-eval X dyn-env)`; an `($unquote-splicing X)` *inside* a list evaluates X and splices its list result via `knl-list-concat`. Nesting depth (`` `\`...\` ``) is not tracked — for Phase-1.5 simplicity, nested quasiquotes flatten. 8 new tests in `tests/standard.sx`. chisel: shapes-reflective. The quoting walker shape is universal across reflective Lisps; sketched the `lib/guest/reflective/quoting.sx` candidate API (`refl-quasi-walk`, `refl-quasi-walk-list`, `refl-list-concat`). 230 tests total.
- 2026-05-11 — Multi-expression body for `$vau`/`$lambda`. Both forms now accept `(formals env-param body1 body2 ...)` / `(formals body1 body2 ...)`. Implementation: `:body` slot now holds a LIST of forms (was a single expression); `kernel-call-operative` calls a new `knl-eval-body` that evaluates each in sequence, returning the last. No dependency on `$sequence` being in static-env — the iteration lives at the host level. 5 new tests in `tests/vau.sx` (multi-body lambda, multi-body vau, sequenced `$define!`, zero-arg multi-body). chisel: nothing (Kernel-internal improvement; doesn't change the reflective API surface). 223 tests total.
- 2026-05-11 — Phase 1 reader macros landed (the deferred checkbox from Phase 1). Parser now recognises four shorthand forms: `'expr` → `($quote expr)`, `` `expr `` → `($quasiquote expr)`, `,expr` → `($unquote expr)`, `,@expr` → `($unquote-splicing expr)`. Delimiter set extended to include `'`, `` ` ``, `,` so they don't slip into adjacent atom tokens. The runtime already has `$quote`; `$quasiquote` / `$unquote` / `$unquote-splicing` are not bound yet (would need a recursive walker for quasi-quote expansion — left for whenever a consumer needs it). 8 new reader-macro tests in `tests/parse.sx` bring parse to 62, total to 218. chisel: consumes-lex (parser still leans on `lib/guest/lex.sx` whitespace + digit predicates only).
- 2026-05-11 — Phase 7 proposal complete (partial extraction per two-consumer rule). Consolidated the four candidate reflective files into the plan's API surface section: `env.sx` (Phase 2), `combiner.sx` (Phase 3), `evaluator.sx` (Phase 4), `hygiene.sx` (Phase 6). Total proposed surface ~25 functions, all sketched with signatures and representation notes. Kernel alone is the first consumer; the *second* consumer must materialise before any actual extraction. Listed candidate second consumers in priority order: metacircular Scheme (highest fit — same scope semantics), CL macro evaluator (medium fit — would drive the deferred hygiene work), Maru/Schemely (eventual). Extraction is estimated at <500 lines moved when the time comes — clean separation of concerns across this loop's six prior commits means the rename-and-move work is mechanical, not a redesign. chisel: proposes-reflective-extraction (the candidate API surface is the entire artefact of this phase). 210 tests across six test files, zero regressions across the loop. The kernel-on-sx loop sustained one feature per commit for seven commits.
- 2026-05-11 — Phase 6 hygiene landed (mostly). Two helpers in `runtime.sx`: `$let` — proper hygienic let; values evaluated in caller env, names bound in fresh child env, body in that child env. `$define-in!` — operative that binds a name in a *specified* env, not the dyn-env. The key insight: hygiene-by-default was already the case from Phase 3's static-env extension semantics — $vau/$lambda close over their static env and bind formals + body $define!s in a CHILD of static-env, so caller's env stays untouched unless explicitly threaded via `eval` or `$define-in!`. The 18 tests in `tests/hygiene.sx` prove this property holds in practice: `$define!` inside an operative body doesn't escape to the caller; `$let`-bound names don't leak after the let; parallel let evaluates RHS in outer scope; `$define-in!` populates the target env without polluting the caller's. Full scope-set / frame-stamp hygiene (Shutt's later research-grade work) is documented in the proposed `lib/guest/reflective/hygiene.sx` notes but deferred — would require lifted symbols with provenance markers, a much larger redesign. chisel: shapes-reflective. The default-hygienic-by-static-env-extension property is itself a chisel finding worth recording — every reflective Lisp would benefit from this design choice, and the `lib/guest/reflective/env.sx` candidate API should make it the default semantic.
- 2026-05-11 — Phase 5 encapsulations landed. `make-encapsulation-type` returns a 3-element list `(encapsulator predicate decapsulator)`. Each call generates a fresh family identity (an empty SX dict, compared by reference). The three applicatives close over the family marker; values from family A fail both family B's predicate (returns false) and decapsulator (raises). 19 tests in `tests/encap.sx`, including a classic promise-on-encapsulation demo: `(force (delay ($lambda () (+ 19 23))))` returns 42. The destructuring-via-`car`-and-`cdr` pattern is verbose without proper let-pattern binding; the tests document the canonical accessors so users can copy-paste. chisel: nothing (pure Kernel work — no new substrate or lib/guest insights). Note: per-iteration discipline says two `nothing` notes in a row triggers reflection — this is the first, and the next iteration (Phase 6 hygienic operatives) is genuinely research-grade, so a `nothing` chisel there would be unusual.
- 2026-05-11 — Phase 4 standard env landed. `kernel-standard-env` extends `kernel-base-env` with: control (`$if`, `$define!`, `$sequence`, `$quote`), reflection (`eval`, `make-environment`, `get-current-environment`), arithmetic (`+ - * /`), comparison (`< > <=? >=? =? eq? equal?`), list/pair (`cons car cdr list length null? pair?`), boolean (`not`). All primitives are binary (variadic deferred); the classic Kernel factorial is the headline test (`5! = 120`, `10! = 3628800`). 49 tests in `tests/standard.sx`, covering $if branching, $define! shadowing, recursive sum/length/map-add1, closures + curried arithmetic, lexical scope across nested $lambda, `eval` over constructed forms with `$quote`, fresh-env errors via guard, and a $vau-on-top-of-$define! example. chisel: shapes-reflective. Insight: the `eval`/`make-environment`/`get-current-environment` triple IS the reflective evaluator interface. Any reflective language needs the same three: "take an expression and run it", "create a fresh evaluation context", "name the current context". That goes in the proposed `lib/guest/reflective/evaluator.sx` candidate. Second chisel — `$define!` was a one-liner because env-bind! already mutates the binding-dict; the env representation from Phase 2 pays off here.
- 2026-05-11 — Phase 3 operatives landed. `lib/kernel/runtime.sx` adds `$vau` (primitive operative that returns a user operative), `$lambda` (sugar for `wrap ∘ $vau`), `wrap` and `unwrap` (Kernel-level applicatives), plus `operative?` and `applicative?` predicates. `kernel-base-env` wires them all into a fresh env. `kernel-eval.sx` now dispatches in `kernel-call-operative` between primitive ops (carry `:impl`) and user ops (carry `:params :env-param :body :static-env`). Parameter binding is a flat list — destructuring/`&rest` deferred. Env-param sentinel: spell `_` or `#ignore` → `:knl-ignore`, which skips the dyn-env bind. 34 tests in `tests/vau.sx`, including the headline custom-operative + custom-applicative composition. chisel: shapes-reflective. Two further reflective-API candidates surfaced: (a) the operative/applicative tag protocol — `make-primitive-operative`, `make-user-operative`, `wrap`, `unwrap` are general for any Lisp-of-fexprs; (b) the call-dispatch fork (primitive vs user) is a *single decision* that every reflective evaluator hits. Both shape go into the proposed `lib/guest/reflective/combiner.sx` candidate.
- 2026-05-10 — Phase 2 evaluator landed. `lib/kernel/eval.sx` is `lookup-and-combine`: zero hardcoded special forms. `kernel-eval EXPR ENV` dispatches on shape — literals self-evaluate, Kernel strings unwrap, symbols lookup, lists evaluate head and combine. `kernel-combine` distinguishes operatives (impl receives un-evaluated args + dynamic env) from applicatives (eval args, recurse into underlying op). `kernel-wrap`/`kernel-unwrap` round-trip cleanly. 36 tests verify literal evaluation, symbol lookup with parent-chain shadowing, tagged-value predicates, and the operative-vs-applicative contract (notably `$if` only evaluates the chosen branch, `$quote` returns its arg unevaluated). chisel: shapes-reflective. Substrate gap surfaced: SX's `make-env` / `env-bind!` family is only registered in HTTP/site mode (`http_setup_platform_constructors`), not in CLI epoch mode used for tests. So Kernel envs are modelled in pure SX as `{:knl-tag :env :bindings DICT :parent P}` — a binding-dict + parent-pointer + recursive lookup walk. This is exactly the `lib/guest/reflective/env.sx` candidate API: any reflective language needs first-class env values that can be extended, queried, and walked. Recording the shape (constructor, extend, bind!, has?, lookup) here for the eventual Phase 7 extraction.
- 2026-05-10 — Phase 1 parser landed. `lib/kernel/parser.sx` reads R-1RK lexical syntax: numbers (int/float/exp), strings (with escapes), symbols (permissive — anything non-delimiting), booleans `#t`/`#f`, the empty list `()`, nested lists, and `;` line comments. Reader macros (`'` `,` `,@`) deferred per plan. AST: numbers/booleans/lists pass through; strings are wrapped as `{:knl-string …}` to distinguish from symbols which are bare SX strings. 54 tests in `lib/kernel/tests/parse.sx` pass via `sx_server.exe` epoch protocol. chisel: consumes-lex (uses `lex-digit?` and `lex-whitespace?` from `lib/guest/lex.sx` — pratt deliberately not consumed because Kernel is plain s-expressions, no precedence climbing).

## Blockers
_(none yet — main risk is substrate gap discovery during Phase 2)_