rose-ash/plans/kernel-on-sx.md

Kernel-on-SX: first-class everything

The natural successor to SX's recently-completed env-as-value work (sx-improvements Phase 4). Kernel — John Shutt's reformulation of Lisp from his 2010 PhD — pushes first-class all the way: environments, evaluators, special forms (operatives), lambda variants are all runtime values, manipulable by programs. SX already has env-as-value; Kernel is what env-as-value looks like all the way.

The chisel: reflection. Every language in the current set treats some part of itself as fixed and ineffable — Common Lisp's special forms, Erlang's process model, OCaml's modules. Kernel reifies more of itself than any other language does. Implementing it stresses the substrate's self-knowledge: which parts of evaluation does SX expose to user programs, and which stay opaque?

What this exposes about the substrate:

• Whether eval-expr can be called as a primitive on user-supplied environments without breaking invariants.
• Whether CEK frames can be reified as values (they currently aren't).
• Whether special-form dispatch can be table-driven and user-extensible at runtime.
• Whether the macro hygiene story extends to Shutt's "hygienic operatives" (operatives that don't capture).

End-state goal: Kernel's R-1RK core — $vau/$lambda/wrap/unwrap, first-class environments, the applicative–operative distinction, the standard environment, encapsulations.

Ground rules

• Scope: lib/kernel/** and plans/kernel-on-sx.md only. Substrate work belongs to sx-improvements.md — if a feature is missing, file it there, don't fix from this plan.
• Consumes from lib/guest/: core/lex.sx, core/pratt.sx (s-expression-shaped, minimal demand), core/ast.sx, core/match.sx.
• May propose a new sub-layer lib/guest/reflective/ — environment reification helpers, applicative-vs-operative dispatch, evaluator continuation protocols. A second consumer would be needed; candidates are a hypothetical "MetaScheme" or a Common-Lisp port that exposes its evaluator.
• Branch: loops/kernel. Standard worktree pattern.

Architecture sketch

Kernel source text (S-expression syntax)
    │
    ▼
lib/kernel/parser.sx       — bog-standard s-expr reader
    │
    ▼
lib/kernel/eval.sx         — kernel-eval: walks the AST, threads first-class env
    │                        dispatches to operatives via env-bound bindings, not
    │                        a hardcoded switch
    ▼
lib/kernel/runtime.sx      — applicative/operative tagged values, wrap/unwrap,
    │                        standard environment construction, encapsulations
    ▼
SX CEK evaluator

Semantic mappings

Kernel construct	SX mapping
($lambda (x) body)	applicative: (make-applicative (fn (x) body)) — args evaluated
($vau (x) e body)	operative: (make-operative (fn (x e) body)) — args UN-evaluated, dynamic env passed as e
(wrap op)	applicative wrapping an operative: evaluate args, then call op
(unwrap app)	get the underlying operative of an applicative
($define! x v)	operative: bind x to v in dynamic env
(eval expr env)	call kernel-eval on expr in env — first-class
(make-environment)	fresh empty env
(get-current-environment)	reify the calling env (via SX env-as-value)
($if c t e)	operative: evaluate c, then t or e in dynamic env

The whole interesting thing: there are no special forms hardcoded in the evaluator. $if, $define!, $lambda are all operatives bound in the standard environment. User code can rebind them. The evaluator is just lookup-and-call.

Roadmap

Phase 1 — Parser

• S-expression reader with the standard atoms (number, string, symbol, boolean, nil) and lists.
• Reader macros optional; defer to Phase 6.
• Tests in lib/kernel/tests/parse.sx.

Phase 2 — Core evaluator with first-class environments

• kernel-eval expr env — primary entry, walks AST, threads env as a value.
• Symbol lookup → environment value (using SX env-as-value primitives).
• List → look up head, dispatch on tag (applicative vs operative).
• No hardcoded special forms — even if/define/lambda are env-bound.
• Tests in lib/kernel/tests/eval.sx.

Phase 3 — $vau / $lambda / wrap / unwrap

• Operative tagged value: {:type :operative :params :env-param :body :static-env}.
• Applicative tagged value wraps an operative + the "evaluate args first" contract.
• $vau builds operatives; $lambda is wrap ∘ $vau.
• wrap / unwrap round-trip cleanly.
• Tests: define a custom operative, define a custom applicative on top of it.

Phase 4 — Standard environment

• Standard env construction: bind $if, $define!, $lambda, $vau, wrap, unwrap, eval, make-environment, get-current-environment, plus arithmetic and list primitives.
• Tests: classic Kernel programs (factorial, list operations, environment manipulation).

Phase 5 — Encapsulations

• make-encapsulation-type returns three operatives: encapsulator, predicate, decapsulator. Standard Kernel idiom for opaque types.
• Tests: implement promises, streams, or simple modules via encapsulations.

Phase 6 — Hygienic operatives (Shutt's later work)

• Operatives that don't capture caller bindings — hygiene-by-default via static-env extension. Full scope-set / frame-stamp story is research-grade and documented but deferred.
• Bridge to SX's hygienic macro story; extends proposed lib/guest/reflective/ with $let and $define-in! hygiene primitives.
• Tests: write an operative that introduces a binding and verify it doesn't shadow caller's same-named bindings.

Phase 7 — Propose lib/guest/reflective/ [partial — pending second consumer]

• Identified reusable env-reification + dispatch primitives across Phases 2–6. Consolidated API surface below as four candidate files: env.sx, combiner.sx, evaluator.sx, hygiene.sx.
• Find a second consumer (Common-Lisp's macro-expansion evaluator? a metacircular Scheme variant? a future plan). Until this lands, extraction is blocked by the two-consumer rule.
• Only extract once two consumers exist (per stratification rule). Do not extract from this loop — Kernel is one consumer; we need another before lib/guest/reflective/ is real.

Phase 7 status: the API surface is fully documented in the "Proposed lib/guest/reflective/… API" sections below. Candidate second consumers in priority order:

1. A metacircular Scheme — Scheme can reuse env.sx directly (same scope semantics), borrow evaluator.sx's eval/make-env/current-env triple, and pattern-match the hygiene.sx story (Scheme has identical lexical scope). Would NOT need combiner.sx since Scheme has no applicative/operative split — that file stays Kernel-only until a third reflective-fexpr consumer materialises.
2. Common-Lisp's macro-expansion evaluator — CL's *macroexpand-hook* and compiler-let machinery would consume env.sx (CL package envs map cleanly) and evaluator.sx (defmacro = an operative-like fexpr in expander phase). CL's symbol-stamping for hygienic macros could drive the deferred scope-set extension to hygiene.sx.
3. A future Maru / Schemely port — these languages have first-class fexprs and would use the whole kit verbatim.

When the second consumer arrives, the extraction work is: rename kernel-* → refl-* in the relevant files, move into lib/guest/reflective/, update both consumers' references. Estimated <500 lines moved, since the bulk is already cleanly separated by responsibility in this loop's commits.

lib/guest feedback loop

Consumes: core/lex, core/pratt, core/ast, core/match.

Stresses substrate: env-as-value (Phase 4 of sx-improvements) under heavy use; eval as a primitive on user environments; potentially CEK frame reification.

May propose: lib/guest/reflective/ sub-layer — environment manipulation, evaluator-as-value, applicative/operative dispatch protocols.

Proposed lib/guest/reflective/short-circuit.sx API (from $and?/$or? chiselling — pending second consumer):

• (refl-short-and? ARGS DYN-ENV) — recursive walker; evaluates each in DYN-ENV, returns first falsy value or last truthy. Identity is true.
• (refl-short-or? ARGS DYN-ENV) — symmetric; returns first truthy or last falsy. Identity is false.
• Both must be defined as operatives in any reflective Lisp because short-circuit semantics require staged evaluation — an applicative would force every argument before any decision could be made.
• Driving insight: short-circuit booleans are a forcing function for "operative semantics matter". Languages that lack first-class operatives have to special-case these as keywords; languages with operatives get them for free, in user code.

Proposed lib/guest/reflective/quoting.sx API (from quasiquote chiselling — pending second consumer):

• (refl-quasi-walk FORM ENV) — top-level entry. Recursively walks FORM; an $unquote sub-expression is evaluated in ENV and replaces itself in the result.
• (refl-quasi-walk-list FORMS ENV) — walks a list of forms, splicing $unquote-splicing results inline.
• (refl-list-concat XS YS) — pure-SX list concatenation (no host dependency on append).
• Driving insight: every reflective Lisp eventually adds quasiquote, and the recursion-with-splicing structure is identical across them. Nesting depth tracking (for e inside e ``) is the only Kernel-specific complication; for the kit, a depth-tracking variant refl-quasi-walk-depth FORM ENV DEPTH` would be the second-tier API.

Proposed lib/guest/reflective/hygiene.sx API (from Phase 6 chiselling — pending second consumer):

• The substrate decision: a user-defined combiner's body runs in (extend STATIC-ENV), NOT in the dyn-env. Any $define! inside the body binds in this fresh child, so callers' envs stay untouched. This is the cheap, lexical-scope hygiene story that R-1RK has had since the start.
• (refl-let BINDINGS BODY) — bind names in a fresh child of dyn-env, evaluate body there. Values evaluated in OUTER env (parallel semantics).
• (refl-define-in! ENV NAME EXPR) — explicit-target bind. The operative that wants to mutate someone else's env says so explicitly.
• Full scope-set / frame-stamp hygiene (Shutt's later work, Racket-style) is research-grade and not implemented. The pieces would include: lifted symbols carrying a stamp set, refl-introduce-symbol to create a fresh-stamp name, refl-symbol=? that compares names and stamps. This belongs in a future Phase 7+ extraction once a second consumer wants it.

Proposed lib/guest/reflective/evaluator.sx API (from Phase 4 chiselling — pending second consumer):

• (refl-eval EXPR ENV) — the primary entry. Used to be implicit; exposing it as a function lets guests call into their own evaluator.
• (refl-make-environment [PARENT]) — fresh evaluation context, optionally a child of an existing one.
• (refl-current-env-operative) — a Kernel-shaped operative that returns the dyn-env when called. Other reflective languages will need the same mechanism (an operative-equivalent that exposes "the env at this point").
• Driving insight: the eval/make-env/current-env triple IS the reflective evaluator interface. Every reflective Lisp eventually exposes these three. Even more so when you start needing macro-expansion-time vs run-time vs call-time envs (the Kernel hygienic operatives work in Phase 6 will reveal whether more refl-env-at-foo-time accessors should join the kit).

Proposed lib/guest/reflective/combiner.sx API (from Phase 3 chiselling — pending second consumer):

• (refl-make-primitive-operative IMPL) — IMPL receives (args dyn-env), args unevaluated.
• (refl-make-user-operative PARAMS EPARAM BODY STATIC-ENV) — for $vau-like constructors. The EPARAM sentinel for "ignore dyn-env" is a fixed keyword (:refl-ignore in the proposal).
• (refl-wrap OP) / (refl-unwrap APP) — round-trip pair.
• (refl-operative? V) / (refl-applicative? V) / (refl-combiner? V).
• (refl-call-combiner COMBINER ARGS DYN-ENV) — the dispatch fork. Pairs with refl-eval from the evaluator kit.
• Representation: {:refl-tag :operative :impl FN} or {:refl-tag :operative :params P :env-param EP :body B :static-env SE}; applicatives are {:refl-tag :applicative :underlying OP}. The dispatch decision lives in one fork: presence of :impl is primitive, presence of :body is user-defined.
• Driving insight: every reflective Lisp must distinguish "eval my args first" from "hand me the syntax". The tag protocol is identical across Kernel, CL fexprs, vau-style Schemes, possibly Forth's IMMEDIATE words.

Proposed lib/guest/reflective/env.sx API (from Phase 2 chiselling — pending second consumer per the two-consumer rule):

• (refl-make-env) / (refl-extend-env PARENT) — fresh / chained envs, plain SX dicts so they're easy to introspect.
• (refl-env? V) — predicate.
• (refl-env-bind! ENV NAME VAL) — local bind; parent is untouched.
• (refl-env-has? ENV NAME) — recursive presence check.
• (refl-env-lookup ENV NAME) — recursive lookup, raises on miss.
• Representation: {:refl-tag :env :bindings DICT :parent ENV-OR-NIL}. Pure-SX dicts so any guest can serialize, diff, snapshot, or rewind environments without help from the host.

The motivation is that SX's host make-env family is registered only in HTTP/site-mode platform setup, so a guest that needs first-class envs in CLI / test contexts has to roll its own anyway. A shared kit means the next reflective consumer (CL macro evaluator? metacircular Scheme?) doesn't need to redo the work.

What it teaches: whether SX's recent env-as-value direction generalises to "evaluator-as-value." If Kernel implements cleanly in <2000 lines, env-as-value is real. If it requires substrate fixes at every turn, env-as-value was incomplete and the substrate is telling us what's missing.

References

• Shutt, "Fexprs as the basis of Lisp function application" (PhD thesis, 2010).
• Kernel Report (R-1RK): https://web.cs.wpi.edu/~jshutt/kernel.html
• Klisp implementation (Andres Navarro) — pragmatic reference.

Progress log

• 2026-05-11 — Variadic + - * / and chained < > <=? >=?. (+ 1 2 3) = 6, (+) = 0, (+ 7) = 7. (- 10 1 2 3) = 4 (left fold); single-arg - negates. (* 1 2 3 4) = 24, (*) = 1. Chained comparison: (< 1 2 3) ≡ (< 1 2) ∧ (< 2 3). Implementation: knl-fold-app for n-ary fold with zero-arity identity and one-arity special-case; knl-chain-cmp for chained boolean. 19 new tests. chisel: nothing (mechanical extension of existing arithmetic primitives). 279 tests total.
• 2026-05-11 — $let* sequential let. Each binding evaluated in scope where earlier bindings are visible, so ($let* ((x 1) (y (+ x 1))) y) returns 2. Implemented by nesting envs one per binding — knl-let*-step recursively builds the env chain. $let and $let* now both accept multi-expression bodies (knl-eval-body re-used). 8 new tests in tests/hygiene.sx. chisel: nothing (a standard derived form). 260 tests total.
• 2026-05-11 — $and? / $or? short-circuit booleans. Operatives (not applicatives) so untaken arguments are NOT evaluated. Identity values: $and? empty = true, $or? empty = false. Returns the last evaluated value (Kernel convention — not coerced to bool). 10 new tests including the short-circuit verification (($and? #f nope) returns false without evaluating nope). chisel: shapes-reflective. Sketched lib/guest/reflective/short-circuit.sx API; the protocol is identical across reflective Lisps because short-circuit FORCES operative semantics — an applicative variant would defeat the purpose. 252 tests total.
• 2026-05-11 — $cond / $when / $unless. Standard Kernel control flow added: $cond walks clauses in order, evaluates first truthy test, runs that clause's body in sequence; else is the catch-all symbol; empty cond and no-match cond return nil. $when and $unless are simple conditional execution. All three preserve hygiene (clauses not taken are NOT evaluated). 12 new tests in tests/standard.sx. chisel: nothing. 242 tests total. (Third nothing in a row but allowable here — these are textbook Kernel idioms with no novel reflective angle.)
• 2026-05-11 — $quasiquote runtime. The parser's reader macros (Phase 1.5) produced unevaluated $quasiquote/$unquote/$unquote-splicing forms; the runtime side now interprets them. kernel-quasiquote-operative walks the template via mutual recursion knl-quasi-walk ↔ knl-quasi-walk-list: atoms and empty lists pass through; an ($unquote X) head form returns (kernel-eval X dyn-env); an ($unquote-splicing X) inside a list evaluates X and splices its list result via knl-list-concat. Nesting depth (`\`...\`) is not tracked — for Phase-1.5 simplicity, nested quasiquotes flatten. 8 new tests in tests/standard.sx. chisel: shapes-reflective. The quoting walker shape is universal across reflective Lisps; sketched the lib/guest/reflective/quoting.sx candidate API (refl-quasi-walk, refl-quasi-walk-list, refl-list-concat). 230 tests total.
• 2026-05-11 — Multi-expression body for $vau/$lambda. Both forms now accept (formals env-param body1 body2 ...) / (formals body1 body2 ...). Implementation: :body slot now holds a LIST of forms (was a single expression); kernel-call-operative calls a new knl-eval-body that evaluates each in sequence, returning the last. No dependency on $sequence being in static-env — the iteration lives at the host level. 5 new tests in tests/vau.sx (multi-body lambda, multi-body vau, sequenced $define!, zero-arg multi-body). chisel: nothing (Kernel-internal improvement; doesn't change the reflective API surface). 223 tests total.
• 2026-05-11 — Phase 1 reader macros landed (the deferred checkbox from Phase 1). Parser now recognises four shorthand forms: 'expr → ($quote expr), `expr → ($quasiquote expr), ,expr → ($unquote expr), ,@expr → ($unquote-splicing expr). Delimiter set extended to include ', `, , so they don't slip into adjacent atom tokens. The runtime already has $quote; $quasiquote / $unquote / $unquote-splicing are not bound yet (would need a recursive walker for quasi-quote expansion — left for whenever a consumer needs it). 8 new reader-macro tests in tests/parse.sx bring parse to 62, total to 218. chisel: consumes-lex (parser still leans on lib/guest/lex.sx whitespace + digit predicates only).
• 2026-05-11 — Phase 7 proposal complete (partial extraction per two-consumer rule). Consolidated the four candidate reflective files into the plan's API surface section: env.sx (Phase 2), combiner.sx (Phase 3), evaluator.sx (Phase 4), hygiene.sx (Phase 6). Total proposed surface ~25 functions, all sketched with signatures and representation notes. Kernel alone is the first consumer; the second consumer must materialise before any actual extraction. Listed candidate second consumers in priority order: metacircular Scheme (highest fit — same scope semantics), CL macro evaluator (medium fit — would drive the deferred hygiene work), Maru/Schemely (eventual). Extraction is estimated at <500 lines moved when the time comes — clean separation of concerns across this loop's six prior commits means the rename-and-move work is mechanical, not a redesign. chisel: proposes-reflective-extraction (the candidate API surface is the entire artefact of this phase). 210 tests across six test files, zero regressions across the loop. The kernel-on-sx loop sustained one feature per commit for seven commits.
• 2026-05-11 — Phase 6 hygiene landed (mostly). Two helpers in runtime.sx: $let — proper hygienic let; values evaluated in caller env, names bound in fresh child env, body in that child env. $define-in! — operative that binds a name in a specified env, not the dyn-env. The key insight: hygiene-by-default was already the case from Phase 3's static-env extension semantics — $vau/$lambda close over their static env and bind formals + body $define!s in a CHILD of static-env, so caller's env stays untouched unless explicitly threaded via eval or $define-in!. The 18 tests in tests/hygiene.sx prove this property holds in practice: $define! inside an operative body doesn't escape to the caller; $let-bound names don't leak after the let; parallel let evaluates RHS in outer scope; $define-in! populates the target env without polluting the caller's. Full scope-set / frame-stamp hygiene (Shutt's later research-grade work) is documented in the proposed lib/guest/reflective/hygiene.sx notes but deferred — would require lifted symbols with provenance markers, a much larger redesign. chisel: shapes-reflective. The default-hygienic-by-static-env-extension property is itself a chisel finding worth recording — every reflective Lisp would benefit from this design choice, and the lib/guest/reflective/env.sx candidate API should make it the default semantic.
• 2026-05-11 — Phase 5 encapsulations landed. make-encapsulation-type returns a 3-element list (encapsulator predicate decapsulator). Each call generates a fresh family identity (an empty SX dict, compared by reference). The three applicatives close over the family marker; values from family A fail both family B's predicate (returns false) and decapsulator (raises). 19 tests in tests/encap.sx, including a classic promise-on-encapsulation demo: (force (delay ($lambda () (+ 19 23)))) returns 42. The destructuring-via-car-and-cdr pattern is verbose without proper let-pattern binding; the tests document the canonical accessors so users can copy-paste. chisel: nothing (pure Kernel work — no new substrate or lib/guest insights). Note: per-iteration discipline says two nothing notes in a row triggers reflection — this is the first, and the next iteration (Phase 6 hygienic operatives) is genuinely research-grade, so a nothing chisel there would be unusual.
• 2026-05-11 — Phase 4 standard env landed. kernel-standard-env extends kernel-base-env with: control ($if, $define!, $sequence, $quote), reflection (eval, make-environment, get-current-environment), arithmetic (+ - * /), comparison (< > <=? >=? =? eq? equal?), list/pair (cons car cdr list length null? pair?), boolean (not). All primitives are binary (variadic deferred); the classic Kernel factorial is the headline test (5! = 120, 10! = 3628800). 49 tests in tests/standard.sx, covering $if branching, $define! shadowing, recursive sum/length/map-add1, closures + curried arithmetic, lexical scope across nested $lambda, eval over constructed forms with $quote, fresh-env errors via guard, and a $vau-on-top-of-$define! example. chisel: shapes-reflective. Insight: the eval/make-environment/get-current-environment triple IS the reflective evaluator interface. Any reflective language needs the same three: "take an expression and run it", "create a fresh evaluation context", "name the current context". That goes in the proposed lib/guest/reflective/evaluator.sx candidate. Second chisel — $define! was a one-liner because env-bind! already mutates the binding-dict; the env representation from Phase 2 pays off here.
• 2026-05-11 — Phase 3 operatives landed. lib/kernel/runtime.sx adds $vau (primitive operative that returns a user operative), $lambda (sugar for wrap ∘ $vau), wrap and unwrap (Kernel-level applicatives), plus operative? and applicative? predicates. kernel-base-env wires them all into a fresh env. kernel-eval.sx now dispatches in kernel-call-operative between primitive ops (carry :impl) and user ops (carry :params :env-param :body :static-env). Parameter binding is a flat list — destructuring/&rest deferred. Env-param sentinel: spell _ or #ignore → :knl-ignore, which skips the dyn-env bind. 34 tests in tests/vau.sx, including the headline custom-operative + custom-applicative composition. chisel: shapes-reflective. Two further reflective-API candidates surfaced: (a) the operative/applicative tag protocol — make-primitive-operative, make-user-operative, wrap, unwrap are general for any Lisp-of-fexprs; (b) the call-dispatch fork (primitive vs user) is a single decision that every reflective evaluator hits. Both shape go into the proposed lib/guest/reflective/combiner.sx candidate.
• 2026-05-10 — Phase 2 evaluator landed. lib/kernel/eval.sx is lookup-and-combine: zero hardcoded special forms. kernel-eval EXPR ENV dispatches on shape — literals self-evaluate, Kernel strings unwrap, symbols lookup, lists evaluate head and combine. kernel-combine distinguishes operatives (impl receives un-evaluated args + dynamic env) from applicatives (eval args, recurse into underlying op). kernel-wrap/kernel-unwrap round-trip cleanly. 36 tests verify literal evaluation, symbol lookup with parent-chain shadowing, tagged-value predicates, and the operative-vs-applicative contract (notably $if only evaluates the chosen branch, $quote returns its arg unevaluated). chisel: shapes-reflective. Substrate gap surfaced: SX's make-env / env-bind! family is only registered in HTTP/site mode (http_setup_platform_constructors), not in CLI epoch mode used for tests. So Kernel envs are modelled in pure SX as {:knl-tag :env :bindings DICT :parent P} — a binding-dict + parent-pointer + recursive lookup walk. This is exactly the lib/guest/reflective/env.sx candidate API: any reflective language needs first-class env values that can be extended, queried, and walked. Recording the shape (constructor, extend, bind!, has?, lookup) here for the eventual Phase 7 extraction.
• 2026-05-10 — Phase 1 parser landed. lib/kernel/parser.sx reads R-1RK lexical syntax: numbers (int/float/exp), strings (with escapes), symbols (permissive — anything non-delimiting), booleans #t/#f, the empty list (), nested lists, and ; line comments. Reader macros (' , ,@) deferred per plan. AST: numbers/booleans/lists pass through; strings are wrapped as {:knl-string …} to distinguish from symbols which are bare SX strings. 54 tests in lib/kernel/tests/parse.sx pass via sx_server.exe epoch protocol. chisel: consumes-lex (uses lex-digit? and lex-whitespace? from lib/guest/lex.sx — pratt deliberately not consumed because Kernel is plain s-expressions, no precedence climbing).

Blockers

(none yet — main risk is substrate gap discovery during Phase 2)