rose-ash/plans/kernel-on-sx.md

Kernel-on-SX: first-class everything

The natural successor to SX's recently-completed env-as-value work (sx-improvements Phase 4). Kernel — John Shutt's reformulation of Lisp from his 2010 PhD — pushes first-class all the way: environments, evaluators, special forms (operatives), lambda variants are all runtime values, manipulable by programs. SX already has env-as-value; Kernel is what env-as-value looks like all the way.

The chisel: reflection. Every language in the current set treats some part of itself as fixed and ineffable — Common Lisp's special forms, Erlang's process model, OCaml's modules. Kernel reifies more of itself than any other language does. Implementing it stresses the substrate's self-knowledge: which parts of evaluation does SX expose to user programs, and which stay opaque?

What this exposes about the substrate:

• Whether eval-expr can be called as a primitive on user-supplied environments without breaking invariants.
• Whether CEK frames can be reified as values (they currently aren't).
• Whether special-form dispatch can be table-driven and user-extensible at runtime.
• Whether the macro hygiene story extends to Shutt's "hygienic operatives" (operatives that don't capture).

End-state goal: Kernel's R-1RK core — $vau/$lambda/wrap/unwrap, first-class environments, the applicative–operative distinction, the standard environment, encapsulations.

Ground rules

• Scope: lib/kernel/** and plans/kernel-on-sx.md only. Substrate work belongs to sx-improvements.md — if a feature is missing, file it there, don't fix from this plan.
• Consumes from lib/guest/: core/lex.sx, core/pratt.sx (s-expression-shaped, minimal demand), core/ast.sx, core/match.sx.
• May propose a new sub-layer lib/guest/reflective/ — environment reification helpers, applicative-vs-operative dispatch, evaluator continuation protocols. A second consumer would be needed; candidates are a hypothetical "MetaScheme" or a Common-Lisp port that exposes its evaluator.
• Branch: loops/kernel. Standard worktree pattern.

Architecture sketch

Kernel source text (S-expression syntax)
    │
    ▼
lib/kernel/parser.sx       — bog-standard s-expr reader
    │
    ▼
lib/kernel/eval.sx         — kernel-eval: walks the AST, threads first-class env
    │                        dispatches to operatives via env-bound bindings, not
    │                        a hardcoded switch
    ▼
lib/kernel/runtime.sx      — applicative/operative tagged values, wrap/unwrap,
    │                        standard environment construction, encapsulations
    ▼
SX CEK evaluator

Semantic mappings

Kernel construct	SX mapping
($lambda (x) body)	applicative: (make-applicative (fn (x) body)) — args evaluated
($vau (x) e body)	operative: (make-operative (fn (x e) body)) — args UN-evaluated, dynamic env passed as e
(wrap op)	applicative wrapping an operative: evaluate args, then call op
(unwrap app)	get the underlying operative of an applicative
($define! x v)	operative: bind x to v in dynamic env
(eval expr env)	call kernel-eval on expr in env — first-class
(make-environment)	fresh empty env
(get-current-environment)	reify the calling env (via SX env-as-value)
($if c t e)	operative: evaluate c, then t or e in dynamic env

The whole interesting thing: there are no special forms hardcoded in the evaluator. $if, $define!, $lambda are all operatives bound in the standard environment. User code can rebind them. The evaluator is just lookup-and-call.

Roadmap

Phase 1 — Parser

• S-expression reader with the standard atoms (number, string, symbol, boolean, nil) and lists.
• Reader macros optional; defer to Phase 6.
• Tests in lib/kernel/tests/parse.sx.

Phase 2 — Core evaluator with first-class environments

• kernel-eval expr env — primary entry, walks AST, threads env as a value.
• Symbol lookup → environment value (using SX env-as-value primitives).
• List → look up head, dispatch on tag (applicative vs operative).
• No hardcoded special forms — even if/define/lambda are env-bound.
• Tests in lib/kernel/tests/eval.sx.

Phase 3 — $vau / $lambda / wrap / unwrap

• Operative tagged value: {:type :operative :params :env-param :body :static-env}.
• Applicative tagged value wraps an operative + the "evaluate args first" contract.
• $vau builds operatives; $lambda is wrap ∘ $vau.
• wrap / unwrap round-trip cleanly.
• Tests: define a custom operative, define a custom applicative on top of it.

Phase 4 — Standard environment

• Standard env construction: bind $if, $define!, $lambda, $vau, wrap, unwrap, eval, make-environment, get-current-environment, plus arithmetic and list primitives.
• Tests: classic Kernel programs (factorial, list operations, environment manipulation).

Phase 5 — Encapsulations

• make-encapsulation-type returns three operatives: encapsulator, predicate, decapsulator. Standard Kernel idiom for opaque types.
• Tests: implement promises, streams, or simple modules via encapsulations.

Phase 6 — Hygienic operatives (Shutt's later work)

• Operatives that don't capture caller bindings — uses scope sets / frame stamps to track provenance.
• Bridge to SX's hygienic macro story; possibly extends lib/guest/reflective/ with hygiene primitives.
• Tests: write an operative that introduces a binding and verify it doesn't shadow caller's same-named bindings.

Phase 7 — Propose lib/guest/reflective/

• Once Phase 3 lands and stabilises, identify which env-reification + dispatch primitives are reusable. Candidate API: make-operative, make-applicative, with-current-env, eval-in-env.
• Find a second consumer (Common-Lisp's macro-expansion evaluator? a metacircular Scheme variant? a future plan).
• Only extract once two consumers exist (per stratification rule).

lib/guest feedback loop

Consumes: core/lex, core/pratt, core/ast, core/match.

Stresses substrate: env-as-value (Phase 4 of sx-improvements) under heavy use; eval as a primitive on user environments; potentially CEK frame reification.

May propose: lib/guest/reflective/ sub-layer — environment manipulation, evaluator-as-value, applicative/operative dispatch protocols.

What it teaches: whether SX's recent env-as-value direction generalises to "evaluator-as-value." If Kernel implements cleanly in <2000 lines, env-as-value is real. If it requires substrate fixes at every turn, env-as-value was incomplete and the substrate is telling us what's missing.

References

• Shutt, "Fexprs as the basis of Lisp function application" (PhD thesis, 2010).
• Kernel Report (R-1RK): https://web.cs.wpi.edu/~jshutt/kernel.html
• Klisp implementation (Andres Navarro) — pragmatic reference.

Progress log

• 2026-05-10 — Phase 1 parser landed. lib/kernel/parser.sx reads R-1RK lexical syntax: numbers (int/float/exp), strings (with escapes), symbols (permissive — anything non-delimiting), booleans #t/#f, the empty list (), nested lists, and ; line comments. Reader macros (' , ,@) deferred per plan. AST: numbers/booleans/lists pass through; strings are wrapped as {:knl-string …} to distinguish from symbols which are bare SX strings. 54 tests in lib/kernel/tests/parse.sx pass via sx_server.exe epoch protocol. chisel: consumes-lex (uses lex-digit? and lex-whitespace? from lib/guest/lex.sx — pratt deliberately not consumed because Kernel is plain s-expressions, no precedence climbing).

Blockers

(none yet — main risk is substrate gap discovery during Phase 2)