coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f0d8db9b68d50d239c1641b4652b655b0b0390c8
rose-ash/sx/sx/plans/mother-language.sx
giles 6f96452f70 Fix empty code blocks: rename ~docs/code param, fix batched IO dispatch 
Two bugs caused code blocks to render empty across the site:

1. ~docs/code component had parameter named `code` which collided with
   the HTML <code> tag name. Renamed to `src` and updated all 57
   callers. Added font-mono class for explicit monospace.

2. Batched IO dispatch in ocaml_bridge.py only skipped one leading
   number (batch ID) but the format has two (epoch + ID):
   (io-request EPOCH ID "name" args...). Changed to skip all leading
   numbers so the string name is correctly found. This fixes highlight
   and other batchable helpers returning empty results.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 18:08:40 +00:00

651 lines
46 KiB
Plaintext
Raw Blame History

;; ---------------------------------------------------------------------------
;; Mother Language — SX as its own compiler, OCaml as the substrate
;; ---------------------------------------------------------------------------
(defcomp ~plans/mother-language/plan-mother-language-content ()
(~docs/page :title "Mother Language"
(p :class "text-stone-500 text-sm italic mb-8"
"The ideal language for evaluating the SX core spec is SX itself. "
"The path: OCaml as the initial substrate (closest existing language to what CEK is), "
"Koka as an alternative (compile-time linearity), ultimately a self-hosting SX compiler "
"that emits machine code directly from the spec. One language. Every target.")
;; -----------------------------------------------------------------------
;; The argument
;; -----------------------------------------------------------------------
(~docs/section :title "The Argument" :id "argument"
(h4 :class "font-semibold mt-4 mb-2" "What the evaluator actually does")
(p "The CEK machine is a " (code "state \u2192 state") " loop over sum types. "
"Each step pattern-matches on the Control register, consults the Environment, "
"and transforms the Kontinuation. Every SX expression, every component render, "
"every signal update goes through this loop. It's the hot path.")
(p "The ideal host language is one where this loop compiles to a tight jump table "
"with minimal allocation. That means: algebraic types, pattern matching, "
"persistent data structures, and a native effect system.")
(h4 :class "font-semibold mt-6 mb-2" "Why multiple hosts is the wrong goal")
(p "The current architecture bootstraps the spec to Python, JavaScript, and Rust. "
"Each host has impedance mismatches:")
(ul :class "list-disc list-inside space-y-1 mt-2"
(li (strong "Python") " \u2014 slow. Tree-walk overhead is 100\u20131000x vs native. "
"The async adapter is complex because Python's async model is cooperative, not effect-based.")
(li (strong "JavaScript") " \u2014 no sum types, prototype-based dispatch, GC pauses unpredictable. "
"The bootstrapper works around JS limitations rather than mapping naturally.")
(li (strong "Rust") " \u2014 ownership fights shared immutable trees. Every " (code "Rc<Vec<Value>>")
" is overhead. Closures capturing environments need " (code "Arc") " gymnastics."))
(p "Each host makes the evaluator work, but none make it " (em "natural") ". "
"The translation is structure-" (em "creating") ", not structure-" (em "preserving") ".")
(h4 :class "font-semibold mt-6 mb-2" "The Mother Language is SX")
(p "The spec defines the semantics. The CEK machine is the most explicit form of those semantics. "
"The ideal \"language\" is one that maps 1:1 onto CEK transitions and compiles them to "
"optimal machine code. That language is SX itself \u2014 compiled, not interpreted.")
(p "The hosts (Python, JS, Haskell, Rust) become " (em "platform layers") " \u2014 "
"they provide IO, DOM, database, GPU access. The evaluator itself is always the same "
"compiled SX core, embedded as a native library or WASM module."))
;; -----------------------------------------------------------------------
;; Why OCaml
;; -----------------------------------------------------------------------
(~docs/section :title "Why OCaml" :id "ocaml"
(p "OCaml is the closest existing language to what the CEK machine is. "
"The translation from " (code "cek.sx") " to OCaml is nearly mechanical.")
(h4 :class "font-semibold mt-4 mb-2" "Natural mapping")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "SX concept")
(th :class "px-3 py-2 font-medium text-stone-600" "OCaml primitive")
(th :class "px-3 py-2 font-medium text-stone-600" "Notes")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Value (Nil | Num | Str | List | ...)")
(td :class "px-3 py-2 text-stone-700" "Algebraic variant type")
(td :class "px-3 py-2 text-stone-600" "Direct mapping, no boxing"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Frame (IfFrame | ArgFrame | MapFrame | ...)")
(td :class "px-3 py-2 text-stone-700" "Algebraic variant type")
(td :class "px-3 py-2 text-stone-600" "20+ variants, pattern match dispatch"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Environment (persistent map)")
(td :class "px-3 py-2 text-stone-700" (code "Map.S"))
(td :class "px-3 py-2 text-stone-600" "Built-in balanced tree, structural sharing"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Continuation (list of frames)")
(td :class "px-3 py-2 text-stone-700" "Immutable list")
(td :class "px-3 py-2 text-stone-600" "cons/match, O(1) push/pop"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "cek-step (pattern match on C)")
(td :class "px-3 py-2 text-stone-700" (code "match") " expression")
(td :class "px-3 py-2 text-stone-600" "Compiles to jump table"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "shift/reset")
(td :class "px-3 py-2 text-stone-700" (code "perform") " / " (code "continue"))
(td :class "px-3 py-2 text-stone-600" "Native in OCaml 5"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Concurrent CEK (fibers)")
(td :class "px-3 py-2 text-stone-700" "Domains + effect handlers")
(td :class "px-3 py-2 text-stone-600" "One fiber per CEK machine"))
(tr
(td :class "px-3 py-2 text-stone-700" "Linear continuations")
(td :class "px-3 py-2 text-stone-700" "One-shot continuations (default)")
(td :class "px-3 py-2 text-stone-600" "Runtime-enforced, not compile-time")))))
(h4 :class "font-semibold mt-4 mb-2" "Compilation targets")
(ul :class "list-disc list-inside space-y-1 mt-2"
(li (strong "Native") " \u2014 OCaml's native compiler produces fast binaries, small footprint. "
"Embed in Python via C ABI (ctypes/cffi). Embed in Node via N-API.")
(li (strong "WASM") " \u2014 " (code "wasm_of_ocaml") " is mature (used by Facebook's Flow/Reason). "
"Produces compact " (code ".wasm") " modules. Loaded via " (code "sx-platform.js") " like the Rust WASM plan.")
(li (strong "JavaScript") " \u2014 " (code "js_of_ocaml") " for legacy browser targets. "
"Falls back to JS when WASM isn't available."))
(h4 :class "font-semibold mt-4 mb-2" "What OCaml replaces")
(p "The Haskell and Rust evaluator implementations become unnecessary. "
"OCaml covers both server (native) and client (WASM) from one codebase. "
"The sx-haskell and sx-rust work proved the spec is host-independent \u2014 "
"OCaml is the convergence point.")
(p "Python and JavaScript evaluators remain as " (em "platform layers") " \u2014 "
"they provide IO primitives, not evaluation logic. The Python web framework calls "
"the OCaml evaluator via FFI for rendering. The browser loads the WASM evaluator "
"and connects it to " (code "sx-platform.js") " for DOM access."))
;; -----------------------------------------------------------------------
;; Koka as alternative
;; -----------------------------------------------------------------------
(~docs/section :title "Koka as Alternative" :id "koka"
(p "Koka (Daan Leijen, MSR) addresses OCaml's one weakness: "
(strong "compile-time linearity") ".")
(h4 :class "font-semibold mt-4 mb-2" "Where Koka wins")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Perceus reference counting") " \u2014 the compiler tracks which values are used linearly. "
"Linear values are mutated in-place (zero allocation). "
"Non-linear values use reference counting (no GC at all).")
(li (strong "Effect types") " \u2014 effects are tracked in the type system. "
"A function's type says exactly which effects it can perform. "
"The type checker enforces that handlers exist for every effect.")
(li (strong "One-shot continuations by default") " \u2014 like OCaml 5, but " (em "statically enforced") ". "
"The type system prevents invoking a linear continuation twice. "
"No runtime check needed."))
(h4 :class "font-semibold mt-4 mb-2" "Where Koka is weaker")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Maturity") " \u2014 research language. Smaller ecosystem, fewer FFI bindings, "
"less battle-tested than OCaml.")
(li (strong "WASM backend") " \u2014 compiles to C \u2192 WASM (via Emscripten). "
"Works but less optimized than " (code "wasm_of_ocaml") " or Rust's " (code "wasm-pack") ".")
(li (strong "Embeddability") " \u2014 C FFI works but less established for Python/Node embedding."))
(p "Koka is the right choice " (em "if") " compile-time linearity proves essential for correctness. "
"The decision point is Step 5 (Linear Effects) of the foundations plan. "
"If SX's own type system can enforce linearity at spec-validation time, "
"OCaml's runtime enforcement is sufficient. If not, Koka becomes the better substrate."))
;; -----------------------------------------------------------------------
;; The path to self-hosting
;; -----------------------------------------------------------------------
(~docs/section :title "The Path to Self-Hosting" :id "self-hosting"
(p "The end state: SX compiles itself. No intermediate language, no general-purpose host.")
(h4 :class "font-semibold mt-4 mb-2" "Phase 1: OCaml bootstrapper")
(p "Write " (code "bootstrap_ml.py") " \u2014 reads " (code "cek.sx") " + " (code "frames.sx")
" + " (code "primitives.sx") " + " (code "eval.sx") ", emits OCaml source. "
"Same pattern as the existing Rust/Python/JS bootstrappers.")
(p "The OCaml output is a standalone module:")
(~docs/code :src (highlight "type value =\n | Nil | Bool of bool | Num of float | Str of string\n | Sym of string | Kw of string\n | List of value list | Dict of (value * value) list\n | Lambda of params * value list * env\n | Component of string * params * value list * env\n | Handle of int (* opaque FFI reference *)\n\ntype frame =\n | IfFrame of value list * value list * env\n | ArgFrame of value list * value list * env\n | MapFrame of value * value list * value list * env\n | ReactiveResetFrame of value\n | DerefFrame of value\n (* ... 20+ frame types from frames.sx *)\n\ntype kont = frame list\ntype state = value * env * kont\n\nlet step ((ctrl, env, kont) : state) : state =\n match ctrl with\n | Lit v -> continue_val v kont\n | Var name -> continue_val (Env.find name env) kont\n | App (f, args) -> (f, env, ArgFrame(args, [], env) :: kont)\n | ..." "ocaml"))
(h4 :class "font-semibold mt-6 mb-2" "Phase 2: Native + WASM builds")
(p "Compile the OCaml output to:")
(ul :class "list-disc list-inside space-y-1 mt-2"
(li (code "sx_core.so") " / " (code "sx_core.dylib") " \u2014 native shared library, C ABI")
(li (code "sx_core.wasm") " \u2014 via " (code "wasm_of_ocaml") " for browser")
(li (code "sx_core.js") " \u2014 via " (code "js_of_ocaml") " as JS fallback"))
(p "Python web framework calls " (code "sx_core.so") " via cffi. "
"Browser loads " (code "sx_core.wasm") " via " (code "sx-platform.js") ".")
(h4 :class "font-semibold mt-6 mb-2" "Phase 3: SX evaluates web framework")
(p "The compiled core evaluator loads web framework " (code ".sx") " at runtime "
"(signals, engine, orchestration, boot). Same as the "
(a :href "/sx/(etc.(plan.isolated-evaluator))" "Isolated Evaluator") " plan, "
"but the evaluator is compiled OCaml/WASM instead of bootstrapped JS.")
(h4 :class "font-semibold mt-6 mb-2" "Phase 4: SX linearity checking")
(p "Extend " (code "types.sx") " with quantity annotations:")
(~docs/code :src (highlight ";; Quantity annotations on types\n(define-type (Signal a) :quantity :affine) ;; use at most once per scope\n(define-type (Channel a) :quantity :linear) ;; must be consumed exactly once\n\n;; Effect declarations with linearity\n(define-io-primitive \"send-message\"\n :params (channel message)\n :quantity :linear\n :effects [io]\n :doc \"Must be handled exactly once.\")\n\n;; The type checker (specced in .sx, compiled to OCaml) validates\n;; linearity at component registration time. Runtime enforcement\n;; by OCaml's one-shot continuations is the safety net." "lisp"))
(p "The type checker runs at spec-validation time. The compiled evaluator "
"executes already-verified code. SX's type system provides the linearity "
"guarantees, not the host language.")
(h4 :class "font-semibold mt-6 mb-2" "Phase 5: Self-hosting compiler")
(p "Write the compiler itself in SX:")
(ul :class "list-disc list-inside space-y-1 mt-2"
(li "Spec the CEK-to-native code generation in " (code ".sx") " files")
(li "The Phase 2 OCaml evaluator compiles the compiler spec")
(li "The compiled compiler can then compile itself")
(li "OCaml becomes the bootstrap language only \u2014 "
"needed once to get the self-hosting loop started, then never touched again"))
(~docs/code :src (highlight ";; The bootstrap chain\n\nStep 0: Python evaluator (existing)\n \u2193 evaluates bootstrap_ml.py\nStep 1: OCaml evaluator (compiled from spec by Python)\n \u2193 evaluates compiler.sx\nStep 2: SX compiler (compiled from .sx by OCaml evaluator)\n \u2193 compiles itself\nStep 3: SX compiler (compiled by itself)\n \u2193 compiles everything\n \u2193 emits native, WASM, JS from .sx spec\n \u2193 OCaml is no longer in the chain" "text"))
(p "At Step 3, the only language is SX. The compiler reads " (code ".sx") " and emits machine code. "
"OCaml was the scaffolding. The scaffolding comes down."))
;; -----------------------------------------------------------------------
;; Concurrent CEK on OCaml 5
;; -----------------------------------------------------------------------
(~docs/section :title "Concurrent CEK on OCaml 5" :id "concurrent-cek"
(p "OCaml 5's concurrency model maps directly onto the "
(a :href "/sx/(etc.(plan.foundations))" "Foundations") " plan's concurrent CEK spec.")
(h4 :class "font-semibold mt-4 mb-2" "Mapping")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "SX primitive")
(th :class "px-3 py-2 font-medium text-stone-600" "OCaml 5")
(th :class "px-3 py-2 font-medium text-stone-600" "Characteristic")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "spawn"))
(td :class "px-3 py-2 text-stone-700" "Fiber via " (code "perform Spawn"))
(td :class "px-3 py-2 text-stone-600" "Lightweight, scheduled by effect handler"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "channel"))
(td :class "px-3 py-2 text-stone-700" (code "Eio.Stream"))
(td :class "px-3 py-2 text-stone-600" "Typed, bounded, backpressure"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "yield!"))
(td :class "px-3 py-2 text-stone-700" (code "perform Yield"))
(td :class "px-3 py-2 text-stone-600" "Cooperative, zero-cost"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "select"))
(td :class "px-3 py-2 text-stone-700" (code "Eio.Fiber.any"))
(td :class "px-3 py-2 text-stone-600" "First-to-complete"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "fork-join"))
(td :class "px-3 py-2 text-stone-700" (code "Eio.Fiber.all"))
(td :class "px-3 py-2 text-stone-600" "Structured concurrency"))
(tr
(td :class "px-3 py-2 text-stone-700" "DAG scheduler")
(td :class "px-3 py-2 text-stone-700" "Domains + fiber pool")
(td :class "px-3 py-2 text-stone-600" "True parallelism across cores")))))
(p "Each concurrent CEK machine is a fiber. The scheduler is an effect handler. "
"This isn't simulating concurrency \u2014 it's using native concurrency whose mechanism " (em "is") " effects.")
(h4 :class "font-semibold mt-4 mb-2" "The Art DAG connection")
(p "Art DAG's 3-phase execution (analyze \u2192 plan \u2192 execute) maps onto "
"concurrent CEK + OCaml domains:")
(ul :class "list-disc list-inside space-y-1 mt-2"
(li "Analyze: single CEK machine walks the DAG graph (one fiber)")
(li "Plan: resolve dependencies, topological sort (pure computation)")
(li "Execute: spawn one fiber per independent node, fan out to domains (true parallelism)")
(li "GPU kernels: fiber performs " (code "Gpu_dispatch") " effect, handler calls into GPU via C FFI")))
;; -----------------------------------------------------------------------
;; Linear effects
;; -----------------------------------------------------------------------
(~docs/section :title "Linear Effects" :id "linear-effects"
(p "The linearity axis from foundations. Two enforcement layers:")
(h4 :class "font-semibold mt-4 mb-2" "Layer 1: SX type system (primary)")
(p "Quantity annotations in " (code "types.sx") " checked at spec-validation time:")
(ul :class "list-disc list-inside space-y-1 mt-2"
(li (code ":linear") " (1) \u2014 must be used exactly once")
(li (code ":affine") " (\u22641) \u2014 may be used at most once (can drop)")
(li (code ":unrestricted") " (\u03c9) \u2014 may be used any number of times"))
(p "Linear effects guarantee: a " (code "send-message") " effect is handled exactly once. "
"A channel is consumed. A resource handle is closed. "
"The type checker proves this before the evaluator ever runs.")
(h4 :class "font-semibold mt-4 mb-2" "Layer 2: Host runtime (safety net)")
(p "OCaml 5's one-shot continuations enforce linearity at runtime. "
"A continuation can only be " (code "continue") "'d once \u2014 second invocation raises an exception. "
"This catches any bugs in the type checker itself.")
(p "If Koka replaces OCaml: compile-time enforcement replaces runtime enforcement. "
"The safety net becomes a proof. Same semantics, stronger guarantees.")
(h4 :class "font-semibold mt-4 mb-2" "Decision point")
(p "When Step 5 (Linear Effects) of the foundations plan is reached:")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li "If SX's type checker can enforce linearity reliably \u2192 "
(strong "stay on OCaml") ". Runtime one-shot is sufficient.")
(li "If linearity bugs keep slipping through \u2192 "
(strong "switch to Koka") ". Compile-time enforcement closes the gap.")
(li "If the self-hosting compiler (Phase 5) reaches maturity \u2192 "
(strong "it doesn't matter") ". SX compiles itself, the substrate is an implementation detail.")))
;; -----------------------------------------------------------------------
;; Compiled all the way down
;; -----------------------------------------------------------------------
(~docs/section :title "Compiled All the Way Down" :id "compiled"
(p "The self-hosting compiler doesn't just compile the evaluator. "
"It compiles " (em "everything") ". Component definitions, page layouts, "
"event handlers, signal computations \u2014 all compiled to native machine code. "
"SX is not an interpreted scripting language with a nice spec. "
"It's a compiled language whose compiler also runs in the browser.")
(h4 :class "font-semibold mt-4 mb-2" "JIT in the browser")
(p "The server sends SX (component definitions, page content). "
"The client receives it and " (strong "compiles to WASM and executes") ". "
"Not interprets. Not dispatches bytecodes. Compiles.")
(~docs/code :src (highlight "Server sends: (defcomp ~card (&key title) (div :class \"card\" (h2 title)))\n\nClient does:\n 1. Parse SX source (fast \u2014 it's s-expressions)\n 2. Hash AST \u2192 CID\n 3. Cache hit? Call the already-compiled WASM function\n 4. Cache miss? Compile to WASM, cache by CID, call it\n\nStep 4 is the JIT. The compiler (itself WASM) emits a WASM\nfunction, instantiates it via WebAssembly.instantiate, caches\nthe module by CID. Next time: direct function call, zero overhead." "text"))
(p "This is what V8 does with JavaScript. What LuaJIT does with Lua. "
"The difference: SX's semantics are simpler (no prototype chains, no " (code "this")
" binding, no implicit coercion), so the compiler is simpler. "
"And content-addressing means compiled artifacts are cacheable by CID \u2014 "
"compile once, store forever.")
(h4 :class "font-semibold mt-4 mb-2" "The compilation tiers")
(~docs/code :src (highlight "Tier 0: .sx source \u2192 tree-walking CEK (correct, slow \u2014 current)\nTier 1: .sx source \u2192 bytecodes \u2192 dispatch loop (correct, fast)\nTier 2: .sx source \u2192 WASM functions \u2192 execute (correct, fastest)\nTier 3: .sx source \u2192 native machine code (ahead-of-time, maximum)" "text"))
(p "Each tier is faster. Tier 1 (bytecodes) is the "
(a :href "/sx/(etc.(plan.wasm-bytecode-vm))" "WASM Bytecode VM")
" plan \u2014 compact wire format, no parse overhead, better cache locality. "
"Tier 2 is JIT \u2014 the compiler emitting WASM functions on the fly. "
"Tier 3 is AOT \u2014 the entire app precompiled. "
"All tiers use the same spec, same platform layer, same platform primitives.")
(h4 :class "font-semibold mt-4 mb-2" "Server-side precompilation")
(p "The server can compile too. Instead of sending SX source for the client to JIT, "
"send precompiled WASM:")
(~docs/code :src (highlight ";; Option A: send SX source, client JIT compiles\nContent-Type: text/sx\n\n(div :class \"card\" (h2 \"hello\"))\n\n;; Option B: send precompiled WASM, client instantiates directly\nContent-Type: application/wasm\nX-Sx-Cid: bafyrei...\n\n<binary WASM module>" "text"))
(p "Option B skips parsing and compilation entirely. The client instantiates "
"the WASM module and calls it. The server did all the work.")
(h4 :class "font-semibold mt-4 mb-2" "Content-addressed compilation cache")
(p "Every " (code ".sx") " expression has a CID. Every compiled artifact has a CID. "
"The mapping is deterministic \u2014 the compiler is a pure function:")
(~docs/code :src (highlight "source CID \u2192 compiled WASM CID\nbafyrei... \u2192 bafyrei...\n\nThis mapping is cacheable everywhere:\n\u2022 Browser cache \u2014 first visitor compiles, second visitor gets cached WASM\n\u2022 CDN \u2014 compiled artifacts served at the edge\n\u2022 IPFS \u2014 content-addressed by definition, globally deduplicated\n\u2022 Local disk \u2014 offline apps work from cached compiled components" "text"))
(h4 :class "font-semibold mt-4 mb-2" "Entire apps as machine code")
(p "The entire application can be ahead-of-time compiled to a WASM binary. "
"Component definitions, page layouts, event handlers, signal computations \u2014 "
"all compiled to native WASM functions. The \"app\" is a " (code ".wasm") " file. "
"The platform layer provides DOM and fetch. Everything in between is compiled machine code.")
(p "The only thing that stays JIT-compiled is truly dynamic content \u2014 "
"user-generated SX, REPL input, " (code "eval") "'d strings. "
"And even those get JIT'd on first use and cached by CID.")
(h4 :class "font-semibold mt-4 mb-2" "The architecture")
(~docs/code :src (highlight "sx-platform.js \u2190 DOM, fetch, timers (the real world)\n \u2191 calls\nsx-compiler.wasm \u2190 the SX compiler (itself compiled to WASM)\n \u2191 compiles\n.sx source \u2190 received from server / cache / inline\n \u2193 emits\nnative WASM functions \u2190 cached by CID, instantiated on demand\n \u2193 executes\nactual DOM mutations via platform primitives" "text"))
(p "The compiler is WASM. The code it produces is WASM. "
"It's compiled code all the way down. "
"The only interpreter in the system is the CPU."))
;; -----------------------------------------------------------------------
;; Security model
;; -----------------------------------------------------------------------
(~docs/section :title "Security Model" :id "security"
(p "Compiled SX running as WASM is " (em "more secure") " than plain JavaScript, "
"not less. JS has ambient access to the full browser API. "
"WASM + the platform layer means compiled SX code has "
(strong "zero ambient capabilities") " \u2014 every capability is explicitly granted.")
(h4 :class "font-semibold mt-4 mb-2" "Five defence layers")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "Layer")
(th :class "px-3 py-2 font-medium text-stone-600" "Enforced by")
(th :class "px-3 py-2 font-medium text-stone-600" "What it prevents")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "1. WASM sandbox")
(td :class "px-3 py-2 text-stone-700" "Browser")
(td :class "px-3 py-2 text-stone-600" "Memory isolation, no system calls, no DOM access except via explicit imports. Validated before execution."))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "2. Platform capabilities")
(td :class "px-3 py-2 text-stone-700" (code "sx-platform.js"))
(td :class "px-3 py-2 text-stone-600" "Compiled code can only call functions you register. No fetch? Can't fetch. No localStorage? Can't read storage. The platform is a capability system."))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "3. Content-addressed verification")
(td :class "px-3 py-2 text-stone-700" "CID determinism")
(td :class "px-3 py-2 text-stone-600" "Compiler is deterministic: same source \u2192 same CID. Client can re-compile and verify. Tampered WASM produces wrong CID \u2192 reject."))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "4. Per-component attenuation")
(td :class "px-3 py-2 text-stone-700" "Platform scoping")
(td :class "px-3 py-2 text-stone-600" "Different components get different capability subsets. User-generated content gets a locked-down platform \u2014 can render DOM but can't fetch or listen to events."))
(tr
(td :class "px-3 py-2 text-stone-700" "5. Source-first fallback")
(td :class "px-3 py-2 text-stone-700" "Client compiler")
(td :class "px-3 py-2 text-stone-600" "Don't trust precompiled WASM? Compile from source locally. The client has the compiler. Precompilation is an optimisation, not a trust requirement.")))))
(h4 :class "font-semibold mt-4 mb-2" "Content-addressed tamper detection")
(p "The server sends both SX source and precompiled WASM CID. The client can verify:")
(~docs/code :src (highlight ";; Server sends:\nContent-Type: application/wasm\nX-Sx-Source-Cid: bafyrei..source\nX-Sx-Compiled-Cid: bafyrei..compiled\n\n;; Client verifies (optional, configurable):\n1. Hash the WASM binary \u2192 matches X-Sx-Compiled-Cid?\n2. Compile source locally \u2192 produces same compiled CID?\n3. Check manifest of pinned CIDs \u2192 CID is expected?\n\n;; Any mismatch = tampered = reject" "text"))
(h4 :class "font-semibold mt-4 mb-2" "Capability attenuation per component")
(p "The platform scopes capabilities per evaluator instance. "
"App shell gets full access. Third-party or user-generated content gets the minimum:")
(~docs/code :src (highlight "// Full capabilities for the app shell\nplatform.registerAll(appShellCompiler);\n\n// Restricted for user-generated content\nplatform.registerSubset(userContentCompiler, {\n allow: [\"dom-create-element\", \"dom-set-attr\", \"dom-append\",\n \"dom-create-text-node\", \"dom-set-text\"],\n deny: [\"fetch\", \"localStorage\", \"dom-listen\",\n \"dom-set-inner-html\", \"eval\"]\n});\n\n// The restricted compiler's WASM module literally doesn't\n// have imports for the denied functions. Not just blocked\n// at runtime \u2014 absent from the binary." "javascript"))
(h4 :class "font-semibold mt-4 mb-2" "Component manifests")
(p "The app ships with a manifest of expected CIDs for its core components. "
"Like subresource integrity (SRI) but for compiled code:")
(~docs/code :src (highlight ";; Component manifest (shipped with the app, signed)\n{\n \"~card\": \"bafyrei..abc\"\n \"~header\": \"bafyrei..def\"\n \"~nav-item\": \"bafyrei..ghi\"\n}\n\n;; On navigation: server sends component update\n;; Client compiles \u2192 checks CID against manifest\n;; Match = trusted, execute\n;; Mismatch = tampered, reject and report" "text"))
(p "The security model is " (em "structural") ", not bolt-on. "
"WASM isolation, platform capabilities, content-addressed verification, "
"and per-component attenuation all arise naturally from the architecture. "
"The platform layer that enables Rust/OCaml interop is the same layer "
"that enforces security boundaries."))
;; -----------------------------------------------------------------------
;; Isomorphic rendering + SEO
;; -----------------------------------------------------------------------
(~docs/section :title "Isomorphic Rendering" :id "isomorphic"
(p "WASM is invisible to search engines. But SX is already isomorphic \u2014 "
"the same spec, the same components, rendered to HTML on the server "
"and to DOM on the client. Compiled WASM doesn't change this. "
"It makes the client side faster without affecting what crawlers see.")
(h4 :class "font-semibold mt-4 mb-2" "The rendering pipeline")
(~docs/code :src (highlight "Crawler visits:\n GET /page\n \u2192 Server compiles SX (native OCaml)\n \u2192 render-to-html (adapter-html.sx)\n \u2192 Full static HTML with semantic markup\n \u2192 Google indexes it\n\nUser first visit:\n GET /page\n \u2192 Server renders HTML (same as crawler)\n \u2192 Browser displays immediately (no JS needed)\n \u2192 Client loads sx-compiler.wasm + sx-platform.js\n \u2192 Hydrates: attaches event handlers, activates islands\n \u2192 Page is interactive\n\nUser navigates (SPA):\n sx-get /next-page\n \u2192 Server sends SX wire format (aser)\n \u2192 Client compiles + renders via WASM\n \u2192 Morph engine patches the DOM" "text"))
(p "The server and client have the " (em "same compiler") " from the " (em "same spec") ". "
(code "adapter-html.sx") " produces HTML strings. "
(code "adapter-dom.sx") " produces DOM nodes. "
"Two rendering modes of one evaluator. The compiled WASM version "
"makes hydration and SPA navigation faster, but the initial HTML "
"is always server-rendered.")
(h4 :class "font-semibold mt-4 mb-2" "What crawlers see")
(ul :class "list-disc list-inside space-y-1 mt-2"
(li "Fully rendered HTML \u2014 no \"loading...\" skeleton, no JS-dependent content")
(li "Semantic markup \u2014 " (code "<h1>") ", " (code "<nav>") ", " (code "<article>") ", " (code "<a href>")
" \u2014 all from the SX component tree")
(li "Meta tags, canonical URLs, structured data \u2014 rendered server-side by " (code "shell.sx"))
(li "No WASM, no JS required \u2014 the HTML is the page, complete on first byte"))
(h4 :class "font-semibold mt-4 mb-2" "Content-addressed prerendering")
(p "The server can prerender every page to static HTML, hash it, "
"and cache it at the CDN edge:")
(~docs/code :src (highlight "Page source CID \u2192 Rendered HTML CID\nbafyrei..source \u2192 bafyrei..html\n\n\u2022 Crawler hits CDN \u2192 instant HTML, no server round-trip\n\u2022 Page content changes \u2192 new source CID \u2192 new HTML CID \u2192 CDN invalidated\n\u2022 Same CID = same HTML forever \u2192 infinite cache, zero revalidation" "text"))
(p "This is the same content-addressed caching as compiled WASM, "
"applied to the HTML output. Both the compiled client code and "
"the server-rendered HTML are cached by CID. "
"The entire delivery pipeline is content-addressed.")
(h4 :class "font-semibold mt-4 mb-2" "Progressive enhancement")
(p "The page works at every level:")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "Client capability")
(th :class "px-3 py-2 font-medium text-stone-600" "Experience")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "No JS (crawler, reader mode)")
(td :class "px-3 py-2 text-stone-600" "Full HTML. Links work. Forms submit. Content is complete."))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "JS, no WASM")
(td :class "px-3 py-2 text-stone-600" "Falls back to js_of_ocaml evaluator or interpreted JS. SPA navigation, islands, signals all work."))
(tr
(td :class "px-3 py-2 text-stone-700" "JS + WASM")
(td :class "px-3 py-2 text-stone-600" "Full compiled pipeline. JIT compilation, cached WASM functions, near-native rendering speed."))))))
;; -----------------------------------------------------------------------
;; WASM responses — htmx with compiled components
;; -----------------------------------------------------------------------
(~docs/section :title "WASM Responses" :id "wasm-responses"
(p "The server doesn't have to send HTML fragments for HTMX-style swaps. "
"It can send compiled WASM functions that produce DOM directly, "
"or \u2014 when the client already has the components cached \u2014 "
"just the data as gzipped bytecode.")
(h4 :class "font-semibold mt-4 mb-2" "Three response tiers")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "Client cache state")
(th :class "px-3 py-2 font-medium text-stone-600" "Server sends")
(th :class "px-3 py-2 font-medium text-stone-600" "Client does")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Nothing cached")
(td :class "px-3 py-2 text-stone-700" "HTML (fallback)")
(td :class "px-3 py-2 text-stone-600" "Parse HTML, morph DOM"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Compiler cached")
(td :class "px-3 py-2 text-stone-700" "SX source")
(td :class "px-3 py-2 text-stone-600" "Compile, execute, morph DOM"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Components cached")
(td :class "px-3 py-2 text-stone-700" "Bytecode (data only)")
(td :class "px-3 py-2 text-stone-600" "Call cached WASM function with args"))
(tr
(td :class "px-3 py-2 text-stone-700" "Components cached + gzip")
(td :class "px-3 py-2 text-stone-700" "Gzipped bytecode")
(td :class "px-3 py-2 text-stone-600" "Decompress, call cached function")))))
(h4 :class "font-semibold mt-4 mb-2" "Why bytecode is tiny")
(p "The bytecode doesn't repeat what the client already knows. "
"Component CID references are 4-byte pointers to already-cached compiled functions. "
"Tag names, class strings, attribute keys \u2014 all baked into the compiled component. "
"The only payload is the data that differs between instances.")
(~docs/code :src (highlight ";; HTML response: ~450 bytes\n<div class=\"card\"><h2>Hello</h2><p class=\"text-stone-600\">World</p></div>\n\n;; SX source response: ~60 bytes\n(~card :title \"Hello\" :body \"World\")\n\n;; Bytecode response: ~18 bytes\n[CALL_CID bafyrei..card] [STR \"Hello\"] [STR \"World\"]\n\n;; Gzipped bytecode: ~12 bytes" "text"))
(p "For a list of 50 cards:")
(~docs/code :src (highlight "HTML: 50\u00d7 <div class=\"card\"><h2>...</h2>...</div> ~22 KB\nSX source: 50\u00d7 (~card :title \"...\" :body \"...\") ~3 KB\nBytecode: [CALL_CID] + 50\u00d7 [STR, STR] ~800 bytes\nGzipped bytecode: ~400 bytes" "text"))
(p "The markup structure is in the compiled component. "
"The class strings are in the compiled component. "
"The only thing on the wire is the data that differs between instances. "
"Gzip crushes the repetitive framing to almost nothing.")
(h4 :class "font-semibold mt-4 mb-2" "Content negotiation")
(p "The client advertises what it has via request headers:")
(~docs/code :src (highlight ";; Client tells server what's cached\nAccept: application/sx-bytecode, text/sx, text/html\nX-Sx-Cached-Components: bafyrei..card, bafyrei..header, bafyrei..nav\n\n;; Server picks the smallest response:\n;; - Client has ~card cached? Send bytecode (data only)\n;; - Client has compiler but not ~card? Send SX source\n;; - Client has nothing? Send HTML" "text"))
(p "The server and client negotiate the optimal response format automatically. "
"First visit is HTML (full progressive enhancement). "
"Subsequent navigations are gzipped bytecode \u2014 "
"just the data, a few hundred bytes, instant render via cached compiled components.")
(h4 :class "font-semibold mt-4 mb-2" "The server becomes a data API")
(p "At the bytecode tier, the server is no longer rendering views. "
"It's serving data \u2014 structured arguments for compiled client-side functions. "
"The rendering logic is cached on the client as compiled WASM. "
"This inverts the traditional server-rendered model without sacrificing "
"progressive enhancement \u2014 the HTML fallback is always available."))
;; -----------------------------------------------------------------------
;; How this changes existing plans
;; -----------------------------------------------------------------------
(~docs/section :title "Impact on Existing Plans" :id "impact"
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "Plan")
(th :class "px-3 py-2 font-medium text-stone-600" "Impact")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700"
(a :href "/sx/(etc.(plan.rust-wasm-host))" "Rust/WASM Host"))
(td :class "px-3 py-2 text-stone-600"
"Superseded. OCaml/WASM replaces Rust/WASM. The handle table and platform layer design carry over."))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700"
(a :href "/sx/(etc.(plan.isolated-evaluator))" "Isolated Evaluator"))
(td :class "px-3 py-2 text-stone-600"
"Architecture preserved. sx-platform.js and evaluator isolation apply to the OCaml evaluator too. The JS evaluator becomes a fallback."))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700"
(a :href "/sx/(etc.(plan.foundations))" "Foundations"))
(td :class "px-3 py-2 text-stone-600"
"Accelerated. OCaml 5 has native effects/continuations/fibers. Steps 4 (Concurrent CEK) and 5 (Linear Effects) map directly."))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700"
(a :href "/sx/(etc.(plan.wasm-bytecode-vm))" "WASM Bytecode VM"))
(td :class "px-3 py-2 text-stone-600"
"Subsumed as Tier 1. Bytecodes become an intermediate step on the path to native WASM compilation. The dispatch loop is Tier 1; JIT to WASM functions is Tier 2; AOT is Tier 3."))
(tr
(td :class "px-3 py-2 text-stone-700"
(a :href "/sx/(etc.(plan.self-hosting-bootstrapper))" "Self-Hosting Bootstrapper"))
(td :class "px-3 py-2 text-stone-600"
"Converges. js.sx and py.sx are self-hosting emitters. The self-hosting SX compiler (Phase 5 here) is the logical endpoint."))))))
;; -----------------------------------------------------------------------
;; Principles
;; -----------------------------------------------------------------------
(~docs/section :title "Principles" :id "principles"
(ul :class "list-disc list-inside space-y-2"
(li (strong "SX is the language.") " Not OCaml, not Rust, not Haskell. "
"Those are substrates. SX defines the semantics, SX checks the types, "
"SX will ultimately compile itself.")
(li (strong "OCaml is scaffolding.") " The closest existing language to CEK. "
"Used to bootstrap the self-hosting compiler. Comes down when the compiler is mature.")
(li (strong "The spec is the compiler's input.") " " (code "cek.sx") ", " (code "frames.sx")
", " (code "primitives.sx") " \u2014 the same files that define the language "
"become the compiler's source. One truth, one artifact.")
(li (strong "Platforms provide effects, not evaluation.") " Python provides database/HTTP. "
"JavaScript provides DOM. GPU provides tensor ops. "
"The evaluator is always compiled SX, embedded via FFI or WASM.")
(li (strong "Linearity belongs in the spec.") " SX's type system checks it. "
"The host provides runtime backup. If the type system is sound, the runtime check never fires.")
(li (strong "One language, every target.") " Not \"one spec, multiple implementations.\" "
"One " (em "compiled") " evaluator, deployed as native/.wasm/.js depending on context. "
"The evaluator binary is the same code everywhere. Only the platform layer changes.")))
;; -----------------------------------------------------------------------
;; Outcome
;; -----------------------------------------------------------------------
(~docs/section :title "Outcome" :id "outcome"
(p "After completion:")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li "SX core compiled from spec via OCaml \u2014 native + WASM from one codebase.")
(li "Python web framework calls native " (code "sx_core.so") " for rendering \u2014 "
"100\u20131000x faster than interpreted Python.")
(li "Browser loads " (code "sx_core.wasm") " \u2014 same compiler, same spec, near-native speed.")
(li "Server sends SX, client JIT-compiles to WASM functions and caches by CID.")
(li "Entire apps AOT-compiled to " (code ".wasm") " binaries. "
"Platform provides DOM/fetch. Everything else is machine code.")
(li "Content-addressed compilation cache: compile once, cache by CID, serve from CDN/IPFS forever.")
(li "Concurrent CEK runs on OCaml 5 fibers/domains \u2014 true parallelism for Art DAG.")
(li "Linear effects validated by SX type system, enforced by OCaml one-shot continuations.")
(li "Self-hosting compiler: SX compiles itself to machine code. OCaml scaffolding removed.")
(li "The only interpreter in the system is the CPU."))
(p :class "text-stone-500 text-sm italic mt-12"
"The Mother Language was always SX. We just needed to find the right scaffolding to stand it up."))))
Reference in New Issue View Git Blame Copy Permalink