coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f0d8db9b68d50d239c1641b4652b655b0b0390c8
rose-ash/sx/sx/plans/rust-wasm-host.sx
giles 6f96452f70 Fix empty code blocks: rename ~docs/code param, fix batched IO dispatch 
Two bugs caused code blocks to render empty across the site:

1. ~docs/code component had parameter named `code` which collided with
   the HTML <code> tag name. Renamed to `src` and updated all 57
   callers. Added font-mono class for explicit monospace.

2. Batched IO dispatch in ocaml_bridge.py only skipped one leading
   number (batch ID) but the format has two (epoch + ID):
   (io-request EPOCH ID "name" args...). Changed to skip all leading
   numbers so the string name is correctly found. This fixes highlight
   and other batchable helpers returning empty results.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 18:08:40 +00:00

264 lines
24 KiB
Plaintext
Raw Blame History

;; ---------------------------------------------------------------------------
;; Rust/WASM Host — Bootstrap the SX spec to Rust, compile to WASM
;; ---------------------------------------------------------------------------
(defcomp ~plans/rust-wasm-host/plan-rust-wasm-host-content ()
(~docs/page :title "Rust/WASM Host"
(~docs/section :title "Context" :id "context"
(p "The SX host architecture says: spec it in " (code ".sx") ", bootstrap to every target. We've now done it for Rust.")
(p "The Rust bootstrapper (" (code "bootstrap_rs.py") ") reads all 20 " (code ".sx") " spec files and emits a complete Rust crate — " (strong "9,781 lines") " of Rust that compiles with zero errors. The test suite has " (strong "92 tests passing") " across parser, evaluator, primitives, and rendering.")
(p "This is distinct from the " (a :href "/sx/(etc.(plan.wasm-bytecode-vm))" "WASM Bytecode VM") " plan. That plan designs a custom bytecode format and VM. This plan bootstraps the " (strong "same tree-walking evaluator") " directly to Rust/WASM from the same " (code ".sx") " specs — no new bytecode, no new VM, just another host.")
(h4 :class "font-semibold mt-4 mb-2" "What exists today")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "Artifact")
(th :class "px-3 py-2 font-medium text-stone-600" "Lines")
(th :class "px-3 py-2 font-medium text-stone-600" "Status")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "bootstrap_rs.py"))
(td :class "px-3 py-2 text-stone-700" "—")
(td :class "px-3 py-2 text-stone-600" "Rust bootstrapper, reads all 20 .sx spec files"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "sx_ref.rs"))
(td :class "px-3 py-2 text-stone-700" "9,781")
(td :class "px-3 py-2 text-stone-600" "Generated Rust — compiles clean, 0 errors"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "platform.rs"))
(td :class "px-3 py-2 text-stone-700" "—")
(td :class "px-3 py-2 text-stone-600" "Rust platform interface (type constructors, env ops)"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" (code "test_parser.rs"))
(td :class "px-3 py-2 text-stone-700" "—")
(td :class "px-3 py-2 text-stone-600" "92 tests passing (parser, eval, primitives, render)"))
(tr
(td :class "px-3 py-2 text-stone-700" "Primitives")
(td :class "px-3 py-2 text-stone-700" "—")
(td :class "px-3 py-2 text-stone-600" "75 real implementations, 154 stubs"))))))
;; -----------------------------------------------------------------------
;; Architecture
;; -----------------------------------------------------------------------
(~docs/section :title "Architecture" :id "architecture"
(p "The key architectural insight: " (strong "factor the browser primitives into a shared platform layer") " that both the JS evaluator and the WASM module consume.")
(h4 :class "font-semibold mt-4 mb-2" "Shared platform layer")
(p (code "platform_js.py") " already contains all DOM, browser, fetch, timer, and storage implementations — bootstrapped from " (code "boundary.sx") ". These are pure JavaScript functions that call browser APIs. They don't depend on the evaluator.")
(p "Extract them into a standalone " (code "sx-platform.js") " module. Both " (code "sx-browser.js") " (the current JS evaluator) and the new " (code "sx-wasm-shim.js") " import from the same platform module:")
(~docs/code :src (highlight " ┌─────────────────┐\n │ sx-platform.js │ ← DOM, fetch, timers, storage\n └────────┬────────┘\n │\n ┌──────────────┼──────────────┐\n │ │\n ┌─────────┴─────────┐ ┌────────┴────────┐\n │ sx-browser.js │ │ sx-wasm-shim.js │\n │ (JS tree-walker) │ │ (WASM instance │\n │ │ │ + handle table) │\n └────────────────────┘ └─────────────────┘" "text"))
(p "One codebase for all browser primitives. Bug fixes apply to both targets. The evaluator is the only thing that changes — JS tree-walker vs Rust/WASM tree-walker.")
(h4 :class "font-semibold mt-4 mb-2" "Opaque handle table")
(p "Rust/WASM can't hold DOM node references directly. Instead, Rust values use " (code "Value::Handle(u32)") " — an opaque integer that indexes into a JavaScript-side handle table:")
(~docs/code :src (highlight "// JS side (in sx-wasm-shim.js)\nconst handles = []; // handle_id → DOM node\n\nfunction allocHandle(node) {\n const id = handles.length;\n handles.push(node);\n return id;\n}\n\nfunction getHandle(id) { return handles[id]; }\nfunction freeHandle(id) { handles[id] = null; }" "javascript"))
(~docs/code :src (highlight "// Rust side\n#[derive(Clone, Debug)]\nenum Value {\n Nil,\n Bool(bool),\n Number(f64),\n Str(String),\n Symbol(String),\n Keyword(String),\n List(Vec<Value>),\n Dict(Vec<(Value, Value)>),\n Lambda(Rc<Closure>),\n Handle(u32), // ← opaque DOM node reference\n}" "rust"))
(p "When Rust calls a DOM primitive (e.g. " (code "createElement") "), it gets back a " (code "Handle(id)") ". When it passes that handle to " (code "appendChild") ", the JS shim looks up the real node. Rust never sees a DOM node — only integer IDs.")
(h4 :class "font-semibold mt-4 mb-2" "The JS shim is thin")
(p "The WASM shim's job is minimal:")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li "Instantiate the WASM module")
(li "Wire up the handle table")
(li "Delegate all browser primitives to " (code "sx-platform.js"))
(li "Provide " (code "#[wasm_bindgen]") " imports for the DOM primitives that Rust calls"))
(p "Everything complex — event dispatch, morph engine, routing, SSE — lives in the shared platform layer. The shim is glue."))
;; -----------------------------------------------------------------------
;; Stubs breakdown
;; -----------------------------------------------------------------------
(~docs/section :title "Current Stub Breakdown" :id "stubs"
(p "Of the 154 stubbed primitives, most fall into a few categories that map directly to implementation phases:")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "Category")
(th :class "px-3 py-2 font-medium text-stone-600" "Count")
(th :class "px-3 py-2 font-medium text-stone-600" "Examples")
(th :class "px-3 py-2 font-medium text-stone-600" "Phase")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "DOM creation & attrs")
(td :class "px-3 py-2 text-stone-700" "~30")
(td :class "px-3 py-2 text-stone-600" "createElement, setAttribute, appendChild")
(td :class "px-3 py-2 text-stone-600" "Phase 2"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Events & callbacks")
(td :class "px-3 py-2 text-stone-700" "~20")
(td :class "px-3 py-2 text-stone-600" "addEventListener, setTimeout, requestAnimationFrame")
(td :class "px-3 py-2 text-stone-600" "Phase 3"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Fetch & network")
(td :class "px-3 py-2 text-stone-700" "~15")
(td :class "px-3 py-2 text-stone-600" "fetch, XMLHttpRequest, SSE")
(td :class "px-3 py-2 text-stone-600" "Phase 3"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Browser APIs")
(td :class "px-3 py-2 text-stone-700" "~25")
(td :class "px-3 py-2 text-stone-600" "history, location, localStorage, console")
(td :class "px-3 py-2 text-stone-600" "Phase 5"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Morph engine")
(td :class "px-3 py-2 text-stone-700" "~15")
(td :class "px-3 py-2 text-stone-600" "morph, sync-attrs, reconcile")
(td :class "px-3 py-2 text-stone-600" "Phase 3"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 text-stone-700" "Signals & reactivity")
(td :class "px-3 py-2 text-stone-700" "~20")
(td :class "px-3 py-2 text-stone-600" "signal, deref, effect, computed, batch")
(td :class "px-3 py-2 text-stone-600" "Phase 6"))
(tr
(td :class "px-3 py-2 text-stone-700" "Component lifecycle")
(td :class "px-3 py-2 text-stone-700" "~29")
(td :class "px-3 py-2 text-stone-600" "boot, hydrate, register-component")
(td :class "px-3 py-2 text-stone-600" "Phase 4"))))))
;; -----------------------------------------------------------------------
;; Phase 1: WASM build + parse/eval
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 1: WASM Build + Parse/Eval" :id "phase-1"
(p "Get the existing Rust code compiling to WASM and running parse/eval in the browser.")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li "Add " (code "wasm-bindgen") " and " (code "wasm-pack") " to the Rust crate")
(li "Export " (code "#[wasm_bindgen]") " functions: " (code "parse(src) -> JsValue") " and " (code "eval(src) -> JsValue"))
(li "All 75 real primitives work — arithmetic, string ops, list ops, dict ops, comparisons")
(li "Test page: load WASM module, parse SX source, eval expressions, display results")
(li "Benchmark: parse/eval speed vs JS evaluator on the same expressions"))
(p (strong "Milestone:") " SX expressions evaluate identically in JS and WASM. No DOM, no rendering — just computation."))
;; -----------------------------------------------------------------------
;; Phase 2: DOM rendering via handle table
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 2: DOM Rendering via Handle Table" :id "phase-2"
(p "Implement the shared platform layer and handle table. Rust can create and manipulate DOM nodes.")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li "Extract " (code "sx-platform.js") " from " (code "platform_js.py") " output — all DOM primitives as standalone functions")
(li "Implement " (code "sx-wasm-shim.js") " — WASM instantiation + handle table + platform imports")
(li "Wire " (code "Value::Handle(u32)") " through the Rust evaluator for DOM node references")
(li "Implement ~30 DOM stubs: " (code "createElement") ", " (code "setAttribute") ", " (code "appendChild") ", " (code "createTextNode") ", " (code "removeChild"))
(li (code "render-to-dom") " works through WASM — evaluates component tree, produces real DOM nodes via handle table"))
(p (strong "Milestone:") " A component renders to identical DOM whether evaluated by JS or WASM."))
;; -----------------------------------------------------------------------
;; Phase 3: Events + fetch + morph
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 3: Events + Fetch + Morph" :id "phase-3"
(p "The hardest phase — callbacks cross the WASM/JS boundary. Event handlers are Rust closures that JS must be able to invoke.")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Callback table") " — mirror of the handle table but for functions. Rust registers a closure, gets back an ID. JS calls the ID when the event fires. Rust looks up and invokes the closure.")
(li (strong "Event listeners") " — " (code "addEventListener") " stores the callback ID on the element handle. JS dispatches events to the WASM callback table.")
(li (strong "Fetch") " — Rust initiates fetch via JS import, JS calls " (code "sx-platform.js") " " (code "fetch") ", returns result to Rust via callback.")
(li (strong "Morph engine") " — DOM diffing/patching runs in Rust. Morph calls produce a sequence of DOM mutations via handle operations. " (code "sync-attrs") " and " (code "morph-children") " work through the handle table."))
(p (strong "Milestone:") " Interactive pages work — click handlers, form submissions, HTMX-style requests, morph updates. This is the " (strong "MVP") " — a page could ship with the WASM evaluator and function correctly."))
;; -----------------------------------------------------------------------
;; Phase 4: Boot + hydration + components
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 4: Boot + Hydration + Components" :id "phase-4"
(p "Full page lifecycle — the WASM module replaces " (code "sx-browser.js") " for a complete page.")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Boot sequence") " — WASM module loads, reads " (code "data-components") " script tags, registers component definitions, processes " (code "data-sx-page") " content")
(li (strong "Component registration") " — " (code "defcomp") " and " (code "defisland") " evaluated in Rust, stored in the WASM-side environment")
(li (strong "Hydration") " — server-rendered HTML matched against component tree, event handlers attached, islands activated")
(li (strong "CSSX") " — style computation runs in Rust (all CSSX primitives are already in the 75 real implementations)"))
(p (strong "Milestone:") " A full SX page boots and hydrates under WASM. Component definitions, CSSX styles, page content all evaluated by Rust."))
;; -----------------------------------------------------------------------
;; Phase 5: Routing + streaming + SSE
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 5: Routing + Streaming + SSE" :id "phase-5"
(p "Client-side navigation and real-time updates.")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Client routing") " — " (code "navigate-to") ", " (code "popstate") " handling, URL matching from " (code "defpage") " routes")
(li (strong "History API") " — " (code "pushState") " / " (code "replaceState") " via JS imports")
(li (strong "SSE") " — server-sent events for live updates, morph on incoming HTML/SX")
(li (strong "Streaming responses") " — progressive rendering of large page content"))
(p (strong "Milestone:") " Full SPA navigation works under WASM. Pages load, navigate, and receive live updates."))
;; -----------------------------------------------------------------------
;; Phase 6: Signals + reactive islands
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 6: Signals + Reactive Islands" :id "phase-6"
(p "The most architecturally interesting phase — closures-as-values must work across the WASM boundary.")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Signal primitives") " — " (code "signal") ", " (code "deref") ", " (code "reset!") ", " (code "swap!") ", " (code "computed") ", " (code "effect") ", " (code "batch") " all implemented in Rust")
(li (strong "Reactive DOM updates") " — signal changes trigger DOM mutations via handle table operations. No full re-render — fine-grained updates.")
(li (strong "Island scoping") " — " (code "with-island-scope") " manages signal lifecycle. Dispose island = drop all signals, effects, and handles.")
(li (strong "Computed chains") " — dependency graph tracks which signals feed which computed values. Topological update order."))
(p "The closure challenge: " (code "effect") " and " (code "computed") " take closures that capture reactive context. In Rust, these are " (code "Rc<dyn Fn(...)>") " values that the signal graph holds. The " (code "with-island-scope") " arena pattern handles cleanup — drop the arena, drop all closures.")
(p (strong "Milestone:") " Reactive islands work under WASM. Signals, computed values, effects, fine-grained DOM updates — all in Rust."))
;; -----------------------------------------------------------------------
;; Phase 7: Full parity + gradual rollout
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 7: Full Parity + Gradual Rollout" :id "phase-7"
(p "Shadow-compare and feature-flagged rollout.")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Shadow compare") " — run both JS and WASM evaluators in parallel on every page render. Assert identical DOM output. Log divergences. Same principle as async eval convergence.")
(li (strong "Feature flag") " — server sets " (code "data-sx-runtime=\"wasm\"") " or " (code "\"js\"") " on the page. Boot script loads the corresponding evaluator. Flag can be per-page, per-user, or global.")
(li (strong "Progressive enhancement") " — try WASM first, fall back to JS if WASM instantiation fails. Ship both " (code "sx-browser.js") " and " (code "sx-wasm.wasm") ".")
(li (strong "Gradual rollout") " — start with simple pages (documentation, static content). Move to interactive pages. Finally, reactive islands. Each phase validates correctness before advancing."))
(p (strong "Milestone:") " Full parity. Any SX page renders identically under JS or WASM. The runtime is a deployment choice, not an architectural one."))
;; -----------------------------------------------------------------------
;; Shared platform layer
;; -----------------------------------------------------------------------
(~docs/section :title "Shared Platform Layer" :id "shared-platform"
(p "The architectural heart of this plan. " (code "platform_js.py") " already generates all browser primitive implementations — DOM manipulation, fetch wrappers, timer management, storage access, history API, SSE handling. Currently these are inlined into " (code "sx-browser.js") ".")
(p "The extraction:")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li (strong "Step 1") " — refactor " (code "platform_js.py") " to emit a standalone " (code "sx-platform.js") " module (ES module exports)")
(li (strong "Step 2") " — " (code "sx-browser.js") " imports from " (code "sx-platform.js") " instead of containing the implementations inline")
(li (strong "Step 3") " — " (code "sx-wasm-shim.js") " imports from the same " (code "sx-platform.js") " and wires the functions as WASM imports"))
(p "Result: one implementation of every browser primitive, shared by both evaluator targets. Fix a bug in " (code "sx-platform.js") " and both JS and WASM evaluators get the fix.")
(~docs/code :src (highlight "// sx-platform.js (extracted from platform_js.py output)\nexport function createElement(tag) {\n return document.createElement(tag);\n}\nexport function setAttribute(el, key, val) {\n el.setAttribute(key, val);\n}\nexport function appendChild(parent, child) {\n parent.appendChild(child);\n}\nexport function addEventListener(el, event, callback) {\n el.addEventListener(event, callback);\n}\n// ... ~150 more browser primitives\n\n// sx-browser.js\nimport * as platform from './sx-platform.js';\n// Uses platform functions directly — evaluator is JS\n\n// sx-wasm-shim.js\nimport * as platform from './sx-platform.js';\n// Wraps platform functions for WASM import — evaluator is Rust" "javascript")))
;; -----------------------------------------------------------------------
;; Interaction with other plans
;; -----------------------------------------------------------------------
(~docs/section :title "Interaction with Other Plans" :id "interactions"
(ul :class "list-disc list-inside space-y-2"
(li (strong "WASM Bytecode VM") " — complementary, not competing. This plan bootstraps the tree-walking evaluator to Rust/WASM. The bytecode VM plan compiles SX to a custom bytecode format and runs it in a dispatch loop. Tree-walking comes first (it's working now). Bytecode VM is a future optimisation on top of the Rust host.")
(li (strong "Runtime Slicing") " — the WASM module can be tiered. L0 pages need no WASM at all. L1 pages need a minimal WASM module (just parse + eval, no DOM). L2+ pages need the full module with DOM and signals. Tree-shake unused primitives per tier.")
(li (strong "Content-Addressed Components") " — deterministic Rust compilation means the same " (code ".sx") " source always produces the same WASM binary. CID-addressable WASM modules. Fetch the evaluator itself by content hash.")
(li (strong "Async Eval Convergence") " — must complete first. The spec must be the single source of truth before we add another compilation target. The Rust bootstrapper reads the same spec files that the Python and JS bootstrappers read.")))
;; -----------------------------------------------------------------------
;; Principles
;; -----------------------------------------------------------------------
(~docs/section :title "Principles" :id "principles"
(ul :class "list-disc list-inside space-y-2"
(li (strong "Same spec, another host.") " The Rust target is not special — it's the same architecture as Python and JavaScript. " (code "bootstrap_rs.py") " reads the same " (code ".sx") " files and emits Rust instead of Python or JS. The spec doesn't know which host runs it.")
(li (strong "Platform primitives stay in JavaScript.") " DOM, fetch, timers, storage — these are browser APIs. Rust doesn't reimplement them. It calls them through the shared platform layer via the handle table.")
(li (strong "Shared platform, not duplicated platform.") " The key win over a pure-WASM approach. Browser primitives exist once in " (code "sx-platform.js") ". Both evaluators use them. No divergence, no duplicate bugs.")
(li (strong "Progressive, not all-or-nothing.") " WASM is an enhancement. JS remains the fallback. Pages can opt in per-page. The server decides which runtime to ship. Rollout is gradual and reversible.")
(li (strong "Handle table is the boundary.") " Rust holds integer IDs. JavaScript holds real objects. The handle table is the only bridge. This keeps the WASM module platform-independent — swap the handle table implementation for a different host (Node, Deno, native webview) and the Rust code doesn't change.")))
;; -----------------------------------------------------------------------
;; Outcome
;; -----------------------------------------------------------------------
(~docs/section :title "Outcome" :id "outcome"
(p "After completion:")
(ul :class "list-disc list-inside space-y-2 mt-2"
(li "SX bootstraps to four hosts: JavaScript, Python, Rust (native), Rust (WASM)")
(li "Browser evaluation runs at near-native speed via WASM tree-walking")
(li "All browser primitives shared between JS and WASM evaluators — zero duplication")
(li "Gradual rollout: feature flag per-page, shadow-compare for correctness, progressive enhancement for compatibility")
(li "The architecture proof is complete: one spec, every host, deployment-time target selection")
(li "Future bytecode VM plan builds on the Rust host — the platform layer and handle table are already in place")))))
Reference in New Issue View Git Blame Copy Permalink