Named let's sf-named-let used call-lambda which returns a thunk that
was never trampolined. The body executed in a disconnected env, so
set! couldn't reach outer let bindings. Fixed by using cek-call which
evaluates through the full CEK machine with proper env chain.
Also converted test-named-let-set.sx from assert= (uses broken = for
lists) to deftest/assert-equal (uses deep equal?).
JS standard: 1120/1120, JS full: 1600/1600. Zero failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Named let creates a loop continuation but set! inside the loop body
does not mutate bindings in the enclosing let scope. Affects both
the WASM kernel and native OCaml CEK evaluator.
6 failing Node tests cover:
- set! counter (simplest case)
- set! counter with named let params
- set! list accumulator via append
- append! + set! counter combo
- set! string concatenation
- nested named let set!
3 baselines pass: plain let set!, functional named let, plain append!
Also adds spec/tests/test-named-let-set.sx (7 assertions, first
fails and aborts — confirms bug exists in spec test suite too).
This is the root cause of empty source code blocks on all example
pages: tokenize-sx uses set! in named let → empty tokens →
highlight returns "(<> )" → empty <code> blocks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
component-source from data/helpers.sx was overriding the native
OCaml version. The SX version calls env-get with wrong arity (1 arg
vs required 2), producing empty source. Re-bind the native version
in SSR overrides after file loading.
Note: source code still not visible because highlight function
returns empty — separate issue in the aser rendering pipeline.
Also adds:
- spec/tests/test-reactive-islands.sx — 22 SX-native tests for all
14 reactive island demos (render + signal logic + DOM)
- tests/node/run-sx-tests.js — Node runner for SX test files
- tests/node/test-reactive-islands.js — 39 Node/happy-dom tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three root causes for reactive attribute updates not propagating in WASM:
1. `context` CEK special form only searched kont provide frames, missing
`scope-push!` entries in the native scope_stacks hashtable. Unified by
adding scope_stacks fallback to step_sf_context.
2. `flush-subscribers` used bare `(sub)` call which failed to invoke
complex closures in for-each HO callbacks. Changed to `(cek-call sub nil)`.
3. Test eagerly evaluated `(deref s)` before render-to-dom saw it.
Fixed tests to use quoted expressions matching real browser boot.
WASM native: 10/10, WASM shell: 26/26.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Revert (when (client?) ...) guard in signals.sx — it broke JS tests
since client? is false in Node.js too.
Instead, rebind effect and register-in-scope as no-ops in sx_server.ml
AFTER all .sx files load. The SX definition from signals.sx is replaced
only in the OCaml SSR context. JS tests and WASM browser keep the real
effect implementation.
Remove redundant browser primitive stubs from sx_primitives.ml — only
resource SSR stub needed (effect override moved to server setup).
JS tests: 1582/1585 (3 VM closure interop remain)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- signals.sx: guard effect body with (when (client?) ...) so effects
are no-op during SSR — only 2 stubs needed (effect, register-in-scope)
- sx_primitives.ml: add resource SSR stub (returns signal {loading: true}),
remove 27 unnecessary browser primitive stubs
- sx_server.ml: native component-source that looks up Component/Island
from env and pretty-prints the definition (replaces broken Python helper)
- reactive-islands/index.sx: Examples section with all 15 live demos
inline + highlighted source via component-source
- reactive-islands/demo.sx: replace 14 hardcoded highlight strings with
(component-source "~name") calls for always-current source
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
signal-add-sub! used (append! subscribers f) which returns a new list
for immutable List but discards the result — after signal-remove-sub!
replaces the subscribers list via dict-set!, re-adding subscribers
silently fails. Counter island only worked once (0→1 then stuck).
Fix: use (dict-set! s "subscribers" (append ...)) to explicitly update
the dict field, matching signal-remove-sub!'s pattern.
Build pipeline fixes:
- sx-build-all.sh now bundles spec→dist and recompiles .sxbc bytecode
- compile-modules.js syncs .sx source files alongside .sxbc to wasm/sx/
- Per-file cache busting: wasm, platform JS, and sxbc each get own hash
- bundle.sh adds cssx.sx to dist
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: _env_bind_hook mirrored ALL env_bind calls (including
lambda parameter bindings) to the shared VM globals table. Factory
functions like make-page-fn that return closures capturing different
values for the same param names (default-name, prefix, suffix) would
have the last call's values overwrite all previous closures' captured
state in globals. OP_GLOBAL_GET reads globals first, so all closures
returned the last factory call's values.
Fix: only sync root-env bindings (parent=None) to VM globals. Lambda
parameter bindings stay in their local env, found via vm_closure_env
fallback in OP_GLOBAL_GET.
Also in this commit:
- OP_CLOSURE propagates parent vm_closure_env to child closures
- Remove JIT globals injection (closure vars found via env chain)
- sx_server.ml: SX-Request header → returns text/sx (aser only)
- sx_server.ml: diagnostic endpoint GET /sx/_debug/{env,eval,route}
- sx_server.ml: page helper stubs for deep page rendering
- sx_server.ml: skip client-libs/ dir (browser-only definitions)
- adapter-html.sx: unknown components → HTML comment (not error)
- sx-platform.js: .sxbc fallback loader for bytecode modules
- Delete sx_http.ml (standalone HTTP server, unused)
- Delete stale .sxbc.json files (arity=0 bug, replaced by .sxbc)
- 7 new closure isolation tests in test-closure-isolation.sx
- mcp_tree.ml: emit arity + upvalue-count in .sxbc.json output
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OCaml HTML renderer (sx_render.ml) silently returned "" when env_get
failed for primitive function calls (str, +, len, etc.) inside HTML
elements. The Eval_error catch now falls through to eval_expr which
resolves primitives correctly. Fixes 21 rendering tests.
Rename condition system special form from "signal" to "signal-condition"
in evaluator.sx, matching the OCaml bootstrapped evaluator (sx_ref.ml).
This avoids clashing with the reactive signal function. Fixes 9
condition system tests.
1166 passed, 0 failed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Convert large cond chains doing string equality dispatch to use the
match special form: step-eval-list (42 arms), step-continue (31 arms),
compile-list (30 arms), ho-setup-dispatch (7 arms), value-matches-type?
(10 arms). Also fix test-canonical.sx to use defsuite/deftest format
and load canonical.sx in both test runners.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spec/canonical.sx defines:
- canonical-serialize: deterministic s-expression serialization
(sorted dict keys, normalized numbers, minimal escaping)
- content-id: SHA3-256 of canonical form = CID of any s-expression
- Bytecode module format: (sxbc version source-hash (code ...))
- Provenance records linking source CID → bytecode CID → compiler CID
The CID is the identity model for SX. A component, a bytecode module,
a test suite — anything expressed as an s-expression — is addressable
by content hash. Annotation layers (source maps, variable names, test
results, documentation) reference CIDs without polluting the artifacts.
Requires host primitives: sha3-256, sort. Tests in test-canonical.sx.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Machine-readable SX semantics reference with 35 evaluation rules
covering literals, lookup, special forms, definitions, higher-order
forms, scopes, continuations, and reactive primitives.
New sx_explain MCP tool: query by form name ("let", "map") or
category ("special-form", "higher-order") to get pattern, rule,
effects, and examples.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixed three fundamental issues:
1. cek-try arg passing: handler was called with raw string instead of
(List [String msg]), causing "lambda expects 1 args, got N" errors
2. Silent island hydration failures: hydrate-island now wraps body
render in cek-try, displaying red error box with stack trace instead
of empty div. No more silent failures.
3. swap! thunk leak: apply result wasn't trampolined, storing thunks
as signal values instead of evaluated results
Also fixed: assert= uses = instead of equal? for value comparison,
assert-signal-value uses deref instead of signal-value, HTML entity
decoding in script tag test source via host-call replaceAll.
Temperature converter demo page now shows live test results:
✓ initial celsius is 20
✓ computed fahrenheit = celsius * 1.8 + 32
✓ +5 increments celsius
✓ fahrenheit updates on celsius change
✓ multiple clicks accumulate
1116/1116 OCaml tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- render-html-island wraps body SSR in cek-try (graceful fallback to
empty container when island body has DOM/signal code)
- defcomp placeholder pattern: server renders safe placeholder div
with data-sx-island, browser hydrates the actual island
- cek-try primitive added to both server and browser OCaml kernels
- assert/assert= added to spec/harness.sx for standalone use
- Test source stored in <script type="text/sx-test" data-for="...">
with HTML entity decoding via host-call replaceAll
Temperature converter: 5 tests embedded in demo page. Test runner
hydrates and finds tests but body render is empty — needs debugging
of the specific construct that silently fails in render-to-dom.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Core reactive primitives (signal, deref, reset!, swap!, computed, effect,
batch, notify-subscribers, dispose-computed, with-island-scope,
register-in-scope) moved to spec/signals.sx — pure SX, zero platform
dependencies. Usable by any host: web, CLI, embedded, server, harness.
Web extensions (marsh scopes, stores, event bridge, resource) remain in
web/signals.sx, which now depends on spec/signals.sx.
Updated all load paths:
- hosts/ocaml/bin/sx_server.ml — loads spec/signals.sx before web/signals.sx
- hosts/ocaml/bin/run_tests.ml — loads both in order
- hosts/ocaml/browser/bundle.sh + sx-platform.js — loads core-signals.sx first
- shared/sx/ocaml_bridge.py — loads spec/signals.sx before web extensions
1116/1116 OCaml tests pass. Browser reactive island preview works.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spec/harness.sx — spec-level test harness with:
- Mock platform (30+ default IO mocks: fetch, query, DOM, storage, etc.)
- Session management (make-harness, harness-reset!, harness-set!/get)
- IO interception (make-interceptor, install-interceptors)
- IO log queries (io-calls, io-call-count, io-call-nth, io-call-args)
- IO assertions (assert-io-called, assert-no-io, assert-io-count, etc.)
15 harness tests passing on both OCaml (1116/1116) and JS (15/15).
Loaded automatically by both test runners.
MCP tool: sx_harness_eval — evaluate SX with mock IO, returns result + IO trace.
The harness is extensible: new platforms just add entries to the platform dict.
Components can ship with deftest forms that verify IO behavior against mocks.
Tests are independent objects that can be published separately (by CID).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two paren bugs in home-stepper.sx caused the home page to render blank:
1. Line 222 had one extra ) that prematurely closed the letrec bindings
list — rebuild-preview and do-back became body expressions instead
of bindings, making them undefined in scope.
2. Lines 241-308 were outside the let/letrec scope entirely — the outer
let closed at line 240, so freeze-scope, cookie restore, source
parsing, and the entire div rendering tree had no access to signals
or letrec functions.
Also hardens defisland to wrap multi-expression bodies in (begin ...),
matching the Python-side fix from 9f0c541. Both spec/evaluator.sx and
the OCaml transpiled sx_ref.ml are updated.
Adds SX Tools essay under Applications — the revised plan for structural
tree reading/editing tools for .sx files, motivated by this exact bug.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spec/ now contains only the language definition (5 files):
evaluator.sx, parser.sx, primitives.sx, render.sx, special-forms.sx
lib/ contains code written IN the language (8 files):
stdlib.sx, types.sx, freeze.sx, content.sx,
bytecode.sx, compiler.sx, vm.sx, callcc.sx
Test files follow source: spec/tests/ for core language tests,
lib/tests/ for library tests (continuations, freeze, types, vm).
Updated all consumers:
- JS/Python/OCaml bootstrappers: added lib/ to source search paths
- OCaml bridge: spec_dir for parser/render, lib_dir for compiler/freeze
- JS test runner: scans spec/tests/ (always) + lib/tests/ (--full)
- OCaml test runner: scans spec/tests/, lib tests via explicit request
- Docker dev mounts: added ./lib:/app/lib:ro
Tests: 1041 JS standard, 1322 JS full, 1101 OCaml — all pass
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Regression tests for the silent failure where callLambda returns a
Thunk (TCO) that must be trampolined for side effects to execute.
Without trampoline, event handlers (swap!, reset!) silently did nothing.
5 tests covering: single mutation, event arg passing, multi-statement
body, repeated accumulation, and nested lambda calls — all through
the (trampoline (call-lambda handler args)) pattern that dom-on uses.
Tests: 1322 JS (full), 1114 OCaml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three bugs broke island SSR rendering of the home stepper widget:
1. Inline VM opcodes (OP_ADD..OP_DEC) broke JIT-compiled functions.
The compiler emitted single-byte opcodes for first/rest/len/= etc.
that produced wrong results in complex recursive code (sx-parse
returned nil, split-tag produced 1 step instead of 16). Reverted
compiler to use CALL_PRIM for all primitives. VM opcode handlers
kept for future use.
2. Named let (let loop ((x init)) body) had no compiler support —
silently produced broken bytecode. Added desugaring to letrec.
3. URL-encoded cookie values not decoded server-side. Client set-cookie
uses encodeURIComponent but Werkzeug doesn't decode cookie values.
Added unquote() in bridge cookie injection.
Also: call-lambda used eval_expr which copies Dict values (signals),
breaking mutations through aser lambda calls. Switched to cek_call.
Also: stepper preview now includes ~cssx/tw spreads for SSR styling.
Tests: 1317 JS, 1114 OCaml, 26 integration (2 pre-existing failures)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
16 new opcodes (160-175) bypass the CALL_PRIM hashtable lookup for
the most frequently called primitives:
Arithmetic: OP_ADD, OP_SUB, OP_MUL, OP_DIV, OP_INC, OP_DEC, OP_NEG
Comparison: OP_EQ, OP_LT, OP_GT, OP_NOT
Collection: OP_LEN, OP_FIRST, OP_REST, OP_NTH, OP_CONS
The compiler (compiler.sx) recognizes these names at compile time and
emits the inline opcode instead of CALL_PRIM. The opcode is self-
contained — no constant pool index, no argc byte. Each primitive is
a single byte in the bytecode stream.
Implementation in all three VMs:
- OCaml (sx_vm.ml): direct pattern match, no allocation
- SX spec (vm.sx): delegates to existing primitives
- JS (transpiled): same as SX spec
66 new tests in spec/tests/vm-inline.sx covering arithmetic, comparison,
collection ops, composition, and edge cases.
Tests: 1314 JS (full), 1114 OCaml, 32 Playwright
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These are domain definition forms (same pattern as defhandler, defpage,
etc.), not core language constructs. Moving them to web-forms.sx keeps
the core evaluator + types.sx cleaner for WASM compilation.
web-forms.sx now loaded in both JS and Python build pipelines.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the fragile pattern of capturing and wrapping definition-form?
with a mutable *definition-form-extensions* list in render.sx. Web
modules append names to this list instead of redefining the function.
Survives spec reloads without losing registrations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move defhandler/defquery/defaction/defpage/defrelation from hardcoded
evaluator dispatch to web/web-forms.sx extension module, registered via
register-special-form!. Adapters updated to use definition-form? and
dynamically extended form-name lists.
Fix modifier-key clicks (ctrl-click → new tab) in three click handlers:
bindBoostLink, bindClientRouteClick, and orchestration.sx bind-event.
Add event-modifier-key? primitive (eventModifierKey_p for transpiler).
Fix CSSX SSR: ~cssx/flush no longer drains the collected bucket on the
server, so the shell template correctly emits CSSX rules in <head>.
Add missing server-side DOM stubs (create-text-node, dom-append, etc.)
and SSR passthrough for portal/error-boundary/promise-delayed.
Passive event listeners for touch/wheel/scroll to fix touchpad scrolling.
97/97 Playwright demo tests + 4/4 isomorphic SSR tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spec/tests/test-render-html.sx covers the full HTML serialization surface:
text/literals, content escaping, attribute escaping, normal elements,
all 14 void elements, 18 boolean attributes, regular/data-*/aria-* attrs,
fragments, raw HTML, headings, lists, tables, forms, media, semantic
elements, SVG, control flow (if/when/cond), let bindings, map/for-each,
components (simple/children/keyword+children/nested), macros, begin/do,
letrec, scope/provide, islands with hydration markers, lakes, marshes,
threading, define-in-template.
Validates adapter-html.sx can replace sx_render.ml as the canonical renderer.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
JIT allowlist (sx_server.ml):
- Replace try-every-lambda strategy with StringSet allowlist. Only
functions in the list get JIT compiled (compiler, parser, pure transforms).
Render functions that need dynamic scope skip JIT entirely — no retry
overhead, no silent fallbacks.
- Add (jit-allow name) command for dynamic expansion from Python bridge.
- JIT failures log once with "[jit] DISABLED fn — reason" then go silent.
Standalone --test mode (sx_server.ml):
- New --test flag loads full env (spec + adapters + compiler + signals),
supports --eval and --load flags. Quick kernel testing without Docker.
Example: dune exec bin/sx_server.exe -- --test --eval '(len HTML_TAGS)'
Integration tests (integration_tests.ml):
- New binary exercising the full rendering pipeline: loads spec + adapters
into a server-like env, renders HTML via both native and SX adapter paths.
- 26 tests: HTML tags, special forms (when/if/let), letrec with side
effects, component rendering, eval-expr with HTML tag functions.
- Would have caught the "Undefined symbol: div/lake/init" issues from
the previous commit immediately without Docker.
VM cleanup (sx_vm.ml):
- Remove temporary debug logging (insn counter, call_closure counter,
VmClosure depth tracking) added during debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Core VM changes:
- Add VmClosure value variant — inner closures created by OP_CLOSURE are
first-class VM values, not NativeFn wrappers around call_closure
- Convert `run` from recursive to while-loop — zero OCaml stack growth,
true TCO for VmClosure tail calls
- vm_call handles VmClosure by pushing frame on current VM (no new VM
allocation per call)
- Forward ref _vm_call_closure_ref for cross-boundary calls (CEK/primitives)
Compiler (spec/compiler.sx):
- Define hoisting in compile-begin: pre-allocate local slots for all
define forms before compiling any values. Fixes forward references
between inner functions (e.g. read-expr referencing skip-ws in sx-parse)
- scope-define-local made idempotent (skip if slot already exists)
Server (sx_server.ml):
- JIT fail-once sentinel: mark l_compiled as failed after first VM runtime
error. Eliminates thousands of retry attempts per page render.
- HTML tag bindings: register all HTML tags as pass-through NativeFns so
eval-expr can handle (div ...) etc. in island component bodies.
- Log VM FAIL errors with function name before disabling JIT.
SSR fixes:
- adapter-html.sx letrec handler: evaluate bindings in proper letrec scope
(pre-bind nil, then evaluate), render body with render-to-html instead of
eval-expr. Fixes island SSR for components using letrec.
- Add `init` primitive to OCaml kernel (all-but-last of list).
- VmClosure handling in sx_runtime.ml sx_call dispatch.
Tests: 971/971 OCaml (+19 new), 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: sf-letrec returns a thunk (for TCO) but the CEK dispatch
wrapped it as a value without evaluating. The thunk leaked as the
return value of letrec expressions, breaking sx-parse and any function
using letrec.
Fix: step-sf-letrec unwraps the thunk into a CEK state, so the last
letrec body expression is properly evaluated by the CEK machine.
Also:
- compile-letrec: two-phase (nil-init then assign) for mutual recursion
- Skip JIT for inner functions (closure.bindings != globals) in both
vm_call and JIT hook
- vm-reset-fn for sx-parse removed (no longer needed)
- Parser regression test: letrec with mutable pos + recursive sublists
Test results: JS 943/17, OCaml 955/0, Python 747/0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests: parse "(a (b) (c))" must produce 3 siblings, not nested.
Catches JIT compilation bug where closing parens cause sibling
lists to become children.
Reset sx-parse to CEK on the OCaml kernel — the JIT-compiled
version of sx-parse's complex letrec produces wrong bytecode.
CEK interpretation works correctly (tests pass on all platforms).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The compiler was treating letrec as let — binding values sequentially.
This meant mutually recursive functions (like sx-parse's read-list
calling read-expr and vice versa) couldn't reference each other.
compile-letrec uses two phases:
1. Define all local slots initialized to nil
2. Compile and assign values — all names already in scope
This fixes sx-parse producing wrong ASTs (nested instead of sibling
lists) when JIT-compiled, which caused the stepper's step count to
be 2 instead of 16.
Also: skip JIT for lambdas with closure bindings (inner functions
like read-list-loop) — the closure merging into vm_env_ref produces
incorrect variable resolution.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix infinite recursion in VM JIT: restore sentinel pre-mark in vm_call
and pre-compile loop so recursive compiler functions don't trigger
unbounded compilation cascades. Runtime VM errors fall back to CEK;
compile errors surface visibly (not silently swallowed).
New: compile-quasiquote emits inline code instead of delegating to
qq-expand-runtime. Closure-captured variables merged into VM globals
so compiled closures resolve outer bindings via GLOBAL_GET.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compile-quasiquote, compile-defcomp, compile-defmacro were hardcoding
CALL_PRIM for runtime functions (qq-expand-runtime, eval-defcomp,
eval-defmacro) that aren't in the primitives table. Changed to
GLOBAL_GET + CALL so the VM resolves them from env.bindings at runtime.
The compile-call function already checks (primitive? name) before
emitting CALL_PRIM — only the three special-case compilers were wrong.
Also: register scope-push!/pop! as primitives, add scope-peek/emit!
to OCaml transpiler name mapping, fix sx_runtime.ml scope wrappers
to route through prim_call "scope-push!" etc.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace continuation-based scope frames with hashtable stacks for all
scope operations. The CEK evaluator's scope/provide/context/emit!/emitted
now use scope-push!/pop!/peek/emit! primitives (registered in
sx_primitives table) instead of walking continuation frames.
This eliminates the two-world problem where the aser used hashtable
stacks (scope-push!/pop!) but eval-expr used continuation frames
(ScopeFrame/ScopeAccFrame). Now both paths share the same mechanism.
Benefits:
- scope/context works inside eval-expr calls (e.g. (str ... (context x)))
- O(1) scope lookup vs O(n) continuation walking
- Simpler — no ScopeFrame/ScopeAccFrame/ProvideFrame creation/dispatch
- VM-compiled code and CEK code both see the same scope state
Also registers scope-push!/pop!/peek/emit!/collect!/collected/
clear-collected! as real primitives (sx_primitives table) so the
transpiled evaluator can call them directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- get primitive returns nil for type mismatches (list+string) instead
of raising — matches JS/Python behavior, fixes find-nav-match errors
- scope-peek, collect!, collected, clear-collected! registered as real
primitives in sx_primitives table (not just env bindings) so the CEK
step-sf-context can find them via get-primitive
- step-sf-context checks scope-peek hashtable BEFORE walking CEK
continuation — bridges aser's scope-push!/pop! with CEK's context
- context, emit!, emitted added to SPECIAL_FORM_NAMES and handled in
aser-special (scope operations in aser rendering mode)
- sx-context NativeFn for VM-compiled code paths
- VM execution errors no longer mark functions as permanently failed —
bytecode is correct, errors are from runtime data
- kbd, samp, var added to HTML_TAGS + sx-browser.js rebuilt
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace AOT adapter compilation with lazy JIT — each named lambda is
compiled to VM bytecode on first call, cached in l_compiled field for
subsequent calls. Compilation failures fall back to CEK gracefully.
VM types (vm_code, vm_upvalue_cell, vm_closure) moved to sx_types.ml
mutual recursion block. Lambda and Component records gain mutable
l_compiled/c_compiled cache fields. jit_compile_lambda in sx_vm.ml
wraps body as (fn (params) body), invokes spec/compiler.sx via CEK,
extracts inner closure from OP_CLOSURE constant.
JIT hooks in both paths:
- vm_call: Lambda calls from compiled VM code
- continue_with_call: Lambda calls from CEK step loop (injected by
bootstrap.py post-processing)
Pre-mark sentinel prevents re-entrancy (compile function itself was
hanging when JIT'd mid-compilation). VM execution errors caught and
fall back to CEK with sentinel marking.
Also: add kbd/samp/var to HTML_TAGS, rebuild sx-browser.js, add page
URL to sx-page-full-py timing log.
Performance: first page 28s (JIT compiles 17 functions), subsequent
pages 0.31s home / 0.71s wittgenstein (was 2.3s). All 1945 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compiler fixes:
- Upvalue re-lookup returns own position (uv-index), not parent slot
- Spec: cek-call uses (make-env) not (dict) — OCaml Dict≠Env
- Bootstrap post-processes transpiler Dict→Env for cek_call
VM runtime fixes:
- compile_adapter evaluates constant defines (SPECIAL_FORM_NAMES etc.)
via execute_module instead of wrapping as NativeFn closures
- Native primitives: map-indexed, some, every?
- Nil-safe HO forms: map/filter/for-each/some/every? accept nil as empty
- expand-components? set in kernel env (not just VM globals)
- unwrap_env diagnostic: reports actual type received
sx-page-full command:
- Single OCaml call: aser-slot body + render-to-html shell
- Eliminates two pipe round-trips (was: aser-slot→Python→shell render)
- Shell statics (component_defs, CSS, pages_sx) cached in Python,
injected into kernel once, referenced by symbol in per-request command
- Large blobs use placeholder tokens — Python splices post-render,
pipe transfers ~51KB instead of 2MB
Performance (warm):
- Server total: 0.55s (was ~2s)
- aser-slot VM: 0.3s, shell render: 0.01s, pipe: 0.06s
- kwargs computation: 0.000s (cached)
SX_STANDALONE mode for sx_docs dev (skips fragment fetches).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixed compile-define to skip :effects/:as keyword annotations between
the name and body. (define name :effects [render] (fn ...)) now
correctly compiles the fn body, not the :effects keyword.
Result: adapter-sx.sx compiles to 25 code objects, 4044 bytes of
bytecode. All 12 aser functions (aser, aser-call, aser-list,
aser-fragment, aser-expand-component, etc.) compile successfully.
40/40 VM tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added vm-compile command: iterates env, compiles lambdas to bytecode,
replaces with NativeFn VM wrappers (with CEK fallback on error).
Tested: 3/109 compile, reduces CEK steps 23%.
Disabled auto-compile in production — the compiler doesn't handle
closures with upvalues yet, and compiled functions that reference
dynamic env vars crash. Infrastructure stays for when compiler
handles all SX features.
Also: added set-nth! and mutable-list primitives (needed by
compiler.sx for bytecode patching). Fixed compiler.sx to use
mutable lists on OCaml (ListRef for append!/set-nth! mutation).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
End-to-end pipeline working:
Python compiler.sx → bytecode → OCaml VM → result
Verified: (+ (* 3 4) 2) → 14 ✓
(+ 0 1 2 ... 49) → 1225 ✓
Benchmark (500 iterations, 50 additions each):
CEK machine: 327ms
Bytecode VM: 145ms
Speedup: 2.2x
VM handles: constants, local variables, global variables,
primitive calls, jumps, conditionals, closures (via NativeFn
wrapper), define, return.
Protocol: (vm-exec {:bytecode (...) :constants (...)})
- Compiler outputs clean format (no internal index dict)
- VM converts bytecode list to int array, constants to value array
- Stack-based execution with direct opcode dispatch
The 2.2x speedup is for pure arithmetic. For aser (the real
target), the speedup will be larger because aser involves:
- String building (no CEK frame allocation in VM)
- Map/filter iterations (no frame-per-iteration in VM)
- Closure calls (no thunk/trampoline in VM)
Next: compile and run the aser adapter on the VM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new files forming the bytecode compilation pipeline:
spec/bytecode.sx — opcode definitions (~65 ops):
- Stack/constant ops (CONST, NIL, TRUE, POP, DUP)
- Lexical variable access (LOCAL_GET/SET, UPVALUE_GET/SET, GLOBAL_GET/SET)
- Jump-based control flow (JUMP, JUMP_IF_FALSE/TRUE)
- Function ops (CALL, TAIL_CALL, RETURN, CLOSURE, CALL_PRIM)
- HO form ops (ITER_INIT/NEXT, MAP_OPEN/APPEND/CLOSE)
- Scope/continuation ops (SCOPE_PUSH/POP, RESET, SHIFT)
- Aser specialization (ASER_TAG, ASER_FRAG)
spec/compiler.sx — SX-to-bytecode compiler (SX code, portable):
- Scope analysis: resolve variables to local/upvalue/global at compile time
- Tail position detection for TCO
- Code generation for: if, when, and, or, let, begin, lambda,
define, set!, quote, function calls, primitive calls
- Constant pool with deduplication
- Jump patching for forward references
hosts/ocaml/lib/sx_vm.ml — bytecode interpreter (OCaml):
- Stack-based VM with array-backed operand stack
- Call frames with base pointer for locals
- Direct opcode dispatch via pattern match
- Zero allocation per step (unlike CEK machine's dict-per-step)
- Handles: constants, variables, jumps, calls, primitives,
collections, string concat, define
Architecture: compiler.sx is spec (SX, portable). VM is platform
(OCaml-native). Same bytecode runs on JS/WASM VMs.
Also includes: CekFrame record optimization in transpiler.sx
(29 frame types as records instead of Hashtbl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
