Three bugs broke island SSR rendering of the home stepper widget:
1. Inline VM opcodes (OP_ADD..OP_DEC) broke JIT-compiled functions.
The compiler emitted single-byte opcodes for first/rest/len/= etc.
that produced wrong results in complex recursive code (sx-parse
returned nil, split-tag produced 1 step instead of 16). Reverted
compiler to use CALL_PRIM for all primitives. VM opcode handlers
kept for future use.
2. Named let (let loop ((x init)) body) had no compiler support —
silently produced broken bytecode. Added desugaring to letrec.
3. URL-encoded cookie values not decoded server-side. Client set-cookie
uses encodeURIComponent but Werkzeug doesn't decode cookie values.
Added unquote() in bridge cookie injection.
Also: call-lambda used eval_expr which copies Dict values (signals),
breaking mutations through aser lambda calls. Switched to cek_call.
Also: stepper preview now includes ~cssx/tw spreads for SSR styling.
Tests: 1317 JS, 1114 OCaml, 26 integration (2 pre-existing failures)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
16 new opcodes (160-175) bypass the CALL_PRIM hashtable lookup for
the most frequently called primitives:
Arithmetic: OP_ADD, OP_SUB, OP_MUL, OP_DIV, OP_INC, OP_DEC, OP_NEG
Comparison: OP_EQ, OP_LT, OP_GT, OP_NOT
Collection: OP_LEN, OP_FIRST, OP_REST, OP_NTH, OP_CONS
The compiler (compiler.sx) recognizes these names at compile time and
emits the inline opcode instead of CALL_PRIM. The opcode is self-
contained — no constant pool index, no argc byte. Each primitive is
a single byte in the bytecode stream.
Implementation in all three VMs:
- OCaml (sx_vm.ml): direct pattern match, no allocation
- SX spec (vm.sx): delegates to existing primitives
- JS (transpiled): same as SX spec
66 new tests in spec/tests/vm-inline.sx covering arithmetic, comparison,
collection ops, composition, and edge cases.
Tests: 1314 JS (full), 1114 OCaml, 32 Playwright
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
primitive? in make_server_env was checking env bindings only (NativeFn),
missing all 132 primitives in the Sx_primitives hashtable. Now checks
both primitives table and env. get-primitive similarly fixed.
replace primitive now coerces SxExpr/Thunk/RawHTML/etc to strings instead
of crashing with "replace: 3 string args" — fixes aser JIT DISABLED.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Route all rendering through OCaml bridge — render_to_html no longer uses
Python async_eval. Fix register_components to parse &key params and &rest
children from defcomp forms. Remove all dead sx_ref.py imports.
Epoch protocol (prevents pipe desync):
- Every command prefixed with (epoch N), all responses tagged with epoch
- Both sides discard stale-epoch messages — desync structurally impossible
- OCaml main loop discards stale io-responses between commands
Consolidate scope primitives into sx_scope.ml:
- Single source of truth for scope-push!/pop!/peek, collect!/collected,
emit!/emitted, context, and 12 other scope operations
- Removes duplicate registrations from sx_server.ml (including bugs where
scope-emit! and clear-collected! were registered twice with different impls)
- Bind scope prims into env so JIT VM finds them via OP_GLOBAL_GET
JIT VM fixes:
- Trampoline thunks before passing args to CALL_PRIM
- as_list resolves thunks via _sx_trampoline_fn
- len handles all value types (Bool, Number, RawHTML, SxExpr, Spread, etc.)
Other fixes:
- ~cssx/tw signature: (tokens) → (&key tokens) to match callers
- Minimal Python evaluator in html.py for sync sx() Jinja function
- Python scope primitive stubs (thread-local) for non-OCaml paths
- Reader macro resolution via OcamlSync instead of sx_ref.py
Tests: 1114 OCaml, 1078 JS, 35 Python regression, 6/6 Playwright SSR
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removes the 5993-line bootstrapped Python evaluator (sx_ref.py) and all
code that depended on it exclusively. Both bootstrappers (JS + OCaml)
now use a new synchronous OCaml bridge (ocaml_sync.py) to run the
transpiler. JS build produces identical output; OCaml bootstrap produces
byte-identical sx_ref.ml.
Key changes:
- New shared/sx/ocaml_sync.py: sync subprocess bridge to sx_server.exe
- hosts/javascript/bootstrap.py: serialize defines → temp file → OCaml eval
- hosts/ocaml/bootstrap.py: same pattern for OCaml transpiler
- shared/sx/{html,async_eval,resolver,jinja_bridge,handlers,pages,deps,helpers}:
stub or remove sx_ref imports; runtime uses OCaml bridge (SX_USE_OCAML=1)
- sx/sxc/pages: parse defpage/defhandler from AST instead of Python eval
- hosts/ocaml/lib/sx_primitives.ml: append handles non-list 2nd arg per spec
- Deleted: sx_ref.py, async_eval_ref.py, 6 Python test runners, misc ref/ files
Test results: JS 1078/1078, OCaml 1114/1114.
sx_docs SSR has pre-existing rendering issues to investigate separately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The test runner's component-body/component-params/component-has-children
bindings only handled Component values, not Island. When adapter-html.sx
called (component-body island), it hit the fallback and returned nil,
producing empty island bodies. Also removed debug logging from
component-has-children? primitive.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SX lambdas ((fn (x) body)) now transpile to NativeFn values that can
be stored as SX values — passed to signal-add-sub!, stored in dicts,
used as reactive subscribers. Previously emitted as bare OCaml closures
which couldn't be stored in the SX value type system.
ml-emit-fn → NativeFn("λ", fun args -> match args with [...] -> body)
ml-emit-fn-bare → (fun params -> body) — used by HO inliners and
recursive let bindings (let rec) which call themselves directly.
HO forms (map, filter, reduce, for-each, map-indexed, map-dict) use
cek_call for non-inline function arguments, bare OCaml lambdas for
inline (fn ...) arguments.
Runtime: with_island_scope accepts NativeFn values (pattern match on
value type) since transpiled lambdas are now NativeFn-wrapped.
Unblocks WASM reactive signals — the bootstrap FIXUPS that manually
wrapped reactive_shift_deref's subscriber as NativeFn are no longer
needed when merging to the wasm branch.
1314/1314 JS tests, 4/4 Playwright isomorphic tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace permissive is_symbol_char (negative check — everything not a
delimiter) with spec-compliant is_ident_start/is_ident_char (positive
check matching the exact character sets documented in parser.sx).
Changes:
- ident-start: remove extra chars (|, %, ^, $) not in spec
- ident-char: add comma (,) per spec
- Comma (,) now handled as dedicated unquote case in match, not in
the catch-all fallback — matches spec dispatch order
- Remove ~@ splice-unquote alias (spec only defines ,@)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root causes of missing CSSX classes in SSR:
1. _sx_trampoline_fn in sx_primitives.ml was never wired — call_any in
HO forms (map/filter/for-each) returned unresolved Thunks, so callbacks
like render-lambda-html's param binding never executed. Fixed in
bootstrap.py FIXUPS: wire Sx_primitives._sx_trampoline_fn after eval_expr.
2. adapter-html.sx used (emit! ...) and (emitted ...) which are CEK special
forms (walk kont for ScopeAccFrame), but scope-push!/scope-pop! use the
hashtable. CEK frames and hashtable are two different scope systems.
Fixed: adapter uses scope-emit!/scope-emitted (hashtable primitives).
3. env-* operations (env-has?, env-get, env-bind!, env-set!, env-extend,
env-merge) only accepted Env type. adapter-html.sx passes Dict as env.
Fixed: all env ops go through unwrap_env which handles Dict/Nil.
Also: fix merge conflict in sx/sx/geography/index.sx, remove duplicate
scope primitives from sx_primitives.ml (sx_server.ml registers them).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sx_runtime.ml: unwrap_env now accepts Dict and Nil (converts to Env),
fixing env-merge when adapter-html.sx passes dict-as-env.
sx_server.ml + run_tests.ml: env-merge bindings use Sx_runtime.env_merge
(which handles Dict/Nil) instead of requiring strict Env pattern match.
sx_primitives.ml: Added scope stack (scope-push!/pop!/peek/emit!, emitted),
type predicates (lambda?/island?/component?/macro?), component accessors
(closure/name/params/body/has-children?), lambda accessors, for-each-indexed,
empty-dict?, make-raw-html, raw-html-content, is-else-clause?.
8 OCaml render tests still fail (env propagation in render-lambda-html) —
same adapter code works in JS and in production via Python bridge.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
JIT allowlist (sx_server.ml):
- Replace try-every-lambda strategy with StringSet allowlist. Only
functions in the list get JIT compiled (compiler, parser, pure transforms).
Render functions that need dynamic scope skip JIT entirely — no retry
overhead, no silent fallbacks.
- Add (jit-allow name) command for dynamic expansion from Python bridge.
- JIT failures log once with "[jit] DISABLED fn — reason" then go silent.
Standalone --test mode (sx_server.ml):
- New --test flag loads full env (spec + adapters + compiler + signals),
supports --eval and --load flags. Quick kernel testing without Docker.
Example: dune exec bin/sx_server.exe -- --test --eval '(len HTML_TAGS)'
Integration tests (integration_tests.ml):
- New binary exercising the full rendering pipeline: loads spec + adapters
into a server-like env, renders HTML via both native and SX adapter paths.
- 26 tests: HTML tags, special forms (when/if/let), letrec with side
effects, component rendering, eval-expr with HTML tag functions.
- Would have caught the "Undefined symbol: div/lake/init" issues from
the previous commit immediately without Docker.
VM cleanup (sx_vm.ml):
- Remove temporary debug logging (insn counter, call_closure counter,
VmClosure depth tracking) added during debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Core VM changes:
- Add VmClosure value variant — inner closures created by OP_CLOSURE are
first-class VM values, not NativeFn wrappers around call_closure
- Convert `run` from recursive to while-loop — zero OCaml stack growth,
true TCO for VmClosure tail calls
- vm_call handles VmClosure by pushing frame on current VM (no new VM
allocation per call)
- Forward ref _vm_call_closure_ref for cross-boundary calls (CEK/primitives)
Compiler (spec/compiler.sx):
- Define hoisting in compile-begin: pre-allocate local slots for all
define forms before compiling any values. Fixes forward references
between inner functions (e.g. read-expr referencing skip-ws in sx-parse)
- scope-define-local made idempotent (skip if slot already exists)
Server (sx_server.ml):
- JIT fail-once sentinel: mark l_compiled as failed after first VM runtime
error. Eliminates thousands of retry attempts per page render.
- HTML tag bindings: register all HTML tags as pass-through NativeFns so
eval-expr can handle (div ...) etc. in island component bodies.
- Log VM FAIL errors with function name before disabling JIT.
SSR fixes:
- adapter-html.sx letrec handler: evaluate bindings in proper letrec scope
(pre-bind nil, then evaluate), render body with render-to-html instead of
eval-expr. Fixes island SSR for components using letrec.
- Add `init` primitive to OCaml kernel (all-but-last of list).
- VmClosure handling in sx_runtime.ml sx_call dispatch.
Tests: 971/971 OCaml (+19 new), 0 failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
vm_closure now stores the original closure env (vm_closure_env).
GLOBAL_GET walks the closure env chain when the variable isn't in
vm.globals. GLOBAL_SET writes to the correct env in the chain.
This enables JIT compilation of all named functions regardless of
closure depth. No more closure skip check needed.
Pre-compile time back to ~7s (was 37s with closure skip).
Note: sx-parse sibling list parsing still has issues — the root
cause is in how the JIT-compiled letrec + OP_CLOSURE interacts
with the upvalue cell mechanism. Investigation ongoing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: sf-letrec returns a thunk (for TCO) but the CEK dispatch
wrapped it as a value without evaluating. The thunk leaked as the
return value of letrec expressions, breaking sx-parse and any function
using letrec.
Fix: step-sf-letrec unwraps the thunk into a CEK state, so the last
letrec body expression is properly evaluated by the CEK machine.
Also:
- compile-letrec: two-phase (nil-init then assign) for mutual recursion
- Skip JIT for inner functions (closure.bindings != globals) in both
vm_call and JIT hook
- vm-reset-fn for sx-parse removed (no longer needed)
- Parser regression test: letrec with mutable pos + recursive sublists
Test results: JS 943/17, OCaml 955/0, Python 747/0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The closure merging in jit_compile_lambda (copying globals + injecting
closure bindings into vm_env_ref) produces incorrect variable resolution
for inner functions. Symptoms: sx-parse's read-list-loop mishandles
closing parens (siblings become children), parser produces wrong ASTs.
Fix: vm_call skips JIT compilation for lambdas with non-empty closures.
These run on CEK which handles closures correctly. Top-level defines
(empty closure) are still JIT-compiled.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The CSSX colour resolution failure was NOT a JIT compiler bug.
CALL_PRIM looks up primitives table (not env), and parse-int in
the primitives table only handled 1-arg calls. The 2-arg form
(parse-int "699" nil) returned Nil, causing cssx-resolve's colour
branch to fail its and-condition.
Fix: update Sx_primitives.register "parse-int" with same 2-arg
handling as the env binding. Remove the vm-reset-fn workaround.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Islands now render their initial state as HTML on the server, like
React SSR. The client hydrates with reactive behavior on boot.
Root causes fixed:
- is_signal/signal_value now recognize Dict-based signals (from
signals.sx) in addition to native Signal values
- Register "context" as a primitive so the CEK deref frame handler
can read scope stacks for reactive tracking
- Load adapter-html.sx into kernel for SX-level render-to-html
(islands use this instead of the OCaml render module)
- Component accessors (params, body, has-children?, affinity) handle
Island values with ? suffix aliases
- Add platform primitives: make-raw-html, raw-html-content,
empty-dict?, for-each-indexed, cek-call
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Server now renders page content as HTML inside <div id="sx-root">,
visible immediately before JavaScript loads. The SX source is still
included in a <script data-mount="#sx-root"> tag for client hydration.
SSR pipeline: after aser produces the SX wire format, parse and
render-to-html it (~17ms for a 22KB page). Islands with reactive
state gracefully fall back to empty — client hydrates them.
Supporting changes:
- Load signals.sx into OCaml kernel (reactive primitives for island SSR)
- Add cek-call and context to kernel env (needed by signals/deref)
- Island-aware component accessors in sx_types.ml
- render-to-html handles Island values (renders as component with fallback)
- Fix 431 (Request Header Fields Too Large): replace SX-Components
header (full component name list) with SX-Components-Hash (12 chars)
- CORS allow SX-Components-Hash header
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix infinite recursion in VM JIT: restore sentinel pre-mark in vm_call
and pre-compile loop so recursive compiler functions don't trigger
unbounded compilation cascades. Runtime VM errors fall back to CEK;
compile errors surface visibly (not silently swallowed).
New: compile-quasiquote emits inline code instead of delegating to
qq-expand-runtime. Closure-captured variables merged into VM globals
so compiled closures resolve outer bindings via GLOBAL_GET.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The second get implementation in sx_runtime.ml (used by transpiled code)
was still raising on type mismatches. Now returns nil like sx_primitives.
Remove per-call [vm-call-closure] FAIL logging — the jit-hook already
logs failures at the right level. Reduces 70K log lines to ~5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace continuation-based scope frames with hashtable stacks for all
scope operations. The CEK evaluator's scope/provide/context/emit!/emitted
now use scope-push!/pop!/peek/emit! primitives (registered in
sx_primitives table) instead of walking continuation frames.
This eliminates the two-world problem where the aser used hashtable
stacks (scope-push!/pop!) but eval-expr used continuation frames
(ScopeFrame/ScopeAccFrame). Now both paths share the same mechanism.
Benefits:
- scope/context works inside eval-expr calls (e.g. (str ... (context x)))
- O(1) scope lookup vs O(n) continuation walking
- Simpler — no ScopeFrame/ScopeAccFrame/ProvideFrame creation/dispatch
- VM-compiled code and CEK code both see the same scope state
Also registers scope-push!/pop!/peek/emit!/collect!/collected/
clear-collected! as real primitives (sx_primitives table) so the
transpiled evaluator can call them directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- get primitive returns nil for type mismatches (list+string) instead
of raising — matches JS/Python behavior, fixes find-nav-match errors
- scope-peek, collect!, collected, clear-collected! registered as real
primitives in sx_primitives table (not just env bindings) so the CEK
step-sf-context can find them via get-primitive
- step-sf-context checks scope-peek hashtable BEFORE walking CEK
continuation — bridges aser's scope-push!/pop! with CEK's context
- context, emit!, emitted added to SPECIAL_FORM_NAMES and handled in
aser-special (scope operations in aser rendering mode)
- sx-context NativeFn for VM-compiled code paths
- VM execution errors no longer mark functions as permanently failed —
bytecode is correct, errors are from runtime data
- kbd, samp, var added to HTML_TAGS + sx-browser.js rebuilt
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace AOT adapter compilation with lazy JIT — each named lambda is
compiled to VM bytecode on first call, cached in l_compiled field for
subsequent calls. Compilation failures fall back to CEK gracefully.
VM types (vm_code, vm_upvalue_cell, vm_closure) moved to sx_types.ml
mutual recursion block. Lambda and Component records gain mutable
l_compiled/c_compiled cache fields. jit_compile_lambda in sx_vm.ml
wraps body as (fn (params) body), invokes spec/compiler.sx via CEK,
extracts inner closure from OP_CLOSURE constant.
JIT hooks in both paths:
- vm_call: Lambda calls from compiled VM code
- continue_with_call: Lambda calls from CEK step loop (injected by
bootstrap.py post-processing)
Pre-mark sentinel prevents re-entrancy (compile function itself was
hanging when JIT'd mid-compilation). VM execution errors caught and
fall back to CEK with sentinel marking.
Also: add kbd/samp/var to HTML_TAGS, rebuild sx-browser.js, add page
URL to sx-page-full-py timing log.
Performance: first page 28s (JIT compiles 17 functions), subsequent
pages 0.31s home / 0.71s wittgenstein (was 2.3s). All 1945 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- sx_render.ml: add raw! handler to HTML renderer (inject pre-rendered
content without HTML escaping)
- docker-compose.yml: move SX_USE_OCAML/SX_OCAML_BIN to shared env
(available to all services, not just sx_docs)
- hosts/ocaml/Dockerfile: OCaml kernel build stage
- shared/sx/tests/: golden test data + generator for OCaml render tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Aser serialization: aser-call/fragment now return SxExpr instead of String.
serialize/inspect passes SxExpr through unquoted, preventing the double-
escaping (\" → \\\" ) that broke client-side parsing when aser wire format
was output via raw! into <script> tags. Added make-sx-expr + sx-expr-source
primitives to OCaml and JS hosts.
Binary blob protocol: eval, aser, aser-slot, and sx-page-full now send SX
source as length-prefixed blobs instead of escaped strings. Eliminates pipe
desync from concurrent requests and removes all string-escape round-trips
between Python and OCaml.
Bridge safety: re-entrancy guard (_in_io_handler) raises immediately if an
IO handler tries to call the bridge, preventing silent deadlocks.
Fetch error logging: orchestration.sx error callback now logs method + URL
via log-warn. Platform catches (fetchAndRestore, fetchPreload, bindBoostForm)
also log errors instead of silently swallowing them.
Transpiler fixes: makeEnv, scopePeek, scopeEmit, makeSxExpr added as
platform function definitions + transpiler mappings — were referenced in
transpiled code but never defined as JS functions.
Playwright test infrastructure:
- nav() captures JS errors and fails fast with the actual error message
- Checks for [object Object] rendering artifacts
- New tests: delete-row interaction, full page refresh, back button,
direct load with fresh context, code block content verification
- Default base URL changed to localhost:8013 (standalone dev server)
- docker-compose.dev-sx.yml: port 8013 exposed for local testing
- test-sx-build.sh: build + unit tests + Playwright smoke tests
Geography content: index page component written (sx/sx/geography/index.sx)
describing OCaml evaluator, wire formats, rendering pipeline, and topic
links. Wiring blocked by aser-expand-component children passing issue.
Tests: 1080/1080 JS, 952/952 OCaml, 66/66 Playwright
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compiler fixes:
- Upvalue re-lookup returns own position (uv-index), not parent slot
- Spec: cek-call uses (make-env) not (dict) — OCaml Dict≠Env
- Bootstrap post-processes transpiler Dict→Env for cek_call
VM runtime fixes:
- compile_adapter evaluates constant defines (SPECIAL_FORM_NAMES etc.)
via execute_module instead of wrapping as NativeFn closures
- Native primitives: map-indexed, some, every?
- Nil-safe HO forms: map/filter/for-each/some/every? accept nil as empty
- expand-components? set in kernel env (not just VM globals)
- unwrap_env diagnostic: reports actual type received
sx-page-full command:
- Single OCaml call: aser-slot body + render-to-html shell
- Eliminates two pipe round-trips (was: aser-slot→Python→shell render)
- Shell statics (component_defs, CSS, pages_sx) cached in Python,
injected into kernel once, referenced by symbol in per-request command
- Large blobs use placeholder tokens — Python splices post-render,
pipe transfers ~51KB instead of 2MB
Performance (warm):
- Server total: 0.55s (was ~2s)
- aser-slot VM: 0.3s, shell render: 0.01s, pipe: 0.06s
- kwargs computation: 0.000s (cached)
SX_STANDALONE mode for sx_docs dev (skips fragment fetches).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of for-each failure: CALL_PRIM checked globals before
primitives. Globals had ho_via_cek wrappers that routed for-each
through the CEK machine — which couldn't call VM closures correctly.
Fix: check Sx_primitives.get_primitive FIRST (native call_any that
handles NativeFn directly), fall back to globals for env-specific
bindings like set-render-active!.
Result: (for-each (fn (x) (+ x 1)) (list 1 2 3)) on VM → 42 ✓
Full adapter aser chain executing:
aser → aser-list → aser-call → for-each callback
Fails at UPVALUE_GET idx=6 (have 6) — compiler upvalue count
off by one. Next fix: compiler scope analysis.
Also: floor(0)=-1 bug found and fixed (was round(x-0.5), now
uses OCaml's native floor). This was causing all compile failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aser-slot now routes through the VM when adapter is compiled:
- compile_adapter: compiles each define body, extracts inner code
from OP_CLOSURE wrapper, stores as NativeFn in separate globals
- vm_adapter_globals: isolated from kernel env (no cross-contamination)
- aser-slot checks vm_adapter_globals, calls VM aser directly
Status: 2/12 adapter functions compile and run on VM. 6 fail during
OCaml-side compilation with "index out of bounds" — likely from
set-nth! silent failure on ListRef during bytecode jump patching.
Debug output shows outer code structure is correct (4 bytes, 1 const).
Inner code_from_value conversion needs fixing for nested closures.
Also: vm-compile-adapter command inside _ensure_components lock
(fixes pipe desync from concurrent requests).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
adapter-sx.sx compiles to 25 code objects (4044 bytes bytecode).
vm-load-module loads it. But replacing Lambda values in env.bindings
with NativeFn wrappers breaks the CEK machine for non-aser functions.
Root cause: shared env.bindings between CEK and VM. The CEK needs
Lambda values (for closure merging). The VM needs NativeFn wrappers.
Both can't coexist in the same env.
Fix needed: VM adapter gets its own globals table (with compiled
closures). The aser-slot command routes directly to the VM with
its own globals, not through the CEK with shared env.
Disabled vm-load-module. Pages render correctly via CEK.
Also: OP_CALL_PRIM now logs primitive name + argc in error messages
for easier debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes:
1. HO forms (map/filter/for-each/reduce): registered as Python
primitives so compiler emits OP_CALL_PRIM (direct dispatch to
OCaml primitive) instead of OP_CALL (which routed through CEK
HO special forms and failed on NativeFn closure args).
2. Mutable closures: locals captured by closures now share an
upvalue_cell. OP_LOCAL_GET/SET check frame.local_cells first —
if the slot has a shared cell, read/write through it. OP_CLOSURE
creates or reuses cells for is_local=1 captures. Both parent
and closure see the same mutations.
Frame type extended with local_cells hashtable for captured slots.
40/40 tests pass:
- 12 compiler output tests
- 18 VM execution tests (arithmetic, control flow, closures,
nested let, higher-order, cond, string ops)
- 10 auto-compile pattern tests (recursive, map, filter,
for-each, mutable closures, multiple closures, type dispatch)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added vm-compile command: iterates env, compiles lambdas to bytecode,
replaces with NativeFn VM wrappers (with CEK fallback on error).
Tested: 3/109 compile, reduces CEK steps 23%.
Disabled auto-compile in production — the compiler doesn't handle
closures with upvalues yet, and compiled functions that reference
dynamic env vars crash. Infrastructure stays for when compiler
handles all SX features.
Also: added set-nth! and mutable-list primitives (needed by
compiler.sx for bytecode patching). Fixed compiler.sx to use
mutable lists on OCaml (ListRef for append!/set-nth! mutation).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
End-to-end pipeline working:
Python compiler.sx → bytecode → OCaml VM → result
Verified: (+ (* 3 4) 2) → 14 ✓
(+ 0 1 2 ... 49) → 1225 ✓
Benchmark (500 iterations, 50 additions each):
CEK machine: 327ms
Bytecode VM: 145ms
Speedup: 2.2x
VM handles: constants, local variables, global variables,
primitive calls, jumps, conditionals, closures (via NativeFn
wrapper), define, return.
Protocol: (vm-exec {:bytecode (...) :constants (...)})
- Compiler outputs clean format (no internal index dict)
- VM converts bytecode list to int array, constants to value array
- Stack-based execution with direct opcode dispatch
The 2.2x speedup is for pure arithmetic. For aser (the real
target), the speedup will be larger because aser involves:
- String building (no CEK frame allocation in VM)
- Map/filter iterations (no frame-per-iteration in VM)
- Closure calls (no thunk/trampoline in VM)
Next: compile and run the aser adapter on the VM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new files forming the bytecode compilation pipeline:
spec/bytecode.sx — opcode definitions (~65 ops):
- Stack/constant ops (CONST, NIL, TRUE, POP, DUP)
- Lexical variable access (LOCAL_GET/SET, UPVALUE_GET/SET, GLOBAL_GET/SET)
- Jump-based control flow (JUMP, JUMP_IF_FALSE/TRUE)
- Function ops (CALL, TAIL_CALL, RETURN, CLOSURE, CALL_PRIM)
- HO form ops (ITER_INIT/NEXT, MAP_OPEN/APPEND/CLOSE)
- Scope/continuation ops (SCOPE_PUSH/POP, RESET, SHIFT)
- Aser specialization (ASER_TAG, ASER_FRAG)
spec/compiler.sx — SX-to-bytecode compiler (SX code, portable):
- Scope analysis: resolve variables to local/upvalue/global at compile time
- Tail position detection for TCO
- Code generation for: if, when, and, or, let, begin, lambda,
define, set!, quote, function calls, primitive calls
- Constant pool with deduplication
- Jump patching for forward references
hosts/ocaml/lib/sx_vm.ml — bytecode interpreter (OCaml):
- Stack-based VM with array-backed operand stack
- Call frames with base pointer for locals
- Direct opcode dispatch via pattern match
- Zero allocation per step (unlike CEK machine's dict-per-step)
- Handles: constants, variables, jumps, calls, primitives,
collections, string concat, define
Architecture: compiler.sx is spec (SX, portable). VM is platform
(OCaml-native). Same bytecode runs on JS/WASM VMs.
Also includes: CekFrame record optimization in transpiler.sx
(29 frame types as records instead of Hashtbl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Transpiler detects dict literals with a "type" string field and emits
CekFrame records instead of Dict(Hashtbl). Maps frame-specific fields
to generic record slots:
cf_type, cf_env, cf_name, cf_body, cf_remaining, cf_f,
cf_args (also evaled), cf_results (also raw-args),
cf_extra (ho-type/scheme/indexed/match-val/current-item/...),
cf_extra2 (emitted/effect-list/first-render)
Runtime get_val handles CekFrame with direct field match — O(1)
field access vs Hashtbl.find.
Bootstrapper: skip stdlib.sx entirely (already OCaml primitives).
Result: 29 CekFrame + 2 CekState = 31 record types, only 8
Hashtbl.create remaining (effect-annotations, empty dicts).
Benchmark (200 divs): 2.94s → 1.71s (1.7x speedup from baseline).
Real pages: ~same as CekState-only (frames are <20% of allocations;
states dominate at 199K/page).
Foundation for JIT: record-based value representation enables
typed compilation — JIT can emit direct field access instead of
hash table lookups.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Transpiler (transpiler.sx): detects CEK state dict literals (5 fields:
control/env/kont/phase/value) and emits CekState OCaml record instead
of Dict(Hashtbl). Eliminates 200K Hashtbl allocations per page.
Bootstrapper: skip stdlib.sx (functions already registered as OCaml
primitives). Only transpile evaluator.sx.
Runtime: get_val handles CekState with direct field access. type_of
returns "dict" for CekState (backward compat).
Profiling results (root cause of slowness):
Pure eval: OCaml 1.6x FASTER than Python (expected)
Aser: OCaml 28x SLOWER than Python (unexpected!)
Root cause: Python has a native optimized aser. OCaml runs the SX
adapter-sx.sx through the CEK machine — each aserCall is ~50 CEK
steps with closures, scope operations, string building.
Fix needed: native OCaml aser (like Python's), not SX adapter
through CEK machine.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spec-introspect.sx: pure SX functions that read, parse, and analyze
spec files. No Python. The spec IS data — a macro transforms it into
explorer UI components.
- spec-explore: reads spec file via IO, parses with sx-parse, extracts
sections/defines/effects/params, produces explorer data dict
- spec-form-name/kind/effects/params/source: individual extractors
- spec-group-sections: groups defines into sections
- spec-compute-stats: aggregate effect/define counts
OCaml kernel fixes:
- nth handles strings (character indexing for parser)
- ident-start?, ident-char?, char-numeric?, parse-number: platform
primitives needed by spec/parser.sx when loaded at runtime
- _find_spec_file: searches spec/, web/, shared/sx/ref/ for spec files
83/84 Playwright tests pass. The 1 failure is client-side re-rendering
of the spec explorer (the client evaluates defpage content which calls
find-spec — unavailable on the client).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SX-to-OCaml transpiler (transpiler.sx) generates sx_ref.ml (~90KB, ~135
mutually recursive functions) from the spec evaluator. Foundation tests
all pass: parser, primitives, env operations, type system.
Key design decisions:
- Env variant added to value type for CEK state dict storage
- Continuation carries optional data dict for captured frames
- Dynamic var tracking distinguishes OCaml fn calls from SX value dispatch
- Single let rec...and block for forward references between all defines
- Unused ref pre-declarations eliminated via let-bound name detection
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
