Fix infinite recursion in VM JIT: restore sentinel pre-mark in vm_call
and pre-compile loop so recursive compiler functions don't trigger
unbounded compilation cascades. Runtime VM errors fall back to CEK;
compile errors surface visibly (not silently swallowed).
New: compile-quasiquote emits inline code instead of delegating to
qq-expand-runtime. Closure-captured variables merged into VM globals
so compiled closures resolve outer bindings via GLOBAL_GET.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The second get implementation in sx_runtime.ml (used by transpiled code)
was still raising on type mismatches. Now returns nil like sx_primitives.
Remove per-call [vm-call-closure] FAIL logging — the jit-hook already
logs failures at the right level. Reduces 70K log lines to ~5.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- get primitive returns nil for type mismatches (list+string) instead
of raising — matches JS/Python behavior, fixes find-nav-match errors
- scope-peek, collect!, collected, clear-collected! registered as real
primitives in sx_primitives table (not just env bindings) so the CEK
step-sf-context can find them via get-primitive
- step-sf-context checks scope-peek hashtable BEFORE walking CEK
continuation — bridges aser's scope-push!/pop! with CEK's context
- context, emit!, emitted added to SPECIAL_FORM_NAMES and handled in
aser-special (scope operations in aser rendering mode)
- sx-context NativeFn for VM-compiled code paths
- VM execution errors no longer mark functions as permanently failed —
bytecode is correct, errors are from runtime data
- kbd, samp, var added to HTML_TAGS + sx-browser.js rebuilt
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace AOT adapter compilation with lazy JIT — each named lambda is
compiled to VM bytecode on first call, cached in l_compiled field for
subsequent calls. Compilation failures fall back to CEK gracefully.
VM types (vm_code, vm_upvalue_cell, vm_closure) moved to sx_types.ml
mutual recursion block. Lambda and Component records gain mutable
l_compiled/c_compiled cache fields. jit_compile_lambda in sx_vm.ml
wraps body as (fn (params) body), invokes spec/compiler.sx via CEK,
extracts inner closure from OP_CLOSURE constant.
JIT hooks in both paths:
- vm_call: Lambda calls from compiled VM code
- continue_with_call: Lambda calls from CEK step loop (injected by
bootstrap.py post-processing)
Pre-mark sentinel prevents re-entrancy (compile function itself was
hanging when JIT'd mid-compilation). VM execution errors caught and
fall back to CEK with sentinel marking.
Also: add kbd/samp/var to HTML_TAGS, rebuild sx-browser.js, add page
URL to sx-page-full-py timing log.
Performance: first page 28s (JIT compiles 17 functions), subsequent
pages 0.31s home / 0.71s wittgenstein (was 2.3s). All 1945 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of for-each failure: CALL_PRIM checked globals before
primitives. Globals had ho_via_cek wrappers that routed for-each
through the CEK machine — which couldn't call VM closures correctly.
Fix: check Sx_primitives.get_primitive FIRST (native call_any that
handles NativeFn directly), fall back to globals for env-specific
bindings like set-render-active!.
Result: (for-each (fn (x) (+ x 1)) (list 1 2 3)) on VM → 42 ✓
Full adapter aser chain executing:
aser → aser-list → aser-call → for-each callback
Fails at UPVALUE_GET idx=6 (have 6) — compiler upvalue count
off by one. Next fix: compiler scope analysis.
Also: floor(0)=-1 bug found and fixed (was round(x-0.5), now
uses OCaml's native floor). This was causing all compile failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aser-slot now routes through the VM when adapter is compiled:
- compile_adapter: compiles each define body, extracts inner code
from OP_CLOSURE wrapper, stores as NativeFn in separate globals
- vm_adapter_globals: isolated from kernel env (no cross-contamination)
- aser-slot checks vm_adapter_globals, calls VM aser directly
Status: 2/12 adapter functions compile and run on VM. 6 fail during
OCaml-side compilation with "index out of bounds" — likely from
set-nth! silent failure on ListRef during bytecode jump patching.
Debug output shows outer code structure is correct (4 bytes, 1 const).
Inner code_from_value conversion needs fixing for nested closures.
Also: vm-compile-adapter command inside _ensure_components lock
(fixes pipe desync from concurrent requests).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
adapter-sx.sx compiles to 25 code objects (4044 bytes bytecode).
vm-load-module loads it. But replacing Lambda values in env.bindings
with NativeFn wrappers breaks the CEK machine for non-aser functions.
Root cause: shared env.bindings between CEK and VM. The CEK needs
Lambda values (for closure merging). The VM needs NativeFn wrappers.
Both can't coexist in the same env.
Fix needed: VM adapter gets its own globals table (with compiled
closures). The aser-slot command routes directly to the VM with
its own globals, not through the CEK with shared env.
Disabled vm-load-module. Pages render correctly via CEK.
Also: OP_CALL_PRIM now logs primitive name + argc in error messages
for easier debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes:
1. HO forms (map/filter/for-each/reduce): registered as Python
primitives so compiler emits OP_CALL_PRIM (direct dispatch to
OCaml primitive) instead of OP_CALL (which routed through CEK
HO special forms and failed on NativeFn closure args).
2. Mutable closures: locals captured by closures now share an
upvalue_cell. OP_LOCAL_GET/SET check frame.local_cells first —
if the slot has a shared cell, read/write through it. OP_CLOSURE
creates or reuses cells for is_local=1 captures. Both parent
and closure see the same mutations.
Frame type extended with local_cells hashtable for captured slots.
40/40 tests pass:
- 12 compiler output tests
- 18 VM execution tests (arithmetic, control flow, closures,
nested let, higher-order, cond, string ops)
- 10 auto-compile pattern tests (recursive, map, filter,
for-each, mutable closures, multiple closures, type dispatch)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
End-to-end pipeline working:
Python compiler.sx → bytecode → OCaml VM → result
Verified: (+ (* 3 4) 2) → 14 ✓
(+ 0 1 2 ... 49) → 1225 ✓
Benchmark (500 iterations, 50 additions each):
CEK machine: 327ms
Bytecode VM: 145ms
Speedup: 2.2x
VM handles: constants, local variables, global variables,
primitive calls, jumps, conditionals, closures (via NativeFn
wrapper), define, return.
Protocol: (vm-exec {:bytecode (...) :constants (...)})
- Compiler outputs clean format (no internal index dict)
- VM converts bytecode list to int array, constants to value array
- Stack-based execution with direct opcode dispatch
The 2.2x speedup is for pure arithmetic. For aser (the real
target), the speedup will be larger because aser involves:
- String building (no CEK frame allocation in VM)
- Map/filter iterations (no frame-per-iteration in VM)
- Closure calls (no thunk/trampoline in VM)
Next: compile and run the aser adapter on the VM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new files forming the bytecode compilation pipeline:
spec/bytecode.sx — opcode definitions (~65 ops):
- Stack/constant ops (CONST, NIL, TRUE, POP, DUP)
- Lexical variable access (LOCAL_GET/SET, UPVALUE_GET/SET, GLOBAL_GET/SET)
- Jump-based control flow (JUMP, JUMP_IF_FALSE/TRUE)
- Function ops (CALL, TAIL_CALL, RETURN, CLOSURE, CALL_PRIM)
- HO form ops (ITER_INIT/NEXT, MAP_OPEN/APPEND/CLOSE)
- Scope/continuation ops (SCOPE_PUSH/POP, RESET, SHIFT)
- Aser specialization (ASER_TAG, ASER_FRAG)
spec/compiler.sx — SX-to-bytecode compiler (SX code, portable):
- Scope analysis: resolve variables to local/upvalue/global at compile time
- Tail position detection for TCO
- Code generation for: if, when, and, or, let, begin, lambda,
define, set!, quote, function calls, primitive calls
- Constant pool with deduplication
- Jump patching for forward references
hosts/ocaml/lib/sx_vm.ml — bytecode interpreter (OCaml):
- Stack-based VM with array-backed operand stack
- Call frames with base pointer for locals
- Direct opcode dispatch via pattern match
- Zero allocation per step (unlike CEK machine's dict-per-step)
- Handles: constants, variables, jumps, calls, primitives,
collections, string concat, define
Architecture: compiler.sx is spec (SX, portable). VM is platform
(OCaml-native). Same bytecode runs on JS/WASM VMs.
Also includes: CekFrame record optimization in transpiler.sx
(29 frame types as records instead of Hashtbl).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
