Native functions (NativeFn/VmClosure) called through the CEK evaluator
can now have their Eval_errors caught by guard/handler-bind. The fix is
at the exact OCaml↔CEK boundary in continue-with-call:
- sx_runtime.ml: sx_apply_cek wraps native calls, returns error marker
dict {__eval_error__: true, message: "..."} instead of raising
- sx_runtime.ml: is_eval_error predicate checks for the marker
- spec/evaluator.sx: continue-with-call callable branch uses apply-cek,
detects error markers, converts to raise-eval CEK state
- transpiler.sx: apply-cek and eval-error? emit cases added
No mutable flags, no re-entry risk. Errors flow through the CEK handler
chain naturally. 2798/2800 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The transpiler's append! emit path didn't check ml-is-mutable-global?,
so (append! *provide-batch-queue* sub) wrote to a dead local variable
instead of the global _ref. This caused the combined test suite hang —
fire-provide-subscribers was silently broken before the local-ref shadow
removal, and now correctly modifies the global batch queue.
Also adds run_with_io error-to-raise conversion (kont_has_handler guard)
so native Eval_errors can be caught by CEK guard/handler-bind when running
through the test runner's IO-aware step loop.
2798/2800 tests pass. 2 foreign-type-checking failures remain: guard can't
catch Eval_error from native fns called through cek_run_iterative (the
handler dispatch itself uses cek_call which re-enters cek_run_iterative,
creating an infinite loop). Fix requires spec-level change: make (error)
use CEK raise instead of host-error.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ml-scan-set now checks ml-is-mutable-global? before adding set!/append!
targets to the needs-ref list. Previously, mutable globals like
*bind-tracking* got local `ref Nil` shadows that masked the global _ref,
causing `append!: expected list, got nil` in 43 bind-tracking tests.
Test runner: bind foreign registry functions (foreign-registered?,
foreign-lookup, foreign-names, foreign-register!, foreign-resolve-binding,
foreign-check-args, foreign-build-lambda) + initialize _cek_call_ref for
with-capabilities. 22/24 foreign tests now pass, 8 capabilities tests fixed.
Retranspiled sx_ref.ml — all mutable global shadows eliminated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
FFI: define-foreign special form in evaluator — registry, param parser,
kwargs parser, binding resolver, type checker, lambda builder, dispatcher.
Generates callable lambdas that route through foreign-dispatch to host-call.
24 tests in test-foreign.sx (registry, parsing, resolution, type checking).
Transpiler: fix mutable global ref emission — ml-emit-define now emits
both X_ref = ref <init> and X_ = <init> for starred globals (was missing
the ref definition entirely, broke retranspilation). Add *provide-batch-depth*,
*provide-batch-queue*, *provide-subscribers* to mutable globals list.
Evaluator: add missing (define *provide-batch-queue* (list)) and
(define *provide-subscribers* (dict)) — were only in hand-edited sx_ref.ml.
Known: 36 bind-tracking + 8 capability test failures on retranspilation
(pre-existing transpiler local-ref shadowing bug, not caused by FFI).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tier 1 — Component keyword dispatch on VM:
- Components/islands JIT-compile bodies via jit_compile_comp
- parse_keyword_args matches keyword names against component params
- Added i_compiled field to island type for JIT cache
- Component calls no longer fall back to CEK
Tier 2 — OP_SWAP (opcode 7):
- New stack swap operation for future HO loop compilation
- HO forms already efficient via NativeFn + VmClosure callbacks
Tier 3 — Exception handler stack:
- OP_PUSH_HANDLER (35), OP_POP_HANDLER (36), OP_RAISE (37)
- VM gains handler_stack with frame depth tracking
- Compiler handles guard and raise as bytecode
- Functions with exception handling no longer cause JIT failure
Tier 4 — Scope forms as bytecode:
- Compiler handles provide, context, peek, scope, provide!,
bind, emit!, emitted via CALL_PRIM sequences
- Functions using reactive scope no longer trigger JIT failure
4 new opcodes (SWAP, PUSH_HANDLER, POP_HANDLER, RAISE) → 37 total.
2776/2776 tests pass, zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
run_tests.ml had a local _scope_stacks hash table, separate from
Sx_primitives._scope_stacks used by the CEK evaluator. SX-level
scope-push!/scope-peek used the local table, but step-sf-context's
scope_peek used the global one. Aser's provide handler pushed to
one table, context read from the other — always got nil.
Fix: alias run_tests.ml's _scope_stacks to Sx_primitives._scope_stacks.
2768/2768 OCaml tests pass. Zero failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Provide subscribers stored in global *provide-subscribers* dict (keyed
by name) instead of on provide frames. Fixes subscriber loss when
frames are reconstructed, and enables cross-cek_run notification.
Batch integration: batch-begin!/batch-end! primitives manage
*provide-batch-depth*. fire-provide-subscribers defers to queue when
depth > 0, batch-end! flushes deduped. signals.sx batch calls both.
context now prefers scope-peek over frame value — scope stack is the
source of truth since provide! always updates it (even in nested
cek_run where provide frames aren't on the kont).
2754/2768 OCaml (14 pre-existing). 32/32 WASM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: context called inside lambdas (e.g. swap!) went through
nested cek_run with empty kont, so provide frames weren't found and
never tracked to *bind-tracking*.
Three changes in evaluator.sx:
- step-sf-context: track context names (not frames) to *bind-tracking*
— names work across cek_run boundaries via scope-peek fallback
- bind continue: resolve tracked names to frames via kont-find-provide
on rest-k before registering subscribers
- subscriber: use empty kont instead of kont-extract-provides — old
approach created provide frames whose continue handlers called
scope-pop!, corrupting the scope stack
2752/2768 OCaml tests pass (all 7 bind subscriber tests fixed).
32/32 WASM native tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bind is now a CEK special form that captures its body unevaluated,
establishes a tracking context (*bind-tracking*), and registers
subscribers on provide frames when context reads are tracked.
- bind special form: step-sf-bind, make-bind-frame, bind continue handler
- provide-set frame: provide! evaluates value with kont (fixes peek bug)
- context tracking: step-sf-context appends to *bind-tracking* when active
- scope-stack fallback: provide pushes to scope stack for cek-call contexts
- CekFrame mutation: cf_remaining/cf_results/cf_extra2 now mutable
- Transpiler: subscribers + prev-tracking field mappings, *bind-tracking* in ml-mutable-globals
- Test fixes: string-append → str, restored edge-cases suite
Passing: bind returns initial value, bind with expression, bind with let,
bind no deps is static, bind with conditional deps, provide! updates/multiple/nil,
provide! computed new value, peek read-modify-write, guard inside bind,
bind with string-append, provide! same value does not notify, bind does not
fire on unrelated provide!, bind sees latest value, bind inside provide scope.
Remaining: subscriber re-evaluation on provide! (scope-stack key issue),
batch coalescing (no batch support yet).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- _import_hook verifies library_loaded_p AFTER load_library_file
to catch cases where the file loads but define-library doesn't register
- Re-entry guard (_loading_libs) prevents infinite retry loops
- cek_run import patch deferred — retry approach infinite-loops because
cek_step_loop re-enters deeply nested eval contexts. Root cause:
eval_expr → cek_run → cek_step_loop processes the ENTIRE remaining
kont chain after import resolution, which includes rendering code that
triggers MORE eval_expr calls. Needs architectural solution (step-level
suspension handling, not run-level).
Server runs with 4 harmless IO suspension errors. These don't affect
functionality — symbols load via the global env.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The cek_run import handling (resume after hook loads library) caused
cek_step_loop to infinite-loop during aser page rendering. Root cause
not yet identified — the resumed CEK state never reaches terminal.
Reverted to original cek_run that throws "IO suspension in non-IO
context". The 4 server startup errors are harmless (files load
partially, all needed symbols available via other paths).
Import hook re-entry guard and debug logging retained for future work.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
step_sf_define_library now processes import clauses by evaluating
(import lib-spec) in the library env. Previously import clauses
were silently ignored, so libraries couldn't use symbols from
other libraries.
2694/2694 tests pass (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Capability primitives promoted from mcp_tree.ml to sx_primitives.ml:
- with-capabilities — push cap set, eval body, restore on exit/error
- current-capabilities — returns active capability list (nil = unrestricted)
- has-capability? — check if capability granted (true when unrestricted)
- require-capability! — raise if capability missing
- capability-restricted? — check if any restrictions active
Infrastructure: _cek_call_ref in sx_types.ml (forward ref pattern)
allows primitives to invoke the CEK evaluator without dependency cycles.
10 new tests: unrestricted defaults, scoping, nesting, restore-on-exit.
2693 total tests, 0 regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root causes of server [http-load] errors:
1. _import_hook passed pre-computed string key to library_loaded_p
which calls library_name_key(string) → sx_to_list(string) → crash.
Fix: pass original list spec, not the string key.
2. resolve_library_path didn't check web/lib/ for (sx dom), (sx browser),
(web boot-helpers). These libraries use namespace prefixes that don't
match their file locations.
Server startup errors: 190 → 0.
2683/2684 tests pass (1 known: define-library import clause — spec gap).
New test file: spec/tests/test-import-bind.sx
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
9 new deep recursion tests (100K-200K depth) confirming TCO in:
- match, begin, do, let-match — tail expressions get same continuation
- parameterize — provide frames are contextual, don't block TCO
- guard — handler body in tail position via cond desugaring
- handler-bind — body sequences with rest-k
- and/or — short-circuit preserves tail position
- mutual recursion — 200K depth even/odd
CEK machine correctly preserves tail position in all forms.
2676/2676 standard tests pass (was 2668 + 9 new - 1 pre-existing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- cek_run patched to handle import suspensions via _import_hook.
define-library (import ...) now resolves cleanly on the server.
IO suspension errors: 190 → 0. JIT failures: ~50 → 0.
- _import_hook wired in sx_server.ml to load .sx files on demand.
- compile-modules.js syncs source .sx files to dist/sx/ before
compiling — eliminates stale bytecode from out-of-date copies.
- WASM binary rebuilt with all fixes.
- 2658/2658 tests pass (8 new — previously failing import tests).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the VM or CEK hits an undefined symbol, it checks a symbol→library
index (built from manifest exports at boot), loads the library that
exports it, and returns the value. Execution continues as if the module
was always loaded. No import statements, no load-library! calls, no
Suspense boundaries — just call the function.
This is the same mechanism as IO suspension for data fetching. The
programmer doesn't distinguish between calling a local function and
calling one that needs its module fetched first. The runtime treats
code as just another resource.
Implementation:
- _symbol_resolve_hook in sx_types.ml — called by env_get_id (CEK path)
and vm_global_get (VM path) when a symbol isn't found
- Symbol→library index built from manifest exports in sx-platform.js
- __resolve-symbol native calls __sxLoadLibrary, module loads, symbol
appears in globals, execution resumes
- compile-modules.js extracts export lists into module-manifest.json
- Playground page demonstrates: (freeze-scope) triggers freeze.sxbc
download transparently on first use
2650/2650 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The bytecode compiler now handles let-match (both variants):
- Variant 1: (let-match name expr {:k v} body...) — named binding + destructure
- Variant 2: (let-match {:k v} expr body...) — pattern-only destructure
Desugars to sequential let + get calls — no new opcodes needed.
This was the last blocker for SPA navigation. The bytecoded orchestration
and router modules used let-match which compiled to CALL_PRIM "let-match"
(undefined at runtime). Now desugared at compile time.
Also synced dist/sx/ sources with web/ and recompiled all 26 .sxbc modules.
2650/2650 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- load-library! native: islands can declare module dependencies at
hydration time, triggering on-demand .sxbc loading
- JIT compiler lazy-load: compiler.sxbc loads via setTimeout after boot,
eliminating "JIT: compiler not loaded" errors
- _import_hook on sx_types: infrastructure for hosts to resolve import
suspensions inside eval_expr (server wiring deferred to Step 8)
- Playground page (/sx/(tools.(playground))): REPL island that lazy-loads
the compiler module when navigated to — demonstrates the full
lazy loading pipeline
Known remaining issues:
- SPA navigation broken for pages using let-match (orchestration.sx,
router.sx) — bytecode compiler doesn't handle let-match special form
- Server-side "IO suspension in non-IO context" during http_load_files —
needs cek_run import handling (deferred to Step 8)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bytecode compiler now emits OP_PERFORM for (import ...) and compiles
(define-library ...) bodies. The VM stores the import request in
globals["__io_request"] and stops the run loop — no exceptions needed.
vm-execute-module returns a suspension dict, vm-resume-module continues.
Browser: sx_browser.ml detects suspension dicts from execute_module and
returns JS {suspended, op, request, resume} objects. The sx-platform.js
while loop handles cascading suspensions via handleImportSuspension.
13 modules load via .sxbc bytecode in 226ms (manifest-driven), both
islands hydrate, all handlers wired. 2650/2650 tests pass including
6 new vm-import-suspension tests.
Also: consolidated sx-platform-2.js → sx-platform.js, fixed
vm-execute-module missing code-from-value call, fixed bootstrap.py
protocol registry transpiler issues.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Match exhaustiveness analysis:
- check-match-exhaustiveness function in evaluator.sx
- lint-node in tree-tools.sx checks match forms during format-check
- Warns on: no wildcard/catch-all, boolean missing true/false case
- (match x (true "yes")) → "match may be non-exhaustive"
Evaluator cleanup:
- Added missing step-sf-callcc definition (was in old transpiled output)
- Added missing step-sf-case definition (was in old transpiled output)
- Removed protocol functions from bootstrap skip set (they transpile fine)
- Retranspiled VM (bootstrap_vm.py) for compatibility
2650 tests pass (+5 from new features).
All Step 7 features complete:
7a: ->> |> as-> pipe operators
7b: Dict patterns, &rest, let-match destructuring
7c: define-protocol, implement, satisfies?
7d: Exhaustive match checking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new pattern matching features in evaluator.sx:
1. Dict patterns in match:
(match {:name "Alice" :age 30}
({:name n :age a} (list n a))) ;; => ("Alice" 30)
2. &rest in list patterns:
(match (list 1 2 3 4 5)
((a b &rest tail) tail)) ;; => (3 4 5)
3. let-match form (sugar for match):
(let-match {:x x :y y} {:x 3 :y 4}
(+ (* x x) (* y y))) ;; => 25
Also: transpiler fix — "extra" key added to CekFrame cf_extra mapping
(was the root cause of thread-last mode not being stored).
2644 tests pass, zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new threading operators in evaluator.sx:
- ->> (thread-last): inserts value as last arg
- |> (pipe): alias for ->> (F#/OCaml convention)
- as-> (thread-anywhere): binds value to named variable
(->> 10 (- 3)) ;; => -7 (thread-last: (- 3 10))
(-> 10 (- 3)) ;; => 7 (thread-first: (- 10 3))
(->> 1 (list 2 3)) ;; => (2 3 1)
(as-> 5 x (+ x 1) (* x 2)) ;; => 12
Two transpiler bugs fixed:
1. Non-recursive functions (let without rec) weren't chained as `and`
in the let rec block — became local bindings inside previous function
2. CekFrame "extra" field wasn't in the cf_extra key mapping — mode
was always Nil, making thread-last fall through to thread-first
Also: added missing step-sf-case definition to evaluator.
2644 tests pass, zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add to evaluator.sx:
- step-sf-thread-last: thread-last operator (inserts value at end)
- step-sf-thread-as: thread-anywhere with named binding
- thread-insert-arg-last: last-position insertion function
- step-sf-case: missing function (was in old transpiled output but not spec)
- Register ->>, |>, as-> in step-eval-list dispatch
Status:
- ->> dispatch works (enters thread-last correctly)
- HO forms (map, filter) with ->> work correctly
- Non-HO forms with ->> still use thread-first (transpiler bug)
- as-> binding fails (related transpiler bug)
Transpiler bug: thread_insert_arg_last definition body is merged with
step_continue in the let rec block. The transpiler incorrectly chains
them as one function. Need to investigate the let rec emission logic.
2644 tests still pass (no regressions from new operators).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The transpiled VM (sx_vm_ref.ml, from lib/vm.sx) is now the ACTIVE
bytecode execution engine. sx_server.ml and sx_browser.ml call
Sx_vm_ref.execute_module instead of Sx_vm.execute_module.
Results:
- OCaml tests: 2644 passed, 0 failed
- WASM tests: 32 passed, 0 failed
- Browser: zero errors, zero warnings, islands hydrate
- Server: pages render, JIT compiles, all routes work
The VM logic now lives in ONE place: lib/vm.sx (SX).
OCaml gets it via transpilation (bootstrap_vm.py).
JS/browser gets it via bytecode compilation (compile-modules.js).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add VmFrame/VmMachine types to sx_types.ml (alongside CekState/CekFrame)
- Add VmFrame/VmMachine value variants to the value sum type
- Extend get_val in sx_runtime.ml to dispatch on VmFrame/VmMachine fields
- Extend sx_dict_set_b for VmFrame/VmMachine field mutation
- Extend transpiler ml-emit-dict-native to detect VM dict patterns
and emit native OCaml record construction (same mechanism as CekState)
- Retranspile evaluator — no diff (transpiler extension is additive)
- Update bootstrap_vm.py output location
The transpiler now handles 4 native record types:
CekState (5 fields), CekFrame (10 fields),
VmFrame (4 fields), VmMachine (5 fields)
Full VM replacement (sx_vm.ml → transpiled) still needs vm.sx
feature parity: JIT dispatch, CEK fallback, suspension handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All 26 browser modules recompiled with define-library/import forms.
Compilation works without vm-compile-adapter (JIT pre-compilation
hangs with library wrappers in some JIT paths — skipped for now,
CEK compilation is ~34s total).
Key fixes:
- eval command: import-aware loop that handles define-library/import
locally without touching the Python bridge pipe (avoids deadlock)
- compile-modules.js: skip vm-compile-adapter, bump timeout
2621/2621 OCaml tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sx_server.ml: the "eval" command now uses cek_run_with_io instead of
raw eval_expr. This handles import suspensions during eval-blob
(needed for .sx files with define-library/import wrappers).
compile-modules.js: timeout bumped 5min → 10min for sxbc compilation
with define-library overhead.
2608/2608 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds IO suspension support to the browser WASM kernel. When the VM
hits OP_PERFORM or the CEK suspends (e.g. from import), the kernel
now handles it instead of crashing.
sx_browser.ml additions:
- make_js_suspension: creates JS object {suspended, op, request, resume}
for the platform to handle asynchronously
- handle_import_suspension: checks library registry, returns Some result
if already loaded (immediate resume), None if fetch needed
- api_load_module: catches VmSuspended, resolves imports locally if
library is loaded, returns suspension marker to JS if not
- api_load: IO-aware CEK loop using cek_step_loop/cek_suspended?/
cek_resume — handles import suspensions during .sx file loading
This enables the lazy loading pattern: when a module's (import ...)
encounters an unloaded library, the suspension propagates to JS which
can async-fetch the .sxbc and call resume(). Currently all libraries
are pre-loaded so suspensions resolve immediately.
2608/2608 OCaml tests passing. WASM kernel builds clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compiler (lib/compiler.sx):
- Fix emit-op return type: 8 definition form cases (defstyle,
defhandler, defpage, etc.) and the perform case now return nil
explicitly via (do (emit-op em N) nil) instead of bare emit-op
which transpiled to unit-returning OCaml.
- compile_match PREAMBLE: return Nil instead of unit (was ignore).
- Added init wrapper to PREAMBLE (needed by compile-module).
- All 41 compiler functions re-transpiled cleanly.
Render (bootstrap_render.py): re-transpiled, no changes.
Runtime: restored keyword_p predicate (needed by defio-parse-kwargs!
in the transpiled evaluator).
2608/2608 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
