step_sf_define_library now processes import clauses by evaluating
(import lib-spec) in the library env. Previously import clauses
were silently ignored, so libraries couldn't use symbols from
other libraries.
2694/2694 tests pass (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Capability primitives promoted from mcp_tree.ml to sx_primitives.ml:
- with-capabilities — push cap set, eval body, restore on exit/error
- current-capabilities — returns active capability list (nil = unrestricted)
- has-capability? — check if capability granted (true when unrestricted)
- require-capability! — raise if capability missing
- capability-restricted? — check if any restrictions active
Infrastructure: _cek_call_ref in sx_types.ml (forward ref pattern)
allows primitives to invoke the CEK evaluator without dependency cycles.
10 new tests: unrestricted defaults, scoping, nesting, restore-on-exit.
2693 total tests, 0 regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root causes of server [http-load] errors:
1. _import_hook passed pre-computed string key to library_loaded_p
which calls library_name_key(string) → sx_to_list(string) → crash.
Fix: pass original list spec, not the string key.
2. resolve_library_path didn't check web/lib/ for (sx dom), (sx browser),
(web boot-helpers). These libraries use namespace prefixes that don't
match their file locations.
Server startup errors: 190 → 0.
2683/2684 tests pass (1 known: define-library import clause — spec gap).
New test file: spec/tests/test-import-bind.sx
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- cek_run patched to handle import suspensions via _import_hook.
define-library (import ...) now resolves cleanly on the server.
IO suspension errors: 190 → 0. JIT failures: ~50 → 0.
- _import_hook wired in sx_server.ml to load .sx files on demand.
- compile-modules.js syncs source .sx files to dist/sx/ before
compiling — eliminates stale bytecode from out-of-date copies.
- WASM binary rebuilt with all fixes.
- 2658/2658 tests pass (8 new — previously failing import tests).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the VM or CEK hits an undefined symbol, it checks a symbol→library
index (built from manifest exports at boot), loads the library that
exports it, and returns the value. Execution continues as if the module
was always loaded. No import statements, no load-library! calls, no
Suspense boundaries — just call the function.
This is the same mechanism as IO suspension for data fetching. The
programmer doesn't distinguish between calling a local function and
calling one that needs its module fetched first. The runtime treats
code as just another resource.
Implementation:
- _symbol_resolve_hook in sx_types.ml — called by env_get_id (CEK path)
and vm_global_get (VM path) when a symbol isn't found
- Symbol→library index built from manifest exports in sx-platform.js
- __resolve-symbol native calls __sxLoadLibrary, module loads, symbol
appears in globals, execution resumes
- compile-modules.js extracts export lists into module-manifest.json
- Playground page demonstrates: (freeze-scope) triggers freeze.sxbc
download transparently on first use
2650/2650 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- load-library! native: islands can declare module dependencies at
hydration time, triggering on-demand .sxbc loading
- JIT compiler lazy-load: compiler.sxbc loads via setTimeout after boot,
eliminating "JIT: compiler not loaded" errors
- _import_hook on sx_types: infrastructure for hosts to resolve import
suspensions inside eval_expr (server wiring deferred to Step 8)
- Playground page (/sx/(tools.(playground))): REPL island that lazy-loads
the compiler module when navigated to — demonstrates the full
lazy loading pipeline
Known remaining issues:
- SPA navigation broken for pages using let-match (orchestration.sx,
router.sx) — bytecode compiler doesn't handle let-match special form
- Server-side "IO suspension in non-IO context" during http_load_files —
needs cek_run import handling (deferred to Step 8)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bytecode compiler now emits OP_PERFORM for (import ...) and compiles
(define-library ...) bodies. The VM stores the import request in
globals["__io_request"] and stops the run loop — no exceptions needed.
vm-execute-module returns a suspension dict, vm-resume-module continues.
Browser: sx_browser.ml detects suspension dicts from execute_module and
returns JS {suspended, op, request, resume} objects. The sx-platform.js
while loop handles cascading suspensions via handleImportSuspension.
13 modules load via .sxbc bytecode in 226ms (manifest-driven), both
islands hydrate, all handlers wired. 2650/2650 tests pass including
6 new vm-import-suspension tests.
Also: consolidated sx-platform-2.js → sx-platform.js, fixed
vm-execute-module missing code-from-value call, fixed bootstrap.py
protocol registry transpiler issues.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Match exhaustiveness analysis:
- check-match-exhaustiveness function in evaluator.sx
- lint-node in tree-tools.sx checks match forms during format-check
- Warns on: no wildcard/catch-all, boolean missing true/false case
- (match x (true "yes")) → "match may be non-exhaustive"
Evaluator cleanup:
- Added missing step-sf-callcc definition (was in old transpiled output)
- Added missing step-sf-case definition (was in old transpiled output)
- Removed protocol functions from bootstrap skip set (they transpile fine)
- Retranspiled VM (bootstrap_vm.py) for compatibility
2650 tests pass (+5 from new features).
All Step 7 features complete:
7a: ->> |> as-> pipe operators
7b: Dict patterns, &rest, let-match destructuring
7c: define-protocol, implement, satisfies?
7d: Exhaustive match checking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new pattern matching features in evaluator.sx:
1. Dict patterns in match:
(match {:name "Alice" :age 30}
({:name n :age a} (list n a))) ;; => ("Alice" 30)
2. &rest in list patterns:
(match (list 1 2 3 4 5)
((a b &rest tail) tail)) ;; => (3 4 5)
3. let-match form (sugar for match):
(let-match {:x x :y y} {:x 3 :y 4}
(+ (* x x) (* y y))) ;; => 25
Also: transpiler fix — "extra" key added to CekFrame cf_extra mapping
(was the root cause of thread-last mode not being stored).
2644 tests pass, zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three new threading operators in evaluator.sx:
- ->> (thread-last): inserts value as last arg
- |> (pipe): alias for ->> (F#/OCaml convention)
- as-> (thread-anywhere): binds value to named variable
(->> 10 (- 3)) ;; => -7 (thread-last: (- 3 10))
(-> 10 (- 3)) ;; => 7 (thread-first: (- 10 3))
(->> 1 (list 2 3)) ;; => (2 3 1)
(as-> 5 x (+ x 1) (* x 2)) ;; => 12
Two transpiler bugs fixed:
1. Non-recursive functions (let without rec) weren't chained as `and`
in the let rec block — became local bindings inside previous function
2. CekFrame "extra" field wasn't in the cf_extra key mapping — mode
was always Nil, making thread-last fall through to thread-first
Also: added missing step-sf-case definition to evaluator.
2644 tests pass, zero regressions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add to evaluator.sx:
- step-sf-thread-last: thread-last operator (inserts value at end)
- step-sf-thread-as: thread-anywhere with named binding
- thread-insert-arg-last: last-position insertion function
- step-sf-case: missing function (was in old transpiled output but not spec)
- Register ->>, |>, as-> in step-eval-list dispatch
Status:
- ->> dispatch works (enters thread-last correctly)
- HO forms (map, filter) with ->> work correctly
- Non-HO forms with ->> still use thread-first (transpiler bug)
- as-> binding fails (related transpiler bug)
Transpiler bug: thread_insert_arg_last definition body is merged with
step_continue in the let rec block. The transpiler incorrectly chains
them as one function. Need to investigate the let rec emission logic.
2644 tests still pass (no regressions from new operators).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The transpiled VM (sx_vm_ref.ml, from lib/vm.sx) is now the ACTIVE
bytecode execution engine. sx_server.ml and sx_browser.ml call
Sx_vm_ref.execute_module instead of Sx_vm.execute_module.
Results:
- OCaml tests: 2644 passed, 0 failed
- WASM tests: 32 passed, 0 failed
- Browser: zero errors, zero warnings, islands hydrate
- Server: pages render, JIT compiles, all routes work
The VM logic now lives in ONE place: lib/vm.sx (SX).
OCaml gets it via transpilation (bootstrap_vm.py).
JS/browser gets it via bytecode compilation (compile-modules.js).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add VmFrame/VmMachine types to sx_types.ml (alongside CekState/CekFrame)
- Add VmFrame/VmMachine value variants to the value sum type
- Extend get_val in sx_runtime.ml to dispatch on VmFrame/VmMachine fields
- Extend sx_dict_set_b for VmFrame/VmMachine field mutation
- Extend transpiler ml-emit-dict-native to detect VM dict patterns
and emit native OCaml record construction (same mechanism as CekState)
- Retranspile evaluator — no diff (transpiler extension is additive)
- Update bootstrap_vm.py output location
The transpiler now handles 4 native record types:
CekState (5 fields), CekFrame (10 fields),
VmFrame (4 fields), VmMachine (5 fields)
Full VM replacement (sx_vm.ml → transpiled) still needs vm.sx
feature parity: JIT dispatch, CEK fallback, suspension handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All 26 browser modules recompiled with define-library/import forms.
Compilation works without vm-compile-adapter (JIT pre-compilation
hangs with library wrappers in some JIT paths — skipped for now,
CEK compilation is ~34s total).
Key fixes:
- eval command: import-aware loop that handles define-library/import
locally without touching the Python bridge pipe (avoids deadlock)
- compile-modules.js: skip vm-compile-adapter, bump timeout
2621/2621 OCaml tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sx_server.ml: the "eval" command now uses cek_run_with_io instead of
raw eval_expr. This handles import suspensions during eval-blob
(needed for .sx files with define-library/import wrappers).
compile-modules.js: timeout bumped 5min → 10min for sxbc compilation
with define-library overhead.
2608/2608 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds IO suspension support to the browser WASM kernel. When the VM
hits OP_PERFORM or the CEK suspends (e.g. from import), the kernel
now handles it instead of crashing.
sx_browser.ml additions:
- make_js_suspension: creates JS object {suspended, op, request, resume}
for the platform to handle asynchronously
- handle_import_suspension: checks library registry, returns Some result
if already loaded (immediate resume), None if fetch needed
- api_load_module: catches VmSuspended, resolves imports locally if
library is loaded, returns suspension marker to JS if not
- api_load: IO-aware CEK loop using cek_step_loop/cek_suspended?/
cek_resume — handles import suspensions during .sx file loading
This enables the lazy loading pattern: when a module's (import ...)
encounters an unloaded library, the suspension propagates to JS which
can async-fetch the .sxbc and call resume(). Currently all libraries
are pre-loaded so suspensions resolve immediately.
2608/2608 OCaml tests passing. WASM kernel builds clean.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compiler (lib/compiler.sx):
- Fix emit-op return type: 8 definition form cases (defstyle,
defhandler, defpage, etc.) and the perform case now return nil
explicitly via (do (emit-op em N) nil) instead of bare emit-op
which transpiled to unit-returning OCaml.
- compile_match PREAMBLE: return Nil instead of unit (was ignore).
- Added init wrapper to PREAMBLE (needed by compile-module).
- All 41 compiler functions re-transpiled cleanly.
Render (bootstrap_render.py): re-transpiled, no changes.
Runtime: restored keyword_p predicate (needed by defio-parse-kwargs!
in the transpiled evaluator).
2608/2608 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1: Absorb sx_scope.ml (180 lines) into sx_primitives.ml. Scope stacks,
cookies, and trace infrastructure now live alongside other primitives. All 20
scope primitive registrations moved. References updated in sx_server.ml and
sx_browser.ml. sx_scope.ml deleted.
Phase 2: Transpiler handles (define name :effects (...) (fn ...)) forms.
ml-emit-define and ml-emit-define-body detect keyword at position 2 and use
(last expr) instead. Unblocks transpilation of spec/render.sx and
web/adapter-html.sx which use 4-child defines extensively.
2598/2598 tests passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Delete from sx_types.ml:
- Comment of string variant (no longer needed)
Delete from sx_parser.ml:
- _preserve_comments mutable ref
- collect_comment_node function
- comment-mode branches in read_value, read_list
- ~comments parameter from parse_all and parse_file
- skip_whitespace and read_comment (only used by old comment mode)
Delete from mcp_tree.ml:
- has_interior_comments function
- Comment handling in pretty_print_value
- pretty_print_file function (replaced by CST write-back)
- ~comments parameter from local parse_file
Migrate sx_pretty_print, sx_write_file, sx_doc_gen to CST path.
Net: -69 lines. 24/24 CST round-trips, 2583/2583 evaluator tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the comment preservation workaround (comment_map type,
separate_comments, reinterleave, strip_interior_comments,
extract_fragment_comments, inject_comments — ~170 lines) with
CST-based editing (~80 lines).
write_edit_cst: compares old AST vs new AST per node. Unchanged
nodes keep original source verbatim. Changed nodes are pretty-printed
with original leading trivia preserved. New nodes (insertions) get
minimal formatting.
parse_file_cst: returns (AST tree, CST file). AST goes to tree-tools,
CST is used for write-back.
extract_cst_comments / inject_cst_comments: read comment trivia from
CST nodes for summarise/read_tree display.
Net: -39 lines. 24/24 CST round-trip tests, 2583/2583 evaluator tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parser: read_value/read_list now capture Comment nodes inside lists
when ~comments:true. Module-level _preserve_comments ref threads the
flag through the recursive descent without changing signatures.
Pretty printer: has_interior_comments (recursive) forces multi-line
when any nested list contains comments. Comment nodes inside lists
emit as indented comment lines.
Edit tools: separate_comments strips interior comments recursively
via strip_interior_comments before passing to tree-tools (paths stay
correct). extract_fragment_comments parses new source with comments,
attaches leading comments to the target position in the comment map.
sx_get_siblings: injects comments for top-level siblings.
sx_doc_gen: parses with comments, tracks preceding Comment node,
includes cleaned comment text in generated component documentation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
evaluator.sx: 11 section headers + 27 subgroup/function comments
documenting the CEK machine structure (state, frames, kont ops,
extension points, eval utilities, machine core, special forms,
call dispatch, HO forms, continue phase, entry points).
mcp_tree.ml: sx_summarise and sx_read_tree now inject file comments
into their output — comments appear as un-numbered annotation lines
between indexed entries, so indices stay correct for editing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parser gains Comment(string) AST variant and ~comments:true mode that
captures top-level ;; lines instead of discarding them. All MCP edit
tools (replace_node, insert_child, delete_node, wrap_node, rename_symbol,
replace_by_pattern, insert_near, rename_across, pretty_print, write_file)
now preserve comments: separate before tree-tools operate (so index paths
stay correct), re-interleave after editing, emit in pretty_print_file.
Default parse path (evaluator, runtime, compiler) is unchanged — comments
are still stripped unless explicitly requested.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When aser manages scope via scope_stacks but a sub-expression falls
through to the CEK machine, context/emit!/emitted couldn't find the
scope frames (they're in scope_stacks, not on the kont). Now the CEK
special forms fall back to env-bound primitives when kont lookup fails.
2568/2568 tests pass (was 2566/2568).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bootstrap.py produces correct output with no post-processing.
Browser sx_browser.ml updated to use Sx_runtime._jit_try_call_fn.
2566/2568 tests pass (2 pre-existing scope).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- make-raise-guard-frame: was never defined in spec — added it
- *last-error-kont*: set at error origination (host-error calls), not
wrapped around every cek-run step. Zero overhead on normal path.
- JIT: jit-try-call runtime function called from spec. Platform
registers hook via _jit_try_call_fn ref. No bootstrap patching.
- bootstrap.py compile_spec_to_ml() now returns transpiled output
with zero post-processing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
No more regex fixups for *strict* / *prim-param-types* — transpiler
handles reads (!_ref), writes (_ref :=), and defines natively.
2566/2568 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- guard as CEK special form in evaluator.sx, desugars to call/cc +
handler-bind with sentinel-based re-raise (avoids handler loop)
- bootstrap.py: fix bind_lambda_with_rest type annotations, auto-inject
make_raise_guard_frame when transpiler drops it
- mcp_tree: add timeout param to sx_test (default 300s)
- 2566/2568 tests pass (2 pre-existing scope failures)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
guard macro expansion loops in the transpiled evaluator because
expand_macro→eval_expr→CEK can't handle let+first/rest in macro
bodies. Removed guard macro; will re-add as special form once
transpiler handles runtime AST construction (cons/append/make-symbol).
Fixed null? to handle empty lists (not just nil).
Fixed boolean=? to use = instead of undefined eq?.
2561/2568 tests pass (37 new vs baseline, 5 guard + 2 scope pending).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 engine step 4 — R7RS compatibility primitives for the CEK evaluator.
call/cc: undelimited continuation capture with separate CallccContinuation
type (distinct from delimited shift/reset continuations). Escape semantics —
invoking k replaces the current continuation entirely.
raise/raise-continuable: proper CEK arg evaluation via raise-eval frame.
Non-continuable raise uses raise-guard frame that errors on handler return.
host-error primitive for safe unhandled exception fallback.
Multi-arity map: (map fn list1 list2 ...) zips multiple lists. Single-list
path unchanged for performance. New multi-map frame type.
cond =>: arrow clause syntax (cond (test => fn)) calls fn with test value.
New cond-arrow frame type.
R7RS do: shape-detecting dispatch — (do ((var init step) ...) (test result) body)
desugars to named let. Existing (do expr1 expr2) sequential form preserved.
integer? primitive, host-error alias. Transpiler fixes: match/case routing,
wildcard _ support, nested match arm handling.
2522/2524 OCaml tests pass (2 pre-existing scope failures from transpiler
match codegen, not related to these changes).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Core SX has zero IO — platforms extend __io-registry via (defio name
:category :data/:code/:effect ...). The server web platform declares 44
operations in web/io.sx. batchable_helpers now derived from registry
(:batchable true) instead of hardcoded list. Startup validation warns if
bound IO ops lack registry entries. Browser gets empty registry, ready
for step 5 (IO suspension).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 Step 2 of architecture roadmap. The OCaml HTTP server is now
generic — all sx_docs-specific values (layout components, path prefix,
title, warmup paths, handler prefixes, CSS/JS, client libs) move into
sx/sx/app-config.sx as a __app-config dict. Server reads config at
startup with hardcoded defaults as fallback, so it works with no config,
partial config, or full config.
Removed: 9 demo data stubs, stepper cookie cache logic, page-functions.sx
directory heuristic. Added: 29-test server config test suite covering
standard, custom, no-config, and minimal-config scenarios.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
