Commit Graph

168 Commits

Author SHA1 Message Date
1eadefd0c1 Step 17b: Pretext — DOM-free text layout with otfm font measurement
Pure SX text layout library with one IO boundary (text-measure perform).
Knuth-Plass optimal line breaking, Liang's hyphenation, position calculation.

Library (lib/text-layout.sx):
- break-lines: Knuth-Plass DP over word widths
- break-lines-greedy: simple word-wrap for comparison
- hyphenate-word: Liang's trie algorithm
- position-line/position-lines: running x/y sums
- measure-text: single perform (text-measure IO)

Server font measurement (otfm):
- Reads OpenType cmap + hmtx tables from .ttf files
- DejaVu Serif/Sans bundled in shared/static/fonts/
- _cek_io_resolver hook: perform works inside aser/eval_expr
- JIT VM suspension inline resolution for IO in compiled code

~font component (shared/sx/templates/font.sx):
- Works like ~tw: emits @font-face CSS via cssx scope
- Sets font-family on parent via spread
- Deduplicates font declarations

Infrastructure fixes:
- stdin load command: per-expression error handling (was aborting on first error)
- cek_run IO hook: _cek_io_resolver in sx_types.ml
- JIT VmSuspended: inline IO resolution when resolver installed
- ListRef handling in IO resolver (perform creates ListRef, not List)

Demo page at /sx/(applications.(pretext)):
- Hero: justified paragraph with otfm-measured proportional widths
- Greedy vs Knuth-Plass side-by-side comparison
- Badness scoring visualization
- Hyphenation syllable decomposition

25 new tests (spec/tests/test-text-layout.sx), 3201/3201 passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 15:13:00 +00:00
854ed9c027 Hyperscript conformance: 372→373 — hide/show strategy, generator toEqual
Parser: hide/show handle `with opacity/visibility/display` strategy,
target detection for then-less chaining (add/remove/set/put as boundary).
Generator: inline run().toEqual([...]) pattern for eval-only tests.
Compiler: hide/show emit correct CSS property per strategy.

373/831 (45%)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 11:42:28 +00:00
99c5c44cc1 Step 14: source locations — pos-to-loc, error-loc, sx-parse-loc — 15 tests
Pure SX layer: pos-to-loc (offset→line/col), error-loc (parse result→loc),
format-parse-error (human-readable error with source context line).
OCaml platform: cst_to_ast_loc (CST spans→loc dicts), sx-parse-loc
primitive (parse with locations), source-loc accessor.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:03:45 +00:00
7f273dc7c2 Wire hyperscript activation into browser boot pipeline
- orchestration.sx: add hs-boot-subtree! call to process-elements
- integration.sx: remove load-library! calls (browser loads via manifest)
- sx_vm.ml: add __resolve-symbol hook to OP_GLOBAL_GET for lazy loading
- compile-modules.js: add HS modules as lazy_deps in manifest

HS compilation works in browser (tokenize→parse→compile verified).
Activation pipeline partially working — hs-activate! needs debugging
(dom-get-data/dom-set-data interaction with WASM host-get on functions).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 19:59:04 +00:00
908f4f80d4 Fix bytecode resume mutation order: isolate VM frames in cek_call_or_suspend
When cek_call_or_suspend runs a CEK machine for a non-bytecoded Lambda
(e.g. a thunk), _active_vm still pointed to the caller's VM. VmClosure
calls inside the CEK (e.g. hs-wait) would merge their frames with the
caller's VM via call_closure_reuse, causing the VM to skip the CEK's
remaining continuation on resume — producing wrong DOM mutation order
(+active, +active, -active instead of +active, -active, +active).

Fix: swap _active_vm with an empty isolation VM before running the CEK,
restore after. This keeps VmClosure calls on their own frame stack while
preserving js_of_ocaml exception identity (Some path, not None).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 22:55:26 +00:00
6456bd927a Fix bytecode when/do/perform: snapshot pending_cek in resume closure
Root cause: nested cek_call_or_suspend calls on the same VM (from
synchronous callbacks like dom-listen firing handler immediately)
overwrote pending_cek before the first resume ran.

Fix: _vm_suspension_to_dict snapshots pending_cek at capture time
and restores it in the resume closure before calling resume_vm.
This ensures each suspension's CEK state is preserved regardless
of nested overwrite.

test_bytecode_repeat.js: 4/4 pass (was 3/4).
Source: 6 suspensions ✓  Bytecode: 6 suspensions ✓

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 21:34:10 +00:00
7a1af7a80a WIP: bytecode when/do/perform — host-callback _driveAsync fix + debugging
Root cause identified: nested cek_call_or_suspend calls on same VM
overwrite pending_cek. First call suspends (thunk's hs-wait), second
call from synchronous dom-listen callback overwrites before resume.

sandbox host-callback: removed _driveAsync call to prevent duplicate
resume chains. Still 3/6 in Node.js test — issue is in OCaml call
stack nesting, not JS async.

Next: prevent pending_cek overwrite in nested CEK→VM→CEK→VM chains.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 21:21:08 +00:00
33e8788781 Lambda→CEK dispatch: enable IO suspension through sx_call
Lambda calls in sx_call now go through the CEK machine instead of
returning a Thunk for the tree-walker trampoline. This lets perform/
IO suspension work everywhere — including hyperscript wait/bounce.

Key changes:
- sx_runtime: Lambda case calls _cek_eval_lambda_ref (forward ref)
- sx_vm: initializes ref with cek_step_loop + stub VM for suspension
- sx_apply_cek: VmSuspended → __vm_suspended marker dict (not exception)
- continue_with_call callable path: handles __vm_suspended with
  vm-resume-frame, matching the existing JIT Lambda pattern
- sx_render: let VmSuspended propagate through try_catch
- Remove invalid io-contract test (perform now suspends, not errors)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 16:19:30 +00:00
d715d8c4ac JIT: closure env merge + bytecode locals scan for closure functions
- jit_compile_lambda: merge closure bindings into effective_globals so
  GLOBAL_GET resolves variables from let/define blocks (emit-on, etc.)
- code_from_value: scan bytecode for max LOCAL_GET/SET slot to compute
  vc_locals (fixes LOCAL_GET overflow in large functions like hs-parse)

3127/3127 no-JIT, 3116/3127 JIT (11 hyperscript on-event: specific
bytecode correctness issue in recursive parser — wrong branch taken
strips on/event-name from result).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 11:24:50 +00:00
3155ba47f9 JIT: VM fast path, &rest support, locals scan, test runner fixes
- jit_compile_lambda: call compile directly via VM when it has bytecode
  (100-400x faster JIT compilation, server pre-warm 1.6s vs hung)
- code_from_value: scan bytecode for highest LOCAL_GET/SET slot to
  compute vc_locals correctly (fixes hyperscript LOCAL_GET overflow)
- code_from_value: accept both compiler keys (bytecode) and SX VM
  keys (vc-bytecode) for interop
- jit_compile_lambda: skip &key/:as params (compiler can't emit them)
- Test runner: seed VM globals with primitives + env bindings,
  native vm-execute-module with suspension fallback to SX version,
  _jit_refresh_globals syncs globals after module loading,
  VmSuspended + "VM undefined" caught and sentineled

3127/3127 without JIT, 3116/3127 with JIT (11 hyperscript on-event
parsing — specific closure/scope issue, not infrastructure).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:52:44 +00:00
387a6cb49e Refactor MCP tree server: dispatch table, caching, validation, subprocess cleanup
Break up the 1735-line handle_tool match into 45 individual handler functions
with hashtable-based dispatch. Add mtime-based file parse caching (AST + CST),
consolidated run_command helper replacing 9 bare open_process_in patterns,
require_file/require_dir input validation, and pagination (limit/offset) for
sx_find_across, sx_comp_list, sx_comp_usage. Also includes pending VM changes:
rest-arity support, hyperscript parser, compiler/transpiler updates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:12:57 +00:00
03278c640d Fix JIT compilation cascade + MCP robustness
Three interacting JIT bugs caused infinite loops and server hangs:

1. _jit_compiling cascade: the re-entrancy flag was local to each
   binary's hook. When vm_call triggered JIT compilation internally,
   compiler functions got JIT-compiled during compilation, creating
   infinite cascades. Fix: shared _jit_compiling flag in sx_vm.ml,
   set in jit_compile_lambda itself.

2. call_closure always created new VMs: every HO primitive callback
   (for-each, map, filter) allocated a fresh VM. With 43K+ calls
   during compilation, this was the direct cause of hangs. Fix:
   call_closure_reuse reuses the active VM by isolating frames and
   running re-entrantly. VmSuspended is handled by merging frames
   for proper IO resumption.

3. vm_call for compiled Lambdas: OP_CALL dispatching to a Lambda
   with cached bytecode created a new VM instead of pushing a frame
   on the current one. Fix: push_closure_frame directly.

Additional MCP server fixes:
- Hot-reload: auto-execv when binary on disk is newer (no restart needed)
- Robust JSON: to_int_safe/to_int_or handle null, string, int params
- sx_summarise depth now optional (default 2)
- Per-request error handling (malformed JSON doesn't crash server)
- sx_test uses pre-built binary (skips dune rebuild overhead)
- Timed module loading for startup diagnostics

sx_server.ml fixes:
- Uses shared _jit_compiling flag
- Marks lambdas as jit_failed_sentinel on compile failure (no retry spam)
- call_closure_reuse with VmSuspended frame merging for IO support

Compiled compiler bytecode bug: deeply nested cond/case/let forms
(e.g. tw-resolve-style) cause the compiled compiler to loop.
Workaround: _jit_compiling guard prevents compiled function execution
during compilation. Compilation uses CEK (slower but correct).
Test suite: 3127/3127 passed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 21:59:31 +00:00
577d09f443 Fix vm-global-get in native OCaml VM + transpiled VM ref
The previous commit fixed lib/vm.sx (SX spec) but the server uses
sx_vm.ml (hand-maintained native OCaml) and sx_vm_ref.ml (transpiled).
Both had the same globals-first lookup bug. Now all three implementations
check closure env before vm.globals, matching vm-global-set.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 12:14:30 +00:00
5d88b363e4 Step 13: String/regex primitives — PCRE-compatible, cross-host
New primitives in sx_primitives.ml:
  char-at, char-code, parse-number — string inspection + conversion
  regex-match, regex-match?, regex-find-all — PCRE pattern matching
  regex-replace, regex-replace-first — PCRE substitution
  regex-split — split by PCRE pattern

Uses Re.Pcre (OCaml re library) so regex patterns use the same syntax
as JS RegExp — patterns in .sx files work identically on browser and
server. Replaces the old test-only regex-find-all stub.

Also: split now handles multi-char separators via Re.

176 new tests (10 suites). 2912/2912 total, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 20:38:40 +00:00
1d68f20a37 CEK-safe native call boundary: apply-cek + eval-error? marker
Native functions (NativeFn/VmClosure) called through the CEK evaluator
can now have their Eval_errors caught by guard/handler-bind. The fix is
at the exact OCaml↔CEK boundary in continue-with-call:

- sx_runtime.ml: sx_apply_cek wraps native calls, returns error marker
  dict {__eval_error__: true, message: "..."} instead of raising
- sx_runtime.ml: is_eval_error predicate checks for the marker
- spec/evaluator.sx: continue-with-call callable branch uses apply-cek,
  detects error markers, converts to raise-eval CEK state
- transpiler.sx: apply-cek and eval-error? emit cases added

No mutable flags, no re-entry risk. Errors flow through the CEK handler
chain naturally. 2798/2800 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:31:00 +00:00
7f772e0f23 Fix transpiler append! emit for mutable globals + run_with_io error recovery
The transpiler's append! emit path didn't check ml-is-mutable-global?,
so (append! *provide-batch-queue* sub) wrote to a dead local variable
instead of the global _ref. This caused the combined test suite hang —
fire-provide-subscribers was silently broken before the local-ref shadow
removal, and now correctly modifies the global batch queue.

Also adds run_with_io error-to-raise conversion (kont_has_handler guard)
so native Eval_errors can be caught by CEK guard/handler-bind when running
through the test runner's IO-aware step loop.

2798/2800 tests pass. 2 foreign-type-checking failures remain: guard can't
catch Eval_error from native fns called through cek_run_iterative (the
handler dispatch itself uses cek_call which re-enters cek_run_iterative,
creating an infinite loop). Fix requires spec-level change: make (error)
use CEK raise instead of host-error.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:59:38 +00:00
b61b437ccd Transpiler local-ref shadowing fix + foreign test runner bindings
ml-scan-set now checks ml-is-mutable-global? before adding set!/append!
targets to the needs-ref list. Previously, mutable globals like
*bind-tracking* got local `ref Nil` shadows that masked the global _ref,
causing `append!: expected list, got nil` in 43 bind-tracking tests.

Test runner: bind foreign registry functions (foreign-registered?,
foreign-lookup, foreign-names, foreign-register!, foreign-resolve-binding,
foreign-check-args, foreign-build-lambda) + initialize _cek_call_ref for
with-capabilities. 22/24 foreign tests now pass, 8 capabilities tests fixed.

Retranspiled sx_ref.ml — all mutable global shadows eliminated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:29:13 +00:00
000f285ae8 Step 11: define-foreign FFI + transpiler mutable globals fix
FFI: define-foreign special form in evaluator — registry, param parser,
kwargs parser, binding resolver, type checker, lambda builder, dispatcher.
Generates callable lambdas that route through foreign-dispatch to host-call.
24 tests in test-foreign.sx (registry, parsing, resolution, type checking).

Transpiler: fix mutable global ref emission — ml-emit-define now emits
both X_ref = ref <init> and X_ = <init> for starred globals (was missing
the ref definition entirely, broke retranspilation). Add *provide-batch-depth*,
*provide-batch-queue*, *provide-subscribers* to mutable globals list.

Evaluator: add missing (define *provide-batch-queue* (list)) and
(define *provide-subscribers* (dict)) — were only in hand-edited sx_ref.ml.

Known: 36 bind-tracking + 8 capability test failures on retranspilation
(pre-existing transpiler local-ref shadowing bug, not caused by FFI).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 17:22:33 +00:00
fb30351be2 Post-10d: JIT measurement infrastructure + compiler fixes
Measurement:
- JIT hit/miss/skip counters in sx_runtime.ml (jit_try_call)
- VM instruction counter enabled in run loop
- jit-enable, vm-counters, vm-counters-reset epoch commands
- Test runner --jit flag for opt-in JIT measurement
- Results (132 tests): 5.8% VM hit, 56% evaluator self-calls, 38% anon

Fixes:
- Move compile-provide, compile-scope, compile-guard,
  compile-guard-clauses inside define-library begin block
  (were orphaned outside, causing "Undefined symbol" JIT failures)
- Add deref primitive (signal unwrap with tracking)
- Add deref compiler dispatch
- Fix compile-expr for scope forms to handle non-keyword args

CEK pruning assessment: evaluator self-calls (56%) can't be pruned —
the CEK must evaluate itself. Real pruning requires self-hosting
compiler (Phase 2+). The VM correctly handles user code that
JIT-compiles. 2776/2776 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 14:32:48 +00:00
2cf4c73ab3 Step 10d: bytecode expansion — close the CEK gap
Tier 1 — Component keyword dispatch on VM:
- Components/islands JIT-compile bodies via jit_compile_comp
- parse_keyword_args matches keyword names against component params
- Added i_compiled field to island type for JIT cache
- Component calls no longer fall back to CEK

Tier 2 — OP_SWAP (opcode 7):
- New stack swap operation for future HO loop compilation
- HO forms already efficient via NativeFn + VmClosure callbacks

Tier 3 — Exception handler stack:
- OP_PUSH_HANDLER (35), OP_POP_HANDLER (36), OP_RAISE (37)
- VM gains handler_stack with frame depth tracking
- Compiler handles guard and raise as bytecode
- Functions with exception handling no longer cause JIT failure

Tier 4 — Scope forms as bytecode:
- Compiler handles provide, context, peek, scope, provide!,
  bind, emit!, emitted via CALL_PRIM sequences
- Functions using reactive scope no longer trigger JIT failure

4 new opcodes (SWAP, PUSH_HANDLER, POP_HANDLER, RAISE) → 37 total.
2776/2776 tests pass, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 13:19:25 +00:00
0e311f0c7d Step 10c: fix capabilities, closure-scope, define-library imports
- Initialize _cek_call_ref in sx_ref.ml — fixes 8 capabilities tests
- Rename test variable 'peek' to 'get-val' — collides with new peek
  special form. Fixes closure-scope-edge test.
- Add import clause handling to define-library — was silently skipping
  (import ...) inside library definitions. Fixes 4 define-library tests.

2767/2768 OCaml (1 pre-existing aser/render-to-sx issue).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 11:58:18 +00:00
fb262aa49b Step 10c: batch coalescing + global subscriber registry
Provide subscribers stored in global *provide-subscribers* dict (keyed
by name) instead of on provide frames. Fixes subscriber loss when
frames are reconstructed, and enables cross-cek_run notification.

Batch integration: batch-begin!/batch-end! primitives manage
*provide-batch-depth*. fire-provide-subscribers defers to queue when
depth > 0, batch-end! flushes deduped. signals.sx batch calls both.

context now prefers scope-peek over frame value — scope stack is the
source of truth since provide! always updates it (even in nested
cek_run where provide frames aren't on the kont).

2754/2768 OCaml (14 pre-existing). 32/32 WASM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 11:39:38 +00:00
44b520a9e9 Step 10c: fix bind subscriber re-evaluation — track names not frames
Root cause: context called inside lambdas (e.g. swap!) went through
nested cek_run with empty kont, so provide frames weren't found and
never tracked to *bind-tracking*.

Three changes in evaluator.sx:
- step-sf-context: track context names (not frames) to *bind-tracking*
  — names work across cek_run boundaries via scope-peek fallback
- bind continue: resolve tracked names to frames via kont-find-provide
  on rest-k before registering subscribers
- subscriber: use empty kont instead of kont-extract-provides — old
  approach created provide frames whose continue handlers called
  scope-pop!, corrupting the scope stack

2752/2768 OCaml tests pass (all 7 bind subscriber tests fixed).
32/32 WASM native tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 11:05:17 +00:00
a965731a33 Step 10c: bind CEK special form + provide-set frame + scope-stack integration
bind is now a CEK special form that captures its body unevaluated,
establishes a tracking context (*bind-tracking*), and registers
subscribers on provide frames when context reads are tracked.

- bind special form: step-sf-bind, make-bind-frame, bind continue handler
- provide-set frame: provide! evaluates value with kont (fixes peek bug)
- context tracking: step-sf-context appends to *bind-tracking* when active
- scope-stack fallback: provide pushes to scope stack for cek-call contexts
- CekFrame mutation: cf_remaining/cf_results/cf_extra2 now mutable
- Transpiler: subscribers + prev-tracking field mappings, *bind-tracking* in ml-mutable-globals
- Test fixes: string-append → str, restored edge-cases suite

Passing: bind returns initial value, bind with expression, bind with let,
bind no deps is static, bind with conditional deps, provide! updates/multiple/nil,
provide! computed new value, peek read-modify-write, guard inside bind,
bind with string-append, provide! same value does not notify, bind does not
fire on unrelated provide!, bind sees latest value, bind inside provide scope.

Remaining: subscriber re-evaluation on provide! (scope-stack key issue),
batch coalescing (no batch support yet).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 09:13:33 +00:00
98fd315f14 Step 10c: unified reactive model — peek + provide! special forms + tracking primitives
CEK evaluator integration:
- peek — non-tracking read from provide frame (like context but never subscribes)
- provide! — mutate value in provide frame (cf_extra made mutable)
- Both dispatch as special forms alongside provide/context

Scope-stack primitives (for adapter/island use):
- provide-reactive! / provide-pop-reactive! / provide-set! — signal-backed scope
- peek (primitive) — non-tracking scope read
- context (override) — tracking-aware scope read
- bind — tracked computation with auto-resubscription
- tracking-start! / tracking-stop! / tracking-active? — tracking context

12/13 user-authored peek/provide! tests pass.
bind integration with CEK context pending (scope vs kont gap).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 02:10:26 +00:00
b3e9ebee1d Step 10c: Vector type + unified reactive model test spec (34 tests)
- Vector of value array added to sx_types.ml (prior commit)
- Vector primitives in sx_primitives.ml (make-vector, vector-ref,
  vector-set!, vector-length, vector->list, list->vector)
- R7RS vector tests
- test-unified-reactive.sx: 34 tests specifying the unified reactive
  model (provide/context/peek/bind replacing signal/deref split).
  All 34 currently fail — implementation next.
- WASM binary rebuilt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 01:27:27 +00:00
244c669334 Revert cek_run import patch — caused infinite CEK loop on server
The cek_run import handling (resume after hook loads library) caused
cek_step_loop to infinite-loop during aser page rendering. Root cause
not yet identified — the resumed CEK state never reaches terminal.

Reverted to original cek_run that throws "IO suspension in non-IO
context". The 4 server startup errors are harmless (files load
partially, all needed symbols available via other paths).

Import hook re-entry guard and debug logging retained for future work.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 00:59:45 +00:00
143a2ebefe define-library handles (import ...) clauses
step_sf_define_library now processes import clauses by evaluating
(import lib-spec) in the library env. Previously import clauses
were silently ignored, so libraries couldn't use symbols from
other libraries.

2694/2694 tests pass (11 new).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 23:58:55 +00:00
5df21fca36 Step 10b: capability-based sandboxing
Capability primitives promoted from mcp_tree.ml to sx_primitives.ml:
- with-capabilities — push cap set, eval body, restore on exit/error
- current-capabilities — returns active capability list (nil = unrestricted)
- has-capability? — check if capability granted (true when unrestricted)
- require-capability! — raise if capability missing
- capability-restricted? — check if any restrictions active

Infrastructure: _cek_call_ref in sx_types.ml (forward ref pattern)
allows primitives to invoke the CEK evaluator without dependency cycles.

10 new tests: unrestricted defaults, scoping, nesting, restore-on-exit.
2693 total tests, 0 regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 23:51:25 +00:00
6e216038ba Fix import resolution: correct library paths + hook type mismatch
Root causes of server [http-load] errors:
1. _import_hook passed pre-computed string key to library_loaded_p
   which calls library_name_key(string) → sx_to_list(string) → crash.
   Fix: pass original list spec, not the string key.
2. resolve_library_path didn't check web/lib/ for (sx dom), (sx browser),
   (web boot-helpers). These libraries use namespace prefixes that don't
   match their file locations.

Server startup errors: 190 → 0.
2683/2684 tests pass (1 known: define-library import clause — spec gap).
New test file: spec/tests/test-import-bind.sx

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 23:44:22 +00:00
6fe3476e18 Step 9: mutable data structures — R7RS vectors
New Vector type (value array) with 11 primitives:
- make-vector, vector — constructors
- vector-ref, vector-set! — element access/mutation
- vector-length, vector?, vector-fill! — inspection
- vector->list, list->vector — conversion
- vector-copy — independent copy
- Element-wise equality in safe_eq

10 new tests (2668/2668 pass). Follows Record pattern (value array).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:56:10 +00:00
5ac1ca9756 Fix server import suspension, dist sync, JIT errors
- cek_run patched to handle import suspensions via _import_hook.
  define-library (import ...) now resolves cleanly on the server.
  IO suspension errors: 190 → 0. JIT failures: ~50 → 0.
- _import_hook wired in sx_server.ml to load .sx files on demand.
- compile-modules.js syncs source .sx files to dist/sx/ before
  compiling — eliminates stale bytecode from out-of-date copies.
- WASM binary rebuilt with all fixes.
- 2658/2658 tests pass (8 new — previously failing import tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:52:41 +00:00
b0a4be0f22 Step 8: numeric tower — exact/inexact predicates + truncate/remainder/modulo
7 new R7RS primitives on the float-based tower (Number of float unchanged):
- exact? / inexact? — integer detection via Float.is_integer
- exact->inexact / inexact->exact — identity / round-to-integer
- truncate — toward zero (floor for positive, ceil for negative)
- remainder — sign follows dividend (= Float.rem)
- modulo — sign follows divisor

8 new tests (2658/2658 pass). No type system, VM, compiler, or parser changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:25:40 +00:00
2f3e727a6f Transparent lazy module loading — code loads like data
When the VM or CEK hits an undefined symbol, it checks a symbol→library
index (built from manifest exports at boot), loads the library that
exports it, and returns the value. Execution continues as if the module
was always loaded. No import statements, no load-library! calls, no
Suspense boundaries — just call the function.

This is the same mechanism as IO suspension for data fetching. The
programmer doesn't distinguish between calling a local function and
calling one that needs its module fetched first. The runtime treats
code as just another resource.

Implementation:
- _symbol_resolve_hook in sx_types.ml — called by env_get_id (CEK path)
  and vm_global_get (VM path) when a symbol isn't found
- Symbol→library index built from manifest exports in sx-platform.js
- __resolve-symbol native calls __sxLoadLibrary, module loads, symbol
  appears in globals, execution resumes
- compile-modules.js extracts export lists into module-manifest.json
- Playground page demonstrates: (freeze-scope) triggers freeze.sxbc
  download transparently on first use

2650/2650 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:23:45 +00:00
aee4770a6a Lazy module loading: compiler loads on demand, playground page
- load-library! native: islands can declare module dependencies at
  hydration time, triggering on-demand .sxbc loading
- JIT compiler lazy-load: compiler.sxbc loads via setTimeout after boot,
  eliminating "JIT: compiler not loaded" errors
- _import_hook on sx_types: infrastructure for hosts to resolve import
  suspensions inside eval_expr (server wiring deferred to Step 8)
- Playground page (/sx/(tools.(playground))): REPL island that lazy-loads
  the compiler module when navigated to — demonstrates the full
  lazy loading pipeline

Known remaining issues:
- SPA navigation broken for pages using let-match (orchestration.sx,
  router.sx) — bytecode compiler doesn't handle let-match special form
- Server-side "IO suspension in non-IO context" during http_load_files —
  needs cek_run import handling (deferred to Step 8)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 20:34:08 +00:00
4baed1853c OCaml runtime: R7RS parameters, VM closure introspection, import suspension
- R7RS parameter primitives (make-parameter, parameter?, parameterize support)
- VM closure get_val introspection (vm-code, vm-upvalues, vm-name, vm-globals)
- Lazy list caching on vm_code for transpiled VM performance
- VM import suspension: check_io_suspension + resume_module for browser lazy loading
- 23 new R7RS tests (parameter-basic, parameterize-basic, syntax-rules-basic)
- Playwright bytecode-loading spec + WASM rebuild

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 18:48:51 +00:00
2727577702 VM import suspension for browser lazy loading
Bytecode compiler now emits OP_PERFORM for (import ...) and compiles
(define-library ...) bodies. The VM stores the import request in
globals["__io_request"] and stops the run loop — no exceptions needed.
vm-execute-module returns a suspension dict, vm-resume-module continues.

Browser: sx_browser.ml detects suspension dicts from execute_module and
returns JS {suspended, op, request, resume} objects. The sx-platform.js
while loop handles cascading suspensions via handleImportSuspension.

13 modules load via .sxbc bytecode in 226ms (manifest-driven), both
islands hydrate, all handlers wired. 2650/2650 tests pass including
6 new vm-import-suspension tests.

Also: consolidated sx-platform-2.js → sx-platform.js, fixed
vm-execute-module missing code-from-value call, fixed bootstrap.py
protocol registry transpiler issues.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 17:11:12 +00:00
efd0d9168f Step 7d complete: exhaustive match checking + evaluator cleanup
Match exhaustiveness analysis:
- check-match-exhaustiveness function in evaluator.sx
- lint-node in tree-tools.sx checks match forms during format-check
- Warns on: no wildcard/catch-all, boolean missing true/false case
- (match x (true "yes")) → "match may be non-exhaustive"

Evaluator cleanup:
- Added missing step-sf-callcc definition (was in old transpiled output)
- Added missing step-sf-case definition (was in old transpiled output)
- Removed protocol functions from bootstrap skip set (they transpile fine)
- Retranspiled VM (bootstrap_vm.py) for compatibility

2650 tests pass (+5 from new features).

All Step 7 features complete:
  7a: ->> |> as-> pipe operators
  7b: Dict patterns, &rest, let-match destructuring
  7c: define-protocol, implement, satisfies?
  7d: Exhaustive match checking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 15:43:57 +00:00
653be79c8d Step 7c complete: protocols (define-protocol, implement, satisfies?)
Trait-like dispatch system for record types:

  (define-record-type <point>
    (make-point x y) point? (x point-x) (y point-y))

  (define-protocol Displayable (show self))

  (implement Displayable <point>
    (show self (str (point-x self) "," (point-y self))))

  (show (make-point 3 4))              ;; => "3,4"
  (satisfies? "Displayable" (make-point 1 2))  ;; => true
  (satisfies? "Displayable" 42)        ;; => false

Implementation:
- *protocol-registry* global dict stores protocol specs + implementations
- define-protocol creates dispatch functions via eval-expr (dynamic lambdas)
- implement registers method lambdas keyed by record type name
- Dispatch: (type-of self) → lookup in protocol impls → call method
- satisfies? checks if a record type has implementations for a protocol

2645 tests pass (+1 from protocol self-test).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 15:29:35 +00:00
9607f3c44a Step 7b complete: rich destructuring (dict patterns, &rest, let-match)
Three new pattern matching features in evaluator.sx:

1. Dict patterns in match:
   (match {:name "Alice" :age 30}
     ({:name n :age a} (list n a)))  ;; => ("Alice" 30)

2. &rest in list patterns:
   (match (list 1 2 3 4 5)
     ((a b &rest tail) tail))  ;; => (3 4 5)

3. let-match form (sugar for match):
   (let-match {:x x :y y} {:x 3 :y 4}
     (+ (* x x) (* y y)))  ;; => 25

Also: transpiler fix — "extra" key added to CekFrame cf_extra mapping
(was the root cause of thread-last mode not being stored).

2644 tests pass, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 15:08:36 +00:00
cd414b96a7 Step 7a complete: ->> |> as-> pipe operators + transpiler fixes
Three new threading operators in evaluator.sx:
- ->> (thread-last): inserts value as last arg
- |> (pipe): alias for ->> (F#/OCaml convention)
- as-> (thread-anywhere): binds value to named variable

  (->> 10 (- 3))           ;; => -7  (thread-last: (- 3 10))
  (-> 10 (- 3))            ;; => 7   (thread-first: (- 10 3))
  (->> 1 (list 2 3))       ;; => (2 3 1)
  (as-> 5 x (+ x 1) (* x 2)) ;; => 12

Two transpiler bugs fixed:
1. Non-recursive functions (let without rec) weren't chained as `and`
   in the let rec block — became local bindings inside previous function
2. CekFrame "extra" field wasn't in the cf_extra key mapping — mode
   was always Nil, making thread-last fall through to thread-first

Also: added missing step-sf-case definition to evaluator.

2644 tests pass, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 14:29:40 +00:00
f814193c94 Step 7a WIP: ->> and as-> pipe operators (thread-last has transpiler bug)
Add to evaluator.sx:
- step-sf-thread-last: thread-last operator (inserts value at end)
- step-sf-thread-as: thread-anywhere with named binding
- thread-insert-arg-last: last-position insertion function
- step-sf-case: missing function (was in old transpiled output but not spec)
- Register ->>, |>, as-> in step-eval-list dispatch

Status:
- ->> dispatch works (enters thread-last correctly)
- HO forms (map, filter) with ->> work correctly
- Non-HO forms with ->> still use thread-first (transpiler bug)
- as-> binding fails (related transpiler bug)

Transpiler bug: thread_insert_arg_last definition body is merged with
step_continue in the let rec block. The transpiler incorrectly chains
them as one function. Need to investigate the let rec emission logic.

2644 tests still pass (no regressions from new operators).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 14:03:00 +00:00
e46cdf3d4d Wire transpiled VM as active execute_module — 2644 tests pass
The transpiled VM (sx_vm_ref.ml, from lib/vm.sx) is now the ACTIVE
bytecode execution engine. sx_server.ml and sx_browser.ml call
Sx_vm_ref.execute_module instead of Sx_vm.execute_module.

Results:
- OCaml tests: 2644 passed, 0 failed
- WASM tests: 32 passed, 0 failed
- Browser: zero errors, zero warnings, islands hydrate
- Server: pages render, JIT compiles, all routes work

The VM logic now lives in ONE place: lib/vm.sx (SX).
OCaml gets it via transpilation (bootstrap_vm.py).
JS/browser gets it via bytecode compilation (compile-modules.js).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 13:34:11 +00:00
54ee673050 Transpiled VM compiles with real native preamble
bootstrap_vm.py preamble now has real implementations for all 48
native OCaml functions: stack ops, frame access, upvalue capture,
closure creation, JIT dispatch, CEK fallback, env walking.

The transpiled sx_vm_ref.ml (25KB) compiles cleanly alongside
sx_vm.ml. 7 logic functions transpiled from vm.sx:
  vm-call, vm-resolve-ho-form, vm-call-external,
  vm-run, vm-step, vm-call-closure, vm-execute-module

Next: wire callers to use Sx_vm_ref instead of Sx_vm.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 13:23:27 +00:00
df89d8249b Transpiler native record support for VM types (Step 5.5 unblock)
- Add VmFrame/VmMachine types to sx_types.ml (alongside CekState/CekFrame)
- Add VmFrame/VmMachine value variants to the value sum type
- Extend get_val in sx_runtime.ml to dispatch on VmFrame/VmMachine fields
- Extend sx_dict_set_b for VmFrame/VmMachine field mutation
- Extend transpiler ml-emit-dict-native to detect VM dict patterns
  and emit native OCaml record construction (same mechanism as CekState)
- Retranspile evaluator — no diff (transpiler extension is additive)
- Update bootstrap_vm.py output location

The transpiler now handles 4 native record types:
  CekState (5 fields), CekFrame (10 fields),
  VmFrame (4 fields), VmMachine (5 fields)

Full VM replacement (sx_vm.ml → transpiled) still needs vm.sx
feature parity: JIT dispatch, CEK fallback, suspension handling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 12:39:20 +00:00
7b4c918773 Recompile all 26 .sxbc with define-library wrappers + fix eval/JIT
All 26 browser modules recompiled with define-library/import forms.
Compilation works without vm-compile-adapter (JIT pre-compilation
hangs with library wrappers in some JIT paths — skipped for now,
CEK compilation is ~34s total).

Key fixes:
- eval command: import-aware loop that handles define-library/import
  locally without touching the Python bridge pipe (avoids deadlock)
- compile-modules.js: skip vm-compile-adapter, bump timeout

2621/2621 OCaml tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:08:00 +00:00
2d7dd7d582 Step 5 piece 6: migrate 23 .sx files to define-library/import
Wraps all core .sx files in R7RS define-library with explicit export
lists, plus (import ...) at end for backward-compatible global re-export.

Libraries registered:
  (sx bytecode)      — 83 opcode constants
  (sx render)        — 15 tag registries + render helpers
  (sx signals)       — 23 reactive signal primitives
  (sx r7rs)          — 21 R7RS aliases
  (sx compiler)      — 42 compiler functions
  (sx vm)            — 32 VM functions
  (sx freeze)        — 9 freeze/thaw functions
  (sx content)       — 6 content store functions
  (sx callcc)        — 1 call/cc wrapper
  (sx highlight)     — 13 syntax highlighting functions
  (sx stdlib)        — 47 stdlib functions
  (sx swap)          — 13 swap algebra functions
  (sx render-trace)  — 8 render trace functions
  (sx harness)       — 21 test harness functions
  (sx canonical)     — 12 canonical serialization functions
  (web adapter-html) — 13 HTML renderer functions
  (web adapter-sx)   — 13 SX wire format functions
  (web engine)       — 33 hypermedia engine functions
  (web request-handler) — 4 request handling functions
  (web page-helpers) — 12 page helper functions
  (web router)       — 36 routing functions
  (web deps)         — 19 dependency analysis functions
  (web orchestration) — 59 page orchestration functions

Key changes:
- define-library now inherits parent env (env-extend env instead of
  env-extend make-env) so library bodies can access platform primitives
- sx_server.ml: added resolve_library_path + load_library_file for
  import resolution (maps library specs to file paths)
- cek_run_with_io: handles "import" locally instead of sending to
  Python bridge

2608/2608 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 21:48:54 +00:00
397d0f39c0 Re-bootstrap compiler + render after IO registry and Step 5 changes
Compiler (lib/compiler.sx):
- Fix emit-op return type: 8 definition form cases (defstyle,
  defhandler, defpage, etc.) and the perform case now return nil
  explicitly via (do (emit-op em N) nil) instead of bare emit-op
  which transpiled to unit-returning OCaml.
- compile_match PREAMBLE: return Nil instead of unit (was ignore).
- Added init wrapper to PREAMBLE (needed by compile-module).
- All 41 compiler functions re-transpiled cleanly.

Render (bootstrap_render.py): re-transpiled, no changes.

Runtime: restored keyword_p predicate (needed by defio-parse-kwargs!
in the transpiled evaluator).

2608/2608 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 21:26:20 +00:00
5f72801901 Step 3: IO registry — spec-level defio + io contract dispatch
Promotes defio from native OCaml special form to spec-level CEK
evaluator feature. The IO registry is now the contract layer between
evaluator and platform.

Evaluator additions (spec/evaluator.sx):
- *io-registry* mutable dict global (like *library-registry*)
- io-register!, io-registered?, io-lookup, io-names accessors
- defio-parse-kwargs! recursive keyword parser
- sf-defio processes (defio "name" :category :data :params (...) ...)
- "defio" dispatch in step-eval-list
- step-sf-io: the contract function — validates against registry,
  then delegates to perform for IO suspension
- "io" dispatch in step-eval-list

Native OCaml defio handlers removed from:
- sx_server.ml (~20 lines)
- sx_browser.ml (~20 lines)
- run_tests.ml (~18 lines)
All replaced with __io-registry alias to spec's *io-registry*.

IO accessor functions bound in run_tests.ml env so tests can
call io-registered?, io-lookup, io-names.

10 new tests (spec/tests/test-io-registry.sx):
- defio populates registry
- io-lookup returns spec with name/category/returns/doc
- io-registered?/io-names work correctly
- kwargs parsing (batchable, cacheable, params)
- io contract rejects unregistered ops
- io contract passes validation for registered ops

2608/2608 tests passing (+10 new).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 21:18:04 +00:00
9257b6a2d8 Step 5.5 phases 5-6: primitive :body specs + runtime trimming
Phase 5: Added :body implementations to 13 more primitives in
spec/primitives.sx (26/94 total, up from 13). New bodies for:
- Type predicates: nil?, boolean?, number?, string?, list?, dict?,
  continuation? — all via (= (type-of x) "typename")
- Comparisons: <=, >=, eq?, equal? — composed from <, >, =, identical?
- Logic: not — via (if x false true)
- Collections: empty? — via (or (nil? coll) (= (len coll) 0))

Remaining 68 are genuinely native (host string/list/dict/math ops).

Phase 6: Removed 43 unused wrapper functions from sx_runtime.ml
(489 → 414 lines, -78 lines). Dead code from pre-transpilation era:
predicate wrappers (nil_p, keyword_p, contains_p, etc.), signal
accessors (signal_set_value, notify_subscribers, etc.), scope
delegates (sx_collect, sx_emit, etc.), HO form stubs (map_indexed,
map_dict, for_each), handler def stubs (sf_defquery, sf_defaction,
sf_defpage).

2598/2598 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 20:27:31 +00:00