Commit Graph

232 Commits

Author SHA1 Message Date
6e27442d57 Step 17: streaming render — hyperscript enhancements, WASM builds, live server tests
Streaming chunked transfer with shell-first suspense and resolve scripts.
Hyperscript parser/compiler/runtime expanded for conformance. WASM static
assets added to OCaml host. Playwright streaming and page-level test suites.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 08:41:38 +00:00
7aefe4da8f Fix streaming: resolve scripts inside </body>, live server tests
The shell HTML included closing </body></html> tags. Resolve script
chunks arrived AFTER the document end — browser ignored them
(ERR_INCOMPLETE_CHUNKED_ENCODING). Now strips </body></html> from
shell, sends resolve scripts inside the body, closes document last.

Added live server Playwright tests that hit the actual streaming
endpoint and verify suspense slots resolve with content.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:40:49 +00:00
d4c0be52b1 Fix ListRef handling in streaming data — list from SX is ListRef in OCaml
The streaming render matched `List items` but SX's `(list ...)` produces
`ListRef` (mutable list) in the OCaml runtime. Data items were rejected
with "returned list, expected dict or list" — 0 resolve chunks sent.

Fixed both streaming render and AJAX paths to handle ListRef.
Added sandbox test for streaming-demo-data return type validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:34:50 +00:00
c850737c60 Async IO in streaming render — staggered resolve with io-sleep
Server (sx_server.ml):
- eval_with_io: CEK evaluator with IO suspension handling (io-sleep, import)
- io-sleep platform primitive: raises CekPerformRequest, resolved by eval_with_io
- Streaming render uses eval_with_io for data + content evaluation
- Data items with "delay" field sleep before resolving (async streaming)
- Removed hardcoded streaming-demo-data — application logic belongs in .sx

Application (streaming-demo.sx):
- streaming-demo-data defined in SX: 3 items with 1s/3s/5s delays
- Each item has delay, stream-id, and display data fields
- Shell renders instantly, slots fill progressively as IO completes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:19:21 +00:00
eaf5af4cd8 Step 17: streaming render — chunked transfer, shell-first suspense, resolve scripts
Server (sx_server.ml):
- Chunked HTTP transport (Transfer-Encoding: chunked)
- Streaming page detection via scan_defpages (:stream true)
- Shell-first render: outer layout + shell AST → aser → SSR → flush
- Data resolution: evaluate :data, render :content per slot, flush __sxResolve scripts
- AJAX streaming: synchronous eval + OOB swaps for SPA navigation
- SX URL → flat path conversion for defpage matching
- Error boundaries per resolve section
- streaming-demo-data helper for the demo page

Client (sx-platform.js):
- Sx.resolveSuspense: finds [data-suspense] element, parses SX, renders to DOM
- Fallback define for resolve-suspense when boot.sx imports fail in WASM
- __sxPending drain on boot (queued resolves from before sx.js loads)
- __sxResolve direct dispatch after boot

Tests (streaming.spec.js):
- 5 sandbox tests using real WASM kernel
- Suspense placeholder rendering, __sxResolve replacement, independent slot resolution
- Full layout with gutters, end-to-end resolve with streaming-demo/chunk components

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:12:28 +00:00
bca0d8e4e5 Step 15: bytecode + CEK state serialization — 16 tests
bytecode-serialize/deserialize: sxbc v2 format wrapping compiled code
dicts. cek-serialize/deserialize: cek-state v1 format wrapping suspended
CEK state (phase, request, env, kont). Both use SX s-expression
round-trip via inspect/parse. lib/serialize.sx has pure SX versions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:19:16 +00:00
99c5c44cc1 Step 14: source locations — pos-to-loc, error-loc, sx-parse-loc — 15 tests
Pure SX layer: pos-to-loc (offset→line/col), error-loc (parse result→loc),
format-parse-error (human-readable error with source context line).
OCaml platform: cst_to_ast_loc (CST spans→loc dicts), sx-parse-loc
primitive (parse with locations), source-loc accessor.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:03:45 +00:00
79ba9c2d40 Fix stepper SSR/hydration flash: server reads cookie, cache bypass
Three changes to eliminate the stepper flash:

1. home-stepper.sx: server path reads cookie via (get-cookie) for
   step-idx initial value. Client path reads document.cookie via
   def-store. Both default to 0 when no cookie exists.

2. sx_server.ml: bypass response cache when sx-home-stepper cookie
   is present. Render on main thread (not worker) so get-cookie
   sees the parsed request cookies.

3. site-full.spec.js: flash detection test sets cookie=7 via
   Playwright context, checks SSR HTML matches hydrated state.

Test: "No flash: SSR=7 hydrated=7 (cookie=7)" — passes.
Tested on fresh stack=site server subprocess.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 07:28:47 +00:00
7492ceac4e Restore hyperscript work on stable site base (908f4f80)
Reset to last known-good state (908f4f80) where links, stepper, and
islands all work, then recovered all hyperscript implementation,
conformance tests, behavioral tests, Playwright specs, site sandbox,
IO-aware server loading, and upstream test suite from f271c88a.

Excludes runtime changes (VM resolve hook, VmSuspended browser handler,
sx_ref.ml guard recovery) that need careful re-integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 19:29:56 +00:00
520424954b Sandbox bytecode loading: K.load + load-sxbc, bytecode param, web stack sxbc via loadModule
Bytecode modules now load correctly in sandbox mode. HS .sxbc modules
use K.load('(load-sxbc ...)') which syncs defines to eval env. Web stack
.sxbc modules use K.loadModule with import suspension drive loop.

K.eval used directly for expression eval (not thunk wrapper) so bytecode-
defined symbols are visible. Falls back to callFn thunk on IO suspension.

Sandbox now reproduces the bytecode repeat bug: source gives 6/6
suspensions, bytecode gives 4/6. Bug is in bytecode compilation of
when/do across perform boundaries, not the runtime wrapper.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 20:56:54 +00:00
a9066c0653 Persistent Lisp image for sx_eval: smart file reload + IO tracing
sx_eval now accepts files (smart-loaded by mtime — unchanged files skip),
trace_io (harness-wrapped IO capture), mock (evaluated platform overrides),
and setup params. Definitions survive between calls. sx_harness_eval also
uses smart loading. sx_write_file can create new files.

New lib/hyperscript/debug.sx: mock DOM platform for instant hyperscript
testing — compile and execute HS expressions against simulated elements,
see every DOM mutation and wait in the IO trace.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 19:56:38 +00:00
3155ba47f9 JIT: VM fast path, &rest support, locals scan, test runner fixes
- jit_compile_lambda: call compile directly via VM when it has bytecode
  (100-400x faster JIT compilation, server pre-warm 1.6s vs hung)
- code_from_value: scan bytecode for highest LOCAL_GET/SET slot to
  compute vc_locals correctly (fixes hyperscript LOCAL_GET overflow)
- code_from_value: accept both compiler keys (bytecode) and SX VM
  keys (vc-bytecode) for interop
- jit_compile_lambda: skip &key/:as params (compiler can't emit them)
- Test runner: seed VM globals with primitives + env bindings,
  native vm-execute-module with suspension fallback to SX version,
  _jit_refresh_globals syncs globals after module loading,
  VmSuspended + "VM undefined" caught and sentineled

3127/3127 without JIT, 3116/3127 with JIT (11 hyperscript on-event
parsing — specific closure/scope issue, not infrastructure).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:52:44 +00:00
387a6cb49e Refactor MCP tree server: dispatch table, caching, validation, subprocess cleanup
Break up the 1735-line handle_tool match into 45 individual handler functions
with hashtable-based dispatch. Add mtime-based file parse caching (AST + CST),
consolidated run_command helper replacing 9 bare open_process_in patterns,
require_file/require_dir input validation, and pagination (limit/offset) for
sx_find_across, sx_comp_list, sx_comp_usage. Also includes pending VM changes:
rest-arity support, hyperscript parser, compiler/transpiler updates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:12:57 +00:00
4d1079aa5e Fix JIT server hang: compiled compiler helpers loop on complex ASTs
Root cause: pre-compiled compiler helper functions (compile-expr,
compile-cond, etc.) produce bytecode that loops when processing
deeply nested ASTs like tw-resolve-style. The test suite passes
because _jit_compiling prevents compiled function execution during
compilation — all functions run via CEK. The server pre-compiled
helpers, so they ran as bytecode during compilation, triggering loops.

Fix:
- _jit_compiling guard on the "already compiled" hook branch prevents
  compiled functions from running during JIT compilation. Compilation
  always uses CEK (correct for all AST sizes). Normal execution uses
  bytecode (fast).
- "compile" itself marked as jit_failed_sentinel — never JIT compiled.
  Runs via CEK, while its helpers use bytecode for normal (non-compile)
  execution.
- Server hook uses call_closure (own VM per call) for IO suspension
  safety. MCP uses call_closure_reuse (fast, no IO needed).

The underlying bytecode bug in the compiled helpers remains — fixing
it requires diagnosing which specific helper loops and why. This is
tracked as a separate issue. Server now starts in ~30s (pre-warm)
and serves pages correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:17:51 +00:00
03278c640d Fix JIT compilation cascade + MCP robustness
Three interacting JIT bugs caused infinite loops and server hangs:

1. _jit_compiling cascade: the re-entrancy flag was local to each
   binary's hook. When vm_call triggered JIT compilation internally,
   compiler functions got JIT-compiled during compilation, creating
   infinite cascades. Fix: shared _jit_compiling flag in sx_vm.ml,
   set in jit_compile_lambda itself.

2. call_closure always created new VMs: every HO primitive callback
   (for-each, map, filter) allocated a fresh VM. With 43K+ calls
   during compilation, this was the direct cause of hangs. Fix:
   call_closure_reuse reuses the active VM by isolating frames and
   running re-entrantly. VmSuspended is handled by merging frames
   for proper IO resumption.

3. vm_call for compiled Lambdas: OP_CALL dispatching to a Lambda
   with cached bytecode created a new VM instead of pushing a frame
   on the current one. Fix: push_closure_frame directly.

Additional MCP server fixes:
- Hot-reload: auto-execv when binary on disk is newer (no restart needed)
- Robust JSON: to_int_safe/to_int_or handle null, string, int params
- sx_summarise depth now optional (default 2)
- Per-request error handling (malformed JSON doesn't crash server)
- sx_test uses pre-built binary (skips dune rebuild overhead)
- Timed module loading for startup diagnostics

sx_server.ml fixes:
- Uses shared _jit_compiling flag
- Marks lambdas as jit_failed_sentinel on compile failure (no retry spam)
- call_closure_reuse with VmSuspended frame merging for IO support

Compiled compiler bytecode bug: deeply nested cond/case/let forms
(e.g. tw-resolve-style) cause the compiled compiler to loop.
Workaround: _jit_compiling guard prevents compiled function execution
during compilation. Compilation uses CEK (slower but correct).
Test suite: 3127/3127 passed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 21:59:31 +00:00
d3ff4f7ef3 Fix POST body reading + handler param binding for POST requests
Two fixes in the HTTP server:

1. Read full POST body: the single 8192-byte read() could miss the body
   if it arrived in a separate TCP segment. Now parses Content-Length
   and reads remaining bytes in a loop.

2. Handler param binding: for POST/PUT/PATCH, check request-form before
   request-arg. The old (or (request-arg n) (request-form n)) pattern
   short-circuited on request-arg's "" default (truthy in SX), never
   reaching request-form.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 14:26:05 +00:00
3a9d113537 Fix JIT mutable closure bug: vm-global-get now checks closure env first
vm-global-get checked vm.globals before closure-env, while vm-global-set
wrote to closure-env first. This asymmetry meant set! mutations to mutable
closure variables (e.g. parser position counters) were invisible to sibling
closures reading via JIT — they saw stale snapshots in the globals table.

Reversed vm-global-get lookup order: closure env → globals → primitives,
matching vm-global-set. Also enabled JIT in the MCP harness (compiler.sx
loading, env_bind hook for live globals sync, jit_try_call hook) so
sx_harness_eval exercises the same code path as the server.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 12:08:08 +00:00
cf088a33b4 Step 18 (part 8): Playground page + compile handler + url_decode fix
sx/sx/hyperscript.sx — _hyperscript playground page at
  /sx/(applications.(hyperscript))
  - compile-result defcomp (SSR compilation display)
  - pipeline documentation (tokenize → parse → compile)
  - example showcases with pre-compiled output
  - sx-post form → handler for interactive compilation

sx/sx/handlers/hyperscript-api.sx — POST handler:
  /sx/(applications.(hyperscript.(api.compile)))
  Accepts source param, returns compiled SX + parse tree HTML
  NOTE: hs-parse returns (do) in server context — JIT/CEK runtime
  issue where parser closures don't evaluate correctly. Works in
  test runner (3127/3127). Investigating separately.

sx_server.ml — url_decode fix: decode + as space in form data
  Standard application/x-www-form-urlencoded uses + for spaces.

Nav: _hyperscript added to Applications section.
Config: handler:hs- prefix added for handler dispatch.

3127/3127 tests, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:10:19 +00:00
f5da2bcfd5 Step 18 (part 6): _hyperscript integration — _="..." attribute wiring
lib/hyperscript/integration.sx — connects compiled hyperscript to DOM:
  hs-handler(src) — compile source → callable (fn (me) ...) via eval-expr-cek
  hs-activate!(el) — read _="...", compile, execute with me=element
  hs-boot!() — scan document for [_] elements, activate all
  hs-boot-subtree!(root) — activate within subtree (for HTMX swaps)

Handler wraps compiled SX in (fn (me) (let ((it nil) (event nil)) ...))
so each element gets its own me binding and clean it/event state.
Double-activation prevented via data-hs-active marker.

12 integration tests verify full pipeline: source → compile → eval.
Handlers correctly bind me, support arithmetic, conditionals, sequences,
for loops, and repeat. 3111/3111 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:48:58 +00:00
9a57bd5beb Step 18 (part 5): _hyperscript runtime shims — 25 functions
lib/hyperscript/runtime.sx — thin wrappers over web/lib/dom.sx
primitives implementing hyperscript-specific semantics:

Event handling: hs-on, hs-on-every, hs-init
Async/timing: hs-wait (IO suspend), hs-wait-for, hs-settle
Classes: hs-toggle-class!, hs-toggle-between!, hs-take!
DOM insertion: hs-put! (into/before/after)
Navigation: hs-navigate!, hs-next, hs-previous, hs-query-first/last
Iteration: hs-repeat-times, hs-repeat-forever
Fetch: hs-fetch (json/text/html format dispatch)
Type coercion: hs-coerce (Int/Float/String/Boolean/Array)
Object creation: hs-make (Object/Array/Set/Map)
Behaviors: hs-install
Measurement: hs-measure
Transitions: hs-transition (CSS property + optional duration)

23 runtime + 7 end-to-end pipeline tests.
3099/3099 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:40:55 +00:00
c5d2fa8c96 Step 18 (part 4): _hyperscript compiler — AST → SX expressions
lib/hyperscript/compiler.sx — transforms parsed hyperscript AST into
SX expressions targeting web/lib/dom.sx primitives. Two entry points:
  hs-to-sx    — AST node → SX expression
  hs-to-sx-from-source — source string → SX (tokenize+parse+emit)

Compiler handles:
  Expressions: me/it/event, refs, queries, attrs, styles, locals,
    arithmetic, comparison, boolean, array literals, property access,
    DOM traversal (closest/next/previous/first/last), type conversion,
    membership test, exists/empty/matches/contains predicates
  Commands: add/remove/toggle class, set (var/attr/style/prop dispatch),
    put, if/else, do, wait, wait-for, log, send, trigger, hide, show,
    transition, repeat, fetch, call, return, throw, settle, go, append,
    tell (rebinds me), for, take, make, install, measure, inc/dec
  Features: on (with from/filter/every), init, def, behavior

Maps to SX primitives: dom-add-class, dom-set-attr, dom-set-style,
dom-set-prop, dom-query, dom-closest, dom-dispatch, dom-append, etc.

33 compiler tests across 10 suites. 3076/3076 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:31:32 +00:00
4cd0e77331 Step 18 (part 2): _hyperscript parser — token stream → SX AST
lib/hyperscript/parser.sx — parses token stream from hs-tokenize into
SX AST forms. Covers:
  Commands: add/remove/toggle class, set/put, log, hide/show, settle
  Events: on with from/filter, command sequences
  Sequencing: then, wait (with time units)
  Conditionals: if/then/else/end
  Expressions: property chains, it, comparisons, exists, refs
  DOM traversal: closest, next, previous
  Send/trigger events to targets
  Repeat: forever, N times
  Fetch/call with argument lists

55 tests across 12 suites. 3005/3005 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 07:41:17 +00:00
3336c4e957 Step 18 (part 1): _hyperscript tokenizer — 38 tests
lib/hyperscript/tokenizer.sx — tokenizes real _hyperscript syntax into
typed token stream. Handles:
  Keywords (on, set, add, toggle, if, then, from, etc.)
  DOM literals (.class, #id, @attr, *style, :local, <sel/>)
  Strings (single/double quoted, escapes), template literals
  Numbers (integers, decimals, time units: 100ms, 2s)
  Operators (==, !=, +, -, 's possessive)
  Punctuation (parens, brackets, braces, commas, dots)
  Line comments (// to EOL)

Parser will disambiguate .name as class vs property access from context.
Possessive 's correctly distinguished from single-quote strings.

2952/2952 tests, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 21:49:13 +00:00
5d88b363e4 Step 13: String/regex primitives — PCRE-compatible, cross-host
New primitives in sx_primitives.ml:
  char-at, char-code, parse-number — string inspection + conversion
  regex-match, regex-match?, regex-find-all — PCRE pattern matching
  regex-replace, regex-replace-first — PCRE substitution
  regex-split — split by PCRE pattern

Uses Re.Pcre (OCaml re library) so regex patterns use the same syntax
as JS RegExp — patterns in .sx files work identically on browser and
server. Replaces the old test-only regex-find-all stub.

Also: split now handles multi-char separators via Re.

176 new tests (10 suites). 2912/2912 total, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 20:38:40 +00:00
516f9c7186 Step 12: Parser combinator library — pure SX, 68 tests
lib/parser-combinators.sx — 46 definitions in 8 layers:
  0. Result constructors (make-ok, make-fail, ok?, result-value, etc.)
  1. Primitives (satisfy, parse-char, any-char, parse-string, eof)
  2. Core combinators (fmap, parse-bind, seq, alt, label, lazy-parser)
  3. Repetition (many, many1, optional, skip-many)
  4. Structural (between, sep-by, sep-by1, skip-left, skip-right,
     not-followed-by, look-ahead)
  5. Character classes (digit, letter, alpha-num, whitespace, skip-spaces)
  6. Literal parsers (number-literal, string-literal, identifier)
  7. Run function (run-parser)
  8. SX tokenizer (sx-comment, sx-keyword, sx-symbol, sx-number,
     sx-string, sx-token, sx-tokenize)

Self-tests by tokenizing SX: (define x 42), {:ok true}, (+ 1 (* 2 3)),
comments, negative numbers, nested parens, recursive grammars.

No evaluator changes. Pure HO functions + thunks for lazy recursion.
2868/2868 tests, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 20:14:51 +00:00
67ae88b87f Fix last 2 foreign-type-checking tests: ListRef match + #t→true
- run_tests.ml: foreign-check-args binding now matches ListRef (from
  the list primitive) in addition to List
- test-foreign.sx: replace #t with true in guard clauses — SX parser
  treats #t as a symbol, not a boolean

2800/2800 tests, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:49:37 +00:00
7f772e0f23 Fix transpiler append! emit for mutable globals + run_with_io error recovery
The transpiler's append! emit path didn't check ml-is-mutable-global?,
so (append! *provide-batch-queue* sub) wrote to a dead local variable
instead of the global _ref. This caused the combined test suite hang —
fire-provide-subscribers was silently broken before the local-ref shadow
removal, and now correctly modifies the global batch queue.

Also adds run_with_io error-to-raise conversion (kont_has_handler guard)
so native Eval_errors can be caught by CEK guard/handler-bind when running
through the test runner's IO-aware step loop.

2798/2800 tests pass. 2 foreign-type-checking failures remain: guard can't
catch Eval_error from native fns called through cek_run_iterative (the
handler dispatch itself uses cek_call which re-enters cek_run_iterative,
creating an infinite loop). Fix requires spec-level change: make (error)
use CEK raise instead of host-error.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:59:38 +00:00
b61b437ccd Transpiler local-ref shadowing fix + foreign test runner bindings
ml-scan-set now checks ml-is-mutable-global? before adding set!/append!
targets to the needs-ref list. Previously, mutable globals like
*bind-tracking* got local `ref Nil` shadows that masked the global _ref,
causing `append!: expected list, got nil` in 43 bind-tracking tests.

Test runner: bind foreign registry functions (foreign-registered?,
foreign-lookup, foreign-names, foreign-register!, foreign-resolve-binding,
foreign-check-args, foreign-build-lambda) + initialize _cek_call_ref for
with-capabilities. 22/24 foreign tests now pass, 8 capabilities tests fixed.

Retranspiled sx_ref.ml — all mutable global shadows eliminated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:29:13 +00:00
fb30351be2 Post-10d: JIT measurement infrastructure + compiler fixes
Measurement:
- JIT hit/miss/skip counters in sx_runtime.ml (jit_try_call)
- VM instruction counter enabled in run loop
- jit-enable, vm-counters, vm-counters-reset epoch commands
- Test runner --jit flag for opt-in JIT measurement
- Results (132 tests): 5.8% VM hit, 56% evaluator self-calls, 38% anon

Fixes:
- Move compile-provide, compile-scope, compile-guard,
  compile-guard-clauses inside define-library begin block
  (were orphaned outside, causing "Undefined symbol" JIT failures)
- Add deref primitive (signal unwrap with tracking)
- Add deref compiler dispatch
- Fix compile-expr for scope forms to handle non-keyword args

CEK pruning assessment: evaluator self-calls (56%) can't be pruned —
the CEK must evaluate itself. Real pruning requires self-hosting
compiler (Phase 2+). The VM correctly handles user code that
JIT-compiles. 2776/2776 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 14:32:48 +00:00
2cf4c73ab3 Step 10d: bytecode expansion — close the CEK gap
Tier 1 — Component keyword dispatch on VM:
- Components/islands JIT-compile bodies via jit_compile_comp
- parse_keyword_args matches keyword names against component params
- Added i_compiled field to island type for JIT cache
- Component calls no longer fall back to CEK

Tier 2 — OP_SWAP (opcode 7):
- New stack swap operation for future HO loop compilation
- HO forms already efficient via NativeFn + VmClosure callbacks

Tier 3 — Exception handler stack:
- OP_PUSH_HANDLER (35), OP_POP_HANDLER (36), OP_RAISE (37)
- VM gains handler_stack with frame depth tracking
- Compiler handles guard and raise as bytecode
- Functions with exception handling no longer cause JIT failure

Tier 4 — Scope forms as bytecode:
- Compiler handles provide, context, peek, scope, provide!,
  bind, emit!, emitted via CALL_PRIM sequences
- Functions using reactive scope no longer trigger JIT failure

4 new opcodes (SWAP, PUSH_HANDLER, POP_HANDLER, RAISE) → 37 total.
2776/2776 tests pass, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 13:19:25 +00:00
fb7338c022 Fix last test: unify scope stacks between test runner and evaluator
run_tests.ml had a local _scope_stacks hash table, separate from
Sx_primitives._scope_stacks used by the CEK evaluator. SX-level
scope-push!/scope-peek used the local table, but step-sf-context's
scope_peek used the global one. Aser's provide handler pushed to
one table, context read from the other — always got nil.

Fix: alias run_tests.ml's _scope_stacks to Sx_primitives._scope_stacks.

2768/2768 OCaml tests pass. Zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 12:20:23 +00:00
fb262aa49b Step 10c: batch coalescing + global subscriber registry
Provide subscribers stored in global *provide-subscribers* dict (keyed
by name) instead of on provide frames. Fixes subscriber loss when
frames are reconstructed, and enables cross-cek_run notification.

Batch integration: batch-begin!/batch-end! primitives manage
*provide-batch-depth*. fire-provide-subscribers defers to queue when
depth > 0, batch-end! flushes deduped. signals.sx batch calls both.

context now prefers scope-peek over frame value — scope stack is the
source of truth since provide! always updates it (even in nested
cek_run where provide frames aren't on the kont).

2754/2768 OCaml (14 pre-existing). 32/32 WASM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 11:39:38 +00:00
b8f389ac9b Import hook: verify library registered, re-entry guard
- _import_hook verifies library_loaded_p AFTER load_library_file
  to catch cases where the file loads but define-library doesn't register
- Re-entry guard (_loading_libs) prevents infinite retry loops
- cek_run import patch deferred — retry approach infinite-loops because
  cek_step_loop re-enters deeply nested eval contexts. Root cause:
  eval_expr → cek_run → cek_step_loop processes the ENTIRE remaining
  kont chain after import resolution, which includes rendering code that
  triggers MORE eval_expr calls. Needs architectural solution (step-level
  suspension handling, not run-level).

Server runs with 4 harmless IO suspension errors. These don't affect
functionality — symbols load via the global env.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 01:16:03 +00:00
244c669334 Revert cek_run import patch — caused infinite CEK loop on server
The cek_run import handling (resume after hook loads library) caused
cek_step_loop to infinite-loop during aser page rendering. Root cause
not yet identified — the resumed CEK state never reaches terminal.

Reverted to original cek_run that throws "IO suspension in non-IO
context". The 4 server startup errors are harmless (files load
partially, all needed symbols available via other paths).

Import hook re-entry guard and debug logging retained for future work.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 00:59:45 +00:00
6e216038ba Fix import resolution: correct library paths + hook type mismatch
Root causes of server [http-load] errors:
1. _import_hook passed pre-computed string key to library_loaded_p
   which calls library_name_key(string) → sx_to_list(string) → crash.
   Fix: pass original list spec, not the string key.
2. resolve_library_path didn't check web/lib/ for (sx dom), (sx browser),
   (web boot-helpers). These libraries use namespace prefixes that don't
   match their file locations.

Server startup errors: 190 → 0.
2683/2684 tests pass (1 known: define-library import clause — spec gap).
New test file: spec/tests/test-import-bind.sx

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 23:44:22 +00:00
5ac1ca9756 Fix server import suspension, dist sync, JIT errors
- cek_run patched to handle import suspensions via _import_hook.
  define-library (import ...) now resolves cleanly on the server.
  IO suspension errors: 190 → 0. JIT failures: ~50 → 0.
- _import_hook wired in sx_server.ml to load .sx files on demand.
- compile-modules.js syncs source .sx files to dist/sx/ before
  compiling — eliminates stale bytecode from out-of-date copies.
- WASM binary rebuilt with all fixes.
- 2658/2658 tests pass (8 new — previously failing import tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:52:41 +00:00
2727577702 VM import suspension for browser lazy loading
Bytecode compiler now emits OP_PERFORM for (import ...) and compiles
(define-library ...) bodies. The VM stores the import request in
globals["__io_request"] and stops the run loop — no exceptions needed.
vm-execute-module returns a suspension dict, vm-resume-module continues.

Browser: sx_browser.ml detects suspension dicts from execute_module and
returns JS {suspended, op, request, resume} objects. The sx-platform.js
while loop handles cascading suspensions via handleImportSuspension.

13 modules load via .sxbc bytecode in 226ms (manifest-driven), both
islands hydrate, all handlers wired. 2650/2650 tests pass including
6 new vm-import-suspension tests.

Also: consolidated sx-platform-2.js → sx-platform.js, fixed
vm-execute-module missing code-from-value call, fixed bootstrap.py
protocol registry transpiler issues.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 17:11:12 +00:00
e46cdf3d4d Wire transpiled VM as active execute_module — 2644 tests pass
The transpiled VM (sx_vm_ref.ml, from lib/vm.sx) is now the ACTIVE
bytecode execution engine. sx_server.ml and sx_browser.ml call
Sx_vm_ref.execute_module instead of Sx_vm.execute_module.

Results:
- OCaml tests: 2644 passed, 0 failed
- WASM tests: 32 passed, 0 failed
- Browser: zero errors, zero warnings, islands hydrate
- Server: pages render, JIT compiles, all routes work

The VM logic now lives in ONE place: lib/vm.sx (SX).
OCaml gets it via transpilation (bootstrap_vm.py).
JS/browser gets it via bytecode compilation (compile-modules.js).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 13:34:11 +00:00
7b4c918773 Recompile all 26 .sxbc with define-library wrappers + fix eval/JIT
All 26 browser modules recompiled with define-library/import forms.
Compilation works without vm-compile-adapter (JIT pre-compilation
hangs with library wrappers in some JIT paths — skipped for now,
CEK compilation is ~34s total).

Key fixes:
- eval command: import-aware loop that handles define-library/import
  locally without touching the Python bridge pipe (avoids deadlock)
- compile-modules.js: skip vm-compile-adapter, bump timeout

2621/2621 OCaml tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:08:00 +00:00
ac772ac357 IO-aware eval for server dispatch + compile-modules timeout bump
sx_server.ml: the "eval" command now uses cek_run_with_io instead of
raw eval_expr. This handles import suspensions during eval-blob
(needed for .sx files with define-library/import wrappers).

compile-modules.js: timeout bumped 5min → 10min for sxbc compilation
with define-library overhead.

2608/2608 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 23:31:41 +00:00
2d7dd7d582 Step 5 piece 6: migrate 23 .sx files to define-library/import
Wraps all core .sx files in R7RS define-library with explicit export
lists, plus (import ...) at end for backward-compatible global re-export.

Libraries registered:
  (sx bytecode)      — 83 opcode constants
  (sx render)        — 15 tag registries + render helpers
  (sx signals)       — 23 reactive signal primitives
  (sx r7rs)          — 21 R7RS aliases
  (sx compiler)      — 42 compiler functions
  (sx vm)            — 32 VM functions
  (sx freeze)        — 9 freeze/thaw functions
  (sx content)       — 6 content store functions
  (sx callcc)        — 1 call/cc wrapper
  (sx highlight)     — 13 syntax highlighting functions
  (sx stdlib)        — 47 stdlib functions
  (sx swap)          — 13 swap algebra functions
  (sx render-trace)  — 8 render trace functions
  (sx harness)       — 21 test harness functions
  (sx canonical)     — 12 canonical serialization functions
  (web adapter-html) — 13 HTML renderer functions
  (web adapter-sx)   — 13 SX wire format functions
  (web engine)       — 33 hypermedia engine functions
  (web request-handler) — 4 request handling functions
  (web page-helpers) — 12 page helper functions
  (web router)       — 36 routing functions
  (web deps)         — 19 dependency analysis functions
  (web orchestration) — 59 page orchestration functions

Key changes:
- define-library now inherits parent env (env-extend env instead of
  env-extend make-env) so library bodies can access platform primitives
- sx_server.ml: added resolve_library_path + load_library_file for
  import resolution (maps library specs to file paths)
- cek_run_with_io: handles "import" locally instead of sending to
  Python bridge

2608/2608 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 21:48:54 +00:00
5f72801901 Step 3: IO registry — spec-level defio + io contract dispatch
Promotes defio from native OCaml special form to spec-level CEK
evaluator feature. The IO registry is now the contract layer between
evaluator and platform.

Evaluator additions (spec/evaluator.sx):
- *io-registry* mutable dict global (like *library-registry*)
- io-register!, io-registered?, io-lookup, io-names accessors
- defio-parse-kwargs! recursive keyword parser
- sf-defio processes (defio "name" :category :data :params (...) ...)
- "defio" dispatch in step-eval-list
- step-sf-io: the contract function — validates against registry,
  then delegates to perform for IO suspension
- "io" dispatch in step-eval-list

Native OCaml defio handlers removed from:
- sx_server.ml (~20 lines)
- sx_browser.ml (~20 lines)
- run_tests.ml (~18 lines)
All replaced with __io-registry alias to spec's *io-registry*.

IO accessor functions bound in run_tests.ml env so tests can
call io-registered?, io-lookup, io-names.

10 new tests (spec/tests/test-io-registry.sx):
- defio populates registry
- io-lookup returns spec with name/category/returns/doc
- io-registered?/io-names work correctly
- kwargs parsing (batchable, cacheable, params)
- io contract rejects unregistered ops
- io contract passes validation for registered ops

2608/2608 tests passing (+10 new).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 21:18:04 +00:00
19e7a6ee2d Step 5.5 phases 1-2: merge sx_scope into sx_primitives, 4-child define support
Phase 1: Absorb sx_scope.ml (180 lines) into sx_primitives.ml. Scope stacks,
cookies, and trace infrastructure now live alongside other primitives. All 20
scope primitive registrations moved. References updated in sx_server.ml and
sx_browser.ml. sx_scope.ml deleted.

Phase 2: Transpiler handles (define name :effects (...) (fn ...)) forms.
ml-emit-define and ml-emit-define-body detect keyword at position 2 and use
(last expr) instead. Unblocks transpilation of spec/render.sx and
web/adapter-html.sx which use 4-child defines extensively.

2598/2598 tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 19:43:26 +00:00
1dd4c87d64 Step 5: CEK IO suspension + R7RS modules (define-library/import)
Third CEK phase "io-suspended": perform suspends evaluation, host
resolves IO, cek-resume feeds result back. VM OP_PERFORM (opcode 112)
enables JIT-compiled functions to suspend. VM→CEK→suspend chain
propagates suspension across the JIT/CEK boundary via pending_cek.

R7RS define-library creates isolated environments with export control.
import checks the library registry and suspends for unknown libraries,
enabling lazy on-demand loading. Import qualifiers: only, prefix.

Server-side cek_run_with_io handles suspension by dispatching IO
requests to the Python bridge and resuming. guard composes cleanly
with perform for structured error recovery across IO boundaries.

2598/2598 tests (30 new: 15 core suspension, 3 JIT, 1 cross-boundary,
9 modules, 2 error handling). Zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 18:55:43 +00:00
9b8a8dd272 Remove Comment variant and old comment-mode parser — CST handles all
Delete from sx_types.ml:
- Comment of string variant (no longer needed)

Delete from sx_parser.ml:
- _preserve_comments mutable ref
- collect_comment_node function
- comment-mode branches in read_value, read_list
- ~comments parameter from parse_all and parse_file
- skip_whitespace and read_comment (only used by old comment mode)

Delete from mcp_tree.ml:
- has_interior_comments function
- Comment handling in pretty_print_value
- pretty_print_file function (replaced by CST write-back)
- ~comments parameter from local parse_file

Migrate sx_pretty_print, sx_write_file, sx_doc_gen to CST path.
Net: -69 lines. 24/24 CST round-trips, 2583/2583 evaluator tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 18:19:19 +00:00
af63d49451 Migrate MCP tools from comment_map to CST-based round-tripping
Replace the comment preservation workaround (comment_map type,
separate_comments, reinterleave, strip_interior_comments,
extract_fragment_comments, inject_comments — ~170 lines) with
CST-based editing (~80 lines).

write_edit_cst: compares old AST vs new AST per node. Unchanged
nodes keep original source verbatim. Changed nodes are pretty-printed
with original leading trivia preserved. New nodes (insertions) get
minimal formatting.

parse_file_cst: returns (AST tree, CST file). AST goes to tree-tools,
CST is used for write-back.

extract_cst_comments / inject_cst_comments: read comment trivia from
CST nodes for summarise/read_tree display.

Net: -39 lines. 24/24 CST round-trip tests, 2583/2583 evaluator tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 18:13:34 +00:00
5390df7b0b CST parser: lossless concrete syntax tree for .sx files
New sx_cst.ml: CstAtom, CstList, CstDict node types with leading/trailing
trivia (whitespace + comments). Two projections:
- cst_to_source/cst_file_to_source: exact source reconstruction
- cst_to_ast: strip trivia → Sx_types.value for evaluation

New parse_all_cst/parse_file_cst in sx_parser.ml: parallel CST parser
alongside existing AST parser. Reuses read_string, read_symbol, try_number.
Trivia collected via collect_trivia (replaces skip_whitespace_and_comments).

Round-trip invariant: cst_file_to_source(parse_all_cst(src)) = src
Verified on 13 synthetic tests + 7 real codebase files (101KB evaluator,
parser, primitives, render, tree-tools, engine, io).

CST→AST equivalence: cst_to_ast matches parse_all output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 18:07:35 +00:00
36acb56a3a Quiet noisy JIT compilation logs: only log slow (>500ms) or failed compiles
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 17:08:24 +00:00
38556af423 Interior comments, fragment comments, get_siblings + doc_gen comment support
Parser: read_value/read_list now capture Comment nodes inside lists
when ~comments:true. Module-level _preserve_comments ref threads the
flag through the recursive descent without changing signatures.

Pretty printer: has_interior_comments (recursive) forces multi-line
when any nested list contains comments. Comment nodes inside lists
emit as indented comment lines.

Edit tools: separate_comments strips interior comments recursively
via strip_interior_comments before passing to tree-tools (paths stay
correct). extract_fragment_comments parses new source with comments,
attaches leading comments to the target position in the comment map.

sx_get_siblings: injects comments for top-level siblings.

sx_doc_gen: parses with comments, tracks preceding Comment node,
includes cleaned comment text in generated component documentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 17:00:56 +00:00
033b2cb304 Add section comments to evaluator.sx, show comments in sx_summarise
evaluator.sx: 11 section headers + 27 subgroup/function comments
documenting the CEK machine structure (state, frames, kont ops,
extension points, eval utilities, machine core, special forms,
call dispatch, HO forms, continue phase, entry points).

mcp_tree.ml: sx_summarise and sx_read_tree now inject file comments
into their output — comments appear as un-numbered annotation lines
between indexed entries, so indices stay correct for editing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 16:45:39 +00:00