Commit Graph

243 Commits

Author SHA1 Message Date
4aa49e42e8 htmx demos working: activation, fetch, swap, OOB filtering, test runner page
- htmx-boot-subtree! wired into process-elements for auto-activation
- Fixed cond compilation bug in hx-verb-info (Clojure-style flat cond)
- Platform io-fetch upgraded: method/body/headers support, full response dict
- Replaced perform IO ops with browser primitives (set-timeout, browser-confirm, etc)
- SX→HTML rendering in hx-do-swap with OOB section filtering
- hx-collect-params: collects input name/value for all methods
- Handler naming: ex-{slug} convention, removed perform IO dependencies
- Test runner page at (test.(applications.(htmx))) with iframe-based runner
- Header "test" link on every page linking to test URL
- Page file restructure: 285 files moved to URL-matching paths (a/b/c/index.sx)
- page-functions.sx: ~100 component name references updated
- _test added to skip_dirs, test- file prefix convention for test files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 11:56:15 +00:00
b81c26c45b HS tests: sync textContent when innerHTML is set
When innerHTML is set on a mock element, textContent now updates to
match (with HTML tags stripped). Many HS tests do `put "foo" into me`
(which sets innerHTML) then check textContent. Previously textContent
stayed empty because only innerHTML was updated.

Also fixes innerHTML="" to fully detach children from parent.

393 → 408/831 HS tests (+15).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 12:58:23 +00:00
23e8379622 HS tests: innerHTML/textContent clears children in mock DOM
Setting innerHTML="" on a mock element now detaches and removes all
children, matching browser behavior. Previously hs-cleanup! (which
sets body.innerHTML="") left stale children attached, causing
querySelector to find elements from prior tests.

Also clears children when textContent is set (browser behavior).

375 → 393/831 HS tests (+18).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 12:47:34 +00:00
a6eb125dcc HS tests: stringify DOM properties (innerHTML, textContent, value)
In a real browser, innerHTML/textContent/value are always strings.
The mock was storing raw SX values (Number, Bool, Nil), causing type
mismatches like "Expected 1, got 1" where the value was correct but
Number 1.0 != String "1".

Now coerces to string on host-set! for innerHTML, textContent, value,
outerHTML, innerText. Fixes 10 increment tests that were doing
`put value into me` with numeric results.

367 → 375/831 HS tests (+8 net, +10 new passes, -2 regressions).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 12:38:33 +00:00
e3eb46d0dc HS tests: SIGALRM + raise timeout for native OCaml loops
The infinite loops in the HS parser are in transpiled native OCaml code,
not in the VM or CEK step loop. Neither step counters (in cek_step_loop,
cek_step, trampoline) nor VM instruction checks caught them because
the loops are in direct OCaml recursion.

Fix: SIGALRM handler raises Eval_error to break out of native loops.
Also sets step_limit flag to catch VM loops. Combined approach handles
both native OCaml recursion (alarm+raise) and VM bytecode (step check).

The alarm+raise can become unreliable after ~13 timeouts in a single
process, but handles the common case well. Reverts the fork-based
approach which lost inter-test state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 11:57:33 +00:00
3d7fffe4eb HS tests: host-get method truthiness + fork-based test timeout
Two critical fixes for the mock DOM test runner:

1. host-get returns truthy for DOM method names on mock elements.
   dom.sx guards like `(and el (host-get el "setAttribute"))` were
   silently skipping setAttribute/getAttribute calls because the mock
   dict had no "setAttribute" key. Now returns Bool true for known
   DOM method names, fixing hs-activate! → dom-set-attr → dom-get-attr
   chain. Also adds firstElementChild, nextElementSibling, etc. as
   computed properties.

2. Fork-based per-test timeout (5 seconds). The HS parser has infinite
   loops on certain syntax ([@attr], complex put targets). Signal-based
   alarm doesn't work reliably in OCaml 5. Fork + waitpid + select
   gives hard OS-level timeout protection.

Also adds step_limit/step_count to sx_ref.ml trampoline (currently
unused but available for future CEK-level timeout).

Result: 525/963 total, up from 498. Many more add/remove/toggle/set
tests now pass because hs-activate! actually wires up event handlers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 11:04:03 +00:00
1d83ccba3c Content-addressed on-demand loading: Merkle DAG for all browser assets
Replace the monolithic 500KB <script data-components> block with a 25KB
JSON manifest mapping names to content hashes. Every definition —
components, islands, macros, client libraries, bytecode modules, and
WASM binaries — is now content-addressed and loaded on demand.

Server (sx_server.ml):
- build_hash_index: Merkle DAG over all definitions — topological sort,
  hash leaves first, component refs become @h:{hash} in instantiated form
- /sx/h/{hash} endpoint: serves definitions with Cache-Control: immutable
- Per-page manifest in <script data-sx-manifest> with defs + modules + boot
- Client library .sx files hashed as whole units (tw.sx, tw-layout.sx, etc.)
- .sxbc modules and WASM kernel hashed individually

Browser (sx-platform.js):
- Content-addressed boot: inline script loads kernel + platform by hash
- loadDefinitionByHash: recursive dep resolution with @h: rewriting
- resolveHash: 3-tier cache (memory → localStorage → fetch /sx/h/{hash})
- __resolve-symbol extended for manifest-based component + library loading
- Cache API wrapper intercepts .wasm fetches for offline caching
- Eager pre-loading of plain symbol deps for CEK evaluator compatibility

Shell template (shell.sx):
- Monolithic <script data-components> removed
- data-sx-manifest script with full hash manifest
- Inline bootstrap replaces <script src="...?v="> with CID-based loading

Second visit loads zero bytes from network. Changed content gets a new
hash — only that item refetched (Merkle propagation).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:14:39 +00:00
2cba359fdf HS behavioral tests: mock DOM + eval-hs in OCaml test runner
Add mock DOM layer to run_tests.ml so hyperscript behavioral tests
(spec/tests/test-hyperscript-behavioral.sx) can run in the OCaml test
runner without a browser. Previously these tests required Playwright
which crashed after 10 minutes from WASM page reboots.

Mock DOM implementation:
- host-global, host-get, host-set!, host-call, host-new, host-callback,
  host-typeof, host-await — OCaml primitives operating on SX Dict elements
- Mock elements with classList, style, attributes, event dispatch + bubbling
- querySelector/querySelectorAll with #id, .class, tag, [attr] selectors
- Load web/lib/dom.sx and web/lib/browser.sx for dom-* wrappers
- eval-hs function for expression-only tests (comparisonOperator, etc.)

Result: 367/831 HS tests pass in ~30 seconds (was: Playwright crash).
14 suites at 100%: live, component, liveTemplate, scroll, call, go,
focus, log, reactive-properties, resize, measure, attributeRef,
objectLiteral, queryRef.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 10:03:26 +00:00
1eadefd0c1 Step 17b: Pretext — DOM-free text layout with otfm font measurement
Pure SX text layout library with one IO boundary (text-measure perform).
Knuth-Plass optimal line breaking, Liang's hyphenation, position calculation.

Library (lib/text-layout.sx):
- break-lines: Knuth-Plass DP over word widths
- break-lines-greedy: simple word-wrap for comparison
- hyphenate-word: Liang's trie algorithm
- position-line/position-lines: running x/y sums
- measure-text: single perform (text-measure IO)

Server font measurement (otfm):
- Reads OpenType cmap + hmtx tables from .ttf files
- DejaVu Serif/Sans bundled in shared/static/fonts/
- _cek_io_resolver hook: perform works inside aser/eval_expr
- JIT VM suspension inline resolution for IO in compiled code

~font component (shared/sx/templates/font.sx):
- Works like ~tw: emits @font-face CSS via cssx scope
- Sets font-family on parent via spread
- Deduplicates font declarations

Infrastructure fixes:
- stdin load command: per-expression error handling (was aborting on first error)
- cek_run IO hook: _cek_io_resolver in sx_types.ml
- JIT VmSuspended: inline IO resolution when resolver installed
- ListRef handling in IO resolver (perform creates ListRef, not List)

Demo page at /sx/(applications.(pretext)):
- Hero: justified paragraph with otfm-measured proportional widths
- Greedy vs Knuth-Plass side-by-side comparison
- Badness scoring visualization
- Hyphenation syllable decomposition

25 new tests (spec/tests/test-text-layout.sx), 3201/3201 passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 15:13:00 +00:00
f60d22e86e Hyperscript: focus command, diagnostic test output, blur keyword
Parser/compiler/runtime for focus command. Tokenizer: focus, blur,
precedes, follows, ignoring, case keywords. Test spec: per-test
failure output for diagnosis.

374/831 (45%)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 12:38:05 +00:00
854ed9c027 Hyperscript conformance: 372→373 — hide/show strategy, generator toEqual
Parser: hide/show handle `with opacity/visibility/display` strategy,
target detection for then-less chaining (add/remove/set/put as boundary).
Generator: inline run().toEqual([...]) pattern for eval-only tests.
Compiler: hide/show emit correct CSS property per strategy.

373/831 (45%)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 11:42:28 +00:00
6e27442d57 Step 17: streaming render — hyperscript enhancements, WASM builds, live server tests
Streaming chunked transfer with shell-first suspense and resolve scripts.
Hyperscript parser/compiler/runtime expanded for conformance. WASM static
assets added to OCaml host. Playwright streaming and page-level test suites.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 08:41:38 +00:00
7aefe4da8f Fix streaming: resolve scripts inside </body>, live server tests
The shell HTML included closing </body></html> tags. Resolve script
chunks arrived AFTER the document end — browser ignored them
(ERR_INCOMPLETE_CHUNKED_ENCODING). Now strips </body></html> from
shell, sends resolve scripts inside the body, closes document last.

Added live server Playwright tests that hit the actual streaming
endpoint and verify suspense slots resolve with content.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:40:49 +00:00
d4c0be52b1 Fix ListRef handling in streaming data — list from SX is ListRef in OCaml
The streaming render matched `List items` but SX's `(list ...)` produces
`ListRef` (mutable list) in the OCaml runtime. Data items were rejected
with "returned list, expected dict or list" — 0 resolve chunks sent.

Fixed both streaming render and AJAX paths to handle ListRef.
Added sandbox test for streaming-demo-data return type validation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:34:50 +00:00
c850737c60 Async IO in streaming render — staggered resolve with io-sleep
Server (sx_server.ml):
- eval_with_io: CEK evaluator with IO suspension handling (io-sleep, import)
- io-sleep platform primitive: raises CekPerformRequest, resolved by eval_with_io
- Streaming render uses eval_with_io for data + content evaluation
- Data items with "delay" field sleep before resolving (async streaming)
- Removed hardcoded streaming-demo-data — application logic belongs in .sx

Application (streaming-demo.sx):
- streaming-demo-data defined in SX: 3 items with 1s/3s/5s delays
- Each item has delay, stream-id, and display data fields
- Shell renders instantly, slots fill progressively as IO completes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:19:21 +00:00
eaf5af4cd8 Step 17: streaming render — chunked transfer, shell-first suspense, resolve scripts
Server (sx_server.ml):
- Chunked HTTP transport (Transfer-Encoding: chunked)
- Streaming page detection via scan_defpages (:stream true)
- Shell-first render: outer layout + shell AST → aser → SSR → flush
- Data resolution: evaluate :data, render :content per slot, flush __sxResolve scripts
- AJAX streaming: synchronous eval + OOB swaps for SPA navigation
- SX URL → flat path conversion for defpage matching
- Error boundaries per resolve section
- streaming-demo-data helper for the demo page

Client (sx-platform.js):
- Sx.resolveSuspense: finds [data-suspense] element, parses SX, renders to DOM
- Fallback define for resolve-suspense when boot.sx imports fail in WASM
- __sxPending drain on boot (queued resolves from before sx.js loads)
- __sxResolve direct dispatch after boot

Tests (streaming.spec.js):
- 5 sandbox tests using real WASM kernel
- Suspense placeholder rendering, __sxResolve replacement, independent slot resolution
- Full layout with gutters, end-to-end resolve with streaming-demo/chunk components

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 16:12:28 +00:00
bca0d8e4e5 Step 15: bytecode + CEK state serialization — 16 tests
bytecode-serialize/deserialize: sxbc v2 format wrapping compiled code
dicts. cek-serialize/deserialize: cek-state v1 format wrapping suspended
CEK state (phase, request, env, kont). Both use SX s-expression
round-trip via inspect/parse. lib/serialize.sx has pure SX versions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:19:16 +00:00
99c5c44cc1 Step 14: source locations — pos-to-loc, error-loc, sx-parse-loc — 15 tests
Pure SX layer: pos-to-loc (offset→line/col), error-loc (parse result→loc),
format-parse-error (human-readable error with source context line).
OCaml platform: cst_to_ast_loc (CST spans→loc dicts), sx-parse-loc
primitive (parse with locations), source-loc accessor.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:03:45 +00:00
79ba9c2d40 Fix stepper SSR/hydration flash: server reads cookie, cache bypass
Three changes to eliminate the stepper flash:

1. home-stepper.sx: server path reads cookie via (get-cookie) for
   step-idx initial value. Client path reads document.cookie via
   def-store. Both default to 0 when no cookie exists.

2. sx_server.ml: bypass response cache when sx-home-stepper cookie
   is present. Render on main thread (not worker) so get-cookie
   sees the parsed request cookies.

3. site-full.spec.js: flash detection test sets cookie=7 via
   Playwright context, checks SSR HTML matches hydrated state.

Test: "No flash: SSR=7 hydrated=7 (cookie=7)" — passes.
Tested on fresh stack=site server subprocess.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 07:28:47 +00:00
7492ceac4e Restore hyperscript work on stable site base (908f4f80)
Reset to last known-good state (908f4f80) where links, stepper, and
islands all work, then recovered all hyperscript implementation,
conformance tests, behavioral tests, Playwright specs, site sandbox,
IO-aware server loading, and upstream test suite from f271c88a.

Excludes runtime changes (VM resolve hook, VmSuspended browser handler,
sx_ref.ml guard recovery) that need careful re-integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 19:29:56 +00:00
520424954b Sandbox bytecode loading: K.load + load-sxbc, bytecode param, web stack sxbc via loadModule
Bytecode modules now load correctly in sandbox mode. HS .sxbc modules
use K.load('(load-sxbc ...)') which syncs defines to eval env. Web stack
.sxbc modules use K.loadModule with import suspension drive loop.

K.eval used directly for expression eval (not thunk wrapper) so bytecode-
defined symbols are visible. Falls back to callFn thunk on IO suspension.

Sandbox now reproduces the bytecode repeat bug: source gives 6/6
suspensions, bytecode gives 4/6. Bug is in bytecode compilation of
when/do across perform boundaries, not the runtime wrapper.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 20:56:54 +00:00
a9066c0653 Persistent Lisp image for sx_eval: smart file reload + IO tracing
sx_eval now accepts files (smart-loaded by mtime — unchanged files skip),
trace_io (harness-wrapped IO capture), mock (evaluated platform overrides),
and setup params. Definitions survive between calls. sx_harness_eval also
uses smart loading. sx_write_file can create new files.

New lib/hyperscript/debug.sx: mock DOM platform for instant hyperscript
testing — compile and execute HS expressions against simulated elements,
see every DOM mutation and wait in the IO trace.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 19:56:38 +00:00
3155ba47f9 JIT: VM fast path, &rest support, locals scan, test runner fixes
- jit_compile_lambda: call compile directly via VM when it has bytecode
  (100-400x faster JIT compilation, server pre-warm 1.6s vs hung)
- code_from_value: scan bytecode for highest LOCAL_GET/SET slot to
  compute vc_locals correctly (fixes hyperscript LOCAL_GET overflow)
- code_from_value: accept both compiler keys (bytecode) and SX VM
  keys (vc-bytecode) for interop
- jit_compile_lambda: skip &key/:as params (compiler can't emit them)
- Test runner: seed VM globals with primitives + env bindings,
  native vm-execute-module with suspension fallback to SX version,
  _jit_refresh_globals syncs globals after module loading,
  VmSuspended + "VM undefined" caught and sentineled

3127/3127 without JIT, 3116/3127 with JIT (11 hyperscript on-event
parsing — specific closure/scope issue, not infrastructure).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:52:44 +00:00
387a6cb49e Refactor MCP tree server: dispatch table, caching, validation, subprocess cleanup
Break up the 1735-line handle_tool match into 45 individual handler functions
with hashtable-based dispatch. Add mtime-based file parse caching (AST + CST),
consolidated run_command helper replacing 9 bare open_process_in patterns,
require_file/require_dir input validation, and pagination (limit/offset) for
sx_find_across, sx_comp_list, sx_comp_usage. Also includes pending VM changes:
rest-arity support, hyperscript parser, compiler/transpiler updates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:12:57 +00:00
4d1079aa5e Fix JIT server hang: compiled compiler helpers loop on complex ASTs
Root cause: pre-compiled compiler helper functions (compile-expr,
compile-cond, etc.) produce bytecode that loops when processing
deeply nested ASTs like tw-resolve-style. The test suite passes
because _jit_compiling prevents compiled function execution during
compilation — all functions run via CEK. The server pre-compiled
helpers, so they ran as bytecode during compilation, triggering loops.

Fix:
- _jit_compiling guard on the "already compiled" hook branch prevents
  compiled functions from running during JIT compilation. Compilation
  always uses CEK (correct for all AST sizes). Normal execution uses
  bytecode (fast).
- "compile" itself marked as jit_failed_sentinel — never JIT compiled.
  Runs via CEK, while its helpers use bytecode for normal (non-compile)
  execution.
- Server hook uses call_closure (own VM per call) for IO suspension
  safety. MCP uses call_closure_reuse (fast, no IO needed).

The underlying bytecode bug in the compiled helpers remains — fixing
it requires diagnosing which specific helper loops and why. This is
tracked as a separate issue. Server now starts in ~30s (pre-warm)
and serves pages correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 22:17:51 +00:00
03278c640d Fix JIT compilation cascade + MCP robustness
Three interacting JIT bugs caused infinite loops and server hangs:

1. _jit_compiling cascade: the re-entrancy flag was local to each
   binary's hook. When vm_call triggered JIT compilation internally,
   compiler functions got JIT-compiled during compilation, creating
   infinite cascades. Fix: shared _jit_compiling flag in sx_vm.ml,
   set in jit_compile_lambda itself.

2. call_closure always created new VMs: every HO primitive callback
   (for-each, map, filter) allocated a fresh VM. With 43K+ calls
   during compilation, this was the direct cause of hangs. Fix:
   call_closure_reuse reuses the active VM by isolating frames and
   running re-entrantly. VmSuspended is handled by merging frames
   for proper IO resumption.

3. vm_call for compiled Lambdas: OP_CALL dispatching to a Lambda
   with cached bytecode created a new VM instead of pushing a frame
   on the current one. Fix: push_closure_frame directly.

Additional MCP server fixes:
- Hot-reload: auto-execv when binary on disk is newer (no restart needed)
- Robust JSON: to_int_safe/to_int_or handle null, string, int params
- sx_summarise depth now optional (default 2)
- Per-request error handling (malformed JSON doesn't crash server)
- sx_test uses pre-built binary (skips dune rebuild overhead)
- Timed module loading for startup diagnostics

sx_server.ml fixes:
- Uses shared _jit_compiling flag
- Marks lambdas as jit_failed_sentinel on compile failure (no retry spam)
- call_closure_reuse with VmSuspended frame merging for IO support

Compiled compiler bytecode bug: deeply nested cond/case/let forms
(e.g. tw-resolve-style) cause the compiled compiler to loop.
Workaround: _jit_compiling guard prevents compiled function execution
during compilation. Compilation uses CEK (slower but correct).
Test suite: 3127/3127 passed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 21:59:31 +00:00
d3ff4f7ef3 Fix POST body reading + handler param binding for POST requests
Two fixes in the HTTP server:

1. Read full POST body: the single 8192-byte read() could miss the body
   if it arrived in a separate TCP segment. Now parses Content-Length
   and reads remaining bytes in a loop.

2. Handler param binding: for POST/PUT/PATCH, check request-form before
   request-arg. The old (or (request-arg n) (request-form n)) pattern
   short-circuited on request-arg's "" default (truthy in SX), never
   reaching request-form.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 14:26:05 +00:00
3a9d113537 Fix JIT mutable closure bug: vm-global-get now checks closure env first
vm-global-get checked vm.globals before closure-env, while vm-global-set
wrote to closure-env first. This asymmetry meant set! mutations to mutable
closure variables (e.g. parser position counters) were invisible to sibling
closures reading via JIT — they saw stale snapshots in the globals table.

Reversed vm-global-get lookup order: closure env → globals → primitives,
matching vm-global-set. Also enabled JIT in the MCP harness (compiler.sx
loading, env_bind hook for live globals sync, jit_try_call hook) so
sx_harness_eval exercises the same code path as the server.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 12:08:08 +00:00
cf088a33b4 Step 18 (part 8): Playground page + compile handler + url_decode fix
sx/sx/hyperscript.sx — _hyperscript playground page at
  /sx/(applications.(hyperscript))
  - compile-result defcomp (SSR compilation display)
  - pipeline documentation (tokenize → parse → compile)
  - example showcases with pre-compiled output
  - sx-post form → handler for interactive compilation

sx/sx/handlers/hyperscript-api.sx — POST handler:
  /sx/(applications.(hyperscript.(api.compile)))
  Accepts source param, returns compiled SX + parse tree HTML
  NOTE: hs-parse returns (do) in server context — JIT/CEK runtime
  issue where parser closures don't evaluate correctly. Works in
  test runner (3127/3127). Investigating separately.

sx_server.ml — url_decode fix: decode + as space in form data
  Standard application/x-www-form-urlencoded uses + for spaces.

Nav: _hyperscript added to Applications section.
Config: handler:hs- prefix added for handler dispatch.

3127/3127 tests, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:10:19 +00:00
f5da2bcfd5 Step 18 (part 6): _hyperscript integration — _="..." attribute wiring
lib/hyperscript/integration.sx — connects compiled hyperscript to DOM:
  hs-handler(src) — compile source → callable (fn (me) ...) via eval-expr-cek
  hs-activate!(el) — read _="...", compile, execute with me=element
  hs-boot!() — scan document for [_] elements, activate all
  hs-boot-subtree!(root) — activate within subtree (for HTMX swaps)

Handler wraps compiled SX in (fn (me) (let ((it nil) (event nil)) ...))
so each element gets its own me binding and clean it/event state.
Double-activation prevented via data-hs-active marker.

12 integration tests verify full pipeline: source → compile → eval.
Handlers correctly bind me, support arithmetic, conditionals, sequences,
for loops, and repeat. 3111/3111 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:48:58 +00:00
9a57bd5beb Step 18 (part 5): _hyperscript runtime shims — 25 functions
lib/hyperscript/runtime.sx — thin wrappers over web/lib/dom.sx
primitives implementing hyperscript-specific semantics:

Event handling: hs-on, hs-on-every, hs-init
Async/timing: hs-wait (IO suspend), hs-wait-for, hs-settle
Classes: hs-toggle-class!, hs-toggle-between!, hs-take!
DOM insertion: hs-put! (into/before/after)
Navigation: hs-navigate!, hs-next, hs-previous, hs-query-first/last
Iteration: hs-repeat-times, hs-repeat-forever
Fetch: hs-fetch (json/text/html format dispatch)
Type coercion: hs-coerce (Int/Float/String/Boolean/Array)
Object creation: hs-make (Object/Array/Set/Map)
Behaviors: hs-install
Measurement: hs-measure
Transitions: hs-transition (CSS property + optional duration)

23 runtime + 7 end-to-end pipeline tests.
3099/3099 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:40:55 +00:00
c5d2fa8c96 Step 18 (part 4): _hyperscript compiler — AST → SX expressions
lib/hyperscript/compiler.sx — transforms parsed hyperscript AST into
SX expressions targeting web/lib/dom.sx primitives. Two entry points:
  hs-to-sx    — AST node → SX expression
  hs-to-sx-from-source — source string → SX (tokenize+parse+emit)

Compiler handles:
  Expressions: me/it/event, refs, queries, attrs, styles, locals,
    arithmetic, comparison, boolean, array literals, property access,
    DOM traversal (closest/next/previous/first/last), type conversion,
    membership test, exists/empty/matches/contains predicates
  Commands: add/remove/toggle class, set (var/attr/style/prop dispatch),
    put, if/else, do, wait, wait-for, log, send, trigger, hide, show,
    transition, repeat, fetch, call, return, throw, settle, go, append,
    tell (rebinds me), for, take, make, install, measure, inc/dec
  Features: on (with from/filter/every), init, def, behavior

Maps to SX primitives: dom-add-class, dom-set-attr, dom-set-style,
dom-set-prop, dom-query, dom-closest, dom-dispatch, dom-append, etc.

33 compiler tests across 10 suites. 3076/3076 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 08:31:32 +00:00
4cd0e77331 Step 18 (part 2): _hyperscript parser — token stream → SX AST
lib/hyperscript/parser.sx — parses token stream from hs-tokenize into
SX AST forms. Covers:
  Commands: add/remove/toggle class, set/put, log, hide/show, settle
  Events: on with from/filter, command sequences
  Sequencing: then, wait (with time units)
  Conditionals: if/then/else/end
  Expressions: property chains, it, comparisons, exists, refs
  DOM traversal: closest, next, previous
  Send/trigger events to targets
  Repeat: forever, N times
  Fetch/call with argument lists

55 tests across 12 suites. 3005/3005 full build, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 07:41:17 +00:00
3336c4e957 Step 18 (part 1): _hyperscript tokenizer — 38 tests
lib/hyperscript/tokenizer.sx — tokenizes real _hyperscript syntax into
typed token stream. Handles:
  Keywords (on, set, add, toggle, if, then, from, etc.)
  DOM literals (.class, #id, @attr, *style, :local, <sel/>)
  Strings (single/double quoted, escapes), template literals
  Numbers (integers, decimals, time units: 100ms, 2s)
  Operators (==, !=, +, -, 's possessive)
  Punctuation (parens, brackets, braces, commas, dots)
  Line comments (// to EOL)

Parser will disambiguate .name as class vs property access from context.
Possessive 's correctly distinguished from single-quote strings.

2952/2952 tests, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 21:49:13 +00:00
5d88b363e4 Step 13: String/regex primitives — PCRE-compatible, cross-host
New primitives in sx_primitives.ml:
  char-at, char-code, parse-number — string inspection + conversion
  regex-match, regex-match?, regex-find-all — PCRE pattern matching
  regex-replace, regex-replace-first — PCRE substitution
  regex-split — split by PCRE pattern

Uses Re.Pcre (OCaml re library) so regex patterns use the same syntax
as JS RegExp — patterns in .sx files work identically on browser and
server. Replaces the old test-only regex-find-all stub.

Also: split now handles multi-char separators via Re.

176 new tests (10 suites). 2912/2912 total, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 20:38:40 +00:00
516f9c7186 Step 12: Parser combinator library — pure SX, 68 tests
lib/parser-combinators.sx — 46 definitions in 8 layers:
  0. Result constructors (make-ok, make-fail, ok?, result-value, etc.)
  1. Primitives (satisfy, parse-char, any-char, parse-string, eof)
  2. Core combinators (fmap, parse-bind, seq, alt, label, lazy-parser)
  3. Repetition (many, many1, optional, skip-many)
  4. Structural (between, sep-by, sep-by1, skip-left, skip-right,
     not-followed-by, look-ahead)
  5. Character classes (digit, letter, alpha-num, whitespace, skip-spaces)
  6. Literal parsers (number-literal, string-literal, identifier)
  7. Run function (run-parser)
  8. SX tokenizer (sx-comment, sx-keyword, sx-symbol, sx-number,
     sx-string, sx-token, sx-tokenize)

Self-tests by tokenizing SX: (define x 42), {:ok true}, (+ 1 (* 2 3)),
comments, negative numbers, nested parens, recursive grammars.

No evaluator changes. Pure HO functions + thunks for lazy recursion.
2868/2868 tests, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 20:14:51 +00:00
67ae88b87f Fix last 2 foreign-type-checking tests: ListRef match + #t→true
- run_tests.ml: foreign-check-args binding now matches ListRef (from
  the list primitive) in addition to List
- test-foreign.sx: replace #t with true in guard clauses — SX parser
  treats #t as a symbol, not a boolean

2800/2800 tests, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 19:49:37 +00:00
7f772e0f23 Fix transpiler append! emit for mutable globals + run_with_io error recovery
The transpiler's append! emit path didn't check ml-is-mutable-global?,
so (append! *provide-batch-queue* sub) wrote to a dead local variable
instead of the global _ref. This caused the combined test suite hang —
fire-provide-subscribers was silently broken before the local-ref shadow
removal, and now correctly modifies the global batch queue.

Also adds run_with_io error-to-raise conversion (kont_has_handler guard)
so native Eval_errors can be caught by CEK guard/handler-bind when running
through the test runner's IO-aware step loop.

2798/2800 tests pass. 2 foreign-type-checking failures remain: guard can't
catch Eval_error from native fns called through cek_run_iterative (the
handler dispatch itself uses cek_call which re-enters cek_run_iterative,
creating an infinite loop). Fix requires spec-level change: make (error)
use CEK raise instead of host-error.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:59:38 +00:00
b61b437ccd Transpiler local-ref shadowing fix + foreign test runner bindings
ml-scan-set now checks ml-is-mutable-global? before adding set!/append!
targets to the needs-ref list. Previously, mutable globals like
*bind-tracking* got local `ref Nil` shadows that masked the global _ref,
causing `append!: expected list, got nil` in 43 bind-tracking tests.

Test runner: bind foreign registry functions (foreign-registered?,
foreign-lookup, foreign-names, foreign-register!, foreign-resolve-binding,
foreign-check-args, foreign-build-lambda) + initialize _cek_call_ref for
with-capabilities. 22/24 foreign tests now pass, 8 capabilities tests fixed.

Retranspiled sx_ref.ml — all mutable global shadows eliminated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 18:29:13 +00:00
fb30351be2 Post-10d: JIT measurement infrastructure + compiler fixes
Measurement:
- JIT hit/miss/skip counters in sx_runtime.ml (jit_try_call)
- VM instruction counter enabled in run loop
- jit-enable, vm-counters, vm-counters-reset epoch commands
- Test runner --jit flag for opt-in JIT measurement
- Results (132 tests): 5.8% VM hit, 56% evaluator self-calls, 38% anon

Fixes:
- Move compile-provide, compile-scope, compile-guard,
  compile-guard-clauses inside define-library begin block
  (were orphaned outside, causing "Undefined symbol" JIT failures)
- Add deref primitive (signal unwrap with tracking)
- Add deref compiler dispatch
- Fix compile-expr for scope forms to handle non-keyword args

CEK pruning assessment: evaluator self-calls (56%) can't be pruned —
the CEK must evaluate itself. Real pruning requires self-hosting
compiler (Phase 2+). The VM correctly handles user code that
JIT-compiles. 2776/2776 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 14:32:48 +00:00
2cf4c73ab3 Step 10d: bytecode expansion — close the CEK gap
Tier 1 — Component keyword dispatch on VM:
- Components/islands JIT-compile bodies via jit_compile_comp
- parse_keyword_args matches keyword names against component params
- Added i_compiled field to island type for JIT cache
- Component calls no longer fall back to CEK

Tier 2 — OP_SWAP (opcode 7):
- New stack swap operation for future HO loop compilation
- HO forms already efficient via NativeFn + VmClosure callbacks

Tier 3 — Exception handler stack:
- OP_PUSH_HANDLER (35), OP_POP_HANDLER (36), OP_RAISE (37)
- VM gains handler_stack with frame depth tracking
- Compiler handles guard and raise as bytecode
- Functions with exception handling no longer cause JIT failure

Tier 4 — Scope forms as bytecode:
- Compiler handles provide, context, peek, scope, provide!,
  bind, emit!, emitted via CALL_PRIM sequences
- Functions using reactive scope no longer trigger JIT failure

4 new opcodes (SWAP, PUSH_HANDLER, POP_HANDLER, RAISE) → 37 total.
2776/2776 tests pass, zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 13:19:25 +00:00
fb7338c022 Fix last test: unify scope stacks between test runner and evaluator
run_tests.ml had a local _scope_stacks hash table, separate from
Sx_primitives._scope_stacks used by the CEK evaluator. SX-level
scope-push!/scope-peek used the local table, but step-sf-context's
scope_peek used the global one. Aser's provide handler pushed to
one table, context read from the other — always got nil.

Fix: alias run_tests.ml's _scope_stacks to Sx_primitives._scope_stacks.

2768/2768 OCaml tests pass. Zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 12:20:23 +00:00
fb262aa49b Step 10c: batch coalescing + global subscriber registry
Provide subscribers stored in global *provide-subscribers* dict (keyed
by name) instead of on provide frames. Fixes subscriber loss when
frames are reconstructed, and enables cross-cek_run notification.

Batch integration: batch-begin!/batch-end! primitives manage
*provide-batch-depth*. fire-provide-subscribers defers to queue when
depth > 0, batch-end! flushes deduped. signals.sx batch calls both.

context now prefers scope-peek over frame value — scope stack is the
source of truth since provide! always updates it (even in nested
cek_run where provide frames aren't on the kont).

2754/2768 OCaml (14 pre-existing). 32/32 WASM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 11:39:38 +00:00
b8f389ac9b Import hook: verify library registered, re-entry guard
- _import_hook verifies library_loaded_p AFTER load_library_file
  to catch cases where the file loads but define-library doesn't register
- Re-entry guard (_loading_libs) prevents infinite retry loops
- cek_run import patch deferred — retry approach infinite-loops because
  cek_step_loop re-enters deeply nested eval contexts. Root cause:
  eval_expr → cek_run → cek_step_loop processes the ENTIRE remaining
  kont chain after import resolution, which includes rendering code that
  triggers MORE eval_expr calls. Needs architectural solution (step-level
  suspension handling, not run-level).

Server runs with 4 harmless IO suspension errors. These don't affect
functionality — symbols load via the global env.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 01:16:03 +00:00
244c669334 Revert cek_run import patch — caused infinite CEK loop on server
The cek_run import handling (resume after hook loads library) caused
cek_step_loop to infinite-loop during aser page rendering. Root cause
not yet identified — the resumed CEK state never reaches terminal.

Reverted to original cek_run that throws "IO suspension in non-IO
context". The 4 server startup errors are harmless (files load
partially, all needed symbols available via other paths).

Import hook re-entry guard and debug logging retained for future work.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 00:59:45 +00:00
6e216038ba Fix import resolution: correct library paths + hook type mismatch
Root causes of server [http-load] errors:
1. _import_hook passed pre-computed string key to library_loaded_p
   which calls library_name_key(string) → sx_to_list(string) → crash.
   Fix: pass original list spec, not the string key.
2. resolve_library_path didn't check web/lib/ for (sx dom), (sx browser),
   (web boot-helpers). These libraries use namespace prefixes that don't
   match their file locations.

Server startup errors: 190 → 0.
2683/2684 tests pass (1 known: define-library import clause — spec gap).
New test file: spec/tests/test-import-bind.sx

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 23:44:22 +00:00
5ac1ca9756 Fix server import suspension, dist sync, JIT errors
- cek_run patched to handle import suspensions via _import_hook.
  define-library (import ...) now resolves cleanly on the server.
  IO suspension errors: 190 → 0. JIT failures: ~50 → 0.
- _import_hook wired in sx_server.ml to load .sx files on demand.
- compile-modules.js syncs source .sx files to dist/sx/ before
  compiling — eliminates stale bytecode from out-of-date copies.
- WASM binary rebuilt with all fixes.
- 2658/2658 tests pass (8 new — previously failing import tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 22:52:41 +00:00
2727577702 VM import suspension for browser lazy loading
Bytecode compiler now emits OP_PERFORM for (import ...) and compiles
(define-library ...) bodies. The VM stores the import request in
globals["__io_request"] and stops the run loop — no exceptions needed.
vm-execute-module returns a suspension dict, vm-resume-module continues.

Browser: sx_browser.ml detects suspension dicts from execute_module and
returns JS {suspended, op, request, resume} objects. The sx-platform.js
while loop handles cascading suspensions via handleImportSuspension.

13 modules load via .sxbc bytecode in 226ms (manifest-driven), both
islands hydrate, all handlers wired. 2650/2650 tests pass including
6 new vm-import-suspension tests.

Also: consolidated sx-platform-2.js → sx-platform.js, fixed
vm-execute-module missing code-from-value call, fixed bootstrap.py
protocol registry transpiler issues.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 17:11:12 +00:00
e46cdf3d4d Wire transpiled VM as active execute_module — 2644 tests pass
The transpiled VM (sx_vm_ref.ml, from lib/vm.sx) is now the ACTIVE
bytecode execution engine. sx_server.ml and sx_browser.ml call
Sx_vm_ref.execute_module instead of Sx_vm.execute_module.

Results:
- OCaml tests: 2644 passed, 0 failed
- WASM tests: 32 passed, 0 failed
- Browser: zero errors, zero warnings, islands hydrate
- Server: pages render, JIT compiles, all routes work

The VM logic now lives in ONE place: lib/vm.sx (SX).
OCaml gets it via transpilation (bootstrap_vm.py).
JS/browser gets it via bytecode compilation (compile-modules.js).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 13:34:11 +00:00
7b4c918773 Recompile all 26 .sxbc with define-library wrappers + fix eval/JIT
All 26 browser modules recompiled with define-library/import forms.
Compilation works without vm-compile-adapter (JIT pre-compilation
hangs with library wrappers in some JIT paths — skipped for now,
CEK compilation is ~34s total).

Key fixes:
- eval command: import-aware loop that handles define-library/import
  locally without touching the Python bridge pipe (avoids deadlock)
- compile-modules.js: skip vm-compile-adapter, bump timeout

2621/2621 OCaml tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:08:00 +00:00