Commit Graph

71 Commits

Author SHA1 Message Date
b53a0fabea Inline test runner: 5/5 temperature converter tests passing
Fixed three fundamental issues:
1. cek-try arg passing: handler was called with raw string instead of
   (List [String msg]), causing "lambda expects 1 args, got N" errors
2. Silent island hydration failures: hydrate-island now wraps body
   render in cek-try, displaying red error box with stack trace instead
   of empty div. No more silent failures.
3. swap! thunk leak: apply result wasn't trampolined, storing thunks
   as signal values instead of evaluated results

Also fixed: assert= uses = instead of equal? for value comparison,
assert-signal-value uses deref instead of signal-value, HTML entity
decoding in script tag test source via host-call replaceAll.

Temperature converter demo page now shows live test results:
  ✓ initial celsius is 20
  ✓ computed fahrenheit = celsius * 1.8 + 32
  ✓ +5 increments celsius
  ✓ fahrenheit updates on celsius change
  ✓ multiple clicks accumulate

1116/1116 OCaml tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 01:51:30 +00:00
5754a9ff9f Add inline test runner for temperature converter demo
Temperature converter tests (6 tests): initial value, computed
fahrenheit derivation, +5/-5 click handlers, reactive propagation,
multiple click accumulation.

New components:
- sx/sx/reactive-islands/test-runner.sx — reusable defisland that
  parses test source, runs defsuite/deftest forms via cek-eval, and
  displays pass/fail results with re-run button
- sx/sx/reactive-islands/test-temperature.sx — standalone test file

Added cek-try primitive to both browser (sx_browser.ml) and server
(sx_server.ml) for safe test execution with error catching.

Browser bundle now includes harness files (harness.sx,
harness-reactive.sx, harness-web.sx) for inline test execution.

Known: SSR renders test runner body instead of placeholder, causing
arity error on complex str expressions. Needs island SSR handling fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 01:00:07 +00:00
b104663481 Split signals: core spec (spec/signals.sx) + web extensions (web/signals.sx)
Core reactive primitives (signal, deref, reset!, swap!, computed, effect,
batch, notify-subscribers, dispose-computed, with-island-scope,
register-in-scope) moved to spec/signals.sx — pure SX, zero platform
dependencies. Usable by any host: web, CLI, embedded, server, harness.

Web extensions (marsh scopes, stores, event bridge, resource) remain in
web/signals.sx, which now depends on spec/signals.sx.

Updated all load paths:
- hosts/ocaml/bin/sx_server.ml — loads spec/signals.sx before web/signals.sx
- hosts/ocaml/bin/run_tests.ml — loads both in order
- hosts/ocaml/browser/bundle.sh + sx-platform.js — loads core-signals.sx first
- shared/sx/ocaml_bridge.py — loads spec/signals.sx before web extensions

1116/1116 OCaml tests pass. Browser reactive island preview works.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 00:23:35 +00:00
b1690a92c4 Add SX test harness: mock IO platform for testing components
spec/harness.sx — spec-level test harness with:
- Mock platform (30+ default IO mocks: fetch, query, DOM, storage, etc.)
- Session management (make-harness, harness-reset!, harness-set!/get)
- IO interception (make-interceptor, install-interceptors)
- IO log queries (io-calls, io-call-count, io-call-nth, io-call-args)
- IO assertions (assert-io-called, assert-no-io, assert-io-count, etc.)

15 harness tests passing on both OCaml (1116/1116) and JS (15/15).
Loaded automatically by both test runners.

MCP tool: sx_harness_eval — evaluate SX with mock IO, returns result + IO trace.

The harness is extensible: new platforms just add entries to the platform dict.
Components can ship with deftest forms that verify IO behavior against mocks.
Tests are independent objects that can be published separately (by CID).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 00:00:19 +00:00
7047a5d7f3 Add git-aware SX tools: changed, diff_branch, blame, doc_gen, playwright
- sx_changed: list .sx files changed since a ref with structural summaries
- sx_diff_branch: structural diff of all .sx changes vs base ref
- sx_blame: git blame for .sx files, optionally focused on a tree path
- sx_doc_gen: generate component docs from defcomp/defisland signatures
- sx_playwright: run Playwright browser tests with structured results

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 23:25:00 +00:00
76ce0c3ecb Add 15 new MCP tools to sx-tree: project-wide search, smart editing, dev workflow
New comprehension tools:
- sx_find_across: search pattern across all .sx files in a directory
- sx_comp_list: list all definitions (defcomp/defisland/defmacro/defpage/define)
- sx_comp_usage: find all uses of a component across files
- sx_diff: structural diff between two .sx files (ADDED/REMOVED/CHANGED)
- sx_eval: REPL — evaluate SX expressions in the MCP server env

Smart read_tree enhancements:
- Auto-summarise large files (>200 lines)
- focus param: expand only matching subtrees, collapse rest
- max_depth/max_lines/offset for depth limiting and pagination

Smart editing tools:
- sx_rename_symbol: rename all occurrences of a symbol in a file
- sx_replace_by_pattern: find+replace first/all pattern matches
- sx_insert_near: insert before/after a pattern match (top-level)
- sx_rename_across: rename symbol across all .sx files (with dry_run)
- sx_write_file: create .sx files with parse validation

Development tools:
- sx_pretty_print: reformat .sx files with indentation (also used by all edit tools)
- sx_build: build JS bundle or OCaml binary
- sx_test: run test suites with structured pass/fail results
- sx_format_check: lint for empty bindings, missing bodies, duplicate params
- sx_macroexpand: evaluate expressions with a file's macro definitions loaded

Also: updated hook to block Write on .sx files, added custom explore agent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 23:19:10 +00:00
3559ce44f2 Add named path navigation: "defisland > let > letrec" resolves to index path
Named paths let you navigate by structure name instead of opaque indices.
Both formats work in all MCP tools:
  - Index: "(0 3 2)"
  - Named: "defisland > let > letrec"

The server detects ">" in the path string and calls resolve-named-path
(SX function) which walks the tree matching child names at each level.

New SX functions: resolve-named-path, split-path-string, find-child-by-name.
MCP server: added trim/split primitives, resolve_path dispatcher.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 20:39:12 +00:00
63babc0d2d Add render tab to tree editor, switch MCP paths to SX format
Tree editor island now has 4 tabs: tree, context, validate, render.
The render tab evaluates SX source as live HTML — type a (div (h2 "Hello"))
and see it rendered immediately.

MCP server paths changed from JSON arrays [0,2,1] to SX strings "(0 2 1)".
Fixes serialization issues and is more natural for an SX tool. The
json_to_path function now parses SX via sx-parse.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 20:35:40 +00:00
6d0e512f19 Add interactive tree editor island (Phase 4) + MCP server fixes
Phase 4: defisland ~sx-tools/tree-editor — interactive tree viewer
embedded in the SX Tools page. Features:
- Textarea with :bind for SX source input
- Parse button to re-parse on demand
- Tree view: annotated tree with path labels, clickable nodes
- Context view: enclosing chain from root to selected node
- Validate view: structural integrity checks (catches missing body etc.)

MCP server fixes: added ident-start?, ident-char?, make-keyword,
escape-string, sx-expr-source — needed by parser.sx when loaded
into the MCP evaluator.

Also: .mcp.json for Claude Code MCP server config, CLAUDE.md protocol
for structural .sx file editing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 20:11:27 +00:00
934604c2bd Add SX tree tools: comprehension, editing, and MCP server
Phase 1-3 of the SX Tools plan — structural reading, editing, and
MCP server for .sx files.

lib/tree-tools.sx — Pure SX functions for tree comprehension and editing:
  Comprehension: annotate-tree, summarise, read-subtree, get-context,
    find-all, get-siblings, validate, navigate
  Editing: replace-node, insert-child, delete-node, wrap-node, tree-set
  Helpers: list-replace, list-insert, list-remove, replace-placeholder

lib/tests/test-tree-tools.sx — 107 tests covering all functions.

hosts/ocaml/bin/mcp_tree.ml — MCP server (stdio JSON-RPC) exposing
  11 tools. Loads tree-tools.sx into the OCaml evaluator, parses .sx
  files with the native parser, calls SX functions for tree operations.

The MCP server can be configured in Claude Code's settings.json as:
  "mcpServers": { "sx-tree": { "command": "path/to/mcp_tree.exe" } }

1429 tests passing (1322 existing + 107 new tree-tools).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 19:16:41 +00:00
43cf590541 Expand JIT closure tests: nested, mutual recursion, set!, island, deep
22 JIT closure scoping tests covering:
- Basic closure var in map callback + context switch
- Signal + letrec + map (stepper pattern)
- Nested closures (inner lambda sees outer let var)
- Mutual recursion in letrec (is-even/is-odd)
- set! mutation of closure var after JIT compilation
- defisland with signal + letrec + map
- Deep nesting (for-each inside map inside letrec inside let)

All test the critical invariant: JIT-compiled lambdas must use
their closure's vm_env_ref, not the caller's globals.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 16:16:00 +00:00
57e7ce9fe4 Add JIT closure scoping tests
Tests the exact pattern that broke the home stepper: a component
with letrec bindings referenced inside a map callback. The JIT
compiles the callback with closure vars merged into vm_env_ref.
Subsequent renders must use that env, not the caller's globals.

7 tests covering:
- letrec closure var in map callback (fmt function)
- Render, unrelated render, re-render (env not polluted)
- Signal + letrec + map (the stepper pattern)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 16:13:28 +00:00
759309c5c4 Fix create-text-node on server: return string instead of nil
Server-side create-text-node was returning Nil, causing imperative
text nodes (stopwatch "Start"/"0.0s", imperative counter "0") to
render as empty in SSR HTML. Now returns the text as a String value,
which render-to-html handles via escape-html.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 14:29:06 +00:00
f3f70cc00b Move stdlib out of spec — clean spec/library boundary
spec/ now contains only the language definition (5 files):
  evaluator.sx, parser.sx, primitives.sx, render.sx, special-forms.sx

lib/ contains code written IN the language (8 files):
  stdlib.sx, types.sx, freeze.sx, content.sx,
  bytecode.sx, compiler.sx, vm.sx, callcc.sx

Test files follow source: spec/tests/ for core language tests,
lib/tests/ for library tests (continuations, freeze, types, vm).

Updated all consumers:
- JS/Python/OCaml bootstrappers: added lib/ to source search paths
- OCaml bridge: spec_dir for parser/render, lib_dir for compiler/freeze
- JS test runner: scans spec/tests/ (always) + lib/tests/ (--full)
- OCaml test runner: scans spec/tests/, lib tests via explicit request
- Docker dev mounts: added ./lib:/app/lib:ro

Tests: 1041 JS standard, 1322 JS full, 1101 OCaml — all pass

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 23:18:30 +00:00
57cffb8bcc Fix isomorphic SSR: revert inline opcodes, add named let compilation, fix cookie decode
Three bugs broke island SSR rendering of the home stepper widget:

1. Inline VM opcodes (OP_ADD..OP_DEC) broke JIT-compiled functions.
   The compiler emitted single-byte opcodes for first/rest/len/= etc.
   that produced wrong results in complex recursive code (sx-parse
   returned nil, split-tag produced 1 step instead of 16). Reverted
   compiler to use CALL_PRIM for all primitives. VM opcode handlers
   kept for future use.

2. Named let (let loop ((x init)) body) had no compiler support —
   silently produced broken bytecode. Added desugaring to letrec.

3. URL-encoded cookie values not decoded server-side. Client set-cookie
   uses encodeURIComponent but Werkzeug doesn't decode cookie values.
   Added unquote() in bridge cookie injection.

Also: call-lambda used eval_expr which copies Dict values (signals),
breaking mutations through aser lambda calls. Switched to cek_call.

Also: stepper preview now includes ~cssx/tw spreads for SSR styling.

Tests: 1317 JS, 1114 OCaml, 26 integration (2 pre-existing failures)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 22:32:51 +00:00
91cf39153b Add promise-delayed SSR stub; fix startup JIT DISABLED noise
promise-delayed is a browser-only primitive used by the resource island
demo. The SSR renderer needs it as a stub to avoid "Undefined symbol"
errors during render-to-html JIT compilation.

The stub returns the value argument (skipping the delay), so SSR renders
the resolved state immediately.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 18:39:48 +00:00
13ba5ee423 Unify JIT to lazy-only: remove allowlist, all lambdas compile on first call
Replace the manual jit_allowlist (StringSet of ~40 function names) with
universal lazy compilation. Every named lambda gets one compile attempt
on first call; failures are sentineled and never retried.

Compiler internals are still pre-compiled at startup (bootstrapping the
JIT itself), but everything else compiles lazily — no manual curation.

Remove jit-allow command (no longer needed). Remove StringSet module.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 18:34:27 +00:00
a6e0e84521 Split setup_type_operations into 6 focused functions
125-line monolith split into:
- setup_core_operations (assert, append!, apply, equal?, primitive?)
- setup_type_constructors (make-keyword, make-symbol, escape-string, etc.)
- setup_character_classification (ident-start?, ident-char?, char-numeric?)
- setup_env_operations (env-get, env-has?, env-bind!, env-set!, etc.)
- setup_strict_mode (gradual type system support)
- setup_io_bridges (json-encode, into, sleep, response headers)

make_server_env now calls 12 focused setup functions total.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 18:31:37 +00:00
3ae49b69f5 Fix env-shadowing: rebind host extension points after .sx file load
evaluator.sx defines *custom-special-forms* and register-special-form!
which shadow the host's native bindings when loaded at runtime. The
native bindings route to Sx_ref.custom_special_forms (the dict the CEK
evaluator checks), but the SX-level defines create a separate dict.

Fix: rebind_host_extensions runs after every load command, re-asserting
the native register-special-form! and *custom-special-forms* bindings.

Add regression test: custom form registered before evaluator.sx load
survives and remains callable via CEK dispatch afterward.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 18:29:29 +00:00
33af6b9266 Fix serialize_value for SxExpr/Spread; handle List-of-SxExpr in aser output
serialize_value was falling through to "nil" for SxExpr and Spread values.
Now SxExpr passes through as raw SX text, Spread serializes as make-spread.

The aser command's result handler now joins a List of SxExprs as a
space-separated fragment (from map/filter producing multiple SxExprs).

Investigation ongoing: handler aser responses still have "class" strings
where :class keywords should be — the component expansion path in aser
loses keyword types during CEK evaluation of component bodies.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 18:16:33 +00:00
3df8c41ca1 Split make_server_env, eliminate all runtime sx_ref imports, fix auth-menu tests
make_server_env split into 7 focused setup functions:
- setup_browser_stubs (22 DOM no-ops)
- setup_scope_env (18 scope primitives from sx_scope.ml)
- setup_evaluator_bridge (CEK eval-expr, trampoline, expand-macro, etc.)
- setup_introspection (type predicates, component/lambda accessors)
- setup_type_operations (string/env/dict/equality/parser helpers)
- setup_html_tags (~100 HTML tag functions)
- setup_io_env (query, action, helper IO bridge)

Eliminate ALL runtime sx_ref.py imports:
- sx/sxc/pages/helpers.py: 24 imports → _ocaml_helpers.py bridge
- sx/sxc/pages/sx_router.py: remove SX_USE_REF fallback
- shared/sx/query_registry.py: use register_components instead of eval

Unify JIT compilation: pre-compile list derived from allowlist
(no manual duplication), only compiler internals pre-compiled.

Fix test_components auth-menu: ~auth-menu → ~shared:fragments/auth-menu

Tests: 1114 OCaml, 29/29 components, 35/35 regression, 6/6 Playwright

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 17:23:09 +00:00
6ef9688bd2 Fix primitive? lookup + replace coercion; remove debug output
primitive? in make_server_env was checking env bindings only (NativeFn),
missing all 132 primitives in the Sx_primitives hashtable. Now checks
both primitives table and env. get-primitive similarly fixed.

replace primitive now coerces SxExpr/Thunk/RawHTML/etc to strings instead
of crashing with "replace: 3 string args" — fixes aser JIT DISABLED.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 16:29:52 +00:00
f9f810ffd7 Complete Python eval removal: epoch protocol, scope consolidation, JIT fixes
Route all rendering through OCaml bridge — render_to_html no longer uses
Python async_eval. Fix register_components to parse &key params and &rest
children from defcomp forms. Remove all dead sx_ref.py imports.

Epoch protocol (prevents pipe desync):
- Every command prefixed with (epoch N), all responses tagged with epoch
- Both sides discard stale-epoch messages — desync structurally impossible
- OCaml main loop discards stale io-responses between commands

Consolidate scope primitives into sx_scope.ml:
- Single source of truth for scope-push!/pop!/peek, collect!/collected,
  emit!/emitted, context, and 12 other scope operations
- Removes duplicate registrations from sx_server.ml (including bugs where
  scope-emit! and clear-collected! were registered twice with different impls)
- Bind scope prims into env so JIT VM finds them via OP_GLOBAL_GET

JIT VM fixes:
- Trampoline thunks before passing args to CALL_PRIM
- as_list resolves thunks via _sx_trampoline_fn
- len handles all value types (Bool, Number, RawHTML, SxExpr, Spread, etc.)

Other fixes:
- ~cssx/tw signature: (tokens) → (&key tokens) to match callers
- Minimal Python evaluator in html.py for sync sx() Jinja function
- Python scope primitive stubs (thread-local) for non-OCaml paths
- Reader macro resolution via OcamlSync instead of sx_ref.py

Tests: 1114 OCaml, 1078 JS, 35 Python regression, 6/6 Playwright SSR

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 16:14:40 +00:00
96f2862385 Fix island rendering in OCaml test runner — add Island cases to component accessors
The test runner's component-body/component-params/component-has-children
bindings only handled Component values, not Island. When adapter-html.sx
called (component-body island), it hit the fallback and returned nil,
producing empty island bodies. Also removed debug logging from
component-has-children? primitive.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 12:55:47 +00:00
8a08de26cd Web extension module for def-forms + modifier-key clicks + CSSX SSR fix
Move defhandler/defquery/defaction/defpage/defrelation from hardcoded
evaluator dispatch to web/web-forms.sx extension module, registered via
register-special-form!. Adapters updated to use definition-form? and
dynamically extended form-name lists.

Fix modifier-key clicks (ctrl-click → new tab) in three click handlers:
bindBoostLink, bindClientRouteClick, and orchestration.sx bind-event.
Add event-modifier-key? primitive (eventModifierKey_p for transpiler).

Fix CSSX SSR: ~cssx/flush no longer drains the collected bucket on the
server, so the shell template correctly emits CSSX rules in <head>.

Add missing server-side DOM stubs (create-text-node, dom-append, etc.)
and SSR passthrough for portal/error-boundary/promise-delayed.

Passive event listeners for touch/wheel/scroll to fix touchpad scrolling.

97/97 Playwright demo tests + 4/4 isomorphic SSR tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 10:01:41 +00:00
bf305deae1 Isomorphic cookie support + stepper cookie persistence
get-cookie / set-cookie primitives on both server and client:
  - JS: reads/writes document.cookie
  - OCaml: get-cookie reads from _request_cookies hashtable,
    set-cookie is no-op (server sets cookies via HTTP headers)
  - Python bridge: _inject_request_cookies_locked() sends
    (set-request-cookies {:name "val"}) to kernel before page render

Stepper island (home-stepper.sx):
  - Persistence switched from localStorage to cookie (sx-home-stepper)
  - freeze-scope/thaw-from-sx mechanism preserved, just different storage
  - Server reads cookie → thaw restores step-idx → SSR renders correct step
  - Code highlighting: removed imperative code-spans/build-code-dom/
    update-code-highlight; replaced with live DOM query that survives morphs
  - Removed code-view lake wrapper (now plain reactive DOM)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 04:13:53 +00:00
ce9c5d3a08 Add scope-collected/scope-clear-collected!/scope-emitted primitives
Register hashtable-based scope accessors that bypass the CEK special form
dispatch, for use by adapter-html.sx and shell templates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 02:50:23 +00:00
7d793ec76c Fix CSSX styling: trampoline wiring + scope-emit!/emitted for adapter-html.sx
Root causes of missing CSSX classes in SSR:

1. _sx_trampoline_fn in sx_primitives.ml was never wired — call_any in
   HO forms (map/filter/for-each) returned unresolved Thunks, so callbacks
   like render-lambda-html's param binding never executed. Fixed in
   bootstrap.py FIXUPS: wire Sx_primitives._sx_trampoline_fn after eval_expr.

2. adapter-html.sx used (emit! ...) and (emitted ...) which are CEK special
   forms (walk kont for ScopeAccFrame), but scope-push!/scope-pop! use the
   hashtable. CEK frames and hashtable are two different scope systems.
   Fixed: adapter uses scope-emit!/scope-emitted (hashtable primitives).

3. env-* operations (env-has?, env-get, env-bind!, env-set!, env-extend,
   env-merge) only accepted Env type. adapter-html.sx passes Dict as env.
   Fixed: all env ops go through unwrap_env which handles Dict/Nil.

Also: fix merge conflict in sx/sx/geography/index.sx, remove duplicate
scope primitives from sx_primitives.ml (sx_server.ml registers them).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 02:23:00 +00:00
e4cabcbb59 Fix env-merge for Dict/Nil args + add adapter-html.sx primitives
sx_runtime.ml: unwrap_env now accepts Dict and Nil (converts to Env),
fixing env-merge when adapter-html.sx passes dict-as-env.

sx_server.ml + run_tests.ml: env-merge bindings use Sx_runtime.env_merge
(which handles Dict/Nil) instead of requiring strict Env pattern match.

sx_primitives.ml: Added scope stack (scope-push!/pop!/peek/emit!, emitted),
type predicates (lambda?/island?/component?/macro?), component accessors
(closure/name/params/body/has-children?), lambda accessors, for-each-indexed,
empty-dict?, make-raw-html, raw-html-content, is-else-clause?.

8 OCaml render tests still fail (env propagation in render-lambda-html) —
same adapter code works in JS and in production via Python bridge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 01:49:21 +00:00
284572c7a9 Wire adapter-html.sx into OCaml server, replacing hand-written renderer
sx_server.ml: sx_render_to_html() calls the SX adapter-html.sx render-to-html
via CEK eval, falling back to Sx_render.render_to_html if adapter not loaded.
CLI --render mode now loads render.sx + adapter-html.sx.

sx_primitives.ml: Added ~25 primitives needed by adapter-html.sx:
  scope-push!/pop!/peek/emit!, emitted, provide-push!/pop! (hashtable stack),
  lambda?/island?/component?/macro?, component-closure/name/params/body/
  has-children?, lambda-closure/params/body, is-else-clause?, for-each-indexed,
  empty-dict?, make-raw-html, raw-html-content

run_tests.ml: Loads render.sx + adapter-html.sx for test-render-html.sx.
Registers trampoline, eval-expr, scope stubs, expand-macro, cond-scheme?.

Status: 1105/1114 OCaml tests pass. 8 remaining failures are env-merge
edge cases in render-lambda-html/component-children/island rendering —
same adapter code works in JS (143/143).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 01:38:18 +00:00
23c8b97cb1 VM spec in SX + 72 tests passing on both JS and OCaml
spec/vm.sx — bytecode VM written in SX (the spec):
  - Stack-based interpreter for bytecode from compiler.sx
  - 24 opcodes: constants, variables (local/upvalue/global), control flow,
    function calls (with TCO), closures with upvalue capture, collections,
    string concat, define
  - Upvalue cells for shared mutable closure variables
  - Call dispatch: vm-closure (fast path), native-fn, CEK fallback
  - Platform interface: 7 primitives (vm-stack-*, call-primitive, cek-call,
    get-primitive, env-parent)

spec/tests/test-vm.sx — 72 tests exercising compile→bytecode→VM pipeline:
  constants, arithmetic, comparison, control flow (if/when/cond/case/and/or),
  let bindings, lambda, closures, upvalue mutation, TCO (10K iterations),
  collections, strings, define, letrec, quasiquote, threading, integration
  (fibonacci, recursive map/filter/reduce, compose)

spec/compiler.sx — fix :else keyword detection in case/cond compilation
  (was comparing Keyword object to evaluated string, now checks type)

Platform primitives added (JS + OCaml):
  make-vm-stack, vm-stack-get, vm-stack-set!, vm-stack-length, vm-stack-copy!,
  primitive?, get-primitive, call-primitive, set-nth! (JS)

Test runners updated to load bytecode.sx + compiler.sx + vm.sx for --full.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 01:20:00 +00:00
5270d2e956 JIT allowlist + integration tests + --test mode + clean up debug logging
JIT allowlist (sx_server.ml):
- Replace try-every-lambda strategy with StringSet allowlist. Only
  functions in the list get JIT compiled (compiler, parser, pure transforms).
  Render functions that need dynamic scope skip JIT entirely — no retry
  overhead, no silent fallbacks.
- Add (jit-allow name) command for dynamic expansion from Python bridge.
- JIT failures log once with "[jit] DISABLED fn — reason" then go silent.

Standalone --test mode (sx_server.ml):
- New --test flag loads full env (spec + adapters + compiler + signals),
  supports --eval and --load flags. Quick kernel testing without Docker.
  Example: dune exec bin/sx_server.exe -- --test --eval '(len HTML_TAGS)'

Integration tests (integration_tests.ml):
- New binary exercising the full rendering pipeline: loads spec + adapters
  into a server-like env, renders HTML via both native and SX adapter paths.
- 26 tests: HTML tags, special forms (when/if/let), letrec with side
  effects, component rendering, eval-expr with HTML tag functions.
- Would have caught the "Undefined symbol: div/lake/init" issues from
  the previous commit immediately without Docker.

VM cleanup (sx_vm.ml):
- Remove temporary debug logging (insn counter, call_closure counter,
  VmClosure depth tracking) added during debugging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 23:58:40 +00:00
dd057247a5 VM: VmClosure value type + iterative run loop + define hoisting + SSR fixes
Core VM changes:
- Add VmClosure value variant — inner closures created by OP_CLOSURE are
  first-class VM values, not NativeFn wrappers around call_closure
- Convert `run` from recursive to while-loop — zero OCaml stack growth,
  true TCO for VmClosure tail calls
- vm_call handles VmClosure by pushing frame on current VM (no new VM
  allocation per call)
- Forward ref _vm_call_closure_ref for cross-boundary calls (CEK/primitives)

Compiler (spec/compiler.sx):
- Define hoisting in compile-begin: pre-allocate local slots for all
  define forms before compiling any values. Fixes forward references
  between inner functions (e.g. read-expr referencing skip-ws in sx-parse)
- scope-define-local made idempotent (skip if slot already exists)

Server (sx_server.ml):
- JIT fail-once sentinel: mark l_compiled as failed after first VM runtime
  error. Eliminates thousands of retry attempts per page render.
- HTML tag bindings: register all HTML tags as pass-through NativeFns so
  eval-expr can handle (div ...) etc. in island component bodies.
- Log VM FAIL errors with function name before disabling JIT.

SSR fixes:
- adapter-html.sx letrec handler: evaluate bindings in proper letrec scope
  (pre-bind nil, then evaluate), render body with render-to-html instead of
  eval-expr. Fixes island SSR for components using letrec.
- Add `init` primitive to OCaml kernel (all-but-last of list).
- VmClosure handling in sx_runtime.ml sx_call dispatch.

Tests: 971/971 OCaml (+19 new), 0 failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 23:39:35 +00:00
8958714c85 VM: closure env chain for GLOBAL_GET/SET + remove JIT skip
vm_closure now stores the original closure env (vm_closure_env).
GLOBAL_GET walks the closure env chain when the variable isn't in
vm.globals. GLOBAL_SET writes to the correct env in the chain.

This enables JIT compilation of all named functions regardless of
closure depth. No more closure skip check needed.

Pre-compile time back to ~7s (was 37s with closure skip).

Note: sx-parse sibling list parsing still has issues — the root
cause is in how the JIT-compiled letrec + OP_CLOSURE interacts
with the upvalue cell mechanism. Investigation ongoing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 21:23:27 +00:00
30cfbf777a Fix letrec thunk resolution + compiler letrec support + closure JIT check
Root cause: sf-letrec returns a thunk (for TCO) but the CEK dispatch
wrapped it as a value without evaluating. The thunk leaked as the
return value of letrec expressions, breaking sx-parse and any function
using letrec.

Fix: step-sf-letrec unwraps the thunk into a CEK state, so the last
letrec body expression is properly evaluated by the CEK machine.

Also:
- compile-letrec: two-phase (nil-init then assign) for mutual recursion
- Skip JIT for inner functions (closure.bindings != globals) in both
  vm_call and JIT hook
- vm-reset-fn for sx-parse removed (no longer needed)
- Parser regression test: letrec with mutable pos + recursive sublists

Test results: JS 943/17, OCaml 955/0, Python 747/0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 21:04:47 +00:00
a823e59376 Fix root cause: skip JIT for closure lambdas in BOTH hook and vm_call
The closure check was only in vm_call (sx_vm.ml) but inner functions
like read-list-loop were also compiled through the JIT hook in
sx_server.ml. The hook compiled them with closure merging, producing
incorrect bytecode (read-list-loop mishandled closing parens).

Added the same closure check to the JIT hook: skip lambdas with
non-empty closures. Now sx-parse works correctly:
  (a (b) (c)) → 3 siblings, not (a (b (c)))

Pre-compiled count increased from 17 to 33 — more top-level
functions compiled (inner ones correctly skipped to CEK).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 19:24:14 +00:00
890c472893 Add compile-letrec to pre-compile list
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 19:11:32 +00:00
b1de591e9e Fix CSSX colour rules: reset cssx-resolve JIT to force CEK
cssx-resolve has a complex cond with nested and conditions that the
JIT compiler miscompiles — the colour branch is skipped even when
all conditions are true. Reset to jit_failed_sentinel after loading
so it runs on CEK (which evaluates correctly).

Added vm-reset-fn kernel command for targeted JIT bypass.

All CSSX colour tokens now generate rules: text-violet-699,
text-stone-500, bg-stone-50, etc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 16:49:49 +00:00
364fbac9e1 Fix parse-int to handle 2-arg form (value + default)
cssx-resolve calls (parse-int "699" nil) — the 2-arg form was
falling to the catch-all and returning Nil, causing colour tokens
like text-violet-699 to not generate CSS rules.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 16:31:41 +00:00
f4610e1799 Fix thunk handling for island SSR + effect no-op on server
- trampoline resolves Thunk values (sf-letrec returns them for TCO)
- render-to-html handles "thunk" type by unwrapping expr+env
- effect overridden to no-op after loading signals.sx (prevents
  reactive loops during SSR — effects are DOM side-effects)
- Added thunk?/thunk-expr/thunk-env primitives
- Added DOM API stubs for SSR (dom-query, schedule-idle, etc.)

Header island renders fully with styling. Stepper island still
fails SSR (letrec + complex body hits "Undefined symbol: div"
in eval path — render mode not active during CEK letrec eval).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 15:31:58 +00:00
f3c0cbd8e2 CSSX rules from island SSR: flush collected rules via ~cssx/flush in shell
Added (~cssx/flush) to shell after sx-root div — picks up CSS rules
generated during island SSR via (collect! "cssx" ...). Registered
clear-collected! primitive for the flush component.

Standard CSSX classes now styled server-side. Custom colour shades
(e.g. text-violet-699) still need investigation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:53:31 +00:00
6e1d28d1d7 Load freeze.sx + browser API stubs for complete island SSR
All islands now render server-side:
- freeze.sx loaded into kernel (freeze-scope for home/stepper)
- Browser-only APIs stubbed (local-storage-get/set, dom-listen,
  dom-dispatch, dom-set-data, dom-get-data, promise-then)
  → return nil on server, client hydrates with real behavior

Zero island failures. Both layouts/header and home/stepper render
with hydration markers, CSS classes, and initial signal values.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:44:51 +00:00
2c8afd230d Island SSR: spreads work, CSS classes render, context primitive registered
Root causes:
- make-spread/spread?/spread-attrs were stubbed (always false/empty)
  → now create/detect/unwrap Spread values properly
- "context" primitive missing from Sx_primitives registry
  → CEK deref frame handler couldn't read reactive scope stacks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:38:45 +00:00
92bfef6406 Island SSR: defislands render to HTML server-side with hydration markers
Islands now render their initial state as HTML on the server, like
React SSR. The client hydrates with reactive behavior on boot.

Root causes fixed:
- is_signal/signal_value now recognize Dict-based signals (from
  signals.sx) in addition to native Signal values
- Register "context" as a primitive so the CEK deref frame handler
  can read scope stacks for reactive tracking
- Load adapter-html.sx into kernel for SX-level render-to-html
  (islands use this instead of the OCaml render module)
- Component accessors (params, body, has-children?, affinity) handle
  Island values with ? suffix aliases
- Add platform primitives: make-raw-html, raw-html-content,
  empty-dict?, for-each-indexed, cek-call

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:33:04 +00:00
894321db18 Isomorphic SSR: server renders HTML body, client takes over with SX
Server now renders page content as HTML inside <div id="sx-root">,
visible immediately before JavaScript loads. The SX source is still
included in a <script data-mount="#sx-root"> tag for client hydration.

SSR pipeline: after aser produces the SX wire format, parse and
render-to-html it (~17ms for a 22KB page). Islands with reactive
state gracefully fall back to empty — client hydrates them.

Supporting changes:
- Load signals.sx into OCaml kernel (reactive primitives for island SSR)
- Add cek-call and context to kernel env (needed by signals/deref)
- Island-aware component accessors in sx_types.ml
- render-to-html handles Island values (renders as component with fallback)
- Fix 431 (Request Header Fields Too Large): replace SX-Components
  header (full component name list) with SX-Components-Hash (12 chars)
- CORS allow SX-Components-Hash header

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:01:41 +00:00
9bd4863ce1 Clean up JIT diagnostic logging from pre-compile loop
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 12:49:20 +00:00
2a5ef0ea09 JIT: restore re-entrancy guards, compile quasiquote inline, closure env merging
Fix infinite recursion in VM JIT: restore sentinel pre-mark in vm_call
and pre-compile loop so recursive compiler functions don't trigger
unbounded compilation cascades. Runtime VM errors fall back to CEK;
compile errors surface visibly (not silently swallowed).

New: compile-quasiquote emits inline code instead of delegating to
qq-expand-runtime. Closure-captured variables merged into VM globals
so compiled closures resolve outer bindings via GLOBAL_GET.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 12:22:54 +00:00
e12b2eab6b Compiler: CALL_PRIM only for real primitives, GLOBAL_GET+CALL for runtime fns
compile-quasiquote, compile-defcomp, compile-defmacro were hardcoding
CALL_PRIM for runtime functions (qq-expand-runtime, eval-defcomp,
eval-defmacro) that aren't in the primitives table. Changed to
GLOBAL_GET + CALL so the VM resolves them from env.bindings at runtime.

The compile-call function already checks (primitive? name) before
emitting CALL_PRIM — only the three special-case compilers were wrong.

Also: register scope-push!/pop! as primitives, add scope-peek/emit!
to OCaml transpiler name mapping, fix sx_runtime.ml scope wrappers
to route through prim_call "scope-push!" etc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:02:17 +00:00
09feb51762 Unify scope mechanism: one world (hashtable stacks everywhere)
Replace continuation-based scope frames with hashtable stacks for all
scope operations. The CEK evaluator's scope/provide/context/emit!/emitted
now use scope-push!/pop!/peek/emit! primitives (registered in
sx_primitives table) instead of walking continuation frames.

This eliminates the two-world problem where the aser used hashtable
stacks (scope-push!/pop!) but eval-expr used continuation frames
(ScopeFrame/ScopeAccFrame). Now both paths share the same mechanism.

Benefits:
- scope/context works inside eval-expr calls (e.g. (str ... (context x)))
- O(1) scope lookup vs O(n) continuation walking
- Simpler — no ScopeFrame/ScopeAccFrame/ProvideFrame creation/dispatch
- VM-compiled code and CEK code both see the same scope state

Also registers scope-push!/pop!/peek/emit!/collect!/collected/
clear-collected! as real primitives (sx_primitives table) so the
transpiled evaluator can call them directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 09:45:25 +00:00
4734d38f3b Fix VM correctness: get nil-safe, scope/context/collect! as primitives
- get primitive returns nil for type mismatches (list+string) instead
  of raising — matches JS/Python behavior, fixes find-nav-match errors
- scope-peek, collect!, collected, clear-collected! registered as real
  primitives in sx_primitives table (not just env bindings) so the CEK
  step-sf-context can find them via get-primitive
- step-sf-context checks scope-peek hashtable BEFORE walking CEK
  continuation — bridges aser's scope-push!/pop! with CEK's context
- context, emit!, emitted added to SPECIAL_FORM_NAMES and handled in
  aser-special (scope operations in aser rendering mode)
- sx-context NativeFn for VM-compiled code paths
- VM execution errors no longer mark functions as permanently failed —
  bytecode is correct, errors are from runtime data
- kbd, samp, var added to HTML_TAGS + sx-browser.js rebuilt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 09:33:18 +00:00