Commit Graph

406 Commits

Author SHA1 Message Date
8a08de26cd Web extension module for def-forms + modifier-key clicks + CSSX SSR fix
Move defhandler/defquery/defaction/defpage/defrelation from hardcoded
evaluator dispatch to web/web-forms.sx extension module, registered via
register-special-form!. Adapters updated to use definition-form? and
dynamically extended form-name lists.

Fix modifier-key clicks (ctrl-click → new tab) in three click handlers:
bindBoostLink, bindClientRouteClick, and orchestration.sx bind-event.
Add event-modifier-key? primitive (eventModifierKey_p for transpiler).

Fix CSSX SSR: ~cssx/flush no longer drains the collected bucket on the
server, so the shell template correctly emits CSSX rules in <head>.

Add missing server-side DOM stubs (create-text-node, dom-append, etc.)
and SSR passthrough for portal/error-boundary/promise-delayed.

Passive event listeners for touch/wheel/scroll to fix touchpad scrolling.

97/97 Playwright demo tests + 4/4 isomorphic SSR tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 10:01:41 +00:00
bf305deae1 Isomorphic cookie support + stepper cookie persistence
get-cookie / set-cookie primitives on both server and client:
  - JS: reads/writes document.cookie
  - OCaml: get-cookie reads from _request_cookies hashtable,
    set-cookie is no-op (server sets cookies via HTTP headers)
  - Python bridge: _inject_request_cookies_locked() sends
    (set-request-cookies {:name "val"}) to kernel before page render

Stepper island (home-stepper.sx):
  - Persistence switched from localStorage to cookie (sx-home-stepper)
  - freeze-scope/thaw-from-sx mechanism preserved, just different storage
  - Server reads cookie → thaw restores step-idx → SSR renders correct step
  - Code highlighting: removed imperative code-spans/build-code-dom/
    update-code-highlight; replaced with live DOM query that survives morphs
  - Removed code-view lake wrapper (now plain reactive DOM)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 04:13:53 +00:00
ce9c5d3a08 Add scope-collected/scope-clear-collected!/scope-emitted primitives
Register hashtable-based scope accessors that bypass the CEK special form
dispatch, for use by adapter-html.sx and shell templates.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 02:50:23 +00:00
8958714c85 VM: closure env chain for GLOBAL_GET/SET + remove JIT skip
vm_closure now stores the original closure env (vm_closure_env).
GLOBAL_GET walks the closure env chain when the variable isn't in
vm.globals. GLOBAL_SET writes to the correct env in the chain.

This enables JIT compilation of all named functions regardless of
closure depth. No more closure skip check needed.

Pre-compile time back to ~7s (was 37s with closure skip).

Note: sx-parse sibling list parsing still has issues — the root
cause is in how the JIT-compiled letrec + OP_CLOSURE interacts
with the upvalue cell mechanism. Investigation ongoing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 21:23:27 +00:00
30cfbf777a Fix letrec thunk resolution + compiler letrec support + closure JIT check
Root cause: sf-letrec returns a thunk (for TCO) but the CEK dispatch
wrapped it as a value without evaluating. The thunk leaked as the
return value of letrec expressions, breaking sx-parse and any function
using letrec.

Fix: step-sf-letrec unwraps the thunk into a CEK state, so the last
letrec body expression is properly evaluated by the CEK machine.

Also:
- compile-letrec: two-phase (nil-init then assign) for mutual recursion
- Skip JIT for inner functions (closure.bindings != globals) in both
  vm_call and JIT hook
- vm-reset-fn for sx-parse removed (no longer needed)
- Parser regression test: letrec with mutable pos + recursive sublists

Test results: JS 943/17, OCaml 955/0, Python 747/0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 21:04:47 +00:00
a823e59376 Fix root cause: skip JIT for closure lambdas in BOTH hook and vm_call
The closure check was only in vm_call (sx_vm.ml) but inner functions
like read-list-loop were also compiled through the JIT hook in
sx_server.ml. The hook compiled them with closure merging, producing
incorrect bytecode (read-list-loop mishandled closing parens).

Added the same closure check to the JIT hook: skip lambdas with
non-empty closures. Now sx-parse works correctly:
  (a (b) (c)) → 3 siblings, not (a (b (c)))

Pre-compiled count increased from 17 to 33 — more top-level
functions compiled (inner ones correctly skipped to CEK).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 19:24:14 +00:00
96f50b9dfa Add sibling sublist parser tests + reset JIT sx-parse
Tests: parse "(a (b) (c))" must produce 3 siblings, not nested.
Catches JIT compilation bug where closing parens cause sibling
lists to become children.

Reset sx-parse to CEK on the OCaml kernel — the JIT-compiled
version of sx-parse's complex letrec produces wrong bytecode.
CEK interpretation works correctly (tests pass on all platforms).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 19:17:05 +00:00
2727a2ed8c Skip JIT for lambdas with closure bindings
The closure merging in jit_compile_lambda (copying globals + injecting
closure bindings into vm_env_ref) produces incorrect variable resolution
for inner functions. Symptoms: sx-parse's read-list-loop mishandles
closing parens (siblings become children), parser produces wrong ASTs.

Fix: vm_call skips JIT compilation for lambdas with non-empty closures.
These run on CEK which handles closures correctly. Top-level defines
(empty closure) are still JIT-compiled.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 18:45:40 +00:00
c4224925f9 CSSX flush appends to persistent head stylesheet
~cssx/flush now appends rules to <style id="sx-css"> in <head>
instead of creating ephemeral inline <style> tags that get morphed
away during SPA navigation. Rules accumulate across navigations.

Future: reference-count rules and remove when no elements use them.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 18:19:04 +00:00
fe84b57bed Move CSSX rules to <head>, skip client-affinity components in SSR
- Shell inlines CSSX flush logic in <head> (collected/clear-collected!)
  so island CSS rules survive #main-panel morphs during SPA navigation
- OCaml render_to_html skips :affinity :client components during
  Phase 1b SSR (prevents ~cssx/flush rendering inside body-html)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 18:14:43 +00:00
9ff913c312 Fix root cause: parse-int in primitives table handles 2-arg form
The CSSX colour resolution failure was NOT a JIT compiler bug.
CALL_PRIM looks up primitives table (not env), and parse-int in
the primitives table only handled 1-arg calls. The 2-arg form
(parse-int "699" nil) returned Nil, causing cssx-resolve's colour
branch to fail its and-condition.

Fix: update Sx_primitives.register "parse-int" with same 2-arg
handling as the env binding. Remove the vm-reset-fn workaround.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 17:02:14 +00:00
b1de591e9e Fix CSSX colour rules: reset cssx-resolve JIT to force CEK
cssx-resolve has a complex cond with nested and conditions that the
JIT compiler miscompiles — the colour branch is skipped even when
all conditions are true. Reset to jit_failed_sentinel after loading
so it runs on CEK (which evaluates correctly).

Added vm-reset-fn kernel command for targeted JIT bypass.

All CSSX colour tokens now generate rules: text-violet-699,
text-stone-500, bg-stone-50, etc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 16:49:49 +00:00
f4610e1799 Fix thunk handling for island SSR + effect no-op on server
- trampoline resolves Thunk values (sf-letrec returns them for TCO)
- render-to-html handles "thunk" type by unwrapping expr+env
- effect overridden to no-op after loading signals.sx (prevents
  reactive loops during SSR — effects are DOM side-effects)
- Added thunk?/thunk-expr/thunk-env primitives
- Added DOM API stubs for SSR (dom-query, schedule-idle, etc.)

Header island renders fully with styling. Stepper island still
fails SSR (letrec + complex body hits "Undefined symbol: div"
in eval path — render mode not active during CEK letrec eval).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 15:31:58 +00:00
f3c0cbd8e2 CSSX rules from island SSR: flush collected rules via ~cssx/flush in shell
Added (~cssx/flush) to shell after sx-root div — picks up CSS rules
generated during island SSR via (collect! "cssx" ...). Registered
clear-collected! primitive for the flush component.

Standard CSSX classes now styled server-side. Custom colour shades
(e.g. text-violet-699) still need investigation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:53:31 +00:00
6e1d28d1d7 Load freeze.sx + browser API stubs for complete island SSR
All islands now render server-side:
- freeze.sx loaded into kernel (freeze-scope for home/stepper)
- Browser-only APIs stubbed (local-storage-get/set, dom-listen,
  dom-dispatch, dom-set-data, dom-get-data, promise-then)
  → return nil on server, client hydrates with real behavior

Zero island failures. Both layouts/header and home/stepper render
with hydration markers, CSS classes, and initial signal values.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:44:51 +00:00
92bfef6406 Island SSR: defislands render to HTML server-side with hydration markers
Islands now render their initial state as HTML on the server, like
React SSR. The client hydrates with reactive behavior on boot.

Root causes fixed:
- is_signal/signal_value now recognize Dict-based signals (from
  signals.sx) in addition to native Signal values
- Register "context" as a primitive so the CEK deref frame handler
  can read scope stacks for reactive tracking
- Load adapter-html.sx into kernel for SX-level render-to-html
  (islands use this instead of the OCaml render module)
- Component accessors (params, body, has-children?, affinity) handle
  Island values with ? suffix aliases
- Add platform primitives: make-raw-html, raw-html-content,
  empty-dict?, for-each-indexed, cek-call

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:33:04 +00:00
894321db18 Isomorphic SSR: server renders HTML body, client takes over with SX
Server now renders page content as HTML inside <div id="sx-root">,
visible immediately before JavaScript loads. The SX source is still
included in a <script data-mount="#sx-root"> tag for client hydration.

SSR pipeline: after aser produces the SX wire format, parse and
render-to-html it (~17ms for a 22KB page). Islands with reactive
state gracefully fall back to empty — client hydrates them.

Supporting changes:
- Load signals.sx into OCaml kernel (reactive primitives for island SSR)
- Add cek-call and context to kernel env (needed by signals/deref)
- Island-aware component accessors in sx_types.ml
- render-to-html handles Island values (renders as component with fallback)
- Fix 431 (Request Header Fields Too Large): replace SX-Components
  header (full component name list) with SX-Components-Hash (12 chars)
- CORS allow SX-Components-Hash header

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:01:41 +00:00
318c818728 Lazy JIT compilation: lambdas compile to bytecode on first call
Replace AOT adapter compilation with lazy JIT — each named lambda is
compiled to VM bytecode on first call, cached in l_compiled field for
subsequent calls. Compilation failures fall back to CEK gracefully.

VM types (vm_code, vm_upvalue_cell, vm_closure) moved to sx_types.ml
mutual recursion block. Lambda and Component records gain mutable
l_compiled/c_compiled cache fields. jit_compile_lambda in sx_vm.ml
wraps body as (fn (params) body), invokes spec/compiler.sx via CEK,
extracts inner closure from OP_CLOSURE constant.

JIT hooks in both paths:
- vm_call: Lambda calls from compiled VM code
- continue_with_call: Lambda calls from CEK step loop (injected by
  bootstrap.py post-processing)

Pre-mark sentinel prevents re-entrancy (compile function itself was
hanging when JIT'd mid-compilation). VM execution errors caught and
fall back to CEK with sentinel marking.

Also: add kbd/samp/var to HTML_TAGS, rebuild sx-browser.js, add page
URL to sx-page-full-py timing log.

Performance: first page 28s (JIT compiles 17 functions), subsequent
pages 0.31s home / 0.71s wittgenstein (was 2.3s). All 1945 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 08:18:44 +00:00
bb34b4948b OCaml raw! in HTML renderer + SX_USE_OCAML env promotion + golden tests
- sx_render.ml: add raw! handler to HTML renderer (inject pre-rendered
  content without HTML escaping)
- docker-compose.yml: move SX_USE_OCAML/SX_OCAML_BIN to shared env
  (available to all services, not just sx_docs)
- hosts/ocaml/Dockerfile: OCaml kernel build stage
- shared/sx/tests/: golden test data + generator for OCaml render tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 22:21:04 +00:00
df461beec2 SxExpr aser wire format fix + Playwright test infrastructure + blob protocol
Aser serialization: aser-call/fragment now return SxExpr instead of String.
serialize/inspect passes SxExpr through unquoted, preventing the double-
escaping (\" → \\\" ) that broke client-side parsing when aser wire format
was output via raw! into <script> tags. Added make-sx-expr + sx-expr-source
primitives to OCaml and JS hosts.

Binary blob protocol: eval, aser, aser-slot, and sx-page-full now send SX
source as length-prefixed blobs instead of escaped strings. Eliminates pipe
desync from concurrent requests and removes all string-escape round-trips
between Python and OCaml.

Bridge safety: re-entrancy guard (_in_io_handler) raises immediately if an
IO handler tries to call the bridge, preventing silent deadlocks.

Fetch error logging: orchestration.sx error callback now logs method + URL
via log-warn. Platform catches (fetchAndRestore, fetchPreload, bindBoostForm)
also log errors instead of silently swallowing them.

Transpiler fixes: makeEnv, scopePeek, scopeEmit, makeSxExpr added as
platform function definitions + transpiler mappings — were referenced in
transpiled code but never defined as JS functions.

Playwright test infrastructure:
- nav() captures JS errors and fails fast with the actual error message
- Checks for [object Object] rendering artifacts
- New tests: delete-row interaction, full page refresh, back button,
  direct load with fresh context, code block content verification
- Default base URL changed to localhost:8013 (standalone dev server)
- docker-compose.dev-sx.yml: port 8013 exposed for local testing
- test-sx-build.sh: build + unit tests + Playwright smoke tests

Geography content: index page component written (sx/sx/geography/index.sx)
describing OCaml evaluator, wire formats, rendering pipeline, and topic
links. Wiring blocked by aser-expand-component children passing issue.

Tests: 1080/1080 JS, 952/952 OCaml, 66/66 Playwright

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 22:17:43 +00:00
6d73edf297 Length-prefixed binary framing for OCaml↔Python pipe
Replace newline-delimited text protocol with length-prefixed blobs
for all response data (send_ok_string, send_ok_raw). The OCaml side
sends (ok-len N)\n followed by exactly N raw bytes + \n. Python reads
the length, then readexactly(N).

This eliminates all pipe desync issues:
- No escaping needed for any content (HTML, SX with newlines, quotes)
- No size limits (1MB+ responses work cleanly)
- No multi-line response splitting
- No double-escaping bugs

The old (ok "...") and (ok-raw ...) formats are still parsed as
fallbacks for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:48:52 +00:00
373a4f0134 Fix pipe desync: send_ok_raw escapes newlines, expand-components? in env
- send_ok_raw: when SX wire format contains newlines (string literals),
  fall back to (ok "...escaped...") instead of (ok-raw ...) to keep
  the pipe single-line. Prevents multi-line responses from desyncing
  subsequent requests.
- expand-components? flag set in kernel env (not just VM adapter globals)
  so aser-list's env-has? check finds it during component expansion.
- SX_STANDALONE: restore no_oauth but generate CSRF via session cookie
  so mutation handlers (DELETE etc.) still work without account service.
- Shell statics injection: only inject small values (hashes, URLs) as
  kernel vars. Large blobs (CSS, component_defs) use placeholder tokens.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:32:03 +00:00
ae0e87fbf8 VM aser-slot → sx-page-full: single-call page render, 0.55s warm
Compiler fixes:
- Upvalue re-lookup returns own position (uv-index), not parent slot
- Spec: cek-call uses (make-env) not (dict) — OCaml Dict≠Env
- Bootstrap post-processes transpiler Dict→Env for cek_call

VM runtime fixes:
- compile_adapter evaluates constant defines (SPECIAL_FORM_NAMES etc.)
  via execute_module instead of wrapping as NativeFn closures
- Native primitives: map-indexed, some, every?
- Nil-safe HO forms: map/filter/for-each/some/every? accept nil as empty
- expand-components? set in kernel env (not just VM globals)
- unwrap_env diagnostic: reports actual type received

sx-page-full command:
- Single OCaml call: aser-slot body + render-to-html shell
- Eliminates two pipe round-trips (was: aser-slot→Python→shell render)
- Shell statics (component_defs, CSS, pages_sx) cached in Python,
  injected into kernel once, referenced by symbol in per-request command
- Large blobs use placeholder tokens — Python splices post-render,
  pipe transfers ~51KB instead of 2MB

Performance (warm):
- Server total: 0.55s (was ~2s)
- aser-slot VM: 0.3s, shell render: 0.01s, pipe: 0.06s
- kwargs computation: 0.000s (cached)

SX_STANDALONE mode for sx_docs dev (skips fragment fetches).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 11:06:04 +00:00
231bfbecb5 VM aser-slot routing: isolated globals, inner code extraction, debug
aser-slot now routes through the VM when adapter is compiled:
- compile_adapter: compiles each define body, extracts inner code
  from OP_CLOSURE wrapper, stores as NativeFn in separate globals
- vm_adapter_globals: isolated from kernel env (no cross-contamination)
- aser-slot checks vm_adapter_globals, calls VM aser directly

Status: 2/12 adapter functions compile and run on VM. 6 fail during
OCaml-side compilation with "index out of bounds" — likely from
set-nth! silent failure on ListRef during bytecode jump patching.

Debug output shows outer code structure is correct (4 bytes, 1 const).
Inner code_from_value conversion needs fixing for nested closures.

Also: vm-compile-adapter command inside _ensure_components lock
(fixes pipe desync from concurrent requests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 22:18:21 +00:00
df256b5607 VM adapter: compile works, env isolation needed
adapter-sx.sx compiles to 25 code objects (4044 bytes bytecode).
vm-load-module loads it. But replacing Lambda values in env.bindings
with NativeFn wrappers breaks the CEK machine for non-aser functions.

Root cause: shared env.bindings between CEK and VM. The CEK needs
Lambda values (for closure merging). The VM needs NativeFn wrappers.
Both can't coexist in the same env.

Fix needed: VM adapter gets its own globals table (with compiled
closures). The aser-slot command routes directly to the VM with
its own globals, not through the CEK with shared env.

Disabled vm-load-module. Pages render correctly via CEK.

Also: OP_CALL_PRIM now logs primitive name + argc in error messages
for easier debugging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 21:36:38 +00:00
0ce23521b7 Aser adapter compiles + loads as VM module — first VM execution
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 21:18:34 +00:00
f12bbae6c9 40/40 VM tests pass, auto-compile disabled until full aser compilation
All VM tests green: closures with shared mutable upvalues, map/filter/
for-each via CALL_PRIM, recursive functions, nested closures.

Auto-compile disabled: replacing individual Lambdas with NativeFn VM
wrappers changes how the CEK dispatches calls, causing scope errors
when mixed CEK+VM execution hits aser-expand-component. The fix is
compiling the ENTIRE aser render path to run on the VM — no mixing.

The VM infrastructure is complete and tested. Next step: compile
adapter-sx.sx as a whole module, run the aser on the VM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:57:59 +00:00
c8c4b322a9 All 40 VM tests pass: map/filter/for-each + mutable closures fixed
Two fixes:

1. HO forms (map/filter/for-each/reduce): registered as Python
   primitives so compiler emits OP_CALL_PRIM (direct dispatch to
   OCaml primitive) instead of OP_CALL (which routed through CEK
   HO special forms and failed on NativeFn closure args).

2. Mutable closures: locals captured by closures now share an
   upvalue_cell. OP_LOCAL_GET/SET check frame.local_cells first —
   if the slot has a shared cell, read/write through it. OP_CLOSURE
   creates or reuses cells for is_local=1 captures. Both parent
   and closure see the same mutations.

   Frame type extended with local_cells hashtable for captured slots.

40/40 tests pass:
  - 12 compiler output tests
  - 18 VM execution tests (arithmetic, control flow, closures,
    nested let, higher-order, cond, string ops)
  - 10 auto-compile pattern tests (recursive, map, filter,
    for-each, mutable closures, multiple closures, type dispatch)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:47:40 +00:00
e7da397f8e VM upvalues + HO primitives + 40 tests (36 pass, 4 fail)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:41:23 +00:00
1bb40415a8 VM upvalue support: closures capture variables from enclosing scopes
Compiler (compiler.sx):
- Function scopes marked is-function=true; let scopes share parent frame
- scope-resolve only creates upvalue captures at function boundaries
- Let scope locals use parent's slot numbering (same frame)
- OP_CLOSURE emits upvalue descriptors: (is_local, index) per capture

VM (sx_vm.ml):
- upvalue_cell type: shared mutable reference to captured value
- OP_UPVALUE_GET/SET: read/write from closure's upvalue array
- OP_CLOSURE: reads upvalue descriptors, creates cells from
  enclosing frame's locals (is_local=1) or upvalues (is_local=0)
- vm_closure carries live env_ref (not snapshot)
- vm_call falls back to CEK for Lambda/Component/Island values

Verified: (let ((x 10)) (let ((add-x (fn (y) (+ x y)))) (add-x 5)))
  Compiles to: CONST 10, LOC_SET #0, CLOSURE [UV_GET#0 LOC_GET#0 CPRIM+ RET]
  with upvalue descriptor: is_local=1 index=0
  VM executes → 15 ✓

Auto-compile: 6/117 functions compile (up from 3). Disabled until
compiler handles all features — fallback can't reconstruct closure
scope for variables like nav-state bound in caller's let*.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:13:17 +00:00
a62b7c8a5e Disable auto-compile until CEK fallback is robust
The vm-compile replaces Lambda values with NativeFn wrappers.
When the VM can't execute (missing env vars, unsupported ops),
it falls back to cek_call. But cek_call needs proper Env values
that the snapshot doesn't provide.

Fix needed: VM closures must capture the LIVE env (not snapshot),
or the CEK fallback must construct a proper Env from the globals.
Disabling until this is resolved.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 19:37:23 +00:00
5ca2ee92bc VM auto-compile infrastructure + disable until compiler is complete
Added vm-compile command: iterates env, compiles lambdas to bytecode,
replaces with NativeFn VM wrappers (with CEK fallback on error).
Tested: 3/109 compile, reduces CEK steps 23%.

Disabled auto-compile in production — the compiler doesn't handle
closures with upvalues yet, and compiled functions that reference
dynamic env vars crash. Infrastructure stays for when compiler
handles all SX features.

Also: added set-nth! and mutable-list primitives (needed by
compiler.sx for bytecode patching). Fixed compiler.sx to use
mutable lists on OCaml (ListRef for append!/set-nth! mutation).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 19:30:54 +00:00
e14fc9b0e1 Auto-compile: lambdas → bytecode VM at load time
After loading .sx files, (vm-compile) iterates all named lambdas,
compiles each body to bytecode, replaces with NativeFn VM wrapper.

Results: 3/109 functions compiled (compiler needs more features).
CEK steps: 49911 → 38083 (23% fewer) for home page.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 19:07:42 +00:00
8707f21ca2 Single-pass aser_slot for HTMX path + kernel eval timing + stable hash
Eliminated double-aser for HTMX requests: build OOB wrapper AST
(~shared:layout/oob-sx :content wrapped_ast) and aser_slot in ONE
pass — same pattern as the full-page path. Halves aser_slot calls.

Added kernel-side timing to stderr:
  [aser-slot] eval=3.6s io_flush=0.0s batched=3 result=22235 chars

Results show batch IO works (io_flush=0.0s for 3 highlight calls)
and the bottleneck is pure CEK evaluation time, not IO.

Performance after single-pass fix:
  Home: 0.7s eval (was 2.2s total)
  Reactive: 3.6s eval (was 6.8s total)
  Language: 1.1s eval (was 18.9s total — double-aser eliminated)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 17:03:47 +00:00
96e7bbbac1 Non-blocking batch IO for OCaml kernel + stable component hash
OCaml kernel (sx_server.ml):
- Batch IO mode for aser-slot: batchable helpers (highlight,
  component-source) return placeholders during evaluation instead
  of blocking on stdin. After aser completes, all batched requests
  are flushed to Python at once.
- Python processes them concurrently with asyncio.gather.
- Placeholders (using «IO:N» markers) are replaced with actual
  values in the result string.
- Non-batchable IO (query, action, ctx, request-arg) still uses
  blocking mode — their results drive control flow.

Python bridge (ocaml_bridge.py):
- _read_until_ok handles batched protocol: collects io-request
  lines with numeric IDs, processes on (io-done N) with gather.
- IO result cache for pure helpers — eliminates redundant calls.
- _handle_io_request strips batch ID from request format.

Component caching (jinja_bridge.py):
- Hash computed from FULL component env (all names + bodies),
  not per-page subset. Stable across all pages — browser caches
  once, no re-download on navigation between pages.
- invalidate_component_hash() called on hot-reload.

Tests: 15/15 OCaml helper tests pass (2 new batch IO tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 16:53:01 +00:00
d3b3b4b720 Fix pipe desync: async drain on _send, robust Playwright tests
Root cause: OcamlBridge._send() used write() without drain().
asyncio.StreamWriter buffers writes — without drain(), multiple
commands accumulate and flush as a batch. The kernel processes
them sequentially, sending responses, but Python only reads one
response per command → pipe desync → "unexpected response" errors.

Fix: _send() is now async, calls drain() after every write.
All 14 callers updated to await.

Playwright tests rewritten:
- test_home_has_header: verifies #logo-opacity visible (was only
  checking for "sx" text — never caught missing header)
- test_home_has_nav_children: Geography link must be visible
- test_home_has_main_panel: #main-panel must have child elements
- TestDirectPageLoad: fresh browser.new_context() per test to
  avoid stale component hash in localStorage
- _setup_error_capture + _check_no_fatal_errors helpers

_render_to_sx uses aser_slot (not aser) — layout wrappers contain
re-parsed content that needs full expansion capability.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 16:11:55 +00:00
f819fda587 aser_slot migration: single-pass expansion, pipe desync fix, _render_to_sx
Three fixes completing the aser_slot migration:

1. Single-pass full-page rendering: eval_sx_url builds layout+content
   AST and aser_slots it in ONE call — avoids double-aser where
   re-parsed content hits "Undefined symbol: title/deref" errors.

2. Pipe desync fix: _inject_helpers_locked runs INSIDE the aser_slot
   lock acquisition (not as a separate lock). Prevents interleaved
   commands from other coroutines between injection and aser-slot.

3. _render_to_sx uses aser_slot (not aser): layout wrappers like
   oob_page_sx contain re-parsed content from earlier aser_slot
   calls. Regular aser fails on symbols that were bound during
   the earlier expansion. aser_slot handles them correctly.

HTMX path: aser_slot the content, then oob_page_sx wraps it.
Full page path: build (~shared:layout/app-body :content wrapped_ast),
aser_slot in one pass, pass directly to sx_page.

New Playwright tests: test_navigate_geography_to_reactive,
test_direct_load_reactive_page.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 14:56:55 +00:00
d06de87bca Island guard in aser expansion + page helper IO tests (13 tests)
Fix: islands (defisland) pass component? check but must NEVER be
expanded server-side — they use browser-only reactive primitives
(signal, deref, computed). Added (not (island? comp)) guard in
adapter-sx.sx aser component dispatch.

New test file: shared/sx/tests/test_ocaml_helpers.py
- TestHelperInjection: 5 tests — helper IO proxy, 2-arg calls,
  aser/aser_slot with helpers, undefined helper error
- TestHelperIOPerformance: 2 tests — 20 sequential IO round-trips
  complete in <5s, aser_slot with 5 helpers in <3s
- TestAserSlotClientAffinity: 6 tests — island exclusion, client
  affinity exclusion, server affinity expansion, auto affinity
  behavior in aser vs aser_slot

eval_sx_url stays on bridge.aser() (server-affinity only) for now.
Switching to aser_slot requires fixing the double-aser issue in
_render_to_sx where content gets re-parsed and re-asered.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 12:48:34 +00:00
109ca7c70b Fix aser server-affinity expansion: keyword values, OOB wrapper, page helpers
Three bugs in aser-expand-component (adapter-sx.sx):
- Keyword values were eval'd (eval-expr can't handle <>, HTML tags);
  now asered, matching the aser's rendering capabilities
- Missing default nil binding for unset &key params (caused
  "Undefined symbol" errors for optional params like header-rows)
- aserCall string-quoted keyword values that were already serialized
  SX — now inlines values starting with "(" directly

Server-affinity annotations for layout/nav shells:
- ~shared:layout/app-body, ~shared:layout/oob-sx — page structure
- ~layouts/nav-sibling-row, ~layouts/nav-children — server-side data
- ~layouts/doc already had :affinity :server
- ~cssx/flush marked :affinity :client (browser-only state)

Navigation fix: restore oob_page_sx wrapper for HTMX responses
so #main-panel section exists for sx-select/sx-swap targeting.

OCaml bridge: lazy page helper injection into kernel via IO proxy
(define name (fn (...) (helper "name" ...))) — enables aser_slot
to evaluate highlight/component-source etc. via coroutine bridge.

Playwright tests: added pageerror listener to test_no_console_errors,
new test_navigate_from_home_to_geography for HTMX nav regression.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 12:06:24 +00:00
e61dc4974b Fix readline buffer limit (10MB) and evaluator spec filename
- ocaml_bridge: 10MB readline buffer for large spec explorer responses
- nav-data: evaluator.sx filename (was eval.sx, actual spec file is evaluator.sx)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 19:37:59 +00:00
71c2003a60 OCaml evaluator for page dispatch + handler aser, 83/83 Playwright tests
Major architectural change: page function dispatch and handler execution
now go through the OCaml kernel instead of the Python bootstrapped evaluator.

OCaml integration:
- Page dispatch: bridge.eval() evaluates SX URL expressions (geography, marshes, etc.)
- Handler aser: bridge.aser() serializes handler responses as SX wire format
- _ensure_components loads all .sx files into OCaml kernel (spec, web adapter, handlers)
- defhandler/defpage registered as no-op special forms so handler files load
- helper IO primitive dispatches to Python page helpers + IO handlers
- ok-raw response format for SX wire format (no double-escaping)
- Natural list serialization in eval (no (list ...) wrapper)
- Clean pipe: _read_until_ok always sends io-response on error

SX adapter (aser):
- scope-emit!/scope-peek aliases to avoid CEK special form conflict
- aser-fragment/aser-call: strings starting with "(" pass through unserialized
- Registered cond-scheme?, is-else-clause?, primitive?, get-primitive in kernel
- random-int, parse-int as kernel primitives; json-encode, into via IO bridge

Handler migration:
- All IO calls converted to (helper "name" args...) pattern
- request-arg, request-form, state-get, state-set!, now, component-source etc.
- Fixed bare (effect ...) in island bodies leaking disposer functions as text
- Fixed lower-case → lower, ~search-results → ~examples/search-results

Reactive islands:
- sx-hydrate-islands called after client-side navigation swap
- force-dispose-islands-in for outerHTML swaps (clears hydration markers)
- clear-processed! platform primitive for re-hydration

Content restructuring:
- Design, event bridge, named stores, phase 2 consolidated into reactive overview
- Marshes split into overview + 5 example sub-pages
- Nav links use sx-get/sx-target for client-side navigation

Playwright test suite (sx/tests/test_demos.py):
- 83 tests covering hypermedia demos, reactive islands, marshes, spec explorer
- Server-side rendering, handler interactions, island hydration, navigation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 17:22:51 +00:00
313f7d6be1 OCaml bootstrapper Phase 2: HTML renderer, SX server, Python bridge
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 23:28:48 +00:00
3a268e7277 Data-first HO forms, fix plan pages, aser error handling (1080/1080)
Some checks failed
Build and Deploy / build-and-deploy (push) Has been cancelled
Evaluator: data-first higher-order forms — ho-swap-args auto-detects
(map coll fn) vs (map fn coll), both work. Threading + HO: (-> data
(map fn)) dispatches through CEK HO machinery via quoted-value splice.
17 new tests in test-cek-advanced.sx.

Fix plan pages: add mother-language, isolated-evaluator, rust-wasm-host
to page-functions.sx plan() — were in defpage but missing from URL router.

Aser error handling: pages.py now catches EvalError separately, renders
visible error banner instead of silently sending empty content. All
except blocks include traceback in logs.

Scope primitives: register collect!/collected/clear-collected!/emitted/
emit!/context in shared/sx/primitives.py so hand-written _aser can
resolve them (fixes ~cssx/flush expansion failure).

New test file: shared/sx/tests/test_aser_errors.py — 19 pytest tests
for error propagation through all aser control flow forms.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 18:05:00 +00:00
b4df216fae Phase 2: Remove dead tree-walk code from eval.sx
eval.sx: 1272 → 846 lines (-33%). sx-browser.js: 392KB → 377KB.

Deleted (superseded by CEK step handlers in cek.sx):
- eval-list: tree-walk dispatch table
- eval-call: tree-walk function dispatch
- sf-if, sf-when, sf-cond (3 variants), sf-case (2 variants)
- sf-and, sf-or, sf-let, sf-begin, sf-quote, sf-quasiquote
- sf-thread-first, sf-set!, sf-define
- ho-map, ho-filter, ho-reduce, ho-some, ho-every, ho-for-each,
  ho-map-indexed, call-fn

Kept (still called by CEK as delegates):
- sf-lambda, sf-defcomp, sf-defisland, sf-defmacro, sf-defstyle,
  sf-deftype, sf-defeffect, sf-letrec, sf-named-let
- sf-scope, sf-provide, sf-dynamic-wind
- expand-macro, qq-expand, cond-scheme?
- call-lambda, call-component, parse-keyword-args
- Strict mode, type helpers

eval-expr is now a stub overridden by CEK fixup.
All tests unchanged: JS 747/747, Full 864/870, Python 679/679.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 13:28:09 +00:00
293af75821 Phase 1: CEK is now the sole evaluator on JavaScript
- Override evalExpr/trampoline in CEK_FIXUPS_JS to route through
  cekRun (matching what Python already does)
- Always include frames+cek in JS builds (not just when DOM present)
- Remove CONTINUATIONS_JS extension (CEK handles shift/reset natively)
- Remove Continuation constructor guard (always define it)
- Add strict-mode type checking to CEK call path via head-name
  propagation through ArgFrame

Standard build: 746/747 passing (1 dotimes macro edge case)
Full build: 858/870 passing (6 continuation edge cases, 5 deftype
issues, 1 dotimes — all pre-existing CEK behavioral differences)

The tree-walk eval-expr, eval-list, eval-call, and all sf-*/ho-*
forms in eval.sx are now dead code — never reached at runtime.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 12:49:08 +00:00
7cde140c7e Phase 5-7: Clean up duplicates, verify end-to-end
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 1m50s
Remove all duplicate .sx files from shared/sx/ref/ — originals now in
spec/, web/, hosts/. Only generated runtime (sx_ref.py), async shim,
and theorem prover tools remain in shared/sx/ref/.

Final structure:
  spec/          10 .sx files (core language)
  spec/tests/     8 .sx files (core tests)
  web/           10 .sx files (web framework)
  web/tests/      7 .sx files (web tests)
  hosts/python/   bootstrapper + platform + 5 test runners
  hosts/javascript/ bootstrapper + CLI + platform
  shared/sx/ref/  generated runtime only

All 89 tests pass. Both bootstrappers build fully.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 02:29:37 +00:00
72eaefac13 Phase 4: Move web framework files to web/ and web/tests/
signals.sx, engine.sx, orchestration.sx, boot.sx, router.sx, deps.sx,
forms.sx, page-helpers.sx, adapters, boundary files → web/
Web tests → web/tests/
Test runners updated with _SPEC_TESTS and _WEB_TESTS paths.
All 89 tests pass (20 signal + 43 CEK + 26 CEK reactive).
Both bootstrappers build fully (5993 Python lines, 387KB JS).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 02:26:18 +00:00
7036621be8 Phase 3: Move host files to hosts/python/ and hosts/javascript/
Python: bootstrap.py, platform.py, transpiler.sx, boundary_parser.py, tests/
JavaScript: bootstrap.py, cli.py, platform.py, transpiler.sx
Both bootstrappers verified — build from new locations, output to shared/.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 02:18:56 +00:00
05f7b10864 Phase 2: Move core spec files to spec/ and spec/tests/
git mv eval.sx, parser.sx, primitives.sx, render.sx, cek.sx, frames.sx,
continuations.sx, callcc.sx, types.sx, special-forms.sx → spec/
Tests → spec/tests/
Both bootstrappers verified — find files via spec/ → web/ → shared/sx/ref/

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 02:12:13 +00:00
1a3d7b3d77 Separate core spec from web framework
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 4m49s
Three-layer architecture:
  spec/  — Core language (19 files): evaluator, parser, primitives,
           CEK machine, types, continuations. Host-independent.
  web/   — Web framework (20 files): signals, adapters, engine,
           orchestration, boot, router, CSSX. Built on core spec.
  sx/    — Application (sx-docs website). Built on web framework.

Split boundary.sx into boundary-core.sx (type-of, make-env, identical?)
and boundary-web.sx (IO primitives, signals, spreads, page helpers).

Bootstrappers search spec/ → web/ → shared/sx/ref/ for .sx files.
Original files remain in shared/sx/ref/ as fallback during transition.
All 63 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 01:42:19 +00:00