Commit Graph

36 Commits

Author SHA1 Message Date
2a5ef0ea09 JIT: restore re-entrancy guards, compile quasiquote inline, closure env merging
Fix infinite recursion in VM JIT: restore sentinel pre-mark in vm_call
and pre-compile loop so recursive compiler functions don't trigger
unbounded compilation cascades. Runtime VM errors fall back to CEK;
compile errors surface visibly (not silently swallowed).

New: compile-quasiquote emits inline code instead of delegating to
qq-expand-runtime. Closure-captured variables merged into VM globals
so compiled closures resolve outer bindings via GLOBAL_GET.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 12:22:54 +00:00
1cc3e761a2 Fix get nil-safety in sx_runtime.ml + reduce VM failure log noise
The second get implementation in sx_runtime.ml (used by transpiled code)
was still raising on type mismatches. Now returns nil like sx_primitives.

Remove per-call [vm-call-closure] FAIL logging — the jit-hook already
logs failures at the right level. Reduces 70K log lines to ~5.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:20:13 +00:00
e12b2eab6b Compiler: CALL_PRIM only for real primitives, GLOBAL_GET+CALL for runtime fns
compile-quasiquote, compile-defcomp, compile-defmacro were hardcoding
CALL_PRIM for runtime functions (qq-expand-runtime, eval-defcomp,
eval-defmacro) that aren't in the primitives table. Changed to
GLOBAL_GET + CALL so the VM resolves them from env.bindings at runtime.

The compile-call function already checks (primitive? name) before
emitting CALL_PRIM — only the three special-case compilers were wrong.

Also: register scope-push!/pop! as primitives, add scope-peek/emit!
to OCaml transpiler name mapping, fix sx_runtime.ml scope wrappers
to route through prim_call "scope-push!" etc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:02:17 +00:00
09feb51762 Unify scope mechanism: one world (hashtable stacks everywhere)
Replace continuation-based scope frames with hashtable stacks for all
scope operations. The CEK evaluator's scope/provide/context/emit!/emitted
now use scope-push!/pop!/peek/emit! primitives (registered in
sx_primitives table) instead of walking continuation frames.

This eliminates the two-world problem where the aser used hashtable
stacks (scope-push!/pop!) but eval-expr used continuation frames
(ScopeFrame/ScopeAccFrame). Now both paths share the same mechanism.

Benefits:
- scope/context works inside eval-expr calls (e.g. (str ... (context x)))
- O(1) scope lookup vs O(n) continuation walking
- Simpler — no ScopeFrame/ScopeAccFrame/ProvideFrame creation/dispatch
- VM-compiled code and CEK code both see the same scope state

Also registers scope-push!/pop!/peek/emit!/collect!/collected/
clear-collected! as real primitives (sx_primitives table) so the
transpiled evaluator can call them directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 09:45:25 +00:00
4734d38f3b Fix VM correctness: get nil-safe, scope/context/collect! as primitives
- get primitive returns nil for type mismatches (list+string) instead
  of raising — matches JS/Python behavior, fixes find-nav-match errors
- scope-peek, collect!, collected, clear-collected! registered as real
  primitives in sx_primitives table (not just env bindings) so the CEK
  step-sf-context can find them via get-primitive
- step-sf-context checks scope-peek hashtable BEFORE walking CEK
  continuation — bridges aser's scope-push!/pop! with CEK's context
- context, emit!, emitted added to SPECIAL_FORM_NAMES and handled in
  aser-special (scope operations in aser rendering mode)
- sx-context NativeFn for VM-compiled code paths
- VM execution errors no longer mark functions as permanently failed —
  bytecode is correct, errors are from runtime data
- kbd, samp, var added to HTML_TAGS + sx-browser.js rebuilt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 09:33:18 +00:00
a716e3f745 Pre-compile compiler functions at startup for faster JIT
The SX compiler's own functions (compile, compile-expr, compile-lambda,
etc.) are now JIT-compiled during vm-compile-adapter before any page
renders. This means all subsequent JIT compilations run the compiler
on the VM instead of CEK — aser compilation drops from 1.0s to 0.2s.

15 compiler functions pre-compiled in ~15s at startup. The compile-lambda
function is the largest (6.4s to compile). First page render aser=0.2s
(was 1.0s). Cached pages unchanged at 0.25-0.3s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 08:28:24 +00:00
318c818728 Lazy JIT compilation: lambdas compile to bytecode on first call
Replace AOT adapter compilation with lazy JIT — each named lambda is
compiled to VM bytecode on first call, cached in l_compiled field for
subsequent calls. Compilation failures fall back to CEK gracefully.

VM types (vm_code, vm_upvalue_cell, vm_closure) moved to sx_types.ml
mutual recursion block. Lambda and Component records gain mutable
l_compiled/c_compiled cache fields. jit_compile_lambda in sx_vm.ml
wraps body as (fn (params) body), invokes spec/compiler.sx via CEK,
extracts inner closure from OP_CLOSURE constant.

JIT hooks in both paths:
- vm_call: Lambda calls from compiled VM code
- continue_with_call: Lambda calls from CEK step loop (injected by
  bootstrap.py post-processing)

Pre-mark sentinel prevents re-entrancy (compile function itself was
hanging when JIT'd mid-compilation). VM execution errors caught and
fall back to CEK with sentinel marking.

Also: add kbd/samp/var to HTML_TAGS, rebuild sx-browser.js, add page
URL to sx-page-full-py timing log.

Performance: first page 28s (JIT compiles 17 functions), subsequent
pages 0.31s home / 0.71s wittgenstein (was 2.3s). All 1945 tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 08:18:44 +00:00
bb34b4948b OCaml raw! in HTML renderer + SX_USE_OCAML env promotion + golden tests
- sx_render.ml: add raw! handler to HTML renderer (inject pre-rendered
  content without HTML escaping)
- docker-compose.yml: move SX_USE_OCAML/SX_OCAML_BIN to shared env
  (available to all services, not just sx_docs)
- hosts/ocaml/Dockerfile: OCaml kernel build stage
- shared/sx/tests/: golden test data + generator for OCaml render tests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 22:21:04 +00:00
df461beec2 SxExpr aser wire format fix + Playwright test infrastructure + blob protocol
Aser serialization: aser-call/fragment now return SxExpr instead of String.
serialize/inspect passes SxExpr through unquoted, preventing the double-
escaping (\" → \\\" ) that broke client-side parsing when aser wire format
was output via raw! into <script> tags. Added make-sx-expr + sx-expr-source
primitives to OCaml and JS hosts.

Binary blob protocol: eval, aser, aser-slot, and sx-page-full now send SX
source as length-prefixed blobs instead of escaped strings. Eliminates pipe
desync from concurrent requests and removes all string-escape round-trips
between Python and OCaml.

Bridge safety: re-entrancy guard (_in_io_handler) raises immediately if an
IO handler tries to call the bridge, preventing silent deadlocks.

Fetch error logging: orchestration.sx error callback now logs method + URL
via log-warn. Platform catches (fetchAndRestore, fetchPreload, bindBoostForm)
also log errors instead of silently swallowing them.

Transpiler fixes: makeEnv, scopePeek, scopeEmit, makeSxExpr added as
platform function definitions + transpiler mappings — were referenced in
transpiled code but never defined as JS functions.

Playwright test infrastructure:
- nav() captures JS errors and fails fast with the actual error message
- Checks for [object Object] rendering artifacts
- New tests: delete-row interaction, full page refresh, back button,
  direct load with fresh context, code block content verification
- Default base URL changed to localhost:8013 (standalone dev server)
- docker-compose.dev-sx.yml: port 8013 exposed for local testing
- test-sx-build.sh: build + unit tests + Playwright smoke tests

Geography content: index page component written (sx/sx/geography/index.sx)
describing OCaml evaluator, wire formats, rendering pipeline, and topic
links. Wiring blocked by aser-expand-component children passing issue.

Tests: 1080/1080 JS, 952/952 OCaml, 66/66 Playwright

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 22:17:43 +00:00
6d73edf297 Length-prefixed binary framing for OCaml↔Python pipe
Replace newline-delimited text protocol with length-prefixed blobs
for all response data (send_ok_string, send_ok_raw). The OCaml side
sends (ok-len N)\n followed by exactly N raw bytes + \n. Python reads
the length, then readexactly(N).

This eliminates all pipe desync issues:
- No escaping needed for any content (HTML, SX with newlines, quotes)
- No size limits (1MB+ responses work cleanly)
- No multi-line response splitting
- No double-escaping bugs

The old (ok "...") and (ok-raw ...) formats are still parsed as
fallbacks for backward compatibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:48:52 +00:00
373a4f0134 Fix pipe desync: send_ok_raw escapes newlines, expand-components? in env
- send_ok_raw: when SX wire format contains newlines (string literals),
  fall back to (ok "...escaped...") instead of (ok-raw ...) to keep
  the pipe single-line. Prevents multi-line responses from desyncing
  subsequent requests.
- expand-components? flag set in kernel env (not just VM adapter globals)
  so aser-list's env-has? check finds it during component expansion.
- SX_STANDALONE: restore no_oauth but generate CSRF via session cookie
  so mutation handlers (DELETE etc.) still work without account service.
- Shell statics injection: only inject small values (hashes, URLs) as
  kernel vars. Large blobs (CSS, component_defs) use placeholder tokens.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 12:32:03 +00:00
ae0e87fbf8 VM aser-slot → sx-page-full: single-call page render, 0.55s warm
Compiler fixes:
- Upvalue re-lookup returns own position (uv-index), not parent slot
- Spec: cek-call uses (make-env) not (dict) — OCaml Dict≠Env
- Bootstrap post-processes transpiler Dict→Env for cek_call

VM runtime fixes:
- compile_adapter evaluates constant defines (SPECIAL_FORM_NAMES etc.)
  via execute_module instead of wrapping as NativeFn closures
- Native primitives: map-indexed, some, every?
- Nil-safe HO forms: map/filter/for-each/some/every? accept nil as empty
- expand-components? set in kernel env (not just VM globals)
- unwrap_env diagnostic: reports actual type received

sx-page-full command:
- Single OCaml call: aser-slot body + render-to-html shell
- Eliminates two pipe round-trips (was: aser-slot→Python→shell render)
- Shell statics (component_defs, CSS, pages_sx) cached in Python,
  injected into kernel once, referenced by symbol in per-request command
- Large blobs use placeholder tokens — Python splices post-render,
  pipe transfers ~51KB instead of 2MB

Performance (warm):
- Server total: 0.55s (was ~2s)
- aser-slot VM: 0.3s, shell render: 0.01s, pipe: 0.06s
- kwargs computation: 0.000s (cached)

SX_STANDALONE mode for sx_docs dev (skips fragment fetches).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 11:06:04 +00:00
8dd3eaa1d9 CALL_PRIM: primitives first, then globals — VM for-each works!
Root cause of for-each failure: CALL_PRIM checked globals before
primitives. Globals had ho_via_cek wrappers that routed for-each
through the CEK machine — which couldn't call VM closures correctly.

Fix: check Sx_primitives.get_primitive FIRST (native call_any that
handles NativeFn directly), fall back to globals for env-specific
bindings like set-render-active!.

Result: (for-each (fn (x) (+ x 1)) (list 1 2 3)) on VM → 42 ✓

Full adapter aser chain executing:
  aser → aser-list → aser-call → for-each callback
  Fails at UPVALUE_GET idx=6 (have 6) — compiler upvalue count
  off by one. Next fix: compiler scope analysis.

Also: floor(0)=-1 bug found and fixed (was round(x-0.5), now
uses OCaml's native floor). This was causing all compile failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 22:48:26 +00:00
e6663a74ba floor(0)=-1 bug fixed + 12/12 adapter compiles + primitives
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 22:34:51 +00:00
231bfbecb5 VM aser-slot routing: isolated globals, inner code extraction, debug
aser-slot now routes through the VM when adapter is compiled:
- compile_adapter: compiles each define body, extracts inner code
  from OP_CLOSURE wrapper, stores as NativeFn in separate globals
- vm_adapter_globals: isolated from kernel env (no cross-contamination)
- aser-slot checks vm_adapter_globals, calls VM aser directly

Status: 2/12 adapter functions compile and run on VM. 6 fail during
OCaml-side compilation with "index out of bounds" — likely from
set-nth! silent failure on ListRef during bytecode jump patching.

Debug output shows outer code structure is correct (4 bytes, 1 const).
Inner code_from_value conversion needs fixing for nested closures.

Also: vm-compile-adapter command inside _ensure_components lock
(fixes pipe desync from concurrent requests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 22:18:21 +00:00
df256b5607 VM adapter: compile works, env isolation needed
adapter-sx.sx compiles to 25 code objects (4044 bytes bytecode).
vm-load-module loads it. But replacing Lambda values in env.bindings
with NativeFn wrappers breaks the CEK machine for non-aser functions.

Root cause: shared env.bindings between CEK and VM. The CEK needs
Lambda values (for closure merging). The VM needs NativeFn wrappers.
Both can't coexist in the same env.

Fix needed: VM adapter gets its own globals table (with compiled
closures). The aser-slot command routes directly to the VM with
its own globals, not through the CEK with shared env.

Disabled vm-load-module. Pages render correctly via CEK.

Also: OP_CALL_PRIM now logs primitive name + argc in error messages
for easier debugging.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 21:36:38 +00:00
0ce23521b7 Aser adapter compiles + loads as VM module — first VM execution
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 21:18:34 +00:00
f12bbae6c9 40/40 VM tests pass, auto-compile disabled until full aser compilation
All VM tests green: closures with shared mutable upvalues, map/filter/
for-each via CALL_PRIM, recursive functions, nested closures.

Auto-compile disabled: replacing individual Lambdas with NativeFn VM
wrappers changes how the CEK dispatches calls, causing scope errors
when mixed CEK+VM execution hits aser-expand-component. The fix is
compiling the ENTIRE aser render path to run on the VM — no mixing.

The VM infrastructure is complete and tested. Next step: compile
adapter-sx.sx as a whole module, run the aser on the VM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:57:59 +00:00
c8c4b322a9 All 40 VM tests pass: map/filter/for-each + mutable closures fixed
Two fixes:

1. HO forms (map/filter/for-each/reduce): registered as Python
   primitives so compiler emits OP_CALL_PRIM (direct dispatch to
   OCaml primitive) instead of OP_CALL (which routed through CEK
   HO special forms and failed on NativeFn closure args).

2. Mutable closures: locals captured by closures now share an
   upvalue_cell. OP_LOCAL_GET/SET check frame.local_cells first —
   if the slot has a shared cell, read/write through it. OP_CLOSURE
   creates or reuses cells for is_local=1 captures. Both parent
   and closure see the same mutations.

   Frame type extended with local_cells hashtable for captured slots.

40/40 tests pass:
  - 12 compiler output tests
  - 18 VM execution tests (arithmetic, control flow, closures,
    nested let, higher-order, cond, string ops)
  - 10 auto-compile pattern tests (recursive, map, filter,
    for-each, mutable closures, multiple closures, type dispatch)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:47:40 +00:00
e7da397f8e VM upvalues + HO primitives + 40 tests (36 pass, 4 fail)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:41:23 +00:00
1bb40415a8 VM upvalue support: closures capture variables from enclosing scopes
Compiler (compiler.sx):
- Function scopes marked is-function=true; let scopes share parent frame
- scope-resolve only creates upvalue captures at function boundaries
- Let scope locals use parent's slot numbering (same frame)
- OP_CLOSURE emits upvalue descriptors: (is_local, index) per capture

VM (sx_vm.ml):
- upvalue_cell type: shared mutable reference to captured value
- OP_UPVALUE_GET/SET: read/write from closure's upvalue array
- OP_CLOSURE: reads upvalue descriptors, creates cells from
  enclosing frame's locals (is_local=1) or upvalues (is_local=0)
- vm_closure carries live env_ref (not snapshot)
- vm_call falls back to CEK for Lambda/Component/Island values

Verified: (let ((x 10)) (let ((add-x (fn (y) (+ x y)))) (add-x 5)))
  Compiles to: CONST 10, LOC_SET #0, CLOSURE [UV_GET#0 LOC_GET#0 CPRIM+ RET]
  with upvalue descriptor: is_local=1 index=0
  VM executes → 15 ✓

Auto-compile: 6/117 functions compile (up from 3). Disabled until
compiler handles all features — fallback can't reconstruct closure
scope for variables like nav-state bound in caller's let*.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 20:13:17 +00:00
5ca2ee92bc VM auto-compile infrastructure + disable until compiler is complete
Added vm-compile command: iterates env, compiles lambdas to bytecode,
replaces with NativeFn VM wrappers (with CEK fallback on error).
Tested: 3/109 compile, reduces CEK steps 23%.

Disabled auto-compile in production — the compiler doesn't handle
closures with upvalues yet, and compiled functions that reference
dynamic env vars crash. Infrastructure stays for when compiler
handles all SX features.

Also: added set-nth! and mutable-list primitives (needed by
compiler.sx for bytecode patching). Fixed compiler.sx to use
mutable lists on OCaml (ListRef for append!/set-nth! mutation).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 19:30:54 +00:00
e14fc9b0e1 Auto-compile: lambdas → bytecode VM at load time
After loading .sx files, (vm-compile) iterates all named lambdas,
compiles each body to bytecode, replaces with NativeFn VM wrapper.

Results: 3/109 functions compiled (compiler needs more features).
CEK steps: 49911 → 38083 (23% fewer) for home page.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 19:07:42 +00:00
a8d1163aa6 SX bytecode VM executing: compile → run → correct results
End-to-end pipeline working:
  Python compiler.sx → bytecode → OCaml VM → result

Verified: (+ (* 3 4) 2) → 14 ✓
          (+ 0 1 2 ... 49) → 1225 ✓

Benchmark (500 iterations, 50 additions each):
  CEK machine: 327ms
  Bytecode VM: 145ms
  Speedup: 2.2x

VM handles: constants, local variables, global variables,
primitive calls, jumps, conditionals, closures (via NativeFn
wrapper), define, return.

Protocol: (vm-exec {:bytecode (...) :constants (...)})
  - Compiler outputs clean format (no internal index dict)
  - VM converts bytecode list to int array, constants to value array
  - Stack-based execution with direct opcode dispatch

The 2.2x speedup is for pure arithmetic. For aser (the real
target), the speedup will be larger because aser involves:
- String building (no CEK frame allocation in VM)
- Map/filter iterations (no frame-per-iteration in VM)
- Closure calls (no thunk/trampoline in VM)

Next: compile and run the aser adapter on the VM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 18:52:50 +00:00
40d0f1a438 SX bytecode: format definition, compiler, OCaml VM (Phase 1)
Three new files forming the bytecode compilation pipeline:

spec/bytecode.sx — opcode definitions (~65 ops):
  - Stack/constant ops (CONST, NIL, TRUE, POP, DUP)
  - Lexical variable access (LOCAL_GET/SET, UPVALUE_GET/SET, GLOBAL_GET/SET)
  - Jump-based control flow (JUMP, JUMP_IF_FALSE/TRUE)
  - Function ops (CALL, TAIL_CALL, RETURN, CLOSURE, CALL_PRIM)
  - HO form ops (ITER_INIT/NEXT, MAP_OPEN/APPEND/CLOSE)
  - Scope/continuation ops (SCOPE_PUSH/POP, RESET, SHIFT)
  - Aser specialization (ASER_TAG, ASER_FRAG)

spec/compiler.sx — SX-to-bytecode compiler (SX code, portable):
  - Scope analysis: resolve variables to local/upvalue/global at compile time
  - Tail position detection for TCO
  - Code generation for: if, when, and, or, let, begin, lambda,
    define, set!, quote, function calls, primitive calls
  - Constant pool with deduplication
  - Jump patching for forward references

hosts/ocaml/lib/sx_vm.ml — bytecode interpreter (OCaml):
  - Stack-based VM with array-backed operand stack
  - Call frames with base pointer for locals
  - Direct opcode dispatch via pattern match
  - Zero allocation per step (unlike CEK machine's dict-per-step)
  - Handles: constants, variables, jumps, calls, primitives,
    collections, string concat, define

Architecture: compiler.sx is spec (SX, portable). VM is platform
(OCaml-native). Same bytecode runs on JS/WASM VMs.

Also includes: CekFrame record optimization in transpiler.sx
(29 frame types as records instead of Hashtbl).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 18:25:41 +00:00
d9e80d8544 CEK frame records: eliminate Hashtbl for all 29 frame types
Transpiler detects dict literals with a "type" string field and emits
CekFrame records instead of Dict(Hashtbl). Maps frame-specific fields
to generic record slots:

  cf_type, cf_env, cf_name, cf_body, cf_remaining, cf_f,
  cf_args (also evaled), cf_results (also raw-args),
  cf_extra (ho-type/scheme/indexed/match-val/current-item/...),
  cf_extra2 (emitted/effect-list/first-render)

Runtime get_val handles CekFrame with direct field match — O(1)
field access vs Hashtbl.find.

Bootstrapper: skip stdlib.sx entirely (already OCaml primitives).

Result: 29 CekFrame + 2 CekState = 31 record types, only 8
Hashtbl.create remaining (effect-annotations, empty dicts).

Benchmark (200 divs): 2.94s → 1.71s (1.7x speedup from baseline).
Real pages: ~same as CekState-only (frames are <20% of allocations;
states dominate at 199K/page).

Foundation for JIT: record-based value representation enables
typed compilation — JIT can emit direct field access instead of
hash table lookups.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 17:56:50 +00:00
c16142d14c CekState record optimization + profiling: 1.5x speedup, root cause found
Transpiler (transpiler.sx): detects CEK state dict literals (5 fields:
control/env/kont/phase/value) and emits CekState OCaml record instead
of Dict(Hashtbl). Eliminates 200K Hashtbl allocations per page.

Bootstrapper: skip stdlib.sx (functions already registered as OCaml
primitives). Only transpile evaluator.sx.

Runtime: get_val handles CekState with direct field access. type_of
returns "dict" for CekState (backward compat).

Profiling results (root cause of slowness):
  Pure eval: OCaml 1.6x FASTER than Python (expected)
  Aser: OCaml 28x SLOWER than Python (unexpected!)

Root cause: Python has a native optimized aser. OCaml runs the SX
adapter-sx.sx through the CEK machine — each aserCall is ~50 CEK
steps with closures, scope operations, string building.

Fix needed: native OCaml aser (like Python's), not SX adapter
through CEK machine.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 17:40:34 +00:00
8707f21ca2 Single-pass aser_slot for HTMX path + kernel eval timing + stable hash
Eliminated double-aser for HTMX requests: build OOB wrapper AST
(~shared:layout/oob-sx :content wrapped_ast) and aser_slot in ONE
pass — same pattern as the full-page path. Halves aser_slot calls.

Added kernel-side timing to stderr:
  [aser-slot] eval=3.6s io_flush=0.0s batched=3 result=22235 chars

Results show batch IO works (io_flush=0.0s for 3 highlight calls)
and the bottleneck is pure CEK evaluation time, not IO.

Performance after single-pass fix:
  Home: 0.7s eval (was 2.2s total)
  Reactive: 3.6s eval (was 6.8s total)
  Language: 1.1s eval (was 18.9s total — double-aser eliminated)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 17:03:47 +00:00
96e7bbbac1 Non-blocking batch IO for OCaml kernel + stable component hash
OCaml kernel (sx_server.ml):
- Batch IO mode for aser-slot: batchable helpers (highlight,
  component-source) return placeholders during evaluation instead
  of blocking on stdin. After aser completes, all batched requests
  are flushed to Python at once.
- Python processes them concurrently with asyncio.gather.
- Placeholders (using «IO:N» markers) are replaced with actual
  values in the result string.
- Non-batchable IO (query, action, ctx, request-arg) still uses
  blocking mode — their results drive control flow.

Python bridge (ocaml_bridge.py):
- _read_until_ok handles batched protocol: collects io-request
  lines with numeric IDs, processes on (io-done N) with gather.
- IO result cache for pure helpers — eliminates redundant calls.
- _handle_io_request strips batch ID from request format.

Component caching (jinja_bridge.py):
- Hash computed from FULL component env (all names + bodies),
  not per-page subset. Stable across all pages — browser caches
  once, no re-download on navigation between pages.
- invalidate_component_hash() called on hot-reload.

Tests: 15/15 OCaml helper tests pass (2 new batch IO tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 16:53:01 +00:00
8373c6cf16 SX spec introspection: the spec examines itself via sx-parse
spec-introspect.sx: pure SX functions that read, parse, and analyze
spec files. No Python. The spec IS data — a macro transforms it into
explorer UI components.

- spec-explore: reads spec file via IO, parses with sx-parse, extracts
  sections/defines/effects/params, produces explorer data dict
- spec-form-name/kind/effects/params/source: individual extractors
- spec-group-sections: groups defines into sections
- spec-compute-stats: aggregate effect/define counts

OCaml kernel fixes:
- nth handles strings (character indexing for parser)
- ident-start?, ident-char?, char-numeric?, parse-number: platform
  primitives needed by spec/parser.sx when loaded at runtime
- _find_spec_file: searches spec/, web/, shared/sx/ref/ for spec files

83/84 Playwright tests pass. The 1 failure is client-side re-rendering
of the spec explorer (the client evaluates defpage content which calls
find-spec — unavailable on the client).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 17:57:19 +00:00
fac97883f9 Spec explorer data endpoint, spec file finder, browser render test (failing)
- Add spec-explorer-data-by-slug helper with _SPEC_SLUG_MAP
- _find_spec_file searches spec/, web/, shared/sx/ref/ directories
- defpage specs-explore-page uses :data for server-side data fetch
- test_evaluator_renders_in_browser: failing test for client-side rendering
  (client re-evaluates defpage content, find-spec unavailable — pre-existing)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 17:36:21 +00:00
71c2003a60 OCaml evaluator for page dispatch + handler aser, 83/83 Playwright tests
Major architectural change: page function dispatch and handler execution
now go through the OCaml kernel instead of the Python bootstrapped evaluator.

OCaml integration:
- Page dispatch: bridge.eval() evaluates SX URL expressions (geography, marshes, etc.)
- Handler aser: bridge.aser() serializes handler responses as SX wire format
- _ensure_components loads all .sx files into OCaml kernel (spec, web adapter, handlers)
- defhandler/defpage registered as no-op special forms so handler files load
- helper IO primitive dispatches to Python page helpers + IO handlers
- ok-raw response format for SX wire format (no double-escaping)
- Natural list serialization in eval (no (list ...) wrapper)
- Clean pipe: _read_until_ok always sends io-response on error

SX adapter (aser):
- scope-emit!/scope-peek aliases to avoid CEK special form conflict
- aser-fragment/aser-call: strings starting with "(" pass through unserialized
- Registered cond-scheme?, is-else-clause?, primitive?, get-primitive in kernel
- random-int, parse-int as kernel primitives; json-encode, into via IO bridge

Handler migration:
- All IO calls converted to (helper "name" args...) pattern
- request-arg, request-form, state-get, state-set!, now, component-source etc.
- Fixed bare (effect ...) in island bodies leaking disposer functions as text
- Fixed lower-case → lower, ~search-results → ~examples/search-results

Reactive islands:
- sx-hydrate-islands called after client-side navigation swap
- force-dispose-islands-in for outerHTML swaps (clears hydration markers)
- clear-processed! platform primitive for re-hydration

Content restructuring:
- Design, event bridge, named stores, phase 2 consolidated into reactive overview
- Marshes split into overview + 5 example sub-pages
- Nav links use sx-get/sx-target for client-side navigation

Playwright test suite (sx/tests/test_demos.py):
- 83 tests covering hypermedia demos, reactive islands, marshes, spec explorer
- Server-side rendering, handler interactions, island hydration, navigation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 17:22:51 +00:00
4ce4762237 Add spec/stdlib.sx: 46 primitives become library functions
The irreducible primitive set drops from 79 to 33. Everything that can
be expressed in SX is now a library function in stdlib.sx, loaded after
evaluator.sx and before render.sx.

Moved to stdlib.sx (pure SX, no host dependency):
- Logic: not
- Comparison: != <= >= eq? eqv? equal?
- Predicates: nil? boolean? number? string? list? dict? continuation?
  empty? odd? even? zero? contains?
- Arithmetic: inc dec abs ceil round min max clamp
- Collections: first last rest nth cons append reverse flatten range
  chunk-every zip-pairs vals has-key? merge assoc dissoc into
- Strings: upcase downcase string-length substring string-contains?
  starts-with? ends-with? split join replace
- Text: pluralize escape assert parse-datetime

Remaining irreducible primitives (33):
  + - * / mod floor pow sqrt = < > type-of symbol-name keyword-name
  str slice index-of upper lower trim char-from-code list dict concat
  get len keys dict-set! append! random-int json-encode format-date
  parse-int format-decimal strip-tags sx-parse error apply

All hosts: JS 957+1080, Python 744, OCaml 952 — zero regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 08:55:57 +00:00
06666ac8c4 Decouple core evaluator from web platform, extract libraries
The core evaluator (spec/evaluator.sx) is now the irreducible computational
core with zero web, rendering, or type-system knowledge. 2531 → 2313 lines.

- Add extensible special form registry (*custom-special-forms* + register-special-form!)
- Add render dispatch hooks (*render-check* / *render-fn*) replacing hardcoded render-active?/is-render-expr?/render-expr
- Extract freeze scopes → spec/freeze.sx (library, not core)
- Extract content addressing → spec/content.sx (library, not core)
- Move sf-deftype/sf-defeffect → spec/types.sx (self-registering)
- Move sf-defstyle → web/forms.sx (self-registering with all web forms)
- Move web tests (defpage, streaming) → web/tests/test-forms.sx
- Add is-else-clause? helper (replaces 5 inline patterns)
- Make escape-html/escape-attr library functions in render.sx (pure SX, not platform-provided)
- Add foundations plan: Step 3.5 (data representations), Step 3.7 (verified components), OCaml for Step 4d
- Update all three bootstrappers (JS 957/957, Python 744/744, OCaml 952/952)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 08:37:50 +00:00
313f7d6be1 OCaml bootstrapper Phase 2: HTML renderer, SX server, Python bridge
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 23:28:48 +00:00
818e5d53f0 OCaml bootstrapper: transpiler compiles full CEK evaluator (61/61 tests)
SX-to-OCaml transpiler (transpiler.sx) generates sx_ref.ml (~90KB, ~135
mutually recursive functions) from the spec evaluator. Foundation tests
all pass: parser, primitives, env operations, type system.

Key design decisions:
- Env variant added to value type for CEK state dict storage
- Continuation carries optional data dict for captured frames
- Dynamic var tracking distinguishes OCaml fn calls from SX value dispatch
- Single let rec...and block for forward references between all defines
- Unused ref pre-declarations eliminated via let-bound name detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 20:51:59 +00:00