Removes the 5993-line bootstrapped Python evaluator (sx_ref.py) and all
code that depended on it exclusively. Both bootstrappers (JS + OCaml)
now use a new synchronous OCaml bridge (ocaml_sync.py) to run the
transpiler. JS build produces identical output; OCaml bootstrap produces
byte-identical sx_ref.ml.
Key changes:
- New shared/sx/ocaml_sync.py: sync subprocess bridge to sx_server.exe
- hosts/javascript/bootstrap.py: serialize defines → temp file → OCaml eval
- hosts/ocaml/bootstrap.py: same pattern for OCaml transpiler
- shared/sx/{html,async_eval,resolver,jinja_bridge,handlers,pages,deps,helpers}:
stub or remove sx_ref imports; runtime uses OCaml bridge (SX_USE_OCAML=1)
- sx/sxc/pages: parse defpage/defhandler from AST instead of Python eval
- hosts/ocaml/lib/sx_primitives.ml: append handles non-list 2nd arg per spec
- Deleted: sx_ref.py, async_eval_ref.py, 6 Python test runners, misc ref/ files
Test results: JS 1078/1078, OCaml 1114/1114.
sx_docs SSR has pre-existing rendering issues to investigate separately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace AOT adapter compilation with lazy JIT — each named lambda is
compiled to VM bytecode on first call, cached in l_compiled field for
subsequent calls. Compilation failures fall back to CEK gracefully.
VM types (vm_code, vm_upvalue_cell, vm_closure) moved to sx_types.ml
mutual recursion block. Lambda and Component records gain mutable
l_compiled/c_compiled cache fields. jit_compile_lambda in sx_vm.ml
wraps body as (fn (params) body), invokes spec/compiler.sx via CEK,
extracts inner closure from OP_CLOSURE constant.
JIT hooks in both paths:
- vm_call: Lambda calls from compiled VM code
- continue_with_call: Lambda calls from CEK step loop (injected by
bootstrap.py post-processing)
Pre-mark sentinel prevents re-entrancy (compile function itself was
hanging when JIT'd mid-compilation). VM execution errors caught and
fall back to CEK with sentinel marking.
Also: add kbd/samp/var to HTML_TAGS, rebuild sx-browser.js, add page
URL to sx-page-full-py timing log.
Performance: first page 28s (JIT compiles 17 functions), subsequent
pages 0.31s home / 0.71s wittgenstein (was 2.3s). All 1945 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compiler fixes:
- Upvalue re-lookup returns own position (uv-index), not parent slot
- Spec: cek-call uses (make-env) not (dict) — OCaml Dict≠Env
- Bootstrap post-processes transpiler Dict→Env for cek_call
VM runtime fixes:
- compile_adapter evaluates constant defines (SPECIAL_FORM_NAMES etc.)
via execute_module instead of wrapping as NativeFn closures
- Native primitives: map-indexed, some, every?
- Nil-safe HO forms: map/filter/for-each/some/every? accept nil as empty
- expand-components? set in kernel env (not just VM globals)
- unwrap_env diagnostic: reports actual type received
sx-page-full command:
- Single OCaml call: aser-slot body + render-to-html shell
- Eliminates two pipe round-trips (was: aser-slot→Python→shell render)
- Shell statics (component_defs, CSS, pages_sx) cached in Python,
injected into kernel once, referenced by symbol in per-request command
- Large blobs use placeholder tokens — Python splices post-render,
pipe transfers ~51KB instead of 2MB
Performance (warm):
- Server total: 0.55s (was ~2s)
- aser-slot VM: 0.3s, shell render: 0.01s, pipe: 0.06s
- kwargs computation: 0.000s (cached)
SX_STANDALONE mode for sx_docs dev (skips fragment fetches).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminated double-aser for HTMX requests: build OOB wrapper AST
(~shared:layout/oob-sx :content wrapped_ast) and aser_slot in ONE
pass — same pattern as the full-page path. Halves aser_slot calls.
Added kernel-side timing to stderr:
[aser-slot] eval=3.6s io_flush=0.0s batched=3 result=22235 chars
Results show batch IO works (io_flush=0.0s for 3 highlight calls)
and the bottleneck is pure CEK evaluation time, not IO.
Performance after single-pass fix:
Home: 0.7s eval (was 2.2s total)
Reactive: 3.6s eval (was 6.8s total)
Language: 1.1s eval (was 18.9s total — double-aser eliminated)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: OcamlBridge._send() used write() without drain().
asyncio.StreamWriter buffers writes — without drain(), multiple
commands accumulate and flush as a batch. The kernel processes
them sequentially, sending responses, but Python only reads one
response per command → pipe desync → "unexpected response" errors.
Fix: _send() is now async, calls drain() after every write.
All 14 callers updated to await.
Playwright tests rewritten:
- test_home_has_header: verifies #logo-opacity visible (was only
checking for "sx" text — never caught missing header)
- test_home_has_nav_children: Geography link must be visible
- test_home_has_main_panel: #main-panel must have child elements
- TestDirectPageLoad: fresh browser.new_context() per test to
avoid stale component hash in localStorage
- _setup_error_capture + _check_no_fatal_errors helpers
_render_to_sx uses aser_slot (not aser) — layout wrappers contain
re-parsed content that needs full expansion capability.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three fixes completing the aser_slot migration:
1. Single-pass full-page rendering: eval_sx_url builds layout+content
AST and aser_slots it in ONE call — avoids double-aser where
re-parsed content hits "Undefined symbol: title/deref" errors.
2. Pipe desync fix: _inject_helpers_locked runs INSIDE the aser_slot
lock acquisition (not as a separate lock). Prevents interleaved
commands from other coroutines between injection and aser-slot.
3. _render_to_sx uses aser_slot (not aser): layout wrappers like
oob_page_sx contain re-parsed content from earlier aser_slot
calls. Regular aser fails on symbols that were bound during
the earlier expansion. aser_slot handles them correctly.
HTMX path: aser_slot the content, then oob_page_sx wraps it.
Full page path: build (~shared:layout/app-body :content wrapped_ast),
aser_slot in one pass, pass directly to sx_page.
New Playwright tests: test_navigate_geography_to_reactive,
test_direct_load_reactive_page.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three bugs in aser-expand-component (adapter-sx.sx):
- Keyword values were eval'd (eval-expr can't handle <>, HTML tags);
now asered, matching the aser's rendering capabilities
- Missing default nil binding for unset &key params (caused
"Undefined symbol" errors for optional params like header-rows)
- aserCall string-quoted keyword values that were already serialized
SX — now inlines values starting with "(" directly
Server-affinity annotations for layout/nav shells:
- ~shared:layout/app-body, ~shared:layout/oob-sx — page structure
- ~layouts/nav-sibling-row, ~layouts/nav-children — server-side data
- ~layouts/doc already had :affinity :server
- ~cssx/flush marked :affinity :client (browser-only state)
Navigation fix: restore oob_page_sx wrapper for HTMX responses
so #main-panel section exists for sx-select/sx-swap targeting.
OCaml bridge: lazy page helper injection into kernel via IO proxy
(define name (fn (...) (helper "name" ...))) — enables aser_slot
to evaluate highlight/component-source etc. via coroutine bridge.
Playwright tests: added pageerror listener to test_no_console_errors,
new test_navigate_from_home_to_geography for HTMX nav regression.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
adapter-sx.sx: aser-expand-component expands :affinity :server components
inline during SX wire format serialization. Binds keyword args via
eval-expr, children via aser (handles HTML tags), then asers the body.
ocaml_bridge.py: 10MB readline buffer for large spec responses.
nav-data.sx: evaluator.sx filename fix.
Page rendering stays on Python _eval_slot for now — full OCaml rendering
needs the page shell IO (headers, CSRF, CSS) migrated to OCaml IO bridge.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major architectural change: page function dispatch and handler execution
now go through the OCaml kernel instead of the Python bootstrapped evaluator.
OCaml integration:
- Page dispatch: bridge.eval() evaluates SX URL expressions (geography, marshes, etc.)
- Handler aser: bridge.aser() serializes handler responses as SX wire format
- _ensure_components loads all .sx files into OCaml kernel (spec, web adapter, handlers)
- defhandler/defpage registered as no-op special forms so handler files load
- helper IO primitive dispatches to Python page helpers + IO handlers
- ok-raw response format for SX wire format (no double-escaping)
- Natural list serialization in eval (no (list ...) wrapper)
- Clean pipe: _read_until_ok always sends io-response on error
SX adapter (aser):
- scope-emit!/scope-peek aliases to avoid CEK special form conflict
- aser-fragment/aser-call: strings starting with "(" pass through unserialized
- Registered cond-scheme?, is-else-clause?, primitive?, get-primitive in kernel
- random-int, parse-int as kernel primitives; json-encode, into via IO bridge
Handler migration:
- All IO calls converted to (helper "name" args...) pattern
- request-arg, request-form, state-get, state-set!, now, component-source etc.
- Fixed bare (effect ...) in island bodies leaking disposer functions as text
- Fixed lower-case → lower, ~search-results → ~examples/search-results
Reactive islands:
- sx-hydrate-islands called after client-side navigation swap
- force-dispose-islands-in for outerHTML swaps (clears hydration markers)
- clear-processed! platform primitive for re-hydration
Content restructuring:
- Design, event bridge, named stores, phase 2 consolidated into reactive overview
- Marshes split into overview + 5 example sub-pages
- Nav links use sx-get/sx-target for client-side navigation
Playwright test suite (sx/tests/test_demos.py):
- 83 tests covering hypermedia demos, reactive islands, marshes, spec explorer
- Server-side rendering, handler interactions, island hydration, navigation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents the three orthogonal primitives (spread, collect!, reactive-spread),
their operation across server/client/morph boundaries, CSSX as use case,
semantic style variables, and the planned provide/context/emit! unification.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add url-to-expr, auto-quote-unknowns, prepare-url-expr to router.sx —
the canonical URL-to-expression pipeline. Dots→spaces, parse, then
auto-quote unknown symbols as strings (slugs). The same spec serves
both server (Python) and client (JS) route handling.
- router.sx: three new pure functions for URL evaluation
- bootstrap_py.py: auto-include router module with html adapter
- platform_js.py: export urlToExpr/autoQuoteUnknowns/prepareUrlExpr
- sx_router.py: replace hand-written auto_quote_slugs with bootstrapped
prepare_url_expr — delete ~50 lines of hardcoded function name sets
- Rebootstrap sx_ref.py (4331 lines) and sx-browser.js
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Component names now reflect filesystem location using / as path separator
and : as namespace separator for shared components:
~sx-header → ~layouts/header
~layout-app-body → ~shared:layout/app-body
~blog-admin-dashboard → ~admin/dashboard
209 files, 4,941 replacements across all services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All routes moved under /sx/ prefix:
- / redirects to /sx/
- /sx/ serves home page
- /sx/<path:expr> is the catch-all for SX expression URLs
- Bare /(...) and /~... redirect to /sx/(...) and /sx/~...
- All ~600 hrefs, sx-get attrs, defhandler paths, redirect
targets, and blueprint routes updated across 44 files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New comprehensive documentation for SX URLs at /(applications.(sx-urls))
covering dots-as-spaces, nesting/scoping, relative URLs, keyword ops,
delta values, special forms, hypermedia integration, and GraphSX.
Fix layout tagline: "A" → "The" framework-free reactive hypermedium.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Every URL at sx-web.org now uses bracketed SX expressions — pages AND
API endpoints. defhandler :path values, sx-get/sx-post/sx-delete attrs,
code examples, and Python route decorators all converted.
- Add SxAtomConverter to handlers.py for parameter matching inside
expression URLs (e.g. /(api.(item.<sx:item_id>)))
- Convert ~50 defhandler :path values in ref-api.sx and examples.sx
- Convert ~90 sx-get/sx-post/sx-delete URLs in reference.sx, examples.sx
- Convert ~30 code example URLs in examples-content.sx
- Convert ~30 API URLs in pages.py (Python string code examples)
- Convert ~70 page navigation URLs in pages.py
- Convert 7 Python route decorators in routes.py
- Convert ~10 reactive API URLs in marshes.sx
- Add API redirect patterns to sx_router.py (301 for old paths)
- Remove /api/ skip in app.py redirects (old API paths now redirect)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
