New SX primitive io-select-channels(read-list write-list timeout-ms) wrapping
Unix.select on the registered channel table. Returns {:readable :writable}.
Tcl event loop implemented purely in Tcl (no sx_server.ml changes):
- fileevent $chan readable|writable script (or "" to unregister)
- fileevent $chan event (1 arg) returns the registered script
- after ms script — schedule one-shot timer
- after ms (no script) — sleep, driving event loop in the meantime
- vwait varname — block until var is set/changed, handlers run between polls
- update — non-blocking event drain (poll-timeout=0)
State on interp: :fileevents (list of (chan event script)) and :timers
(sorted list of (expiry-ms script)).
tcl-event-step is the inner loop: expire timers, build fd lists from
:fileevents, call io-select-channels with computed timeout, run ready
handlers. vwait polls every 1000ms or until var changes.
Scoped to script mode by design — vwait from inside a server-handled
command does not interact with sx_server's stdin scheduler.
+5 idiom tests: after-vwait-timer, after-multiple-timers-update,
fileevent-readable-fires, fileevent-query-script,
after-cancel-via-vwait-timing. 354/354 green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a `perform` fired inside a tree-walked eval_expr path — sf_letrec init
exprs / non-last body exprs, expand_macro body, qq_expand unquote,
sf_dynamic_wind / sf_scope / sf_provide bodies — cek_run raised
"IO suspension in non-IO context" and swallowed the suspension. The hook
that converts the CEK suspended state to VmSuspended (so the outer driver
sees it as a resumable suspension object) was defined in sx_vm.ml but
never invoked from cek_run.
Repro in Node.js (hosts/ocaml/browser/test_letrec_resume.js):
(letrec ((x (perform {:op "io"}))) "ok") ;; threw the error
(letrec ((x 1)) (perform {:op "io"}) "after") ;; threw the error
The originally reported browser symptom — "[sx] resume: Not callable: nil"
after hs-wait resumes inside a letrec — was the same root cause showing
through the JIT/VM resume path instead of as a top-level error.
Fix: cek_run and cek_run_iterative now check !_cek_io_suspend_hook and
invoke it when the loop terminates in a suspended state. The hook (set by
sx_vm.ml in the browser, by run_tests.ml in the test runner) converts the
suspension to VmSuspended / resolves IO synchronously. When the hook is
unset (pure-CEK harness), the legacy Eval_error is raised so misuse stays
visible.
Also patches:
- hosts/ocaml/bootstrap.py — regex-patches the transpiled cek_run on regen
so the fix survives a fresh `python3 hosts/ocaml/bootstrap.py` cycle.
- hosts/ocaml/browser/sx_browser.ml — api_eval / api_eval_vm / api_eval_expr
now catch VmSuspended and surface a clean error string (K.eval has no
driver to resume; callers who want resumption use callFn).
Tests:
- spec/tests/test-letrec-resume-treewalk.sx — 7 CEK-level regression tests
covering letrec init / non-last body, scope/provide bodies, sibling
fn-after-perform. All 7 fail in baseline ("IO suspension in non-IO
context"), all 7 pass with the fix.
- hosts/ocaml/browser/test_letrec_resume.js — 13 WASM kernel tests via
callFn driveSync, including the wait-boot pattern from the briefing.
All 13 pass.
Suite results: 4557 pass / 1338 fail (was 4550 / 1339); +7 new passes,
-1 flaky timeout (hs-upstream-if sieve), no regressions.
After the Integer/Number numeric tower split (c70bbdeb), the bytecode
compiler emits :upvalue-count as Integer, but the VM and SXBC loader
only matched Number. The fallback `_ -> 0` made the VM skip reading
upvalue descriptors entirely, so the IP advanced into raw upvalue
bytes which were then misread as opcodes.
Symptom: JIT runs of nested closures (curried functions, Y combinator,
component bodies that close over outer let-bindings) produced "VM:
CONST index N out of bounds (pool size M)" with N values like 256,
4096, 5120, 12800, 13056 — all of the form `byte | (opcode << 8)`,
i.e. an upvalue descriptor (lo) followed by the next instruction's
opcode (hi) being read as a u16 operand.
Fix all five sites that decode upvalue-count to also accept Integer:
- hosts/ocaml/lib/sx_vm.ml: OP_CLOSURE handler, trace_run, disassemble
- hosts/ocaml/lib/sx_vm_ref.ml + hosts/ocaml/sx_vm_ref.ml + bootstrap_vm.py:
vm_create_closure preamble (the bootstrap source-of-truth and both
generated copies)
- hosts/ocaml/browser/sx_browser.ml: SXBC loader's parse_kv
Test impact: JIT 3848 -> 4538 passing (+690). No-JIT unchanged at 4550.
The previously-failing curried/Y/higher-order tests in
spec/tests/test-cek-advanced.sx now pass under --jit and serve as
regression coverage.
This fixes a real current bug. The 28-day-old memory file describing
parser-combinator JIT bugs predates the numeric tower split and
described a different problem; with this fix the parser-combinator
broken-name list (`_jit_is_broken_name` in sx_vm.ml) is no longer
strictly required for correctness, but keeping it avoids a TIMEOUT
regression in one hyperscript test, so it remains in place.
11 new SX primitives in sx_primitives.ml wrapping Unix.openfile/read/write/
lseek/set_nonblock: channel-open/close/read/read-line/write/flush/seek/tell/
eof?/blocking?/set-blocking!.
Tcl runtime now uses real channel ops:
- open ?-mode? returns "fileN" handle (modes r/w/a/r+/w+/a+)
- close/read/gets/puts/seek/tell/eof/flush wired through
- new fconfigure command supports -blocking 0|1
- puts dispatches to channel-write when first arg starts with "file"
- gets command registration fixed (was pointing to old stub)
eof-returns-1 coro test updated to match real Tcl semantics (eof flips
only after a read hits EOF).
Test runner timeout bumped 180s→1200s (post-merge JIT is slow).
+7 idiom tests covering write+read, gets-loop, seek/tell, eof-after-read,
append mode, seek-to-end, fconfigure-blocking. 349/349 green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The bytecode compiler emitted OP_CALL_PRIM (52) for every primitive call, even
for arithmetic and comparison hot-paths. The VM had specialized opcodes
(OP_ADD, OP_SUB, OP_EQ, etc.) defined but unused.
- lib/compiler.sx (compile-call): emit specialized 1-byte opcode when the
primitive name + arity matches one of {+, -, *, /, =, <, >, cons, not, len,
first, rest}. Falls back to CALL_PRIM otherwise. fib bytecode: 50 → 38 bytes.
- hosts/ocaml/lib/sx_compiler.ml: mirror change in the auto-generated OCaml
compiler so SXBC export from mcp_tree uses the same emission.
- hosts/ocaml/lib/sx_vm.ml: extend OP_ADD/SUB/MUL/DIV to handle Integer+Integer
(not just Number+Number). Inline OP_EQ via Sx_runtime._fast_eq. Inline
OP_LT/GT mixed-numeric comparisons. Avoids Hashtbl lookup on the fallback
path for the common integer cases that dominate tight loops.
- hosts/ocaml/bin/bench_vm.ml: VM-only benchmark — loads compiler.sx via CEK,
JIT-compiles each fn, measures Sx_vm.call_closure throughput.
Median improvements (best of 3 runs of 9-min, bench_vm.exe):
fib(22) 107.87ms → 33.13ms -69%
loop(200000) 429.64ms → 161.16ms -62%
sum-to(50000) 72.85ms → 36.74ms -50%
count-lt(20000) 28.44ms → 17.58ms -38%
count-eq(20000) 37.23ms → 15.46ms -58%
Tests: 4550/4550 OCaml passing (unchanged). Zero regressions.
Last step in the sx-improvements roadmap — all 14 steps complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Added short aliases make-buffer / buffer? / buffer-append! / buffer->string /
buffer-length on both OCaml and JS hosts, sharing the existing StringBuffer
value type. buffer-append! auto-coerces non-strings via inspect.
Rewrote the OCaml host inspect function to walk a single shared Buffer.t
instead of allocating O(n) intermediate strings via String.concat at every
recursion level. inspect underlies sx-serialize and error-path formatting,
so this benefits the tightest serialization paths.
Median improvements (bin/bench_inspect.exe, best-of-3 of 9-run min):
tree-d8 (75KB): 5.31ms -> 1.30ms (-76%)
tree-d10 (679KB): 81.89ms -> 16.02ms (-80%)
dict-1000: 0.80ms -> 0.31ms (-61%)
list-2000: 0.74ms -> 0.33ms (-55%)
Tests: OCaml 4545 -> 4550. JS 2591 -> 2596. Zero regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CEK frames were already records (cek_frame in sx_types.ml), so the actual
hot-path bottleneck was prim_call "=" [...] in step_continue/step_eval
dispatch: each step did a Hashtbl lookup + 2x list cons + pattern match
just to compare frame-type strings.
Added a short-circuit fast path in prim_call (sx_runtime.ml) for the
hot operators: =, <, >, <=, >=, empty?, first, rest, len. These bypass
the primitives Hashtbl entirely and dispatch directly on value shape.
Inlined _fast_eq for scalar/string equality, which dominates frame-type
dispatch comparisons.
Added bin/bench_cek.exe with five tight-loop benchmarks (fib, loop,
map, reduce, let-heavy). Median of 7 runs:
fib(18) 2789ms -> 941ms (-66%)
loop(5000) 2018ms -> 620ms (-69%)
map sq xs(1000) 108ms -> 48ms (-56%)
reduce + ys(2000) 72ms -> 10ms (-86%)
let-heavy(2000) 491ms -> 271ms (-45%)
Tests: 4545/4545 passing baseline preserved (1339 pre-existing failures
unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move `hs-prolog-hook` / `hs-set-prolog-hook!` / `prolog` out of
`lib/hyperscript/runtime.sx` into a self-contained plugin file at
`lib/hyperscript/plugins/prolog.sx`. The API surface is preserved —
`lib/prolog/hs-bridge.sx::pl-install-hs-hook!` still calls
`hs-set-prolog-hook!` exactly as before, just resolved to the plugin
file's binding rather than runtime.sx's.
Move the E39 worker stub registration out of `lib/hyperscript/parser.sx`
into `lib/hyperscript/plugins/worker.sx`. The plugin calls
`(hs-register-feature! "worker" ...)` at file load time. Behaviour is
identical — `worker MyWorker ...` raises the same helpful "plugin not
installed" error, just routed through the registry from a separate
file. The pre-existing `behavioral` test for the helpful error
("raises a helpful error when the worker plugin is not installed")
still passes via the new path.
Wire-up:
- OCaml `bin/run_tests.ml`: load `plugins/worker.sx` and
`plugins/prolog.sx` after `runtime.sx`, before `integration.sx`.
- JS `tests/hs-kernel-eval.js`: extend HS module list with
`hs-worker` / `hs-prolog`; add `HS_PLUGINS` resolver branch so the
`hs-` prefix maps to `lib/hyperscript/plugins/`.
- WASM `hosts/ocaml/browser/bundle.sh`: copy plugin files into
`dist/sx/hs-<name>.sx`.
- WASM `hosts/ocaml/browser/compile-modules.js`: add `hs-worker` /
`hs-prolog` to `FILES`, `HS_DEPS`, and `HS_LAZY` so the lazy loader
resolves them on first reference.
- Worker plugin carries a sentinel `(define hs-worker-loaded? true)`
so `extractDefines` indexes it in the module manifest (the lazy
loader skips files with no defines).
Mirrors `shared/static/wasm/sx/hs-{parser,runtime}.sx` are byte-identical
to source; new mirrors `hs-{prolog,worker}.sx` written via sx_write_file.
OCaml: 4545 passed, 1339 failed — matches baseline.
JS: 2591 passed, 2465 failed — matches baseline.
Smoke tests: `(prolog ...)` raises "prolog hook not installed" cleanly,
`(hs-set-prolog-hook! ...)` then `(prolog ...)` returns the hook result,
`(hs-compile "worker MyWorker def noop() end end")` raises the worker
stub error via the registry path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `_hs-command-registry` and `_hs-converter-registry` dicts plus
`hs-register-command!` / `hs-register-converter!` to
`lib/hyperscript/compiler.sx`. Inside `hs-to-sx`, before the existing
`cond` over head symbols, check both registries: an `as` form whose
type-name has a registered converter dispatches to that converter; any
list head whose name (`(str head)`) is in the command registry
dispatches to that compile-fn. On registry miss, the original ~180
hardcoded branches handle the form.
Each registered fn receives a ctx dict (built per call) exposing
`:hs-to-sx` for recursion plus the AST fields the dispatch needs
(`:ast :head` for commands; `:ast :value-ast :type-name` for
converters). Mirrors Step 9's parser feature registry shape.
Smoke tested: register custom command + converter, both dispatch;
built-in `(as x \"Int\")` still produces `(hs-coerce x \"Int\")`.
Mirror `shared/static/wasm/sx/hs-compiler.sx` copied byte-identical.
OCaml: 4545/1339, JS: 2591/2465 — both match baseline, zero regressions.
Second piece of plans/designs/hs-plugin-system.md (Step 11 next).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `_hs-feature-registry` dict and `hs-register-feature!` to
`lib/hyperscript/parser.sx`. Replace `parse-feat`'s hardcoded `cond`
on feature names with a registry lookup; the paren-open and
default-expression branches remain as fallthroughs.
Each parse-fn receives a `ctx` dict (built per call by `parse-feat-ctx`)
exposing parser internals (`:adv!`, `:tp-val`, `:tp-type`, `:at-end?`,
`:parse-cmd-list`, `:parse-expr`) and the per-feature handlers
(`:parse-on-feat` … `:parse-socket-feat`). All nine builtins
(`on`, `init`, `def`, `behavior`, `live`, `when`, `worker`, `bind`,
`socket`) are registered at file load time, so plugins added later via
`hs-register-feature!` persist across `hs-parse` calls.
Worker stub still raises identically. Mirror `shared/static/wasm/sx/hs-parser.sx`
copied byte-identical. OCaml: 4545/1339, JS: 2591/2465 — both match
baseline, zero regressions.
First piece of plans/designs/hs-plugin-system.md (Steps 10/11 follow).
Emit a warning when a `match` expression on an ADT value misses one
or more constructors and lacks an `else`/`_` clause. Behaviour is
non-fatal — the match still runs, the warning goes to stderr.
- spec/evaluator.sx: helpers `match-clause-is-else?`, `match-clause-ctor-name`,
`match-warn-non-exhaustive`, `match-check-exhaustiveness`. The latter
reads the `*adt-registry*` (already populated by `define-type`),
collects constructor patterns from clauses, and dedupes via an
`*adt-warned*` env-bound dict so each (type, missing-set) warns once.
Wired into `step-sf-match` via a `do` block before clause dispatch.
- hosts/javascript/platform.py: `host-warn` primitive (`console.warn`)
+ matching `hostWarn` js-id helper so the JS-transpiled spec code
can call it directly. Spec code reaches JS via `sx_build target=js`.
- hosts/ocaml/lib/sx_runtime.ml + sx_primitives.ml: `host-warn` runtime
helper (`prerr_endline`) and registered primitive.
- hosts/ocaml/lib/sx_ref.ml: HAND-PATCHED. `step_sf_match` now calls
a hand-written `match_check_exhaustiveness` that handles both
`AdtValue` and back-compat dict-shape ADT values. The OCaml side
is *not* retranspiled because regenerating sx_ref.ml drops
several preamble fixes (seq_to_list, string->symbol mangling,
empty-dict literal bug). Future retranspile must reapply this patch.
- spec/tests/test-adt.sx: 5 new tests covering exhaustive,
non-exhaustive (warning is non-fatal), `else` suppression,
partial coverage with one missing constructor, and `_` wildcard
suppression. Tests assert return values only — warnings go to
stderr and are not captured.
Warning format: `[sx] match: non-exhaustive — TypeName: missing Ctor1, Ctor2`
Both hosts emit identical messages.
Tests: OCaml 4540 → 4545 (+5), JS 2586 → 2591 (+5). Zero regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the ADT test suite with nested-pattern coverage. The spec-level
match-pattern function in spec/evaluator.sx already recurses through
constructor sub-patterns via the dict-shape shim ((get value :_adt|
:_ctor|:_fields)), and already handles _ wildcards, quoted literals,
and bare-symbol variable bindings. Step 5+6 added the AdtValue native
type with the same dict-key access surface, so no host changes are
needed for nesting.
Added 8 new deftests covering:
- nested constructor sanity (Just x / Nothing)
- nested constructor binds inner fields ((Just (Pair a b)) -> a+b)
- nested wildcard ((Just _) -> "yes")
- nested literal equality ((Just 42) literal vs (else) var)
- nested literal-vs-var fall-through (literal fails, var binds)
- deeply nested constructors (W1(W2(L3 n)) -> n)
- mixed bind+wildcard ((BoxM (PairM x _)) -> x)
- nested ctor fail-through (WX (LeftX) vs WX (RightX))
Tests: OCaml 4532 -> 4540 (+8), JS 2578 -> 2586 (+8). Zero regressions
on either host (failures unchanged at 1339 / 2465 baselines).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror of OCaml Step 5 to the JavaScript host. Native ADT representation
for define-type instances, with the same dict-shaped shim approach so
spec-level match-pattern code in evaluator.sx works without changes.
- platform.py typeOf: recognize ._adtv tag, return ._type (so
(type-of (Just 42)) returns "Maybe" not "dict").
- platform.py adds makeAdtValue/isAdtValue helpers and registers
PRIMITIVES["adt?"], "make-adt-value", "adt-value?".
- platform.py inspect: format AdtValue as "(Ctor f1 f2 ...)" and
register as a primitive (was missing entirely on JS).
- fixups_js: hand-written define-type override that constructs
AdtValue via makeAdtValue, with arity check, type/ctor predicates,
and field accessors. Re-registered via registerSpecialForm so the
CEK dispatch routes through it.
- dict? unchanged: AdtValue still passes (no _adtv exclusion) so
the existing (and (dict? v) (get v :_adt) ...) checks in spec
predicates keep working.
Tests: 2578 pass (was 2575), zero regressions. All 43 ADT tests
pass on the JS host (was 40, the 3 new Step 5 tests for type-of /
adt? / inspect are now green).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
