Adds Sx_vm.bytecode_uses_extension_opcodes — an operand-aware
bytecode scanner that walks past CONST u16, CALL_PRIM u16+u8, and
CLOSURE u16+dynamic upvalue descriptors so operand bytes that happen
to be ≥200 don't false-positive as extension opcodes.
jit_compile_lambda calls the scanner on the inner closure's bytecode.
On hit it returns None — the lambda then runs through CEK
interpretation. The VM's dispatch fallthrough still routes the
extension opcodes themselves through the registry; this change just
prevents the JIT from claiming code it has no plan for.
Tests: 7 new foundation cases — pure core eligible, head/middle/
post-CLOSURE detection, CONST + CALL_PRIM + CLOSURE-descriptor false-
positive avoidance. +7 pass vs Phase D baseline, no regressions
across 11 conformance suites.
Loop complete: acceptance criteria 1-4 met. Hand-off to the Erlang
loop — lib/erlang/vm/dispatcher.sx's Phase 9b stub can now be
replaced with a real hosts/ocaml/lib/extensions/erlang.ml consumer.
lib/extensions/ becomes the new home for VM extensions, wired in via
(include_subdirs unqualified). README documents the registration
pattern, opcode-ID range conventions (200-209 guest_vm, 210-219
inline test, 220-229 test_ext, 230-247 ports), and naming rules.
extensions/test_ext.ml is the canonical worked example — two
operand-less opcodes (220 push 42, 221 double TOS) carrying a per-
extension state slot (TestExtState invocation counter). Test_ext.register
called from run_tests.ml at the start of the Phase D suite, on top of
the inline test_reg from earlier suites (disjoint opcode IDs).
Sx_vm.opcode_name now consults extension_opcode_name_ref (forward ref
in the same style as extension_dispatch_ref), so disassemble shows
extension opcodes by name instead of UNKNOWN_n. Registry maintains
name_of_id_table and installs the lookup at module init.
Tests: 5 new foundation cases — primitive resolves test_ext name,
end-to-end bytecode (push + double + return → 84), disassemble shows
"test_ext.OP_TEST_PUSH_42" / "test_ext.OP_TEST_DOUBLE_TOS",
unregistered ext opcodes still fall back to UNKNOWN_n, invocation
counter records the two dispatches. +5 pass vs Phase C baseline, no
regressions across 11 conformance suites.
Registers extension-opcode-id from sx_vm_extensions.ml module init.
Lives downstream of both sx_primitives and sx_vm to avoid a build
cycle. Accepts a string or symbol; returns Integer id when the opcode
is registered, Nil otherwise.
Compilers (lib/compiler.sx) call this to emit extension opcodes by
name. Returning Nil rather than failing on unknown names lets a port's
optimization opt in per-build — missing extensions degrade to slower
correct execution.
Tests: 5 new foundation cases — registered lookup, unknown → nil,
symbol arg, zero-arg + integer-arg rejection. +5 pass vs Phase B
baseline, no regressions across 11 conformance suites.
sx_vm_extension.ml: handler type, extensible extension_state variant,
EXTENSION first-class module signature.
sx_vm_extensions.ml: register / dispatch / id_of_name /
state_of_extension. install_dispatch () runs at module init,
swapping Phase A's stub for the real registry. Rejects out-of-range
opcode IDs (must be 200-247), duplicate IDs, duplicate names, and
duplicate extension names.
Tests: 9 new foundation cases — lookup hits/misses, end-to-end VM
dispatch including opcode composition, all four rejection paths.
+9 pass vs Phase A baseline, no regressions across 11 conformance
suites.
Adds Invalid_opcode of int exception and extension_dispatch_ref forward
ref (default raises Invalid_opcode op), plus the |op when op >= 200 arm
before the catch-all in the bytecode dispatch loop. Partition comment
documents 1-199 core / 200-247 extensions / 248-255 reserved.
Phase B will install the real registry's dispatch into the ref at module
init, replacing this stub.
Tests: 4 new foundation cases (Invalid_opcode for 200/224/247, Eval_error
for 199 to pin the threshold). +4 pass vs baseline, no regressions.
Adds Sx_vm.bytecode_uses_extension_opcodes — an operand-aware
bytecode scanner that walks past CONST u16, CALL_PRIM u16+u8, and
CLOSURE u16+dynamic upvalue descriptors so operand bytes that happen
to be ≥200 don't false-positive as extension opcodes.
jit_compile_lambda calls the scanner on the inner closure's bytecode.
On hit it returns None — the lambda then runs through CEK
interpretation. The VM's dispatch fallthrough still routes the
extension opcodes themselves through the registry; this change just
prevents the JIT from claiming code it has no plan for.
Tests: 7 new foundation cases — pure core eligible, head/middle/
post-CLOSURE detection, CONST + CALL_PRIM + CLOSURE-descriptor false-
positive avoidance. +7 pass vs Phase D baseline, no regressions
across 11 conformance suites.
Loop complete: acceptance criteria 1-4 met. Hand-off to the Erlang
loop — lib/erlang/vm/dispatcher.sx's Phase 9b stub can now be
replaced with a real hosts/ocaml/lib/extensions/erlang.ml consumer.
lib/extensions/ becomes the new home for VM extensions, wired in via
(include_subdirs unqualified). README documents the registration
pattern, opcode-ID range conventions (200-209 guest_vm, 210-219
inline test, 220-229 test_ext, 230-247 ports), and naming rules.
extensions/test_ext.ml is the canonical worked example — two
operand-less opcodes (220 push 42, 221 double TOS) carrying a per-
extension state slot (TestExtState invocation counter). Test_ext.register
called from run_tests.ml at the start of the Phase D suite, on top of
the inline test_reg from earlier suites (disjoint opcode IDs).
Sx_vm.opcode_name now consults extension_opcode_name_ref (forward ref
in the same style as extension_dispatch_ref), so disassemble shows
extension opcodes by name instead of UNKNOWN_n. Registry maintains
name_of_id_table and installs the lookup at module init.
Tests: 5 new foundation cases — primitive resolves test_ext name,
end-to-end bytecode (push + double + return → 84), disassemble shows
"test_ext.OP_TEST_PUSH_42" / "test_ext.OP_TEST_DOUBLE_TOS",
unregistered ext opcodes still fall back to UNKNOWN_n, invocation
counter records the two dispatches. +5 pass vs Phase C baseline, no
regressions across 11 conformance suites.
Registers extension-opcode-id from sx_vm_extensions.ml module init.
Lives downstream of both sx_primitives and sx_vm to avoid a build
cycle. Accepts a string or symbol; returns Integer id when the opcode
is registered, Nil otherwise.
Compilers (lib/compiler.sx) call this to emit extension opcodes by
name. Returning Nil rather than failing on unknown names lets a port's
optimization opt in per-build — missing extensions degrade to slower
correct execution.
Tests: 5 new foundation cases — registered lookup, unknown → nil,
symbol arg, zero-arg + integer-arg rejection. +5 pass vs Phase B
baseline, no regressions across 11 conformance suites.
sx_vm_extension.ml: handler type, extensible extension_state variant,
EXTENSION first-class module signature.
sx_vm_extensions.ml: register / dispatch / id_of_name /
state_of_extension. install_dispatch () runs at module init,
swapping Phase A's stub for the real registry. Rejects out-of-range
opcode IDs (must be 200-247), duplicate IDs, duplicate names, and
duplicate extension names.
Tests: 9 new foundation cases — lookup hits/misses, end-to-end VM
dispatch including opcode composition, all four rejection paths.
+9 pass vs Phase A baseline, no regressions across 11 conformance
suites.
Adds Invalid_opcode of int exception and extension_dispatch_ref forward
ref (default raises Invalid_opcode op), plus the |op when op >= 200 arm
before the catch-all in the bytecode dispatch loop. Partition comment
documents 1-199 core / 200-247 extensions / 248-255 reserved.
Phase B will install the real registry's dispatch into the ref at module
init, replacing this stub.
Tests: 4 new foundation cases (Invalid_opcode for 200/224/247, Eval_error
for 199 to pin the threshold). +4 pass vs baseline, no regressions.
sx_types.ml:
- Add l_uid field on lambda (unique identity for cache tracking)
- Add lambda_uid_counter + next_lambda_uid () minted on construction
- Add jit_budget (default 5000) and jit_evicted_count counter
- Add jit_cache_queue : (int * value) Queue.t — FIFO of compiled lambdas
- jit_cache_size () helper for stats
sx_vm.ml:
- On successful JIT compile, push (uid, Lambda l) onto jit_cache_queue
- While queue length exceeds jit_budget, pop head (oldest entry) and
clear that lambda's l_compiled slot — evicted entries fall through
to cek_call_or_suspend on next call (correct, just slower)
- Guard JIT trigger by !jit_budget > 0 (budget=0 disables JIT entirely)
sx_primitives.ml:
Phase 2:
- jit-set-budget! N — change cache budget at runtime
- jit-stats includes budget, cache-size, evicted
Phase 3:
- jit-reset-cache! — clear all compiled VmClosures (hot paths re-JIT
on next threshold crossing)
- jit-reset-counters! also resets evicted counter
run_tests.ml:
- Update test-fixture lambda construction to include l_uid
Effect: cache size bounded regardless of input pattern. The HS test harness
compiles ~3000 distinct one-shot lambdas, but tiered compilation (Phase 1)
keeps most below threshold so they never enter the cache. Steady-state count
stays in single digits for typical workloads. When a misbehaving caller
saturates the cache (eval-hs in a tight loop, REPL-style host), LRU
eviction caps memory at jit_budget compiled closures × ~1KB each.
Verification: 4771 passed, 1111 failed in run_tests — identical to
pre-Phase-2 baseline. No regressions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OCaml kernel changes:
sx_types.ml:
- Add l_call_count : int field to lambda type — counts how many times
a named lambda has been invoked through the VM dispatch path.
- Add module-level refs jit_threshold (default 4), jit_compiled_count,
jit_skipped_count, jit_threshold_skipped_count for stats.
Refs live here (not sx_vm) so sx_primitives can read them without
creating a sx_primitives → sx_vm dependency cycle.
sx_vm.ml:
- In the Lambda case of cek_call_or_suspend, before triggering the JIT,
increment l.l_call_count. Only call jit_compile_ref if count >= the
runtime-tunable threshold. Below threshold, fall through to the
existing cek_call_or_suspend path (interpreter-style).
sx_primitives.ml:
- Register jit-stats — returns dict {threshold, compiled, compile-failed,
below-threshold}.
- Register jit-set-threshold! N — change threshold at runtime.
- Register jit-reset-counters! — zero the stats counters.
bin/run_tests.ml:
- Add l_call_count = 0 to the test-fixture lambda construction.
Effect: lambdas only get JIT-compiled after the 4th invocation. One-shot
lambdas (test harness wrappers, eval-hs throwaways, REPL inputs) never enter
the JIT cache, eliminating the cumulative slowdown that the batched runner
currently works around. Hot paths (component renders, event handlers) cross
the threshold within a handful of calls and get the full JIT speed.
Phase 2 (LRU eviction) and Phase 3 (jit-reset! / jit-clear-cold!) follow.
Verified: 4771 passed, 1111 failed in OCaml run_tests.exe — identical to
baseline before this change. No regressions; tiered logic is correct.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The bytecode compiler emitted OP_CALL_PRIM (52) for every primitive call, even
for arithmetic and comparison hot-paths. The VM had specialized opcodes
(OP_ADD, OP_SUB, OP_EQ, etc.) defined but unused.
- lib/compiler.sx (compile-call): emit specialized 1-byte opcode when the
primitive name + arity matches one of {+, -, *, /, =, <, >, cons, not, len,
first, rest}. Falls back to CALL_PRIM otherwise. fib bytecode: 50 → 38 bytes.
- hosts/ocaml/lib/sx_compiler.ml: mirror change in the auto-generated OCaml
compiler so SXBC export from mcp_tree uses the same emission.
- hosts/ocaml/lib/sx_vm.ml: extend OP_ADD/SUB/MUL/DIV to handle Integer+Integer
(not just Number+Number). Inline OP_EQ via Sx_runtime._fast_eq. Inline
OP_LT/GT mixed-numeric comparisons. Avoids Hashtbl lookup on the fallback
path for the common integer cases that dominate tight loops.
- hosts/ocaml/bin/bench_vm.ml: VM-only benchmark — loads compiler.sx via CEK,
JIT-compiles each fn, measures Sx_vm.call_closure throughput.
Median improvements (best of 3 runs of 9-min, bench_vm.exe):
fib(22) 107.87ms → 33.13ms -69%
loop(200000) 429.64ms → 161.16ms -62%
sum-to(50000) 72.85ms → 36.74ms -50%
count-lt(20000) 28.44ms → 17.58ms -38%
count-eq(20000) 37.23ms → 15.46ms -58%
Tests: 4550/4550 OCaml passing (unchanged). Zero regressions.
Last step in the sx-improvements roadmap — all 14 steps complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Added short aliases make-buffer / buffer? / buffer-append! / buffer->string /
buffer-length on both OCaml and JS hosts, sharing the existing StringBuffer
value type. buffer-append! auto-coerces non-strings via inspect.
Rewrote the OCaml host inspect function to walk a single shared Buffer.t
instead of allocating O(n) intermediate strings via String.concat at every
recursion level. inspect underlies sx-serialize and error-path formatting,
so this benefits the tightest serialization paths.
Median improvements (bin/bench_inspect.exe, best-of-3 of 9-run min):
tree-d8 (75KB): 5.31ms -> 1.30ms (-76%)
tree-d10 (679KB): 81.89ms -> 16.02ms (-80%)
dict-1000: 0.80ms -> 0.31ms (-61%)
list-2000: 0.74ms -> 0.33ms (-55%)
Tests: OCaml 4545 -> 4550. JS 2591 -> 2596. Zero regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CEK frames were already records (cek_frame in sx_types.ml), so the actual
hot-path bottleneck was prim_call "=" [...] in step_continue/step_eval
dispatch: each step did a Hashtbl lookup + 2x list cons + pattern match
just to compare frame-type strings.
Added a short-circuit fast path in prim_call (sx_runtime.ml) for the
hot operators: =, <, >, <=, >=, empty?, first, rest, len. These bypass
the primitives Hashtbl entirely and dispatch directly on value shape.
Inlined _fast_eq for scalar/string equality, which dominates frame-type
dispatch comparisons.
Added bin/bench_cek.exe with five tight-loop benchmarks (fib, loop,
map, reduce, let-heavy). Median of 7 runs:
fib(18) 2789ms -> 941ms (-66%)
loop(5000) 2018ms -> 620ms (-69%)
map sq xs(1000) 108ms -> 48ms (-56%)
reduce + ys(2000) 72ms -> 10ms (-86%)
let-heavy(2000) 491ms -> 271ms (-45%)
Tests: 4545/4545 passing baseline preserved (1339 pre-existing failures
unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move `hs-prolog-hook` / `hs-set-prolog-hook!` / `prolog` out of
`lib/hyperscript/runtime.sx` into a self-contained plugin file at
`lib/hyperscript/plugins/prolog.sx`. The API surface is preserved —
`lib/prolog/hs-bridge.sx::pl-install-hs-hook!` still calls
`hs-set-prolog-hook!` exactly as before, just resolved to the plugin
file's binding rather than runtime.sx's.
Move the E39 worker stub registration out of `lib/hyperscript/parser.sx`
into `lib/hyperscript/plugins/worker.sx`. The plugin calls
`(hs-register-feature! "worker" ...)` at file load time. Behaviour is
identical — `worker MyWorker ...` raises the same helpful "plugin not
installed" error, just routed through the registry from a separate
file. The pre-existing `behavioral` test for the helpful error
("raises a helpful error when the worker plugin is not installed")
still passes via the new path.
Wire-up:
- OCaml `bin/run_tests.ml`: load `plugins/worker.sx` and
`plugins/prolog.sx` after `runtime.sx`, before `integration.sx`.
- JS `tests/hs-kernel-eval.js`: extend HS module list with
`hs-worker` / `hs-prolog`; add `HS_PLUGINS` resolver branch so the
`hs-` prefix maps to `lib/hyperscript/plugins/`.
- WASM `hosts/ocaml/browser/bundle.sh`: copy plugin files into
`dist/sx/hs-<name>.sx`.
- WASM `hosts/ocaml/browser/compile-modules.js`: add `hs-worker` /
`hs-prolog` to `FILES`, `HS_DEPS`, and `HS_LAZY` so the lazy loader
resolves them on first reference.
- Worker plugin carries a sentinel `(define hs-worker-loaded? true)`
so `extractDefines` indexes it in the module manifest (the lazy
loader skips files with no defines).
Mirrors `shared/static/wasm/sx/hs-{parser,runtime}.sx` are byte-identical
to source; new mirrors `hs-{prolog,worker}.sx` written via sx_write_file.
OCaml: 4545 passed, 1339 failed — matches baseline.
JS: 2591 passed, 2465 failed — matches baseline.
Smoke tests: `(prolog ...)` raises "prolog hook not installed" cleanly,
`(hs-set-prolog-hook! ...)` then `(prolog ...)` returns the hook result,
`(hs-compile "worker MyWorker def noop() end end")` raises the worker
stub error via the registry path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
28 tests, passes on both JS and OCaml.
- spec/stdlib.sx: pure SX format function
- spec/primitives.sx: format primitive declaration
- lib/r7rs.sx: fix number->string to support optional radix arg
- hosts/ocaml: add format-decimal primitive, load stdlib.sx in test runner
- hosts/javascript: load stdlib.sx in test runner
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SxRational type in OCaml (Rational of int * int, stored reduced, denom>0)
and JS (SxRational class with _rational marker). n/d reader syntax in
spec/parser.sx. Arithmetic contagion: int op rational → rational, rational
op float → float. JS keeps int/int → float for CSS backward compatibility.
OCaml as_number + safe_eq extended for cross-type rational equality so
(= 2.5 5/2) → true. 62 tests in test-rationals.sx, all pass.
JS: 2232 passed. OCaml: 4532 passed (+11 vs pre-fix baseline).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Eof sentinel and Port{PortInput/PortOutput} in sx_types.ml. All 15 port
primitives in sx_primitives.ml. type_of/inspect updated. 39/39 port tests
pass (4532 total, +39, zero regressions).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
25 tests pass on both JS and OCaml hosts. Uses dict marker
{:_values true :_list [...]} for 0/2+ values; 1 value passes
through directly. step-sf-define extended to desugar shorthand
(define (name params) body) forms on both hosts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add integer?/float?/exact?/inexact? predicates (Number.isInteger check).
Add truncate/remainder/modulo/random-int/exact->inexact/inexact->exact/parse-number.
inexact->exact uses Math.round (rounds to nearest, matching OCaml).
Fix sx_server.ml epoch/blob/io-response protocol to accept Integer as
well as Number — parser now produces Integer for whole-number literals.
JS: 60 new passing tests (1880→1940). OCaml: 4874/394 baseline unchanged.
Note: 6 tests fail in JS due to platform limitation (JS cannot distinguish
float 2.0 from integer 2).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Shared formatter in sx_types.ml. Small integer-valued floats still print
as plain ints; floats outside safe-int range (|n| >= 1e16) now print as
%.17g (full precision) instead of silently wrapping to negative or 0.
Non-integer values keep %g 6-digit behavior — no existing SX tests regress.
Unblocks Number.MAX_VALUE / Math.pow(2,N) style tests in js-on-sx where
iterative float loops were collapsing to 0 at ~2^63.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- parser `empty` no-target → (ref "me") (was bogus (sym "me"))
- parser `halt` modes distinguish: "all"/"bubbling"/"default" halt execution
(raise hs-return), "the-event"/"the event's" only stop propagation/default.
"'s" now matched as op token, not keyword.
- parser `get` cmd: dispatch + cmd-kw list + parse-get-cmd (parses expr with
optional `as TYPE`). Required for `get result as JSON` in fetch chains.
- compiler empty-target for (local X): emit (set! X (hs-empty-like X)) so
arrays/sets/maps clear the variable, not call DOM empty on the value.
- runtime hs-empty-like: container-of-same-type empty value.
- runtime hs-empty-target!: drop dead FORM branch that was short-circuiting
to innerHTML=""; the querySelectorAll-over-inputs branch now runs.
- runtime hs-halt!: take ev param (was free `event` lookup); raise hs-return
to stop execution unless mode is "the-event".
- runtime hs-reset!: type-aware — FORM → reset, INPUT/TEXTAREA → value/checked
from defaults, SELECT → defaultSelected option.
- runtime hs-open!/hs-close!: toggle `open` attribute on details elements
(not just the prop) so dom-has-attr? assertions work.
- runtime hs-coerce JSON: json-stringify dict/list (was str).
- test-runner mock: host-get on List + "length"/"size" (was only Dict);
dom-set-attr tracks defaultChecked / defaultSelected / defaultValue;
mock_query_all supports comma-separated selector groups.
- generator: emit boolean attrs (checked/selected/etc) even with null value;
drop overcautious "skip HS with bare quotes or embedded HTML" guard so
morph tests (source contains embedded <div>) emit properly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Load sx/sx/geography/cek/ recursively so content/demo/freeze index.sx
pages bind as ~geography/cek/{content,demo,freeze}. Update docs.sx
cek-page dispatch + test-examples cek:content-pages suite to reference
those real names (were stale ~geography/cek/cek-content etc.).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
