After the Integer/Number numeric tower split (c70bbdeb), the bytecode
compiler emits :upvalue-count as Integer, but the VM and SXBC loader
only matched Number. The fallback `_ -> 0` made the VM skip reading
upvalue descriptors entirely, so the IP advanced into raw upvalue
bytes which were then misread as opcodes.
Symptom: JIT runs of nested closures (curried functions, Y combinator,
component bodies that close over outer let-bindings) produced "VM:
CONST index N out of bounds (pool size M)" with N values like 256,
4096, 5120, 12800, 13056 — all of the form `byte | (opcode << 8)`,
i.e. an upvalue descriptor (lo) followed by the next instruction's
opcode (hi) being read as a u16 operand.
Fix all five sites that decode upvalue-count to also accept Integer:
- hosts/ocaml/lib/sx_vm.ml: OP_CLOSURE handler, trace_run, disassemble
- hosts/ocaml/lib/sx_vm_ref.ml + hosts/ocaml/sx_vm_ref.ml + bootstrap_vm.py:
vm_create_closure preamble (the bootstrap source-of-truth and both
generated copies)
- hosts/ocaml/browser/sx_browser.ml: SXBC loader's parse_kv
Test impact: JIT 3848 -> 4538 passing (+690). No-JIT unchanged at 4550.
The previously-failing curried/Y/higher-order tests in
spec/tests/test-cek-advanced.sx now pass under --jit and serve as
regression coverage.
This fixes a real current bug. The 28-day-old memory file describing
parser-combinator JIT bugs predates the numeric tower split and
described a different problem; with this fix the parser-combinator
broken-name list (`_jit_is_broken_name` in sx_vm.ml) is no longer
strictly required for correctness, but keeping it avoids a TIMEOUT
regression in one hyperscript test, so it remains in place.
The bytecode compiler emitted OP_CALL_PRIM (52) for every primitive call, even
for arithmetic and comparison hot-paths. The VM had specialized opcodes
(OP_ADD, OP_SUB, OP_EQ, etc.) defined but unused.
- lib/compiler.sx (compile-call): emit specialized 1-byte opcode when the
primitive name + arity matches one of {+, -, *, /, =, <, >, cons, not, len,
first, rest}. Falls back to CALL_PRIM otherwise. fib bytecode: 50 → 38 bytes.
- hosts/ocaml/lib/sx_compiler.ml: mirror change in the auto-generated OCaml
compiler so SXBC export from mcp_tree uses the same emission.
- hosts/ocaml/lib/sx_vm.ml: extend OP_ADD/SUB/MUL/DIV to handle Integer+Integer
(not just Number+Number). Inline OP_EQ via Sx_runtime._fast_eq. Inline
OP_LT/GT mixed-numeric comparisons. Avoids Hashtbl lookup on the fallback
path for the common integer cases that dominate tight loops.
- hosts/ocaml/bin/bench_vm.ml: VM-only benchmark — loads compiler.sx via CEK,
JIT-compiles each fn, measures Sx_vm.call_closure throughput.
Median improvements (best of 3 runs of 9-min, bench_vm.exe):
fib(22) 107.87ms → 33.13ms -69%
loop(200000) 429.64ms → 161.16ms -62%
sum-to(50000) 72.85ms → 36.74ms -50%
count-lt(20000) 28.44ms → 17.58ms -38%
count-eq(20000) 37.23ms → 15.46ms -58%
Tests: 4550/4550 OCaml passing (unchanged). Zero regressions.
Last step in the sx-improvements roadmap — all 14 steps complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Added short aliases make-buffer / buffer? / buffer-append! / buffer->string /
buffer-length on both OCaml and JS hosts, sharing the existing StringBuffer
value type. buffer-append! auto-coerces non-strings via inspect.
Rewrote the OCaml host inspect function to walk a single shared Buffer.t
instead of allocating O(n) intermediate strings via String.concat at every
recursion level. inspect underlies sx-serialize and error-path formatting,
so this benefits the tightest serialization paths.
Median improvements (bin/bench_inspect.exe, best-of-3 of 9-run min):
tree-d8 (75KB): 5.31ms -> 1.30ms (-76%)
tree-d10 (679KB): 81.89ms -> 16.02ms (-80%)
dict-1000: 0.80ms -> 0.31ms (-61%)
list-2000: 0.74ms -> 0.33ms (-55%)
Tests: OCaml 4545 -> 4550. JS 2591 -> 2596. Zero regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CEK frames were already records (cek_frame in sx_types.ml), so the actual
hot-path bottleneck was prim_call "=" [...] in step_continue/step_eval
dispatch: each step did a Hashtbl lookup + 2x list cons + pattern match
just to compare frame-type strings.
Added a short-circuit fast path in prim_call (sx_runtime.ml) for the
hot operators: =, <, >, <=, >=, empty?, first, rest, len. These bypass
the primitives Hashtbl entirely and dispatch directly on value shape.
Inlined _fast_eq for scalar/string equality, which dominates frame-type
dispatch comparisons.
Added bin/bench_cek.exe with five tight-loop benchmarks (fib, loop,
map, reduce, let-heavy). Median of 7 runs:
fib(18) 2789ms -> 941ms (-66%)
loop(5000) 2018ms -> 620ms (-69%)
map sq xs(1000) 108ms -> 48ms (-56%)
reduce + ys(2000) 72ms -> 10ms (-86%)
let-heavy(2000) 491ms -> 271ms (-45%)
Tests: 4545/4545 passing baseline preserved (1339 pre-existing failures
unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move `hs-prolog-hook` / `hs-set-prolog-hook!` / `prolog` out of
`lib/hyperscript/runtime.sx` into a self-contained plugin file at
`lib/hyperscript/plugins/prolog.sx`. The API surface is preserved —
`lib/prolog/hs-bridge.sx::pl-install-hs-hook!` still calls
`hs-set-prolog-hook!` exactly as before, just resolved to the plugin
file's binding rather than runtime.sx's.
Move the E39 worker stub registration out of `lib/hyperscript/parser.sx`
into `lib/hyperscript/plugins/worker.sx`. The plugin calls
`(hs-register-feature! "worker" ...)` at file load time. Behaviour is
identical — `worker MyWorker ...` raises the same helpful "plugin not
installed" error, just routed through the registry from a separate
file. The pre-existing `behavioral` test for the helpful error
("raises a helpful error when the worker plugin is not installed")
still passes via the new path.
Wire-up:
- OCaml `bin/run_tests.ml`: load `plugins/worker.sx` and
`plugins/prolog.sx` after `runtime.sx`, before `integration.sx`.
- JS `tests/hs-kernel-eval.js`: extend HS module list with
`hs-worker` / `hs-prolog`; add `HS_PLUGINS` resolver branch so the
`hs-` prefix maps to `lib/hyperscript/plugins/`.
- WASM `hosts/ocaml/browser/bundle.sh`: copy plugin files into
`dist/sx/hs-<name>.sx`.
- WASM `hosts/ocaml/browser/compile-modules.js`: add `hs-worker` /
`hs-prolog` to `FILES`, `HS_DEPS`, and `HS_LAZY` so the lazy loader
resolves them on first reference.
- Worker plugin carries a sentinel `(define hs-worker-loaded? true)`
so `extractDefines` indexes it in the module manifest (the lazy
loader skips files with no defines).
Mirrors `shared/static/wasm/sx/hs-{parser,runtime}.sx` are byte-identical
to source; new mirrors `hs-{prolog,worker}.sx` written via sx_write_file.
OCaml: 4545 passed, 1339 failed — matches baseline.
JS: 2591 passed, 2465 failed — matches baseline.
Smoke tests: `(prolog ...)` raises "prolog hook not installed" cleanly,
`(hs-set-prolog-hook! ...)` then `(prolog ...)` returns the hook result,
`(hs-compile "worker MyWorker def noop() end end")` raises the worker
stub error via the registry path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `_hs-command-registry` and `_hs-converter-registry` dicts plus
`hs-register-command!` / `hs-register-converter!` to
`lib/hyperscript/compiler.sx`. Inside `hs-to-sx`, before the existing
`cond` over head symbols, check both registries: an `as` form whose
type-name has a registered converter dispatches to that converter; any
list head whose name (`(str head)`) is in the command registry
dispatches to that compile-fn. On registry miss, the original ~180
hardcoded branches handle the form.
Each registered fn receives a ctx dict (built per call) exposing
`:hs-to-sx` for recursion plus the AST fields the dispatch needs
(`:ast :head` for commands; `:ast :value-ast :type-name` for
converters). Mirrors Step 9's parser feature registry shape.
Smoke tested: register custom command + converter, both dispatch;
built-in `(as x \"Int\")` still produces `(hs-coerce x \"Int\")`.
Mirror `shared/static/wasm/sx/hs-compiler.sx` copied byte-identical.
OCaml: 4545/1339, JS: 2591/2465 — both match baseline, zero regressions.
Second piece of plans/designs/hs-plugin-system.md (Step 11 next).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `_hs-feature-registry` dict and `hs-register-feature!` to
`lib/hyperscript/parser.sx`. Replace `parse-feat`'s hardcoded `cond`
on feature names with a registry lookup; the paren-open and
default-expression branches remain as fallthroughs.
Each parse-fn receives a `ctx` dict (built per call by `parse-feat-ctx`)
exposing parser internals (`:adv!`, `:tp-val`, `:tp-type`, `:at-end?`,
`:parse-cmd-list`, `:parse-expr`) and the per-feature handlers
(`:parse-on-feat` … `:parse-socket-feat`). All nine builtins
(`on`, `init`, `def`, `behavior`, `live`, `when`, `worker`, `bind`,
`socket`) are registered at file load time, so plugins added later via
`hs-register-feature!` persist across `hs-parse` calls.
Worker stub still raises identically. Mirror `shared/static/wasm/sx/hs-parser.sx`
copied byte-identical. OCaml: 4545/1339, JS: 2591/2465 — both match
baseline, zero regressions.
First piece of plans/designs/hs-plugin-system.md (Steps 10/11 follow).
Emit a warning when a `match` expression on an ADT value misses one
or more constructors and lacks an `else`/`_` clause. Behaviour is
non-fatal — the match still runs, the warning goes to stderr.
- spec/evaluator.sx: helpers `match-clause-is-else?`, `match-clause-ctor-name`,
`match-warn-non-exhaustive`, `match-check-exhaustiveness`. The latter
reads the `*adt-registry*` (already populated by `define-type`),
collects constructor patterns from clauses, and dedupes via an
`*adt-warned*` env-bound dict so each (type, missing-set) warns once.
Wired into `step-sf-match` via a `do` block before clause dispatch.
- hosts/javascript/platform.py: `host-warn` primitive (`console.warn`)
+ matching `hostWarn` js-id helper so the JS-transpiled spec code
can call it directly. Spec code reaches JS via `sx_build target=js`.
- hosts/ocaml/lib/sx_runtime.ml + sx_primitives.ml: `host-warn` runtime
helper (`prerr_endline`) and registered primitive.
- hosts/ocaml/lib/sx_ref.ml: HAND-PATCHED. `step_sf_match` now calls
a hand-written `match_check_exhaustiveness` that handles both
`AdtValue` and back-compat dict-shape ADT values. The OCaml side
is *not* retranspiled because regenerating sx_ref.ml drops
several preamble fixes (seq_to_list, string->symbol mangling,
empty-dict literal bug). Future retranspile must reapply this patch.
- spec/tests/test-adt.sx: 5 new tests covering exhaustive,
non-exhaustive (warning is non-fatal), `else` suppression,
partial coverage with one missing constructor, and `_` wildcard
suppression. Tests assert return values only — warnings go to
stderr and are not captured.
Warning format: `[sx] match: non-exhaustive — TypeName: missing Ctor1, Ctor2`
Both hosts emit identical messages.
Tests: OCaml 4540 → 4545 (+5), JS 2586 → 2591 (+5). Zero regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the ADT test suite with nested-pattern coverage. The spec-level
match-pattern function in spec/evaluator.sx already recurses through
constructor sub-patterns via the dict-shape shim ((get value :_adt|
:_ctor|:_fields)), and already handles _ wildcards, quoted literals,
and bare-symbol variable bindings. Step 5+6 added the AdtValue native
type with the same dict-key access surface, so no host changes are
needed for nesting.
Added 8 new deftests covering:
- nested constructor sanity (Just x / Nothing)
- nested constructor binds inner fields ((Just (Pair a b)) -> a+b)
- nested wildcard ((Just _) -> "yes")
- nested literal equality ((Just 42) literal vs (else) var)
- nested literal-vs-var fall-through (literal fails, var binds)
- deeply nested constructors (W1(W2(L3 n)) -> n)
- mixed bind+wildcard ((BoxM (PairM x _)) -> x)
- nested ctor fail-through (WX (LeftX) vs WX (RightX))
Tests: OCaml 4532 -> 4540 (+8), JS 2578 -> 2586 (+8). Zero regressions
on either host (failures unchanged at 1339 / 2465 baselines).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror of OCaml Step 5 to the JavaScript host. Native ADT representation
for define-type instances, with the same dict-shaped shim approach so
spec-level match-pattern code in evaluator.sx works without changes.
- platform.py typeOf: recognize ._adtv tag, return ._type (so
(type-of (Just 42)) returns "Maybe" not "dict").
- platform.py adds makeAdtValue/isAdtValue helpers and registers
PRIMITIVES["adt?"], "make-adt-value", "adt-value?".
- platform.py inspect: format AdtValue as "(Ctor f1 f2 ...)" and
register as a primitive (was missing entirely on JS).
- fixups_js: hand-written define-type override that constructs
AdtValue via makeAdtValue, with arity check, type/ctor predicates,
and field accessors. Re-registered via registerSpecialForm so the
CEK dispatch routes through it.
- dict? unchanged: AdtValue still passes (no _adtv exclusion) so
the existing (and (dict? v) (get v :_adt) ...) checks in spec
predicates keep working.
Tests: 2578 pass (was 2575), zero regressions. All 43 ADT tests
pass on the JS host (was 40, the 3 new Step 5 tests for type-of /
adt? / inspect are now green).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lib/guest/prefix.sx defines a single (defmacro prefix-rename PREFIX ENTRIES)
form that takes a prefix string and a quoted list of entries. Each entry
is either a bare symbol (same-name alias: cl-foo = foo) or a 2-element
list (alias target) for renames (cl-mod = modulo).
Ported lib/common-lisp/runtime.sx: 47 hand-written (define cl-X Y) lines
across 13 contiguous groups now collapse into prefix-rename calls. Loaded
lib/guest/prefix.sx in the conformance preamble so the macro is available
when runtime.sx is parsed.
Verification: cl scoreboard 518/518, up from a stale baseline of 309/309
— Phase 2 (evaluator, +182) and Phase 6 (stdlib, +27) had under-counted
historical results, not affected by this change. No regressions; baseline
updated to reflect true counts.
PARTIAL — pending second consumer. lua/runtime.sx (the brief's specified
second consumer) has zero pure same-name aliases — every lua- definition
wraps custom logic. Step left [partial — pending lua] until a consumer
fits, or the second-consumer choice is revisited (js/runtime.sx has 2
candidates: isFinite/isNaN).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Native algebraic data type representation in the OCaml SX evaluator.
Replaces the dict-based shim that simulated ADT values via tagged dicts.
- sx_types.ml: add AdtValue variant + adt_value record (av_type, av_ctor,
av_fields). type_of returns the type name (e.g. "Maybe"); inspect renders
as a constructor call (e.g. "(Just 42)" or "(Nothing)").
- sx_runtime.ml: get_val handles AdtValue with :_adt/:_type/:_ctor/:_fields
keys for back-compat with spec-level match-pattern code.
- sx_primitives.ml: dict? returns true for AdtValue (so existing match
dispatch keeps working); new adt? predicate distinguishes ADT values.
- sx_ref.ml: sf_define_type now constructs AdtValue instead of Dict.
Predicates (Name?, Ctor?) and accessors (Ctor-field) match on AdtValue
with proper type/ctor name and field index checks.
- spec/tests/test-adt.sx: 3 new tests covering type-of, adt?, and inspect.
Tests: 4532 passed (was 4529 + 3 new), 1339 failed (unchanged baseline).
All 43 ADT tests pass on the native representation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extracted the duplicated conformance plumbing into a single driver:
- lib/guest/conformance.sx — two helper fns that emit (gc-result NAME P F T)
lines for the bash side to grep: gc-dict-result for runners returning
a {:passed :failed :total} dict, and gc-counters-result for guests that
bump a global pass/fail counter from a test file load.
- lib/guest/conformance.sh — config-driven bash driver. Sources a per-lang
conf, locates sx_server, runs sx_server in either single-session "dict"
mode (one preload + many suite evals) or per-suite "counters" mode
(fresh sx_server per suite, with shared preloads). Aggregates and writes
scoreboard.{json,md} via per-lang emit_scoreboard_* functions.
- Ported lib/prolog/conformance.sh and lib/haskell/conformance.sh down to
one-line wrappers that exec the shared driver against their .conf file.
Verification:
- Prolog: 590/590 — diff vs baseline is timestamp-only.
- Haskell: 156/156 — significantly higher than the 0/18 in baseline. The
old conformance.sh was buggy (its `(ok-len 3 ...)` grep never matched,
defaulting every program to 0 pass / 1 fail). Updated baseline to the
true count; no actual test regressed. Plan baseline cell updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verified all 4 hs-upstream-core/sourceInfo tests now pass without any parser
changes. The parser already had `link-next-cmds` (sets `:next` on each command
in a CommandList when hs-span-mode is true) and `:true-branch` extraction in
`parse-cmd` for if statements; Step 3's `:end`/`:line` token fields were the
only missing pieces.
Probed via sx_eval against the parser:
(hs-line-at "if true\n log 'it was true'\n log 'it was true'"
(list :true-branch :next))
returns " log 'it was true'" — matches the expected upstream behaviour.
Test runner output:
PASS: hs-upstream-core/sourceInfo > debug
PASS: hs-upstream-core/sourceInfo > get line works for statements
PASS: hs-upstream-core/sourceInfo > get source works for expressions
PASS: hs-upstream-core/sourceInfo > get source works for statements
Extend hs-make-token to (type value pos &rest extras) producing dicts
{:pos :end :line :value :type}. End defaults to pos+len(value); line
defaults to 1. Both tokenize loops now track current-line via newline
counting in advance!. hs-emit! and t-emit! pass the right end and
start-line to the constructor; redundant dict-set! after construction
removed.
Mirror copied to shared/static/wasm/sx/hs-tokenizer.sx (byte-identical).
Verify: (hs-make-token "NUMBER" "1" 0) returns
{:pos 0 :end 1 :line 1 :value "1" :type "NUMBER"}.
OCaml suite: 4529 pass, 1339 pre-existing failures (baseline). All
4/4 hs-upstream-core/sourceInfo tests now pass (was 2/4 — closes E38).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In `resume_vm`'s `restore_reuse`, the saved sp captured by
`call_closure_reuse` was ignored when restoring the caller frame after the
async callback finished. The suspended callee's locals/temps stayed on the
value stack above saved_sp, so subsequent LOCAL_GET/SET in the caller
frame (e.g. letrec sibling bindings waiting on the suspending call) read
stale callee data instead of their own slots. Sibling bindings appeared
nil after a perform/resume cycle on the JIT path used by the WASM
browser kernel.
Fix: after popping the callback result and restoring saved_frames, reset
`vm.sp <- saved_sp` (when sp is above), then push the callback result.
Mirrors the OP_RETURN+sp-reset discipline that sync `call_closure_reuse`
already follows.
New tests in `spec/tests/test-letrec-resume.sx` cover single binding,
sibling bindings, mutual recursion siblings, and nested letrec —
all four pass. Full OCaml run_tests: 4529/5868 (was 4525/5864), zero
regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In `call_closure_reuse`, the success path used a bare `pop vm` that relied on
OP_RETURN having left the stack at exactly `saved_sp + 1`. When the callee
returns a closure (or hits the bytecode-exhausted fallback path), `vm.sp` can
end up inconsistent with the parent frame's expected layout, corrupting
intermediate values such as parser combinator state in `parse-bind`/`many`/
`seq`.
Fix: read the result at the expected slot, then explicitly reset
`vm.sp <- saved_sp` before returning so the parent frame sees a clean stack
regardless of what the callee left behind.
OCaml run_tests baseline: 4525/5864 unchanged. WASM kernel tests: 24/29
unchanged. No regressions.
