rose-ash/plans/sx-review/conformance.md

SX Conformance Review — cross-host agreement + test adequacy

Axis: CONFORMANCE (do the hosts agree, and would the suite catch it if not?). Sibling lanes: core semantics, host implementations. Date: 2026-07-03. All test runs and probes executed this session (bounded with timeout; no shared sx_server touched — probes used freshly-spawned bounded sx_server.exe instances). CONFIRMED = reproduced here; SUSPECTED = static reasoning, probe proposed. Severity S1 critical → S4 minor. Probe artifacts: /tmp/claude-0/-root-rose-ash/9a04ba52-7bf4-476d-99ea-04f84bff1359/scratchpad/{probes,prims}/; raw suite logs /tmp/sx-review/*.log.

---

0. Ground truth: what the hosts actually are (the briefed picture is stale)

• OCaml native kernel (hosts/ocaml, sx_server.exe) — canonical evaluator, essentially green on the corpus.
• OCaml WASM kernel (shared/static/wasm/sx_browser.bc.wasm.js, js_of_ocaml build of the same kernel) — what production browsers actually run (served at lib/host/blog.sx:1910). Served artifact verified identical (md5) to freshly built one — no artifact staleness.
• JS-transpiled bundle (hosts/javascript → shared/static/scripts/sx-browser.js) — legacy bootstrapped evaluator; still built and gated by scripts/sx-build-all.sh; catastrophically red (below). Not referenced by any served page found.
• Python host (hosts/python) — vestigial: test runner deleted in d735e28b; SX_USE_OCAML=1 in every compose file; bootstrap output consumed by nothing. NOT a live evaluator target. BUT shared/sx/parser.py (an independent Python SX reader/serializer) is live in production plumbing — see F-8.
• hosts/native — Cairo pixel-renderer (separate host concept), single smoke test, not a corpus runner.

Suite results (this session)

Runner	Briefed	Actual
JS standard (node hosts/javascript/run_tests.js)	~747 green	2596 pass / 2490 FAIL (5086)
JS full (--full)	~870 green	2453 pass / 3203 FAIL (5656)
Python	~744 green	runner deleted (d735e28b) — zero tests
OCaml (run_tests.exe; no --full flag exists)	~1080 green	5762 pass / 274 FAIL
OCaml WASM kernel	—	corpus never runs on it (F-2)

Rebuilding the JS bundle fresh from today's spec reproduces the JS failures exactly (2596/2490) → red JS is real, not a stale artifact.

---

CONFIRMED findings (most severe first)

F-1 [S1, confidence high] Server and browser disagree on integer arithmetic — silently

The WASM kernel (the artifact every production browser loads) has 32-bit int semantics (js_of_ocaml); native is 63-bit. Same expression, same "canonical kernel", different answers, no error:

Expr	native sx_server	WASM SxKernel (shipped)	legacy JS bundle
(* 99999999 99999999)	9999999800000001	1674919425 (32-bit wrap)	9999999800000000 (float64)
(+ 9007199254740992 1)	9007199254740993	0 (literal truncated mod 2^32)	9007199254740992
(expt 2 62)	-4611686018427387904 (int63 wrap)	0	4611686018427388000 (float)
(* 999999999999 999999999999)	2003762205206896641 (silent int63 wrap)	9.99999999998e+23 (goes float)	9.99999999998e+23

Three hosts, three different answers on the same input. Repro: scratchpad/probes/run_wasm.js probes.txt vs sx_server.exe (eval ...). Any SSR-computed value re-derived client-side (pagination totals, ids, hashes-by-arithmetic, seat counts) can differ. Note native has two hazards of its own: silent int63 wraparound and reduced float print precision ((+ 0.1 0.2) prints 0.3). Fix ownership: hosts lane (int64/boxed ints or explicit overflow policy in the WASM build); THIS lane's ask: a numeric-tower differential suite that runs on every shipped artifact.

F-2 [S1, high] The shipped browser kernel never runs the corpus

No runner feeds spec/tests through sx_browser.bc.js/.wasm.js. Its entire test surface: test_boot.sh (require() smoke), test_wasm_native.js (test-framework.sx + web/tests/test-wasm-browser.sx only), tests/node/run-sx-tests.js (one deftest file). Both "conformance" runners test kernels that are NOT the shipped browser artifact. F-1 and F-3 existed undetected precisely because of this. The probe harness at scratchpad/probes/run_wasm.js shows a corpus runner on the WASM kernel is trivially buildable (SxKernel.eval works headless in node with the test_boot.sh stub block).

F-3 [S1, high] Native and WASM disagree on apply and dict key order (same kernel family!)

• (apply + (list 1 2 3)) → native: error "Expected number, got list" (apply does not spread); WASM: 6 (spreads, 2-arg form only; (apply + 1 (list 2 3)) errors "apply: function and list"). Legacy JS bundle spreads fully (both forms → 6). Three different apply semantics.
• (assoc {:a 1} :b 2) → native {:b 2 :a 1} (new keys prepend); WASM {:a 1 :b 2} (insertion order); JS bundle insertion order. merge same. Dict iteration/serialization order differs server-vs-browser — anything ordering-dependent (rendered attr order, keys/vals iteration, golden-file comparisons) diverges.
• Defused non-finding: cid-from-sx canonicalizes — CIDs of {:a 1 :b 2} and (assoc {:a 1} :b 2) are identical on native AND WASM (probe verified). Content addressing is order-safe.

F-4 [S1, high] JS host fails ~half the shared corpus; the documented build gate is red

spec/tests discovery is byte-identical in both runners (82 files both), so results are directly comparable: OCaml ≈ green, JS 2490 FAIL / 5086. scripts/sx-build-all.sh (set -euo pipefail) runs node hosts/javascript/run_tests.js --full as a gate → the documented full pipeline currently exits FAIL (run_tests.js exits 1). Either the JS host is dead (remove from gate/docs) or alive (≈2500 tests behind). Today no automated gate enforces cross-host agreement at all.

F-5 [S1, high] The corpus is not host-neutral: OCaml runner preloads libraries + services import; JS runner does neither

• OCaml make_test_env (run_tests.ml:3700-3806) unconditionally preloads r7rs/render/canonical/adapters, forms/engine/router/orchestration, bytecode/compiler/vm, stdlib, signals, freeze, content, parser-combinators, graphql, graphql-exec, dom/browser, and the full hyperscript stack.
• OCaml services (import ...) IO-suspensions at runtime (run_tests.ml:2148-2169, 2178-2272). The JS runner calls Sx.eval directly — no suspension/resume loop, so import can never complete on JS.
• Result: hundreds of "Undefined symbol: gql-tokenize / pc-* / hs-*" JS failures (js-standard.log) that are runner-environment artifacts layered on top of real gaps. Tests exercising import/suspension (test-import-bind, test-io-suspension, test-coroutines, test-modules) do not test the same thing per host.

F-6 [S1, high] Whole test directories are gated on only one host — or none

From run_tests.js:300-448 and run_tests.ml:3959-4001:

• lib/tests (11 files) — continuations, continuations-advanced, freeze, signals-advanced, stdlib, stepper, tree-tools, types, vm, vm-closures, vm-primitives — auto-run only on JS --full (OCaml runs them only if named explicitly). Their only automated gate is the runner that's red: e.g. test-continuations.sx on JS: 475 pass / 54 FAIL; algebraic-data-types: 53 FAIL. Effectively ungated.
• web/tests (13 runnable) — adapter-html, aser, deps, engine, examples, forms, orchestration, page-helpers, relate-picker, router, signals, swap-integration, tw-layout — auto-run only on OCaml.
• 6 web/tests run on NO standard runner: test-adapter-dom, test-boot-helpers, test-cek-reactive, test-handlers (module-loaded, not run as suite), test-layout, test-wasm-browser (only via test_wasm_native.js).
• OCaml foundation tests (run_tests.ml:1178+) — native-only unit tests, no cross-host equivalent (fine, but they're invisible to other hosts by design).
• spec/tests/test.sx matches neither filter (test-*.sx) → runs nowhere.
• Guest-language suites (lib/scheme/tests/records.sx, lib/haskell/tests/records.sx, …) sit outside all discovery — each loop self-tests ad hoc, no aggregate gate.

F-7 [S1, high] Features pass their tests only inside the test runner's private environment — both directions

• OCaml side: values, promise?, make-promise, force are bound only in run_tests.ml (lines 1131-1167). Probes: (force (delay (+ 1 2))), (values ...), (let-values ...) → "Undefined symbol" on BOTH production surfaces (native sx_server epoch env AND WASM SxKernel). ~271 promises/values assertions pass against an environment that exists nowhere in production.
• JS side: run_tests.js injects equal?, apply, env-*, render-html, make-continuation, upcase/downcase, and a fake sha3-256 stub (lines 80-160) into the test env. The real browser bundle lacks these (probe: equal? → Undefined symbol on the bundle). Coverage numbers overstate both shipped artifacts.

F-8 [S1→S3 itemized, high] Differential probes, OCaml native vs JS bundle: 130 exprs, 98 identical, real divergences

(harness: scratchpad/probes/; numeric rows already in F-1)

Core data semantics:

Expr	OCaml native	JS bundle
(cons 1 nil)	(1) (nil ≡ empty list)	(1 nil) (nil is an element) — foundational
(range 3)	(0 1 2)	() — single-arg range returns empty on JS
(round -2.5)	-3 (half away from zero)	-2 (JS half-up)
(str (list 1 2 3))	"(1 2 3)"	"1,2,3" (Array.toString leak)
(str {:a 1})	"{:a 1}"	"[object Object]"
(len "héllo") / (len "👍")	6 / 4 (bytes)	5 / 2 (UTF-16 units) — string indexing math differs on all non-ASCII; neither counts codepoints
(upper "straße")	"STRAßE" (ASCII-only)	"STRASSE" (Unicode)
(reverse "abc")	error (lists only)	"cba"
(sort lst cmp-fn)	error "sort: 1 list"	sorts with comparator
(max)	error	-Infinity
(/ 1 0)	inf	Infinity (print)
(+ 0.1 0.2)	prints 0.3	prints 0.30000000000000004

Agreements worth recording (hosts agree, docs/spec don't): case uses flat pairs (case x 1 "one" 2 "two" :else ...) — the CLAUDE.md-documented clause-list syntax errors identically on all three kernels; (join sep coll) is sep-first on both; int? undefined on both (it's integer?); string->number radix arg unsupported on all hosts (and the corpus asserts it — a live OCaml FAIL, see F-10).

F-9 [S2, high] Primitive-set parity is broken at scale (full lists in scratchpad/prims/report.txt)

• 194 OCaml-core names missing from the JS bundle, incl. language-level: take drop zip unique init char-at substr upcase downcase equal? identical? dict-get dict-has? dict-delete! parse parse-safe parse-float escape-string compile compile-module + records constructors + math extras. Dynamically verified samples throw "Undefined symbol" on JS.
• Naming splits where both have the capability: OCaml regex-* vs JS regexp-*; OCaml parse vs JS sx-parse; OCaml json-encode vs JS json-stringify.
• 24 user-facing JS names missing on the OCaml kernel: format values force promise? make-promise call-with-values json-parse json-stringify format-date parse-datetime pluralize escape strip-tags error-message env-parent char-code-at now-ms satisfies? … (deps-check verified). Note the epoch-mode server doesn't load spec/stdlib.sx, so even format is unresolved in production server env.
• Declared in spec/primitives.sx but implemented NOWHERE: eq?, eqv? (202 spec-declared total; 9 missing from OCaml, 6 from JS).
• JS bundle oddities: 6 statically-assigned PRIMITIVES keys absent at runtime (loop raw-loop reactive-text read-char-name-loop read-map-loop scan).

F-10 [S2, high] The OCaml suite is not green, and a permanent red band is normalized

274 FAIL on the canonical host: 272 hs-upstream-* (fetch/socket/runtimeErrors/asExpression/...). If browser-only, they should be skip-listed like the 6 web tests — a permanently red FAIL column trains everyone to ignore failures. The 2 core failures are live: can-map-an-array ("map with block") and string->number radix (corpus asserts a feature no host implements). Also note hs-upstream pass/fail sets differ wildly between OCaml (272 fail) and JS (~1900 fail) — the suites' shared-corpus value is currently nil for hyperscript. UPDATE (S-2 probe, F-18): ≥118 of the 272 red hs tests PASS on the shipped WASM kernel + happy-dom — the red band is mostly mock-DOM environment deficiency, not engine failure.

F-11 [S2, high] Boundary validation silently disabled in production; stale imports broke the Python-side test files

• shared/sx/boundary.py:34 imports .ref.boundary_parser — moved to hosts/python in 7036621b → ImportError swallowed (boundary.py:44-49) → boundary type validation is a silent no-op.
• shared/sx/tests/test_bootstrapper.py:149, test_parity.py:733 import deleted shared.sx.ref.* — those tests cannot even load.
• shared/sx/async_eval.py (fallback evaluator) is macro-crippled by design (raises "sx_ref.py has been removed"); resolver.py fully stubbed. Fallback path SX_USE_OCAML=0 is non-functional — fine if intentional, but it's still importable wiring. (Hand-off: hosts lane for the fix; kept here because it is the boundary-conformance enforcement.)

F-12 [S2, high] The live Python SX reader diverges from the OCaml reader

shared/sx/parser.py runs on every production request (serialize/parse plumbing under SX_USE_OCAML=1: handlers.py:191,211; helpers.py:449). Probes vs OCaml reader:

• Python CANNOT read: #t/#f (reader-macro error), #\a chars, dotted pairs (a . b), .5 — all valid on OCaml. If any OCaml-serialized value containing a char (#\a) crosses the Python plumbing, Python errors.
• Dict serialization order differs (Python insertion {:a 1 :b "two"} vs OCaml reversed {:b "two" :a 1}) — hazard for any wire-text comparison/caching (CIDs themselves are safe, F-3).
• Agreements: escapes (\n \t " \), floats/exponents, keywords, [..]-as-list, quote/quasiquote sugar.

F-13 [S3, med] The checked-in generated kernel is not reproducible from today's spec+bootstrap

python3 hosts/ocaml/bootstrap.py --output <scratch> vs checked-in hosts/ocaml/lib/sx_ref.ml: 360 diff lines. Function inventory is identical (the delta is let vs and restructuring + block moves — an older generator produced the checked-in file); no semantic difference detected, and tree==_build. But CI (.gitea/Dockerfile.test steps 3-4) re-bootstraps sx_ref.ml from spec and recompiles — so CI's binary is built from different generated source than the dev tree's. Reproducibility should be a checked invariant (regen + diff in CI).

F-14 [S3, high] Doc drift — CLAUDE.md describes a system that no longer exists

• "Canonical SX semantics in shared/sx/ref/*.sx" — FIXED 2026-07-03: CLAUDE.md now points at spec/ and documents the bootstrap chain + corrected island rules (sequential let, implicit begin). Remaining items below still stand.
• Documents a Python host + ~744 tests — deleted.
• Documents case clause-list syntax — all hosts reject it (flat-pair syntax is real, spec/tests/test-cek-advanced.sx:549).
• Briefed suite counts (747/870/744/1080) are months stale; corpus is 5-6k assertions.
• spec/primitives.sx header claims stdlib functions "moved to stdlib" but spec/stdlib.sx defines only format (and production doesn't load it).

F-15 [S4, med] Housekeeping

• Stray recursive tree hosts/ocaml/hosts/ocaml/hosts/ocaml/bin/sx_server.ml (accidental copy) — confuses greps and could get compiled/edited by mistake.
• spec/tests/test.sx dead filename (see F-6).
• JS runner's fake sha3-256 stub returns non-SHA3 values — any hash-shape-only assertion passes; any real-value assertion would mysteriously fail JS-only.

---

Hyperscript on the shipped WASM kernel (S-2 probe, executed 2026-07-03)

Method: new probe harness scratchpad/hsprobe/run_hs_wasm.js — loads the SHIPPED kernel (shared/static/wasm/sx_browser.bc.js) + shipped platform in node/happy-dom via tests/node/sx-harness.js, preloads the same files as the OCaml runner (test-framework, spec/harness, web/lib/dom, the lib/hyperscript stack), and runs the hs spec corpus with streamed per-test output. Per-test diff vs the native run in scratchpad/hsprobe/compare2.py; raw logs /tmp/sx-review/wasm-hs-*.log, comparisons /tmp/sx-review/wasm-hs-compare.txt, wasm-hs-pure2-compare.txt. Coverage: 1,553 tests measured (tokenizer/parser/compiler/runtime 206; behavioral 1,250 of 1,514 — one shard timed out; conformance 97 of 222 — timed out; conformance-dev/sandbox/diag/integration/htmx unmeasured). Bottom line: the kernel itself conforms; the packaging and the test environments are where the defects are.

F-16 [S1, high, CONFIRMED] Shipped browser hyperscript cannot call functions: host-call-fn is referenced but defined nowhere in the shipped stack

• The shipped stack ships a full hs engine (shared/static/wasm/sx/hs-{tokenizer,parser,compiler,runtime,integration,htmx,worker,prolog}.sx(+.sxbc)), but hs-runtime.sx(bc)/hs-integration.sx(bc) reference host-call-fn/host-call-fn-raising, which only run_tests.ml:3564 defines (the test runner's mock host bridge). The shipped platform (sx-platform.js) registers host-call / host-new / host-get etc. — NOT host-call-fn.
• Reproduced: running the behavioral corpus on the shipped stack as-is → 536 pass / 978 fail, 906 of them "Undefined symbol: host-call-fn" (wasm-hs-behavioral.log). Any hyperscript that invokes a function or method would fail the same way in a real browser.
• The OCaml runner's own comment calls this binding "the single biggest gap: ~900 behavioral tests failed" — the fix was made in the test runner only, never in the shipped platform. Textbook case of F-7 (feature exists only in the runner's private environment).

F-17 [S1, high, CONFIRMED] Shipped hs-runtime.sx is missing the (jit-exclude! "hs-*") guard

• lib/hyperscript/runtime.sx ends with (jit-exclude! "hs-*") + a comment explaining the hs recursion web miscompiles under the bytecode JIT (the parser-combinator JIT bug) and must stay CEK-interpreted.
• shared/static/wasm/sx/hs-runtime.sx is byte-identical EXCEPT this guard is absent (10-line diff; the other 5 hs modules are identical copies). The browser is precisely the sxbc/JIT-heavy environment. So the tested configuration (JIT-excluded) is not the shipped configuration (JIT-eligible). The lib→wasm/sx sync that copied these files dropped exactly the safety-critical line.

F-18 [S2, high, CONFIRMED] The kernel conforms; the native runner's mock DOM is the outlier — and the "272 red" band mostly indicts the test env, not the code

Per-test diff, shipped-kernel+happy-dom (with the host bridge mirrored) vs native mock-DOM run:

• Pure pipeline (tokenizer/parser/compiler/runtime, 206 tests): 0 WASM-only failures; all 51 failures shared with native; 2 native-only.
• Behavioral (1,250 matched): 9 WASM-only failures vs 118 native-only failures (WASM+happy-dom passes where the mock DOM fails: fetch 22, toggle 10, on 10, make 8, repeat 6, append 6, ...). Conformance (97 matched): 0 WASM-only, 4 native-only.
• Consequence for F-10: the permanently-red 272 hs failures on the canonical host are largely mock-DOM environment artifacts — the engine passes those tests on the shipped kernel with a more realistic DOM. The red band hides real information in both directions.
• The 9 WASM-only failures (listed in wasm-hs-compare.txt): behavior scoping ×4 ("Expected 10/20, got "), as-fragment conversions ×3 (a {:__host_handle N} leaks where a list is expected — happy-dom NodeList boundary), init/where sequencing ×2. Candidates for bisection; none look like core-semantics bugs.
• Also measured: the js_of_ocaml kernel is ~1-2 orders of magnitude slower on this corpus than native (conformance ≈24s/test vs seconds for the whole file natively) — worth knowing before making WASM corpus runs a gate.

F-19 [S3, med, CONFIRMED] hs corpus drift + inverted assert labels

• The shared-failure bucket (~50 behavioral + parser/tokenizer suites) is corpus drift: generated tests (generate-sx-tests.py, "DO NOT EDIT") still expect the old parser AST — (me) implicit target and (. obj k) — while the current parser emits (beingTold) / poss (probed identically on native sx_server AND WASM: (hs-compile "add .foo") → (add-class "foo" (beingTold)) on both). Tests were not regenerated after the parser change.
• assert= (spec/harness.sx:31) is (actual expected msg) but the generated corpus calls it as (assert= expected actual) — every failure message prints Expected/got swapped, which materially misled diagnosis during this probe (it makes current-parser output look like the "expected" value).
• hypersx.sx in the shipped boot list is NOT the hyperscript engine (it's an sx→hypersx-notation pretty-printer); the actual hs engine modules are shipped but absent from the boot module list (loadWebStackFallback, sx-platform.js:670) — load path for hs in production is unclear/on-demand only.

---

SUSPECTED findings / gaps (probe proposed, not yet reproduced)

S-1 [S1, med] Everything F-8 lists between native and the JS bundle likely also splits native vs WASM in web-stack .sx code paths not covered by my 130 probes (strings near render, regex at the sxbc layer, signals timing). Probe: run the full probe corpus + web adapter smoke through run_wasm.js and diff (harness exists; only 130 exprs run so far).

S-2 — RESOLVED, promoted to F-16/F-17/F-18 below (hs corpus WAS run against the shipped WASM kernel; see "Hyperscript on the shipped kernel" section).

S-3 [S2, med] import-dependent components behave differently in production browser vs test: OCaml runner resolves imports synchronously from disk; browser resolves via fetch + sxbc bundles (compile-modules.js). No test asserts the browser-side import path resolves the same module set. Probe: compare sx_build_manifest/dist sxbc contents vs run_tests preload list.

S-4 [S3, med] Float printing (0.3 vs 0.30000000000000004) means any golden-string test containing floats validates different precision per host — some current "passes" may be precision-coincidences. Probe: grep corpus asserts for float literals with >6 significant digits.

S-5 [S3, med — partially confirmed] The two JS kernel BUILDS disagree with each other: spec/tests/test-adt.sx (algebraic-data-types) passes on the standard bundle but fails 53 assertions on the --full build (--extensions continuations --spec-modules types). Confirmed from js-standard.log (0 FAIL, PASSes present) vs js-full.log (53 FAIL). So build flags change language behavior within one host — the types/continuations extension modules interfere with ADT/match. Suspected mechanism unverified; probe: bisect by building with each flag alone and running test-adt.sx.

---

Suite asymmetry — why the counts differ (summary answer to the brief)

• Corpus exploded (hyperscript upstream ports, GQL, parser combinators, regex, records, chars, bytevectors, numeric tower, values/promises...) to ~5-6k assertions; briefed counts are months stale.
• JS standard (5086) = spec/tests only; JS full (5656) = + lib/tests + bigger kernel build.
• OCaml (6036) = spec/tests + web/tests + native foundation tests, but NOT lib/tests.
• Python = 0 (deleted).
• WASM (shipped browser artifact) = ~0 (boot smoke + 2 files).
• No two hosts run the same file set; only spec/tests overlaps fully — and even there the runner environments differ (F-5, F-7), so "same test passing on two hosts" does not mean "same behavior".

Recommended conformance gate (one paragraph for the maintainer)

Run the spec/tests corpus on: native run_tests.exe, the WASM kernel via node (harness pattern: scratchpad/probes/run_wasm.js), and (if kept) the JS bundle — from the SAME preload manifest, with runner shims deleted or mirrored into all three; add lib/tests to the OCaml runner; skip-list the browser-only hs suites instead of letting them fail; add a numeric/string/dict-order differential probe file (seed: scratchpad/probes/probes.txt, 130 exprs, 32 divergences today) that must be output-identical across kernels; regen sx_ref.ml in CI and diff against checked-in.