erlang: Phase 9g — ring bench on integrated binary (no regression); scope Phase 10

lib/erlang/bench_ring_results.md

@@ -51,3 +51,36 @@ at hot call sites, the real speedup story (~3000× ring throughput,
 ~1000× spawn) starts. Until then this file documents the
 pre-integration ceiling. 72 vm-suite tests guard the stub correctness;
 full conformance is **709/709** with the stub infrastructure loaded.
+
+## Phase 9g — post-integration bench (2026-05-15)
+
+9a (vm-ext mechanism), 9h (`erlang_ext.ml` registering `erlang.OP_*`
+ids 222-239), and 9i (SX dispatcher consulting `extension-opcode-id`)
+are now integrated and built into `hosts/ocaml/_build/default/bin/sx_server.exe`.
+Re-ran the ring ladder on that binary:
+
+| N (processes) | Hops | Wall-clock | Throughput |
+|---|---|---|---|
+| 10 | 10 | 938ms | 11 hops/s |
+| 100 | 100 | 2772ms | 36 hops/s |
+| 500 | 500 | 14190ms | 35 hops/s |
+| 1000 | 1000 | 31814ms | 31 hops/s |
+
+**Numbers are unchanged from the pre-integration baseline** — and that
+is the expected, correct result. The opcode handlers (both the SX stub
+dispatcher and the OCaml `erlang_ext` module) wrap the existing
+`er-match-*` / `er-bif-*` / scheduler implementations 1-to-1, and the
+**bytecode compiler does not yet emit `erlang.OP_*` opcodes**, so every
+hop still goes through the general CEK path exactly as before. The
+unchanged numbers therefore double as a no-regression check: the full
+extension wiring (cherry-picked vm-ext A-E + force-link + erlang_ext +
+SX bridge) added zero per-hop cost. Conformance **715/715** on this
+binary.
+
+The ~3000×/~1000× targets remain gated on a **future phase (Phase 10 —
+bytecode emission)**: teach `lib/compiler.sx` (or the Erlang
+transpiler) to emit `erlang.OP_PATTERN_TUPLE` etc. at hot call sites,
+then give `erlang_ext.ml` real register-machine handlers instead of the
+current honest not-wired raise. That is a substantial standalone phase,
+tracked in `plans/erlang-on-sx.md`. 9g's deliverable — *honest
+measurement + recorded numbers on the integrated binary* — is complete.

plans/erlang-on-sx.md

@@ -138,7 +138,15 @@ Replace today's hardcoded BIF dispatch (`er-apply-bif`/`er-apply-remote-bif` in
 - [x] **9f — BIF dispatch table** — **+18 vm tests** (709/709 total). 10 hot BIFs get their own opcode IDs (136-145) in `lib/erlang/vm/dispatcher.sx`: `OP_BIF_LENGTH`, `OP_BIF_HD`, `OP_BIF_TL`, `OP_BIF_ELEMENT`, `OP_BIF_TUPLE_SIZE`, `OP_BIF_LISTS_REVERSE`, `OP_BIF_IS_INTEGER`, `OP_BIF_IS_ATOM`, `OP_BIF_IS_LIST`, `OP_BIF_IS_TUPLE`. Each opcode's handler IS the underlying `er-bif-*` impl directly (no registry-string-lookup), so cost is opcode-id → handler one-hop. Cold BIFs continue through `er-apply-bif` / `er-lookup-bif` as before. IDs 136-159 reserved for future hot-BIF additions.
 - [x] **9h — `erlang_ext.ml`** — OCaml extension at `hosts/ocaml/lib/extensions/erlang_ext.ml` registering the 18-opcode Erlang namespace (ids **222-239**, names `erlang.OP_*` mirroring the SX stub dispatcher). Registered at sx_server startup via `Erlang_ext.register ()` (guarded against double-register Failure). `extension-opcode-id "erlang.OP_PATTERN_TUPLE"` → 222 … `OP_BIF_IS_TUPLE` → 239, unknown → nil. Handlers raise a descriptive not-wired `Eval_error` (bytecode emission is a later phase; SX stub dispatcher remains the working specialization path) — keeps the extension honest rather than silently corrupting the VM stack. id range 222+ dodges test_reg (210/211) + test_ext (220/221) so all three coexist in run_tests. **+5 OCaml ext tests** (run_tests `Suite: extensions/erlang_ext`); Erlang conformance held **709/709**.
 - [x] **9i — wire SX dispatcher to real ids** — `lib/erlang/vm/dispatcher.sx` gains `er-vm-host-opcode-id` (thin `extension-opcode-id` wrapper) and `er-vm-effective-opcode-id name stub-id` (host id when non-nil, else stub-id). `extension-opcode-id` resolves lazily at call time so loading the file is safe even on a binary lacking the primitive; only invoking the resolver there would raise (documented prereq — the loop builds + runs against the binary that has it). **+6 vm tests** (715/715): OP_PATTERN_TUPLE→222, OP_BIF_IS_TUPLE→239, unknown→nil, effective prefers host (OP_BIF_LENGTH→230), effective falls back to stub on nil (999), and a sweep asserting the whole 18-name namespace maps contiguously to 222..239. Stub-local ids (128-145) registration untouched so the prior 72 vm tests stay green.
-- [ ] **9g — Conformance + perf bench** — Conformance half satisfied: **709/709** with stub infra. Perf-bench half: after 9h+9i, re-run `lib/erlang/bench_ring.sh`, record real numbers in `lib/erlang/bench_ring_results.md`. Targets: 100k+ hops/sec at N=1000, 1M-process spawn under 30s.
+- [x] **9g — Conformance + perf bench** — Ran `lib/erlang/bench_ring.sh 10 100 500 1000` on the integrated binary (9a+9h+9i built in): 11/36/35/31 hops/s — **unchanged from the pre-integration baseline**, which is the correct expected result and doubles as a no-regression proof (the full extension wiring added zero per-hop cost). Conformance **715/715** on the same binary. Numbers recorded in `lib/erlang/bench_ring_results.md` with the rationale. The ~3000×/~1000× targets are gated on Phase 10 (bytecode emission) — the compiler doesn't emit `erlang.OP_*` yet, so every hop still takes the general CEK path. 9g's deliverable (honest measurement on the integrated binary) is complete.
+
+### Phase 10 — bytecode emission (unlock the speedup)
+
+The Phase 9 opcodes are registered, tested, and bridged SX↔OCaml, but inert: nothing emits them. Phase 10 makes the speedup real.
+
+- [ ] **10a — compiler emits `erlang.OP_*` at hot sites**: teach the Erlang transpiler / `lib/compiler.sx` to emit `erlang.OP_PATTERN_TUPLE` etc. (resolved via `extension-opcode-id`) for hot `case`/`receive`/BIF call sites instead of generic dispatch.
+- [ ] **10b — real `erlang_ext.ml` handlers**: replace the not-wired raises with register-machine handlers operating on the VM stack (pattern match, perform/handle, receive-scan, spawn/send, hot BIFs).
+- [ ] **10c — perf validation**: re-run `bench_ring.sh`; target 100k+ hops/sec at N=1000, 1M-process spawn < 30s; record in `bench_ring_results.md`. Conformance must stay green.
 
 **Acceptance:** ring benchmark hits the 100k hops/sec target. All prior phase tests pass. Two opcodes chiselled to `lib/guest/vm/` (or annotated as candidates with a written rationale).
 
@@ -146,6 +154,8 @@ Replace today's hardcoded BIF dispatch (`er-apply-bif`/`er-apply-remote-bif` in
 
 _Newest first._
 
+- **2026-05-15 Phase 9g — perf bench recorded on integrated binary; Phase 10 scoped** — Built the fresh `sx_server.exe` (9a+9h+9i wired in), ran `lib/erlang/bench_ring.sh 10 100 500 1000`: 11/36/35/31 hops/s — statistically identical to the pre-9a baseline (11/24/26/29/34). This is the *expected* outcome and the iteration's actual deliverable: it proves the entire extension stack (vm-ext A-E cherry-pick + `Sx_vm_extensions` force-link + `erlang_ext.ml` + SX dispatcher bridge) added **zero per-hop overhead** — a clean no-regression result — while honestly showing the speedup hasn't arrived because the bytecode compiler still doesn't emit `erlang.OP_*` (every hop takes the general CEK path). Updated `bench_ring_results.md` with a "Phase 9g" section: the table + the rationale that unchanged numbers = correct + no-regression. Conformance **715/715** on the integrated binary. Added **Phase 10 — bytecode emission** to the roadmap (10a compiler emits opcodes at hot sites, 10b real register-machine `erlang_ext.ml` handlers replacing the not-wired raises, 10c perf validation against the 100k-hops/1M-spawn targets). Phase 9 is now fully ticked (9a-9i); the actual speedup is honestly deferred to Phase 10 rather than faked. No code change this iteration — measurement + documentation + roadmap.
+
 - **2026-05-15 Phase 9i — SX dispatcher consults host opcode ids** — `lib/erlang/vm/dispatcher.sx` now bridges SX↔OCaml opcode ids. Two new functions: `er-vm-host-opcode-id` (wraps `extension-opcode-id`) and `er-vm-effective-opcode-id name stub-id` (host id if the OCaml `erlang_ext` registered it, else the stub-local id). Key SX-runtime fact established this iteration: symbol resolution is **lazy/call-time** — `(define f (fn () (extension-opcode-id "x")))` does NOT raise at load even when the primitive is absent; only calling `f` does. Combined with the earlier findings (guard can't catch undefined-symbol; no symbol-existence reflection), this means graceful in-SX degradation is impossible — so the design instead documents the binary prerequisite and relies on the loop building+running the freshly-built `hosts/ocaml/_build/default/bin/sx_server.exe` (conformance.sh's default, which has the vm-ext mechanism + erlang_ext). Stub-local registration (128-145) deliberately left intact so the 72 pre-existing vm tests don't move. 6 new vm tests: 222/239 lookups, unknown→nil, effective-prefers-host (230), effective-fallback (999), and a contiguity sweep over all 18 `erlang.OP_*` names asserting they map to 222..239 in order. vm suite 72→78. Total **715/715** on the fresh binary. Next: 9g — re-run ring bench, record numbers (note: stubs still wrap existing impls 1-to-1 so numbers won't move until the compiler emits these opcodes — a later phase).
 
 - **2026-05-15 Phase 9h — erlang_ext.ml registered, opcode namespace live** — New `hosts/ocaml/lib/extensions/erlang_ext.ml` modelled on `test_ext.ml`: an `EXTENSION` module `name="erlang"`, per-instance `ErlangExtState` (dispatch counter), 18 opcodes ids 222-239 named `erlang.OP_*` exactly mirroring the SX stub dispatcher. Registered at sx_server startup with a second guarded line in `bin/sx_server.ml` (`try Erlang_ext.register () with Failure _ -> ()` — survives a re-entered server). `include_subdirs unqualified` in `lib/dune` already pulls `lib/extensions/*.ml` into the `sx` lib, so no dune edit needed. Handlers deliberately raise a descriptive `Eval_error` ("bytecode emission not yet wired (Phase 9j) — Erlang runs via CEK; specialization path is the SX stub dispatcher") rather than fake stack ops — the compiler doesn't emit these yet, so an honest loud failure beats silent corruption. Hit and fixed an opcode-id collision: the original 200-217 range clashed with run_tests' inline test_reg (210/211); relocated to 222-239 (clears test_reg + test_ext 220/221, all coexist; production sx_server only registers erlang). 5 new OCaml tests in run_tests `Suite: extensions/erlang_ext`: opcode-id 222 + 239 resolve, unknown→nil, dispatch raises not-wired (substring check, no Str dep since run_tests doesn't link str), dispatch_count state ≥1. Built via `eval $(opam env --switch=5.2.0); dune build bin/run_tests.exe bin/sx_server.exe`. Erlang conformance **709/709** on the rebuilt binary (the broad run_tests 1110 failures are loops/erlang's pre-existing months-old divergence from architecture — run_tests was never built on this branch before; my changes are isolated additive). Next: 9i — wire the SX stub dispatcher to consult `extension-opcode-id`.