From 12b56afcd3ec7af4739aa648389b695cadcd1054 Mon Sep 17 00:00:00 2001 From: giles Date: Fri, 15 May 2026 08:11:55 +0000 Subject: [PATCH] erlang: Phase 9a integrated (cherry-pick + force-link); plan 9h/9i added --- plans/erlang-on-sx.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/plans/erlang-on-sx.md b/plans/erlang-on-sx.md index 6919d6f1..670f8d6b 100644 --- a/plans/erlang-on-sx.md +++ b/plans/erlang-on-sx.md @@ -130,13 +130,15 @@ Replace today's hardcoded BIF dispatch (`er-apply-bif`/`er-apply-remote-bif` in **Shared-opcode discipline:** opcodes that another language port could plausibly use (pattern match, perform/handle, record access) get prepared for **chiselling out to `lib/guest/vm/`** when a second use materialises. Same lib/guest pattern, applied at the bytecode layer. Don't pre-extract; do annotate candidates in commit messages. -- [x] **9a — Opcode extension mechanism** — **LOGGED AS BLOCKER** (the contract for this bullet was "Log as Blocker"). Implementation lives in `hosts/ocaml/evaluator/`, out of scope for this loop. Design at `plans/sx-vm-opcode-extension.md`. Sub-phases 9b-9g proceed against a stub dispatcher in `lib/erlang/vm/` and will integrate when 9a lands on the architecture branch. See Blockers for the explicit dependency. +- [x] **9a — Opcode extension mechanism** — **INTEGRATED** (scope widened by user 2026-05-15: hosts/ in scope, merging back). Cherry-picked the 5 vm-ext commits (phases A-E: dispatch fallthrough for opcodes ≥200, `Sx_vm_extension` interface, `Sx_vm_extensions` registry, `extension-opcode-id` SX primitive, JIT skip path) onto loops/erlang. Force-linked `Sx_vm_extensions` into `bin/sx_server.ml` so its module-init runs (was dead-code-eliminated — only `run_tests` referenced it). `extension-opcode-id` is now live in the runtime: returns the registered opcode id, or nil for unknown names. Built clean; conformance held at **709/709** on the freshly built binary. Design: `plans/sx-vm-opcode-extension.md`. - [x] **9b — `OP_PATTERN_TUPLE` / `OP_PATTERN_LIST` / `OP_PATTERN_BINARY`** — **+19 vm tests** (656/656 total). Stub dispatcher in `lib/erlang/vm/dispatcher.sx` mirrors the OCaml extension shape from `plans/sx-vm-opcode-extension.md`: `er-vm-register-opcode!`/`er-vm-lookup-opcode-by-id`/`er-vm-lookup-opcode-by-name`/`er-vm-dispatch`. Opcode IDs 128 (TUPLE), 129 (LIST), 130 (BINARY) per the guest-tier partition (128-199). Handlers are thin wrappers over the existing `er-match-tuple`/`er-match-cons`/`er-match-binary` for now; the real specialization (skip AST walk, register-machine operands) lands when 9a integrates. Conformance must remain unchanged — **656/656** preserved. Candidate for chiselling to `lib/guest/vm/match.sx` once a second port (Prolog? miniKanren?) wants the same opcodes. - [x] **9c — `OP_PERFORM` / `OP_HANDLE`** — **+9 vm tests** (665/665 total). Stubs in `lib/erlang/vm/dispatcher.sx`: `OP_PERFORM` (id 131) raises `{:tag "vm-effect" :effect :args }`; `OP_HANDLE` (id 132) wraps a thunk in `guard`, catches matching effects (by `:effect` name), passes args to the handler, returns the handler's result. Non-matching effects rethrow to outer handlers (verified by a nested-handle test). Pure Erlang `receive` interface unchanged; this is the substrate for the eventual call/cc-free implementation when 9a integrates. Candidate for chiselling (Scheme call/cc, OCaml 5 effects, miniKanren all want the same shape). - [x] **9d — `OP_RECEIVE_SCAN`** — **+10 vm tests** (675/675 total). Stub at id 133 in `lib/erlang/vm/dispatcher.sx`. Operand contract: `(clauses mbox-list env)` where each clause is `{:pattern :guards :body}`, mbox-list is a plain SX list (not a queue — caller does queue→list before invoking and queue-delete after). Walks mbox in arrival order; tries each clause per message; first match returns `{:matched true :index N :body B}` (env mutated with bindings, body NOT evaluated — caller chooses when); no match returns `{:matched false}`. Pure pattern scan; suspension is the caller's job (compose with OP_PERFORM "receive-suspend" once 9a integrates). The real opcode will skip the AST walk by JIT-compiling each clause's match expr; this stub re-uses `er-match!` for correctness. - [x] **9e — `OP_SPAWN` / `OP_SEND` + lightweight scheduler** — **+16 vm tests** (691/691 total). Stubs at ids 134 (SPAWN) and 135 (SEND) in `lib/erlang/vm/dispatcher.sx`, plus the VM-process registry: `er-vm-procs` (dict pid → proc record), `er-vm-next-pid`, `er-vm-procs-reset!`, `er-vm-proc-new!`/`get`/`send!`/`mailbox`/`state`/`count`. Process record shape is the register-machine layout the real scheduler will use: `{:id :registers (list of 8 nil slots) :mailbox (SX list) :state ("runnable"/"waiting"/"dead") :initial-fn :initial-args}`. OP_SPAWN returns a numeric pid and allocates a fresh record; OP_SEND appends to the target's mailbox, flipping `:state` from "waiting" → "runnable" if needed (returns true on success, false on unknown pid — no crash). Sits parallel to `er-scheduler` (the language-level scheduler from Phase 3); the real VM scheduler will take over once 9a integrates and Erlang programs compile to bytecode. Perf targets in the bullet (spawn <50µs, send <5µs) defer to the integration step. - [x] **9f — BIF dispatch table** — **+18 vm tests** (709/709 total). 10 hot BIFs get their own opcode IDs (136-145) in `lib/erlang/vm/dispatcher.sx`: `OP_BIF_LENGTH`, `OP_BIF_HD`, `OP_BIF_TL`, `OP_BIF_ELEMENT`, `OP_BIF_TUPLE_SIZE`, `OP_BIF_LISTS_REVERSE`, `OP_BIF_IS_INTEGER`, `OP_BIF_IS_ATOM`, `OP_BIF_IS_LIST`, `OP_BIF_IS_TUPLE`. Each opcode's handler IS the underlying `er-bif-*` impl directly (no registry-string-lookup), so cost is opcode-id → handler one-hop. Cold BIFs continue through `er-apply-bif` / `er-lookup-bif` as before. IDs 136-159 reserved for future hot-BIF additions. -- [ ] **9g — Conformance + perf bench** — **PARTIALLY BLOCKED on 9a**. Conformance half is satisfied trivially: **709/709** with the stub opcode infrastructure loaded — Phase 1-8 tests don't touch the new opcodes, so they pass alongside. Perf-bench half is unmeasurable until 9a integrates: the stubs wrap existing implementations 1-to-1, so a benchmark run today would measure the same numbers already in `lib/erlang/bench_ring_results.md` (34 hops/s at N=1000, no spawn measurement). Annotation added to bench_ring_results.md documenting the pre-integration state. Re-fire this checkbox after 9a lands. +- [ ] **9h — `erlang_ext.ml`**: OCaml extension module in `hosts/ocaml/lib/extensions/` registering Erlang's opcodes with real VM handlers (pattern tuple/list/binary, perform/handle, receive-scan, spawn/send, hot BIFs). Register it at sx_server startup. Opcode IDs in the 200-247 port tier per `plans/sx-vm-opcode-extension.md`. +- [ ] **9i — wire SX dispatcher to real ids**: `lib/erlang/vm/dispatcher.sx` consults `(extension-opcode-id "erlang.OP_*")` and uses the host id when present, falling back to the stub-local id otherwise. Tests verify both paths. +- [ ] **9g — Conformance + perf bench** — Conformance half satisfied: **709/709** with stub infra. Perf-bench half: after 9h+9i, re-run `lib/erlang/bench_ring.sh`, record real numbers in `lib/erlang/bench_ring_results.md`. Targets: 100k+ hops/sec at N=1000, 1M-process spawn under 30s. **Acceptance:** ring benchmark hits the 100k hops/sec target. All prior phase tests pass. Two opcodes chiselled to `lib/guest/vm/` (or annotated as candidates with a written rationale). @@ -144,6 +146,8 @@ Replace today's hardcoded BIF dispatch (`er-apply-bif`/`er-apply-remote-bif` in _Newest first._ +- **2026-05-15 Phase 9a integrated — scope widened to hosts/** — User lifted the hosts/ scope restriction ("we are going to merge this back anyhow"). Cherry-picked the 5 `vm-ext` commits (phases A-E) from `loops/sx-vm-extensions` onto `loops/erlang` — only conflict was `plans/sx-vm-opcode-extension.md` (already had architecture's final copy from an earlier iteration; resolved `-X ours`, OCaml files auto-merged clean since loops/erlang never touched hosts/). Discovered `extension-opcode-id` was still "Undefined symbol" even on a fresh build: `Sx_vm_extensions`'s module-init (`install_dispatch` + primitive registration) only runs if the module is linked, and `sx_server.ml` never referenced it (only `run_tests.ml` did), so OCaml dead-code-eliminated it. Fix: added `let () = ignore (Sx_vm_extensions.id_of_name "")` force-link reference near the top of `bin/sx_server.ml`. Rebuilt with `dune build` (opam switch 5.2.0; `dune` not on PATH by default — `eval $(opam env --switch=5.2.0)` first). `extension-opcode-id` now live: returns nil for unregistered names, will return real ids once an extension registers. Conformance **709/709** on the freshly built binary (cherry-picked sx_vm.ml dispatch changes + force-link, zero regressions). 9a checkbox flipped from BLOCKED to INTEGRATED; Blockers entry resolved; added 9h (erlang_ext.ml) + 9i (wire SX dispatcher to real ids) as ordinary in-scope checkboxes, reordered 9g after them. Next: write `hosts/ocaml/lib/extensions/erlang_ext.ml`. + - **2026-05-14 Phase 9g logged as partially BLOCKED — perf bench waits on 9a** — Conformance half satisfied: 709/709 with all Phase 9 stub infrastructure loaded (10 opcode IDs registered, 72 vm-suite tests passing, zero regressions in tokenize/parse/eval/runtime/ring/ping-pong/bank/echo/fib/ffi suites). Perf-bench half can't move forward in this worktree because the stub handlers wrap the existing `er-bif-*` / `er-match-*` / scheduler impls 1-to-1; a ring benchmark with the new opcodes "active" would measure the same 34 hops/s already documented in `bench_ring_results.md`. Updated `bench_ring_results.md` with a Phase 9 status section explaining the pre-integration state (stubs ready, real measurement gated on 9a's bytecode compiler emitting these IDs at hot sites). Blockers entry added pairing 9g with the existing 9a Blocker. No code change; total **709/709** unchanged. Phase 9 stub work (9b-9f) is complete from this loop's vantage point — 9a and 9g remain BLOCKED on a `hosts/ocaml/` iteration. - **2026-05-14 Phase 9f — hot-BIF opcode table green** — Ten hot BIFs get direct opcode IDs in `lib/erlang/vm/dispatcher.sx` so the bytecode compiler can emit them at hot call sites without paying the registry string-key hash: `OP_BIF_LENGTH (136)`, `OP_BIF_HD (137)`, `OP_BIF_TL (138)`, `OP_BIF_ELEMENT (139)`, `OP_BIF_TUPLE_SIZE (140)`, `OP_BIF_LISTS_REVERSE (141)`, `OP_BIF_IS_INTEGER (142)`, `OP_BIF_IS_ATOM (143)`, `OP_BIF_IS_LIST (144)`, `OP_BIF_IS_TUPLE (145)`. Implementation is one line per opcode: the handler IS the existing `er-bif-*` function directly — same `(vs)` signature as the dispatcher's `(operands)`, so the registration is `(er-vm-register-opcode! ID "NAME" er-bif-FOO)`. IDs 136-159 reserved for future hot-BIF additions; cold BIFs continue through `er-apply-bif`/`er-lookup-bif`. 18 new tests in `tests/vm.sx`: opcode-by-id verification (LENGTH), one positive test per BIF (length on 3-cons, hd, tl-is-cons, element index 2, tuple_size 4, lists:reverse preserves length AND actually reverses [head check], is_integer pos+neg, is_atom pos+neg, is_list pos+nil pos+tuple neg, is_tuple pos+neg), opcode-list-grew-to-16+. vm suite 54 → 72. Total **709/709** (+18 vm). Real perf benefit lands when 9a integrates and the compiler emits these IDs at hot sites. @@ -218,6 +222,6 @@ _Newest first._ - **Phase 9g — Perf bench gated on 9a** (2026-05-14). The conformance half of 9g (709/709 with stub VM loaded) is satisfied; the perf-bench half requires 9a's bytecode compiler to actually emit the new opcodes at hot call sites. Until then a benchmark would measure today's `er-bif-*` / `er-match-*` numbers unchanged (since the stub handlers wrap them 1-to-1). Re-fire 9g after 9a lands. -- **Phase 9a — Opcode extension mechanism** (2026-05-14). Plan: `plans/sx-vm-opcode-extension.md`. Lives in `hosts/ocaml/evaluator/` which is out of scope for this loop (briefing rule: "do **not** edit `spec/`, `hosts/`, `shared/`..."). Sub-phases 9b-9g design + test opcodes against a stub SX dispatcher in `lib/erlang/vm/`; integration happens when 9a lands on the architecture branch. **Fix path:** a `hosts/` session implements opcode-registration shape per `plans/sx-vm-opcode-extension.md`, then a follow-up iteration here wires the stub dispatcher to the real one. +- **Phase 9a — Opcode extension mechanism** — **RESOLVED 2026-05-15.** User widened scope to include hosts/ (merging back anyhow). Cherry-picked vm-ext phases A-E + force-linked `Sx_vm_extensions` into sx_server.exe. `extension-opcode-id` live; conformance 709/709. Remaining integration work (erlang_ext.ml + wiring the SX stub dispatcher to consult real ids) tracked as ordinary in-scope checkboxes now, not blockers. - **SX runtime lacks platform primitives for crypto / dir-listing / HTTP / SQLite** (2026-05-14). Probed in `mcp_tree.exe`'s embedded `sx_server.exe`: `(sha256 "x")`, `(blake3 "x")`, `(hash "sha256" "x")`, `(file-list-dir "plans")`, `(http-get "url")`, `(fetch "url")` all return `Undefined symbol`. Only file-byte-level primitives exist: `file-read` ✓, `file-write` ✓, `file-delete` ✓, `file-exists?` ✓. Out-of-scope to add these (they live in `hosts/` per ground rules). Blocked Phase 8 BIFs: `crypto:hash/2`, `cid:from_bytes/1`, `cid:to_string/1`, `file:list_dir/1`, `httpc:request/4`, `sqlite:open/exec/query/close`. **Fix path:** a future iteration on the architecture branch can register host primitives (e.g. expose OCaml's `Digestif` for hashes, `Sys.readdir` for list_dir, `cohttp` for httpc); the BIF wrappers here will then become one-line registrations against `er-bif-registry`.