HS-design: E36 WebSocket + socket + RPC proxy

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 06:55:23 +00:00
parent 6b7559fcaf
commit 3587443742
1 changed files with 317 additions and 0 deletions
--- a/plans/designs/e36-websocket.md
+++ b/plans/designs/e36-websocket.md
@@ -0,0 +1,317 @@
+# E36 — WebSocket + `socket` feature + RPC proxy
+
+Design for cluster 36 of `plans/hs-conformance-to-100.md`. Scope: +16 tests in `hs-upstream-socket`, all currently `SKIP (untranslated)`.
+
+## 1. Failing tests (all SKIP'd — untranslated)
+
+Exact names from `spec/tests/test-hyperscript-behavioral.sx:11354`:
+
+1. `converts relative URL to ws:// on http pages`
+2. `converts relative URL to wss:// on https pages`
+3. `dispatchEvent sends JSON-encoded event over the socket`
+4. `namespaced sockets work`
+5. `on message as JSON handler decodes JSON payload`
+6. `on message as JSON throws on non-JSON payload`
+7. `on message handler fires on incoming text message`
+8. `parses socket with absolute ws:// URL`
+9. `rpc proxy blacklists then/catch/length/toJSON`
+10. `rpc proxy default timeout rejects the promise`
+11. `rpc proxy noTimeout avoids timeout rejection`
+12. `rpc proxy reply with throw rejects the promise`
+13. `rpc proxy sends a message and resolves the reply`
+14. `rpc proxy timeout(n) rejects after a custom window`
+15. `rpc reconnects after the underlying socket closes`
+16. `with timeout parses and uses the configured timeout`
+
+## 2. Upstream semantics (stock _hyperscript)
+
+Verified against the JSON fixture and `hyperscript.org/features/socket/`. The prompt's "`with proxy { send, receive }`" wording is **not** part of the upstream socket feature — there is no `with proxy` clause. The shape actually under test is:
+
+```
+socket <Name[.dotted.path]> <url> [with timeout <ms>]
+  [on message [as JSON]
+     <cmd-list>]
+end
+```
+
+**URL scheme normalisation.** Bare paths get a scheme prepended from `location.protocol`: `http:` → `ws://host[:port]<path>`, `https:` → `wss://…`. Absolute `ws://` / `wss://` URLs pass through unchanged.
+
+**Named binding.** `socket Foo url end` assigns `window.Foo = socketObj`. A dotted name `socket MyApp.chat url end` walks/creates `window.MyApp` then sets `window.MyApp.chat = socketObj`.
+
+**Socket object shape** (what the tests assert against):
+```
+{ raw: WebSocket,           // underlying ws
+  rpc: <Proxy>,              // see below
+  dispatchEvent(evt),        // JSON-encodes the event and sends over ws
+  // 'on message' handler registered internally via ws.onmessage
+}
+```
+
+**`dispatchEvent` protocol.** On `send NAME(a:1) to Foo` or `Foo.dispatchEvent(evt)`:
+- Outgoing payload: `JSON.stringify({ type: evt.type, ...evt.detail })` with `sender` and `_namedArgList_` stripped.
+
+**`on message [as JSON]`.** `ws.onmessage({data})` fires the cmd-list. `as JSON` calls `JSON.parse(data)` first and the parsed object becomes the event; non-JSON throws `"Received non-JSON message"`.
+
+**RPC proxy (`socket.rpc`).** ES6 `Proxy` over an object. Accessing `.anyName` normally returns a callable; four property names are blacklisted and return `null`: `then`, `catch`, `length`, `toJSON` (so the proxy isn't mistaken for a thenable and JSON-serialises cleanly). Two reserved chainable properties:
+- `.noTimeout` — returns a proxy that will never time out.
+- `.timeout(n)` — returns a proxy that times out after `n` ms.
+
+Calling `proxy.fnName(arg1, arg2)` performs:
+1. Allocate a unique `iid` (string).
+2. Send `JSON.stringify({iid, function: "fnName", args: [arg1, arg2]})` over ws.
+3. Return a Promise. The promise is tracked by `iid`.
+4. Incoming messages with matching `iid` are routed: `{iid, return: v}` resolves with `v`; `{iid, throw: e}` rejects with `e`.
+5. Default/custom timeout (from `with timeout N`, default per stock is ~0 i.e. "Timed out" reject; override via `.timeout(n)` or `.noTimeout`).
+
+**Reconnect.** When the underlying socket fires `close`, the socket wrapper marks itself dead. The *next* RPC (or `dispatchEvent`) call lazily reconstructs a fresh `new WebSocket(url)`. Test 15 asserts `created.length === 2` after one close + one RPC.
+
+## 3. Proposed user-facing shape
+
+### 3a. Tokens
+
+No new tokens needed. `socket` is a bare identifier that becomes a keyword in `parse-feat`. `end` is already known. `with`, `timeout`, `on`, `message`, `as`, `JSON` are already existing keyword/identifier tokens. Dotted names (`MyApp.chat`) already tokenise as `local` + `dot` + `local`.
+
+### 3b. Parse tree
+
+A new feature AST node, peer to `(def ...)` / `(on ...)`:
+
+```
+(socket <name-path> <url-expr> :timeout <expr-or-nil> :on-message <handler-or-nil>)
+```
+
+- `<name-path>` is a list of symbol strings: `["Foo"]` or `["MyApp" "chat"]`.
+- `<url-expr>` is any parseable expression (usually a string literal).
+- `:timeout` is the parsed ms expression, or `nil` if absent.
+- `:on-message` is `nil`, `(on-message :json? false <body>)`, or `(on-message :json? true <body>)`.
+
+Parser function: `parse-socket-feat` dispatched from `parse-feat` on `val == "socket"`. Reuses `parse-cmd-list` for handler bodies. `as JSON` is detected inline before collecting the body.
+
+### 3c. Compile output
+
+The compiler emits a call to a runtime registration primitive that carries everything the runtime needs:
+
+```
+(hs-socket-register!
+  (list "MyApp" "chat")            ; name path
+  <url-compiled>                   ; usually a string
+  <timeout-ms-or-nil>              ; number or nil
+  (fn (event) <on-message-body>)   ; handler closure or nil
+  :json-message? true|false)
+```
+
+The runtime owns all WebSocket construction, URL normalisation, window binding, promise/timeout bookkeeping, and the RPC proxy — keeping the compiler output small and re-usable.
+
+### 3d. Worked example
+
+Source (from fixture test 13, simplified):
+```
+socket RpcSocket ws://localhost/ws end
+```
+
+Tokens: `socket` `RpcSocket` `ws` `:` `//localhost/ws` `end` (URL tokenising is already handled by the string-like path reader; see how `fetch` URLs parse today).
+
+AST:
+```
+(socket ("RpcSocket") "ws://localhost/ws" :timeout nil :on-message nil)
+```
+
+Compiled SX (emitted into the feature body):
+```
+(hs-socket-register!
+  (list "RpcSocket")
+  "ws://localhost/ws"
+  nil
+  nil
+  :json-message? false)
+```
+
+Fuller example:
+```
+socket MyApp.chat /ws with timeout 1500
+  on message as JSON
+    set window.got to the event
+end
+```
+→
+```
+(socket ("MyApp" "chat") "/ws"
+        :timeout 1500
+        :on-message (on-message :json? true
+                                ((set! window.got event))))
+```
+→
+```
+(hs-socket-register!
+  (list "MyApp" "chat")
+  "/ws"
+  1500
+  (fn (event) (host-set! (host-global "window") "got" event))
+  :json-message? true)
+```
+
+## 4. Runtime architecture
+
+All in `lib/hyperscript/runtime.sx`. One big primitive plus small helpers.
+
+### 4a. `hs-socket-register!` — lifecycle
+
+1. **Normalise URL**: if starts with `ws://`/`wss://` use as-is; else derive scheme from `(host-get (host-global "location") "protocol")` and prepend `ws://host[:port]` or `wss://…`. Preserves path.
+2. **Construct** `(host-new "WebSocket" url)`.
+3. **Build wrapper object** (SX dict): `{:raw ws :url url :timeout <ms> :pending <dict-of-iid->resolver> :handler <closure> :json? <bool> :closed? false}`.
+4. **Install `ws.onmessage`** via `host-set!` to a native callback (`host-callback`) that:
+   - Reads `data = event.data`.
+   - If JSON-looking (starts with `{` or `[`) tries `json-parse`; if parse OK and has `iid` field → RPC reply dispatch (see 4c).
+   - Otherwise → fire user's `on message` handler with `event` (optionally JSON-parsing first if `:json? true`; non-JSON there → `(error "Received non-JSON message")`).
+5. **Install `ws.addEventListener("close", …)`** (via `host-call`) that sets `:closed? true` so next RPC reconnects.
+6. **Install `.dispatchEvent`** method on wrapper via `host-set!`: takes an `Event`, builds `{type, ...detail minus sender,_namedArgList_}`, `JSON.stringify`s, `ws.send`s.
+7. **Install `.rpc`** — the proxy (see 4b).
+8. **Bind** on `window`: walk `name-path` creating intermediate `{}` dicts as needed; final segment assigned the wrapper.
+
+### 4b. RPC proxy
+
+Implemented as a native JS object constructed via a new FFI primitive we already have: `host-call` to a global helper we register **inside the test mock and inside the real runtime**. Concretely the runtime calls `(host-call (host-global "_hs_make_rpc_proxy") "call" nil wrapper)` → returns a `Proxy`. In prod `_hs_make_rpc_proxy` is shipped in `shared/static/scripts/sx-platform-*.js`; in the test mock it's inlined into `tests/hs-run-filtered.js`.
+
+Pseudo-JS for the proxy factory (this is what lives in the platform/mock, not in `.sx`):
+
+```js
+function makeRpcProxy(wrapper, overrides = {}){
+  return new Proxy({}, {
+    get(_, name) {
+      if (['then','catch','length','toJSON'].includes(name)) return null;
+      if (name === 'noTimeout') return makeRpcProxy(wrapper, {...overrides, timeout: Infinity});
+      if (name === 'timeout')   return (n) => makeRpcProxy(wrapper, {...overrides, timeout: n});
+      return (...args) => hsRpcCall(wrapper, name, args, overrides.timeout);
+    }
+  });
+}
+```
+
+`hsRpcCall` returns a native `Promise` whose resolver is stored in `wrapper.pending[iid]`. A `setTimeout` (skipped when `Infinity`) rejects with `"Timed out"`. The sx runtime never calls `hsRpcCall` directly — the proxy does, in JS, because tests read `.then(...)` on the returned promise.
+
+**Why this split**: the test fixtures all go through `.then`/`.catch` in JS, so the proxy and its promise plumbing *must* be real JS. The SX runtime only wires up message routing.
+
+### 4c. Inbound dispatch pipeline
+
+`onmessage` in the runtime (SX):
+
+```
+(fn (event)
+  (let ((data (host-get event "data")))
+    (let ((parsed (hs-try-json-parse data)))
+      (cond
+        ;; RPC reply: has :iid
+        ((and parsed (host-get parsed "iid"))
+          (hs-socket-resolve-rpc! wrapper parsed))
+        ;; User 'on message as JSON'
+        (user-handler-json?
+          (if parsed
+              (user-handler parsed)
+              (error "Received non-JSON message")))
+        ;; User 'on message' plain
+        (user-handler
+          (user-handler event))
+        (true nil)))))
+```
+
+`hs-socket-resolve-rpc!` pulls the resolver/rejecter pair out of `:pending` by iid and calls the JS-side `resolve(ret)` or `reject(thrown)`.
+
+### 4d. `send NAME(a:1) to Foo` integration
+
+Today `emit-send` in compiler.sx always emits `dom-dispatch`. We widen it to `hs-dispatch-to` (new runtime helper) which dispatches:
+- If target exposes `_hs_socket? === true` → JSON-encode + `ws.send`.
+- Else → falls through to `dom-dispatch` (existing behaviour).
+
+This keeps backwards compat — all non-socket send targets continue to use DOM events.
+
+### 4e. `call Foo.rpc.greet("world")` integration
+
+No compiler changes needed. `call` already compiles property+method chains via `method-call`. `Foo.rpc.greet` resolves to the proxy method which returns a Promise. Hyperscript's existing await semantics (via `io-*` suspension or `then` absorption in `result`) then wait on it. For the non-Playwright test harness, the promise is resolved synchronously inside the mock's `_driveAsync` loop — see §5.
+
+### 4f. IO suspension
+
+Sockets do **not** need a new `io-*` operation for the basic tests: `onmessage` is a callback that fires user cmd-lists directly via `host-callback` → `K.callFn` → `_driveAsync` (this already works for `hs-on-every`). RPC promise resolution similarly runs inside a native callback. The only place we might want IO suspension is if someone writes `wait for message in Foo` — **not in this cluster**. Keep `hs-socket-register!` synchronous and callback-driven; defer any `(perform …)` integration to a later cluster.
+
+## 5. Test mock strategy
+
+Minimum surface in `tests/hs-run-filtered.js`:
+
+### 5a. `WebSocket` mock (test-keyed)
+
+Each test self-describes via its test name. We key behaviour off `_current-test-name`. A single `globalThis.WebSocket` constructor looks up the script-driven scenario. But the fixture's tests all re-mock `window.WebSocket` in-test — they are self-contained. That means **the mock runner merely needs a default that records constructions and lets test code override it**.
+
+```js
+globalThis.WebSocket = function(url) {
+  const sock = { url, onmessage: null, _listeners: {},
+                 send(msg){ sock._sent = sock._sent || []; sock._sent.push(msg); },
+                 addEventListener(t, h){ (sock._listeners[t]=sock._listeners[t]||[]).push(h); },
+                 close(){} };
+  globalThis.__hs_ws_created = globalThis.__hs_ws_created || [];
+  globalThis.__hs_ws_created.push(sock);
+  return sock;
+};
+```
+
+### 5b. `_hs_make_rpc_proxy` in the mock
+
+Ship the factory from §4b as a top-level function in `hs-run-filtered.js`. It must use the test-runtime `Promise` (real JS Promise), and `setTimeout` (real host — Node's, already global). Timeout rejection must fire inside `_driveAsync`'s execution window; a short `setImmediate`/microtask tick suffices because the tests all `await evaluate(() => new Promise(resolve => …))`.
+
+### 5c. Generator patterns (`tests/playwright/generate-sx-tests.py`)
+
+The socket fixtures are unusual: they directly evaluate browser-side JS that attaches a `<script type="text/hyperscript">` then calls `_hyperscript.processNode`. We do **not** try to translate that path verbatim. Instead the generator gets a `_socket_setup_ops` recogniser (peer to `_hs_config_setup_ops`) that:
+
+1. Recognises the pattern `var OrigWS = window.WebSocket; … window.WebSocket = function() {...}; … script.textContent = 'socket …'; …`.
+2. Extracts the inner hyperscript source from the `script.textContent` string.
+3. Emits an SX test body that:
+   - Pre-installs a matching WS mock (`host-set! window "WebSocket" ...`) — parameterised by which behaviour the test needs: (a) record URL, (b) record sent messages, (c) record close handlers, (d) provide an `onmessage` hook.
+   - Activates the hyperscript on a `div` (so the socket feature runs through the existing `hs-activate!` pipeline).
+   - Invokes whatever the test asserted against: reads `urls[0]`, calls `.dispatchEvent`, drives `mockSocket.onmessage(…)`, etc.
+   - Asserts the final value (translated from `expect(…).toBe(…)` via the existing `pw_assertion_to_sx`).
+
+A single shared **socket test mini-DSL** — 5 behaviour codes — covers all 16 tests:
+
+| Code | Records | Injects | Asserts |
+|------|---------|---------|---------|
+| `url-capture` | constructor URL | — | URL string |
+| `send-capture` | outgoing frames | — | JSON payload fields |
+| `inbound-text` | — | `mockSocket.onmessage({data:"…"})` | window flag set |
+| `inbound-json` | — | `mockSocket.onmessage({data:JSON…})` | window flag set |
+| `rpc-roundtrip` | outgoing frames | onmessage with `{iid,return:v}` or `{iid,throw:e}` | promise resolves/rejects |
+| `rpc-timeout` | — | timer advance | promise rejects with `"Timed out"` |
+| `close-reconnect` | close handlers + construction count | fire all close handlers then call rpc | `created.length === 2` |
+
+The generator dispatches tests 1,2,8 → `url-capture`; 3 → `send-capture`; 5,6,7 → inbound; 10,11,12,13,14 → rpc; 15 → close-reconnect; 4,9,16 → small probes of wrapper shape (`.rpc typeof === "object"`, namespaced binding exists).
+
+## 6. Test delta estimate
+
+- **Best case:** 16/16. All tests are self-contained scenarios; with the WebSocket mock, proxy factory, and generator recogniser they all translate.
+- **Realistic:** **12–14**. Timeout/reconnect tests (10, 11, 14, 15) need real-time behaviour from the Node mock; if `setTimeout` behaviour is flaky under the step-limited driver, they may go partial.
+- **Worst case:** **8**. Only the synchronous URL/shape/inbound tests land (1, 2, 4, 5, 6, 7, 8, 9, 16); the five full RPC round-trip tests stay SKIP and get handled in a follow-up.
+
+Target commit should aim for **+12 tests, partial on RPC timeout/reconnect**. Mark any stragglers `partial` per rule 7.
+
+## 7. Risks / open questions (human decisions)
+
+1. **Reconnect scope.** The fixture only tests lazy reconnect on next RPC. Do we also reconnect on `dispatchEvent` after close? (Stock yes; tests don't cover it. Decision: yes, keep parity.)
+2. **Buffering while disconnected.** Between `close` and first reconnecting call, does `dispatchEvent` drop the frame or queue it? Stock *drops*. Recommendation: drop, log once.
+3. **Binary frames.** None of the 16 tests exercise `Blob`/`ArrayBuffer`. Explicitly out of scope — document as `not yet supported` in runtime.
+4. **Close-code handling.** `close(code, reason)` / CloseEvent — no test exercises it. Recommendation: ignore code/reason, just mark closed.
+5. **Interaction with IO suspension.** RPC promises live in JS, not SX; a hyperscript `call Foo.rpc.x()` under the current runtime returns the JS Promise and hyperscript's `then` absorption takes over. Verify that the CEK/VM `hs-safe-call` path doesn't eagerly `(host-await)`.
+6. **Proxy implementation language.** The proxy *must* be JS (ES6 `Proxy`) — SX has no equivalent and the tests read `.then` etc. as literal JS property accesses. Means the proxy factory lives **both** in the test mock and in a shipped platform JS file. Do we want to factor out a shared `platform/sockets.js`? (Recommendation: yes, land it in `shared/static/scripts/sx-platform-sockets.js` in a follow-up commit after the mock lands; the mock inline version stays as the reference.)
+7. **Namespace dotted binding collisions.** If `window.MyApp` already exists as a non-dict, the runtime should refuse overwriting it (tests only use fresh names). Recommendation: silently overwrite with a dict — matches stock.
+8. **`emit-send` widening.** Changing send to branch on `_hs_socket?` touches a code path used by every existing send test. Needs a smoke-regression check against `hs-upstream-send` before landing.
+
+## 8. Implementation checklist (one commit per step; must land +N)
+
+Each step is scoped to be independently testable. `sync-wasm` means `cp lib/hyperscript/X.sx shared/static/wasm/sx/hs-X.sx`.
+
+1. **Mock groundwork** *(0 tests)*. Add `WebSocket` constructor + `_hs_make_rpc_proxy` factory to `tests/hs-run-filtered.js`. No runtime/compiler changes. Verify no regressions on smoke 0–195. Commit: `HS-prep: WebSocket + RPC proxy mock`.
+2. **Parser: `socket` feature** *(0 tests)*. Add `parse-socket-feat` in `parser.sx` dispatched from `parse-feat`. Parses name-path, URL, `with timeout N`, `on message [as JSON] … end`, trailing `end`. Emits `(socket …)` AST. Unit-verify via `hs-compile` eval trace. sync-wasm. Commit: `HS: parse socket feature`.
+3. **Compiler + runtime: basic socket registration** *(+3 tests: 1, 2, 8)*. Implement `hs-socket-register!` runtime primitive (URL normalise, construct WS, window bind — **no** RPC/on-message yet). Compiler emits `hs-socket-register!` call from `(socket …)`. sync-wasm. Land tests: `converts relative URL to ws://`, `…to wss://`, `parses socket with absolute ws:// URL`. Commit: `HS: socket URL binding (+3)`.
+4. **Namespaced binding + wrapper shape** *(+2 tests: 4, 16)*. Walk name-path assigning intermediate dicts. Ensure wrapper exposes `.raw` and `.rpc` objects (rpc can be the bare proxy for now). sync-wasm. Land: `namespaced sockets work`, `with timeout parses and uses the configured timeout`. Commit: `HS: socket namespaced names + timeout plumbing (+2)`.
+5. **Inbound `on message` dispatch** *(+3 tests: 5, 6, 7)*. Wire `ws.onmessage` → SX handler. Support `as JSON` parse branch + non-JSON error. Generator: `inbound-text` + `inbound-json` dispatch codes. sync-wasm. Land: `on message handler fires…`, `on message as JSON … decodes…`, `…throws on non-JSON`. Commit: `HS: socket on-message + as JSON (+3)`.
+6. **RPC proxy — happy path** *(+2 tests: 9, 13)*. Wire `.rpc` proxy with blacklist, `iid` allocation, send JSON, resolve on matching `iid.return`. Uses real JS `Promise` from the mock. Generator `rpc-roundtrip` code. sync-wasm. Land: `rpc proxy blacklists …`, `rpc proxy sends a message and resolves…`. Commit: `HS: socket RPC proxy (+2)`.
+7. **RPC error + reconnect** *(+3 tests: 12, 15, plus `dispatchEvent`=3)*. Add `{iid, throw: e}` path. Add close-handler tracking + lazy reconnect on next RPC/dispatch. Add `dispatchEvent` JSON-encode (strips `sender`/`_namedArgList_`). Generator `close-reconnect` + `send-capture`. sync-wasm. Land: `rpc proxy reply with throw rejects`, `rpc reconnects after … closes`, `dispatchEvent sends JSON-encoded event`. Commit: `HS: socket dispatchEvent + reconnect + throw (+3)`.
+8. **RPC timeouts** *(+3 tests: 10, 11, 14)*. `.timeout(n)` / `.noTimeout` chainable; default timeout from `with timeout N`. `setTimeout` reject with `"Timed out"`. Generator `rpc-timeout` code — may need a virtual-clock helper in the mock if real `setTimeout` is flaky under the step limiter. sync-wasm. Land: `…default timeout rejects`, `…noTimeout avoids timeout rejection`, `…timeout(n) rejects after a custom window`. Commit: `HS: socket RPC timeouts (+3)`.
+9. **Polish + scoreboard** *(0 tests)*. Update `plans/hs-conformance-to-100.md` row 36 to `done (+N)`, bump scoreboard. Smoke 0–195 + `hs-upstream-send` regression check. Commit: `HS-plan: E36 complete (+N)`.
+
+Total runway: **8 feature commits** to move cluster 36 from `blocked` → `done (+12–16)`. No commit may land a regression in smoke 0–195 or in `hs-upstream-send`.