From 3587443742b2927a4f81ca3508a0802a889335c4 Mon Sep 17 00:00:00 2001 From: giles Date: Fri, 24 Apr 2026 06:55:23 +0000 Subject: [PATCH] HS-design: E36 WebSocket + socket + RPC proxy Co-Authored-By: Claude Opus 4.7 (1M context) --- plans/designs/e36-websocket.md | 317 +++++++++++++++++++++++++++++++++ 1 file changed, 317 insertions(+) create mode 100644 plans/designs/e36-websocket.md diff --git a/plans/designs/e36-websocket.md b/plans/designs/e36-websocket.md new file mode 100644 index 00000000..4dd9df36 --- /dev/null +++ b/plans/designs/e36-websocket.md @@ -0,0 +1,317 @@ +# E36 — WebSocket + `socket` feature + RPC proxy + +Design for cluster 36 of `plans/hs-conformance-to-100.md`. Scope: +16 tests in `hs-upstream-socket`, all currently `SKIP (untranslated)`. + +## 1. Failing tests (all SKIP'd — untranslated) + +Exact names from `spec/tests/test-hyperscript-behavioral.sx:11354`: + +1. `converts relative URL to ws:// on http pages` +2. `converts relative URL to wss:// on https pages` +3. `dispatchEvent sends JSON-encoded event over the socket` +4. `namespaced sockets work` +5. `on message as JSON handler decodes JSON payload` +6. `on message as JSON throws on non-JSON payload` +7. `on message handler fires on incoming text message` +8. `parses socket with absolute ws:// URL` +9. `rpc proxy blacklists then/catch/length/toJSON` +10. `rpc proxy default timeout rejects the promise` +11. `rpc proxy noTimeout avoids timeout rejection` +12. `rpc proxy reply with throw rejects the promise` +13. `rpc proxy sends a message and resolves the reply` +14. `rpc proxy timeout(n) rejects after a custom window` +15. `rpc reconnects after the underlying socket closes` +16. `with timeout parses and uses the configured timeout` + +## 2. Upstream semantics (stock _hyperscript) + +Verified against the JSON fixture and `hyperscript.org/features/socket/`. The prompt's "`with proxy { send, receive }`" wording is **not** part of the upstream socket feature — there is no `with proxy` clause. The shape actually under test is: + +``` +socket [with timeout ] + [on message [as JSON] + ] +end +``` + +**URL scheme normalisation.** Bare paths get a scheme prepended from `location.protocol`: `http:` → `ws://host[:port]`, `https:` → `wss://…`. Absolute `ws://` / `wss://` URLs pass through unchanged. + +**Named binding.** `socket Foo url end` assigns `window.Foo = socketObj`. A dotted name `socket MyApp.chat url end` walks/creates `window.MyApp` then sets `window.MyApp.chat = socketObj`. + +**Socket object shape** (what the tests assert against): +``` +{ raw: WebSocket, // underlying ws + rpc: , // see below + dispatchEvent(evt), // JSON-encodes the event and sends over ws + // 'on message' handler registered internally via ws.onmessage +} +``` + +**`dispatchEvent` protocol.** On `send NAME(a:1) to Foo` or `Foo.dispatchEvent(evt)`: +- Outgoing payload: `JSON.stringify({ type: evt.type, ...evt.detail })` with `sender` and `_namedArgList_` stripped. + +**`on message [as JSON]`.** `ws.onmessage({data})` fires the cmd-list. `as JSON` calls `JSON.parse(data)` first and the parsed object becomes the event; non-JSON throws `"Received non-JSON message"`. + +**RPC proxy (`socket.rpc`).** ES6 `Proxy` over an object. Accessing `.anyName` normally returns a callable; four property names are blacklisted and return `null`: `then`, `catch`, `length`, `toJSON` (so the proxy isn't mistaken for a thenable and JSON-serialises cleanly). Two reserved chainable properties: +- `.noTimeout` — returns a proxy that will never time out. +- `.timeout(n)` — returns a proxy that times out after `n` ms. + +Calling `proxy.fnName(arg1, arg2)` performs: +1. Allocate a unique `iid` (string). +2. Send `JSON.stringify({iid, function: "fnName", args: [arg1, arg2]})` over ws. +3. Return a Promise. The promise is tracked by `iid`. +4. Incoming messages with matching `iid` are routed: `{iid, return: v}` resolves with `v`; `{iid, throw: e}` rejects with `e`. +5. Default/custom timeout (from `with timeout N`, default per stock is ~0 i.e. "Timed out" reject; override via `.timeout(n)` or `.noTimeout`). + +**Reconnect.** When the underlying socket fires `close`, the socket wrapper marks itself dead. The *next* RPC (or `dispatchEvent`) call lazily reconstructs a fresh `new WebSocket(url)`. Test 15 asserts `created.length === 2` after one close + one RPC. + +## 3. Proposed user-facing shape + +### 3a. Tokens + +No new tokens needed. `socket` is a bare identifier that becomes a keyword in `parse-feat`. `end` is already known. `with`, `timeout`, `on`, `message`, `as`, `JSON` are already existing keyword/identifier tokens. Dotted names (`MyApp.chat`) already tokenise as `local` + `dot` + `local`. + +### 3b. Parse tree + +A new feature AST node, peer to `(def ...)` / `(on ...)`: + +``` +(socket :timeout :on-message ) +``` + +- `` is a list of symbol strings: `["Foo"]` or `["MyApp" "chat"]`. +- `` is any parseable expression (usually a string literal). +- `:timeout` is the parsed ms expression, or `nil` if absent. +- `:on-message` is `nil`, `(on-message :json? false )`, or `(on-message :json? true )`. + +Parser function: `parse-socket-feat` dispatched from `parse-feat` on `val == "socket"`. Reuses `parse-cmd-list` for handler bodies. `as JSON` is detected inline before collecting the body. + +### 3c. Compile output + +The compiler emits a call to a runtime registration primitive that carries everything the runtime needs: + +``` +(hs-socket-register! + (list "MyApp" "chat") ; name path + ; usually a string + ; number or nil + (fn (event) ) ; handler closure or nil + :json-message? true|false) +``` + +The runtime owns all WebSocket construction, URL normalisation, window binding, promise/timeout bookkeeping, and the RPC proxy — keeping the compiler output small and re-usable. + +### 3d. Worked example + +Source (from fixture test 13, simplified): +``` +socket RpcSocket ws://localhost/ws end +``` + +Tokens: `socket` `RpcSocket` `ws` `:` `//localhost/ws` `end` (URL tokenising is already handled by the string-like path reader; see how `fetch` URLs parse today). + +AST: +``` +(socket ("RpcSocket") "ws://localhost/ws" :timeout nil :on-message nil) +``` + +Compiled SX (emitted into the feature body): +``` +(hs-socket-register! + (list "RpcSocket") + "ws://localhost/ws" + nil + nil + :json-message? false) +``` + +Fuller example: +``` +socket MyApp.chat /ws with timeout 1500 + on message as JSON + set window.got to the event +end +``` +→ +``` +(socket ("MyApp" "chat") "/ws" + :timeout 1500 + :on-message (on-message :json? true + ((set! window.got event)))) +``` +→ +``` +(hs-socket-register! + (list "MyApp" "chat") + "/ws" + 1500 + (fn (event) (host-set! (host-global "window") "got" event)) + :json-message? true) +``` + +## 4. Runtime architecture + +All in `lib/hyperscript/runtime.sx`. One big primitive plus small helpers. + +### 4a. `hs-socket-register!` — lifecycle + +1. **Normalise URL**: if starts with `ws://`/`wss://` use as-is; else derive scheme from `(host-get (host-global "location") "protocol")` and prepend `ws://host[:port]` or `wss://…`. Preserves path. +2. **Construct** `(host-new "WebSocket" url)`. +3. **Build wrapper object** (SX dict): `{:raw ws :url url :timeout :pending resolver> :handler :json? :closed? false}`. +4. **Install `ws.onmessage`** via `host-set!` to a native callback (`host-callback`) that: + - Reads `data = event.data`. + - If JSON-looking (starts with `{` or `[`) tries `json-parse`; if parse OK and has `iid` field → RPC reply dispatch (see 4c). + - Otherwise → fire user's `on message` handler with `event` (optionally JSON-parsing first if `:json? true`; non-JSON there → `(error "Received non-JSON message")`). +5. **Install `ws.addEventListener("close", …)`** (via `host-call`) that sets `:closed? true` so next RPC reconnects. +6. **Install `.dispatchEvent`** method on wrapper via `host-set!`: takes an `Event`, builds `{type, ...detail minus sender,_namedArgList_}`, `JSON.stringify`s, `ws.send`s. +7. **Install `.rpc`** — the proxy (see 4b). +8. **Bind** on `window`: walk `name-path` creating intermediate `{}` dicts as needed; final segment assigned the wrapper. + +### 4b. RPC proxy + +Implemented as a native JS object constructed via a new FFI primitive we already have: `host-call` to a global helper we register **inside the test mock and inside the real runtime**. Concretely the runtime calls `(host-call (host-global "_hs_make_rpc_proxy") "call" nil wrapper)` → returns a `Proxy`. In prod `_hs_make_rpc_proxy` is shipped in `shared/static/scripts/sx-platform-*.js`; in the test mock it's inlined into `tests/hs-run-filtered.js`. + +Pseudo-JS for the proxy factory (this is what lives in the platform/mock, not in `.sx`): + +```js +function makeRpcProxy(wrapper, overrides = {}){ + return new Proxy({}, { + get(_, name) { + if (['then','catch','length','toJSON'].includes(name)) return null; + if (name === 'noTimeout') return makeRpcProxy(wrapper, {...overrides, timeout: Infinity}); + if (name === 'timeout') return (n) => makeRpcProxy(wrapper, {...overrides, timeout: n}); + return (...args) => hsRpcCall(wrapper, name, args, overrides.timeout); + } + }); +} +``` + +`hsRpcCall` returns a native `Promise` whose resolver is stored in `wrapper.pending[iid]`. A `setTimeout` (skipped when `Infinity`) rejects with `"Timed out"`. The sx runtime never calls `hsRpcCall` directly — the proxy does, in JS, because tests read `.then(...)` on the returned promise. + +**Why this split**: the test fixtures all go through `.then`/`.catch` in JS, so the proxy and its promise plumbing *must* be real JS. The SX runtime only wires up message routing. + +### 4c. Inbound dispatch pipeline + +`onmessage` in the runtime (SX): + +``` +(fn (event) + (let ((data (host-get event "data"))) + (let ((parsed (hs-try-json-parse data))) + (cond + ;; RPC reply: has :iid + ((and parsed (host-get parsed "iid")) + (hs-socket-resolve-rpc! wrapper parsed)) + ;; User 'on message as JSON' + (user-handler-json? + (if parsed + (user-handler parsed) + (error "Received non-JSON message"))) + ;; User 'on message' plain + (user-handler + (user-handler event)) + (true nil))))) +``` + +`hs-socket-resolve-rpc!` pulls the resolver/rejecter pair out of `:pending` by iid and calls the JS-side `resolve(ret)` or `reject(thrown)`. + +### 4d. `send NAME(a:1) to Foo` integration + +Today `emit-send` in compiler.sx always emits `dom-dispatch`. We widen it to `hs-dispatch-to` (new runtime helper) which dispatches: +- If target exposes `_hs_socket? === true` → JSON-encode + `ws.send`. +- Else → falls through to `dom-dispatch` (existing behaviour). + +This keeps backwards compat — all non-socket send targets continue to use DOM events. + +### 4e. `call Foo.rpc.greet("world")` integration + +No compiler changes needed. `call` already compiles property+method chains via `method-call`. `Foo.rpc.greet` resolves to the proxy method which returns a Promise. Hyperscript's existing await semantics (via `io-*` suspension or `then` absorption in `result`) then wait on it. For the non-Playwright test harness, the promise is resolved synchronously inside the mock's `_driveAsync` loop — see §5. + +### 4f. IO suspension + +Sockets do **not** need a new `io-*` operation for the basic tests: `onmessage` is a callback that fires user cmd-lists directly via `host-callback` → `K.callFn` → `_driveAsync` (this already works for `hs-on-every`). RPC promise resolution similarly runs inside a native callback. The only place we might want IO suspension is if someone writes `wait for message in Foo` — **not in this cluster**. Keep `hs-socket-register!` synchronous and callback-driven; defer any `(perform …)` integration to a later cluster. + +## 5. Test mock strategy + +Minimum surface in `tests/hs-run-filtered.js`: + +### 5a. `WebSocket` mock (test-keyed) + +Each test self-describes via its test name. We key behaviour off `_current-test-name`. A single `globalThis.WebSocket` constructor looks up the script-driven scenario. But the fixture's tests all re-mock `window.WebSocket` in-test — they are self-contained. That means **the mock runner merely needs a default that records constructions and lets test code override it**. + +```js +globalThis.WebSocket = function(url) { + const sock = { url, onmessage: null, _listeners: {}, + send(msg){ sock._sent = sock._sent || []; sock._sent.push(msg); }, + addEventListener(t, h){ (sock._listeners[t]=sock._listeners[t]||[]).push(h); }, + close(){} }; + globalThis.__hs_ws_created = globalThis.__hs_ws_created || []; + globalThis.__hs_ws_created.push(sock); + return sock; +}; +``` + +### 5b. `_hs_make_rpc_proxy` in the mock + +Ship the factory from §4b as a top-level function in `hs-run-filtered.js`. It must use the test-runtime `Promise` (real JS Promise), and `setTimeout` (real host — Node's, already global). Timeout rejection must fire inside `_driveAsync`'s execution window; a short `setImmediate`/microtask tick suffices because the tests all `await evaluate(() => new Promise(resolve => …))`. + +### 5c. Generator patterns (`tests/playwright/generate-sx-tests.py`) + +The socket fixtures are unusual: they directly evaluate browser-side JS that attaches a `