318 lines
21 KiB
Markdown
318 lines
21 KiB
Markdown
# E36 — WebSocket + `socket` feature + RPC proxy
|
||
|
||
Design for cluster 36 of `plans/hs-conformance-to-100.md`. Scope: +16 tests in `hs-upstream-socket`, all currently `SKIP (untranslated)`.
|
||
|
||
## 1. Failing tests (all SKIP'd — untranslated)
|
||
|
||
Exact names from `spec/tests/test-hyperscript-behavioral.sx:11354`:
|
||
|
||
1. `converts relative URL to ws:// on http pages`
|
||
2. `converts relative URL to wss:// on https pages`
|
||
3. `dispatchEvent sends JSON-encoded event over the socket`
|
||
4. `namespaced sockets work`
|
||
5. `on message as JSON handler decodes JSON payload`
|
||
6. `on message as JSON throws on non-JSON payload`
|
||
7. `on message handler fires on incoming text message`
|
||
8. `parses socket with absolute ws:// URL`
|
||
9. `rpc proxy blacklists then/catch/length/toJSON`
|
||
10. `rpc proxy default timeout rejects the promise`
|
||
11. `rpc proxy noTimeout avoids timeout rejection`
|
||
12. `rpc proxy reply with throw rejects the promise`
|
||
13. `rpc proxy sends a message and resolves the reply`
|
||
14. `rpc proxy timeout(n) rejects after a custom window`
|
||
15. `rpc reconnects after the underlying socket closes`
|
||
16. `with timeout parses and uses the configured timeout`
|
||
|
||
## 2. Upstream semantics (stock _hyperscript)
|
||
|
||
Verified against the JSON fixture and `hyperscript.org/features/socket/`. The prompt's "`with proxy { send, receive }`" wording is **not** part of the upstream socket feature — there is no `with proxy` clause. The shape actually under test is:
|
||
|
||
```
|
||
socket <Name[.dotted.path]> <url> [with timeout <ms>]
|
||
[on message [as JSON]
|
||
<cmd-list>]
|
||
end
|
||
```
|
||
|
||
**URL scheme normalisation.** Bare paths get a scheme prepended from `location.protocol`: `http:` → `ws://host[:port]<path>`, `https:` → `wss://…`. Absolute `ws://` / `wss://` URLs pass through unchanged.
|
||
|
||
**Named binding.** `socket Foo url end` assigns `window.Foo = socketObj`. A dotted name `socket MyApp.chat url end` walks/creates `window.MyApp` then sets `window.MyApp.chat = socketObj`.
|
||
|
||
**Socket object shape** (what the tests assert against):
|
||
```
|
||
{ raw: WebSocket, // underlying ws
|
||
rpc: <Proxy>, // see below
|
||
dispatchEvent(evt), // JSON-encodes the event and sends over ws
|
||
// 'on message' handler registered internally via ws.onmessage
|
||
}
|
||
```
|
||
|
||
**`dispatchEvent` protocol.** On `send NAME(a:1) to Foo` or `Foo.dispatchEvent(evt)`:
|
||
- Outgoing payload: `JSON.stringify({ type: evt.type, ...evt.detail })` with `sender` and `_namedArgList_` stripped.
|
||
|
||
**`on message [as JSON]`.** `ws.onmessage({data})` fires the cmd-list. `as JSON` calls `JSON.parse(data)` first and the parsed object becomes the event; non-JSON throws `"Received non-JSON message"`.
|
||
|
||
**RPC proxy (`socket.rpc`).** ES6 `Proxy` over an object. Accessing `.anyName` normally returns a callable; four property names are blacklisted and return `null`: `then`, `catch`, `length`, `toJSON` (so the proxy isn't mistaken for a thenable and JSON-serialises cleanly). Two reserved chainable properties:
|
||
- `.noTimeout` — returns a proxy that will never time out.
|
||
- `.timeout(n)` — returns a proxy that times out after `n` ms.
|
||
|
||
Calling `proxy.fnName(arg1, arg2)` performs:
|
||
1. Allocate a unique `iid` (string).
|
||
2. Send `JSON.stringify({iid, function: "fnName", args: [arg1, arg2]})` over ws.
|
||
3. Return a Promise. The promise is tracked by `iid`.
|
||
4. Incoming messages with matching `iid` are routed: `{iid, return: v}` resolves with `v`; `{iid, throw: e}` rejects with `e`.
|
||
5. Default/custom timeout (from `with timeout N`, default per stock is ~0 i.e. "Timed out" reject; override via `.timeout(n)` or `.noTimeout`).
|
||
|
||
**Reconnect.** When the underlying socket fires `close`, the socket wrapper marks itself dead. The *next* RPC (or `dispatchEvent`) call lazily reconstructs a fresh `new WebSocket(url)`. Test 15 asserts `created.length === 2` after one close + one RPC.
|
||
|
||
## 3. Proposed user-facing shape
|
||
|
||
### 3a. Tokens
|
||
|
||
No new tokens needed. `socket` is a bare identifier that becomes a keyword in `parse-feat`. `end` is already known. `with`, `timeout`, `on`, `message`, `as`, `JSON` are already existing keyword/identifier tokens. Dotted names (`MyApp.chat`) already tokenise as `local` + `dot` + `local`.
|
||
|
||
### 3b. Parse tree
|
||
|
||
A new feature AST node, peer to `(def ...)` / `(on ...)`:
|
||
|
||
```
|
||
(socket <name-path> <url-expr> :timeout <expr-or-nil> :on-message <handler-or-nil>)
|
||
```
|
||
|
||
- `<name-path>` is a list of symbol strings: `["Foo"]` or `["MyApp" "chat"]`.
|
||
- `<url-expr>` is any parseable expression (usually a string literal).
|
||
- `:timeout` is the parsed ms expression, or `nil` if absent.
|
||
- `:on-message` is `nil`, `(on-message :json? false <body>)`, or `(on-message :json? true <body>)`.
|
||
|
||
Parser function: `parse-socket-feat` dispatched from `parse-feat` on `val == "socket"`. Reuses `parse-cmd-list` for handler bodies. `as JSON` is detected inline before collecting the body.
|
||
|
||
### 3c. Compile output
|
||
|
||
The compiler emits a call to a runtime registration primitive that carries everything the runtime needs:
|
||
|
||
```
|
||
(hs-socket-register!
|
||
(list "MyApp" "chat") ; name path
|
||
<url-compiled> ; usually a string
|
||
<timeout-ms-or-nil> ; number or nil
|
||
(fn (event) <on-message-body>) ; handler closure or nil
|
||
:json-message? true|false)
|
||
```
|
||
|
||
The runtime owns all WebSocket construction, URL normalisation, window binding, promise/timeout bookkeeping, and the RPC proxy — keeping the compiler output small and re-usable.
|
||
|
||
### 3d. Worked example
|
||
|
||
Source (from fixture test 13, simplified):
|
||
```
|
||
socket RpcSocket ws://localhost/ws end
|
||
```
|
||
|
||
Tokens: `socket` `RpcSocket` `ws` `:` `//localhost/ws` `end` (URL tokenising is already handled by the string-like path reader; see how `fetch` URLs parse today).
|
||
|
||
AST:
|
||
```
|
||
(socket ("RpcSocket") "ws://localhost/ws" :timeout nil :on-message nil)
|
||
```
|
||
|
||
Compiled SX (emitted into the feature body):
|
||
```
|
||
(hs-socket-register!
|
||
(list "RpcSocket")
|
||
"ws://localhost/ws"
|
||
nil
|
||
nil
|
||
:json-message? false)
|
||
```
|
||
|
||
Fuller example:
|
||
```
|
||
socket MyApp.chat /ws with timeout 1500
|
||
on message as JSON
|
||
set window.got to the event
|
||
end
|
||
```
|
||
→
|
||
```
|
||
(socket ("MyApp" "chat") "/ws"
|
||
:timeout 1500
|
||
:on-message (on-message :json? true
|
||
((set! window.got event))))
|
||
```
|
||
→
|
||
```
|
||
(hs-socket-register!
|
||
(list "MyApp" "chat")
|
||
"/ws"
|
||
1500
|
||
(fn (event) (host-set! (host-global "window") "got" event))
|
||
:json-message? true)
|
||
```
|
||
|
||
## 4. Runtime architecture
|
||
|
||
All in `lib/hyperscript/runtime.sx`. One big primitive plus small helpers.
|
||
|
||
### 4a. `hs-socket-register!` — lifecycle
|
||
|
||
1. **Normalise URL**: if starts with `ws://`/`wss://` use as-is; else derive scheme from `(host-get (host-global "location") "protocol")` and prepend `ws://host[:port]` or `wss://…`. Preserves path.
|
||
2. **Construct** `(host-new "WebSocket" url)`.
|
||
3. **Build wrapper object** (SX dict): `{:raw ws :url url :timeout <ms> :pending <dict-of-iid->resolver> :handler <closure> :json? <bool> :closed? false}`.
|
||
4. **Install `ws.onmessage`** via `host-set!` to a native callback (`host-callback`) that:
|
||
- Reads `data = event.data`.
|
||
- If JSON-looking (starts with `{` or `[`) tries `json-parse`; if parse OK and has `iid` field → RPC reply dispatch (see 4c).
|
||
- Otherwise → fire user's `on message` handler with `event` (optionally JSON-parsing first if `:json? true`; non-JSON there → `(error "Received non-JSON message")`).
|
||
5. **Install `ws.addEventListener("close", …)`** (via `host-call`) that sets `:closed? true` so next RPC reconnects.
|
||
6. **Install `.dispatchEvent`** method on wrapper via `host-set!`: takes an `Event`, builds `{type, ...detail minus sender,_namedArgList_}`, `JSON.stringify`s, `ws.send`s.
|
||
7. **Install `.rpc`** — the proxy (see 4b).
|
||
8. **Bind** on `window`: walk `name-path` creating intermediate `{}` dicts as needed; final segment assigned the wrapper.
|
||
|
||
### 4b. RPC proxy
|
||
|
||
Implemented as a native JS object constructed via a new FFI primitive we already have: `host-call` to a global helper we register **inside the test mock and inside the real runtime**. Concretely the runtime calls `(host-call (host-global "_hs_make_rpc_proxy") "call" nil wrapper)` → returns a `Proxy`. In prod `_hs_make_rpc_proxy` is shipped in `shared/static/scripts/sx-platform-*.js`; in the test mock it's inlined into `tests/hs-run-filtered.js`.
|
||
|
||
Pseudo-JS for the proxy factory (this is what lives in the platform/mock, not in `.sx`):
|
||
|
||
```js
|
||
function makeRpcProxy(wrapper, overrides = {}){
|
||
return new Proxy({}, {
|
||
get(_, name) {
|
||
if (['then','catch','length','toJSON'].includes(name)) return null;
|
||
if (name === 'noTimeout') return makeRpcProxy(wrapper, {...overrides, timeout: Infinity});
|
||
if (name === 'timeout') return (n) => makeRpcProxy(wrapper, {...overrides, timeout: n});
|
||
return (...args) => hsRpcCall(wrapper, name, args, overrides.timeout);
|
||
}
|
||
});
|
||
}
|
||
```
|
||
|
||
`hsRpcCall` returns a native `Promise` whose resolver is stored in `wrapper.pending[iid]`. A `setTimeout` (skipped when `Infinity`) rejects with `"Timed out"`. The sx runtime never calls `hsRpcCall` directly — the proxy does, in JS, because tests read `.then(...)` on the returned promise.
|
||
|
||
**Why this split**: the test fixtures all go through `.then`/`.catch` in JS, so the proxy and its promise plumbing *must* be real JS. The SX runtime only wires up message routing.
|
||
|
||
### 4c. Inbound dispatch pipeline
|
||
|
||
`onmessage` in the runtime (SX):
|
||
|
||
```
|
||
(fn (event)
|
||
(let ((data (host-get event "data")))
|
||
(let ((parsed (hs-try-json-parse data)))
|
||
(cond
|
||
;; RPC reply: has :iid
|
||
((and parsed (host-get parsed "iid"))
|
||
(hs-socket-resolve-rpc! wrapper parsed))
|
||
;; User 'on message as JSON'
|
||
(user-handler-json?
|
||
(if parsed
|
||
(user-handler parsed)
|
||
(error "Received non-JSON message")))
|
||
;; User 'on message' plain
|
||
(user-handler
|
||
(user-handler event))
|
||
(true nil)))))
|
||
```
|
||
|
||
`hs-socket-resolve-rpc!` pulls the resolver/rejecter pair out of `:pending` by iid and calls the JS-side `resolve(ret)` or `reject(thrown)`.
|
||
|
||
### 4d. `send NAME(a:1) to Foo` integration
|
||
|
||
Today `emit-send` in compiler.sx always emits `dom-dispatch`. We widen it to `hs-dispatch-to` (new runtime helper) which dispatches:
|
||
- If target exposes `_hs_socket? === true` → JSON-encode + `ws.send`.
|
||
- Else → falls through to `dom-dispatch` (existing behaviour).
|
||
|
||
This keeps backwards compat — all non-socket send targets continue to use DOM events.
|
||
|
||
### 4e. `call Foo.rpc.greet("world")` integration
|
||
|
||
No compiler changes needed. `call` already compiles property+method chains via `method-call`. `Foo.rpc.greet` resolves to the proxy method which returns a Promise. Hyperscript's existing await semantics (via `io-*` suspension or `then` absorption in `result`) then wait on it. For the non-Playwright test harness, the promise is resolved synchronously inside the mock's `_driveAsync` loop — see §5.
|
||
|
||
### 4f. IO suspension
|
||
|
||
Sockets do **not** need a new `io-*` operation for the basic tests: `onmessage` is a callback that fires user cmd-lists directly via `host-callback` → `K.callFn` → `_driveAsync` (this already works for `hs-on-every`). RPC promise resolution similarly runs inside a native callback. The only place we might want IO suspension is if someone writes `wait for message in Foo` — **not in this cluster**. Keep `hs-socket-register!` synchronous and callback-driven; defer any `(perform …)` integration to a later cluster.
|
||
|
||
## 5. Test mock strategy
|
||
|
||
Minimum surface in `tests/hs-run-filtered.js`:
|
||
|
||
### 5a. `WebSocket` mock (test-keyed)
|
||
|
||
Each test self-describes via its test name. We key behaviour off `_current-test-name`. A single `globalThis.WebSocket` constructor looks up the script-driven scenario. But the fixture's tests all re-mock `window.WebSocket` in-test — they are self-contained. That means **the mock runner merely needs a default that records constructions and lets test code override it**.
|
||
|
||
```js
|
||
globalThis.WebSocket = function(url) {
|
||
const sock = { url, onmessage: null, _listeners: {},
|
||
send(msg){ sock._sent = sock._sent || []; sock._sent.push(msg); },
|
||
addEventListener(t, h){ (sock._listeners[t]=sock._listeners[t]||[]).push(h); },
|
||
close(){} };
|
||
globalThis.__hs_ws_created = globalThis.__hs_ws_created || [];
|
||
globalThis.__hs_ws_created.push(sock);
|
||
return sock;
|
||
};
|
||
```
|
||
|
||
### 5b. `_hs_make_rpc_proxy` in the mock
|
||
|
||
Ship the factory from §4b as a top-level function in `hs-run-filtered.js`. It must use the test-runtime `Promise` (real JS Promise), and `setTimeout` (real host — Node's, already global). Timeout rejection must fire inside `_driveAsync`'s execution window; a short `setImmediate`/microtask tick suffices because the tests all `await evaluate(() => new Promise(resolve => …))`.
|
||
|
||
### 5c. Generator patterns (`tests/playwright/generate-sx-tests.py`)
|
||
|
||
The socket fixtures are unusual: they directly evaluate browser-side JS that attaches a `<script type="text/hyperscript">` then calls `_hyperscript.processNode`. We do **not** try to translate that path verbatim. Instead the generator gets a `_socket_setup_ops` recogniser (peer to `_hs_config_setup_ops`) that:
|
||
|
||
1. Recognises the pattern `var OrigWS = window.WebSocket; … window.WebSocket = function() {...}; … script.textContent = 'socket …'; …`.
|
||
2. Extracts the inner hyperscript source from the `script.textContent` string.
|
||
3. Emits an SX test body that:
|
||
- Pre-installs a matching WS mock (`host-set! window "WebSocket" ...`) — parameterised by which behaviour the test needs: (a) record URL, (b) record sent messages, (c) record close handlers, (d) provide an `onmessage` hook.
|
||
- Activates the hyperscript on a `div` (so the socket feature runs through the existing `hs-activate!` pipeline).
|
||
- Invokes whatever the test asserted against: reads `urls[0]`, calls `.dispatchEvent`, drives `mockSocket.onmessage(…)`, etc.
|
||
- Asserts the final value (translated from `expect(…).toBe(…)` via the existing `pw_assertion_to_sx`).
|
||
|
||
A single shared **socket test mini-DSL** — 5 behaviour codes — covers all 16 tests:
|
||
|
||
| Code | Records | Injects | Asserts |
|
||
|------|---------|---------|---------|
|
||
| `url-capture` | constructor URL | — | URL string |
|
||
| `send-capture` | outgoing frames | — | JSON payload fields |
|
||
| `inbound-text` | — | `mockSocket.onmessage({data:"…"})` | window flag set |
|
||
| `inbound-json` | — | `mockSocket.onmessage({data:JSON…})` | window flag set |
|
||
| `rpc-roundtrip` | outgoing frames | onmessage with `{iid,return:v}` or `{iid,throw:e}` | promise resolves/rejects |
|
||
| `rpc-timeout` | — | timer advance | promise rejects with `"Timed out"` |
|
||
| `close-reconnect` | close handlers + construction count | fire all close handlers then call rpc | `created.length === 2` |
|
||
|
||
The generator dispatches tests 1,2,8 → `url-capture`; 3 → `send-capture`; 5,6,7 → inbound; 10,11,12,13,14 → rpc; 15 → close-reconnect; 4,9,16 → small probes of wrapper shape (`.rpc typeof === "object"`, namespaced binding exists).
|
||
|
||
## 6. Test delta estimate
|
||
|
||
- **Best case:** 16/16. All tests are self-contained scenarios; with the WebSocket mock, proxy factory, and generator recogniser they all translate.
|
||
- **Realistic:** **12–14**. Timeout/reconnect tests (10, 11, 14, 15) need real-time behaviour from the Node mock; if `setTimeout` behaviour is flaky under the step-limited driver, they may go partial.
|
||
- **Worst case:** **8**. Only the synchronous URL/shape/inbound tests land (1, 2, 4, 5, 6, 7, 8, 9, 16); the five full RPC round-trip tests stay SKIP and get handled in a follow-up.
|
||
|
||
Target commit should aim for **+12 tests, partial on RPC timeout/reconnect**. Mark any stragglers `partial` per rule 7.
|
||
|
||
## 7. Risks / open questions (human decisions)
|
||
|
||
1. **Reconnect scope.** The fixture only tests lazy reconnect on next RPC. Do we also reconnect on `dispatchEvent` after close? (Stock yes; tests don't cover it. Decision: yes, keep parity.)
|
||
2. **Buffering while disconnected.** Between `close` and first reconnecting call, does `dispatchEvent` drop the frame or queue it? Stock *drops*. Recommendation: drop, log once.
|
||
3. **Binary frames.** None of the 16 tests exercise `Blob`/`ArrayBuffer`. Explicitly out of scope — document as `not yet supported` in runtime.
|
||
4. **Close-code handling.** `close(code, reason)` / CloseEvent — no test exercises it. Recommendation: ignore code/reason, just mark closed.
|
||
5. **Interaction with IO suspension.** RPC promises live in JS, not SX; a hyperscript `call Foo.rpc.x()` under the current runtime returns the JS Promise and hyperscript's `then` absorption takes over. Verify that the CEK/VM `hs-safe-call` path doesn't eagerly `(host-await)`.
|
||
6. **Proxy implementation language.** The proxy *must* be JS (ES6 `Proxy`) — SX has no equivalent and the tests read `.then` etc. as literal JS property accesses. Means the proxy factory lives **both** in the test mock and in a shipped platform JS file. Do we want to factor out a shared `platform/sockets.js`? (Recommendation: yes, land it in `shared/static/scripts/sx-platform-sockets.js` in a follow-up commit after the mock lands; the mock inline version stays as the reference.)
|
||
7. **Namespace dotted binding collisions.** If `window.MyApp` already exists as a non-dict, the runtime should refuse overwriting it (tests only use fresh names). Recommendation: silently overwrite with a dict — matches stock.
|
||
8. **`emit-send` widening.** Changing send to branch on `_hs_socket?` touches a code path used by every existing send test. Needs a smoke-regression check against `hs-upstream-send` before landing.
|
||
|
||
## 8. Implementation checklist (one commit per step; must land +N)
|
||
|
||
Each step is scoped to be independently testable. `sync-wasm` means `cp lib/hyperscript/X.sx shared/static/wasm/sx/hs-X.sx`.
|
||
|
||
1. **Mock groundwork** *(0 tests)*. Add `WebSocket` constructor + `_hs_make_rpc_proxy` factory to `tests/hs-run-filtered.js`. No runtime/compiler changes. Verify no regressions on smoke 0–195. Commit: `HS-prep: WebSocket + RPC proxy mock`.
|
||
2. **Parser: `socket` feature** *(0 tests)*. Add `parse-socket-feat` in `parser.sx` dispatched from `parse-feat`. Parses name-path, URL, `with timeout N`, `on message [as JSON] … end`, trailing `end`. Emits `(socket …)` AST. Unit-verify via `hs-compile` eval trace. sync-wasm. Commit: `HS: parse socket feature`.
|
||
3. **Compiler + runtime: basic socket registration** *(+3 tests: 1, 2, 8)*. Implement `hs-socket-register!` runtime primitive (URL normalise, construct WS, window bind — **no** RPC/on-message yet). Compiler emits `hs-socket-register!` call from `(socket …)`. sync-wasm. Land tests: `converts relative URL to ws://`, `…to wss://`, `parses socket with absolute ws:// URL`. Commit: `HS: socket URL binding (+3)`.
|
||
4. **Namespaced binding + wrapper shape** *(+2 tests: 4, 16)*. Walk name-path assigning intermediate dicts. Ensure wrapper exposes `.raw` and `.rpc` objects (rpc can be the bare proxy for now). sync-wasm. Land: `namespaced sockets work`, `with timeout parses and uses the configured timeout`. Commit: `HS: socket namespaced names + timeout plumbing (+2)`.
|
||
5. **Inbound `on message` dispatch** *(+3 tests: 5, 6, 7)*. Wire `ws.onmessage` → SX handler. Support `as JSON` parse branch + non-JSON error. Generator: `inbound-text` + `inbound-json` dispatch codes. sync-wasm. Land: `on message handler fires…`, `on message as JSON … decodes…`, `…throws on non-JSON`. Commit: `HS: socket on-message + as JSON (+3)`.
|
||
6. **RPC proxy — happy path** *(+2 tests: 9, 13)*. Wire `.rpc` proxy with blacklist, `iid` allocation, send JSON, resolve on matching `iid.return`. Uses real JS `Promise` from the mock. Generator `rpc-roundtrip` code. sync-wasm. Land: `rpc proxy blacklists …`, `rpc proxy sends a message and resolves…`. Commit: `HS: socket RPC proxy (+2)`.
|
||
7. **RPC error + reconnect** *(+3 tests: 12, 15, plus `dispatchEvent`=3)*. Add `{iid, throw: e}` path. Add close-handler tracking + lazy reconnect on next RPC/dispatch. Add `dispatchEvent` JSON-encode (strips `sender`/`_namedArgList_`). Generator `close-reconnect` + `send-capture`. sync-wasm. Land: `rpc proxy reply with throw rejects`, `rpc reconnects after … closes`, `dispatchEvent sends JSON-encoded event`. Commit: `HS: socket dispatchEvent + reconnect + throw (+3)`.
|
||
8. **RPC timeouts** *(+3 tests: 10, 11, 14)*. `.timeout(n)` / `.noTimeout` chainable; default timeout from `with timeout N`. `setTimeout` reject with `"Timed out"`. Generator `rpc-timeout` code — may need a virtual-clock helper in the mock if real `setTimeout` is flaky under the step limiter. sync-wasm. Land: `…default timeout rejects`, `…noTimeout avoids timeout rejection`, `…timeout(n) rejects after a custom window`. Commit: `HS: socket RPC timeouts (+3)`.
|
||
9. **Polish + scoreboard** *(0 tests)*. Update `plans/hs-conformance-to-100.md` row 36 to `done (+N)`, bump scoreboard. Smoke 0–195 + `hs-upstream-send` regression check. Commit: `HS-plan: E36 complete (+N)`.
|
||
|
||
Total runway: **8 feature commits** to move cluster 36 from `blocked` → `done (+12–16)`. No commit may land a regression in smoke 0–195 or in `hs-upstream-send`.
|