# E36 — WebSocket + `socket` feature + RPC proxy

Design for cluster 36 of `plans/hs-conformance-to-100.md`. Scope: +16 tests in `hs-upstream-socket`, all currently `SKIP (untranslated)`.

## 1. Failing tests (all SKIP'd — untranslated)

Exact names from `spec/tests/test-hyperscript-behavioral.sx:11354`:

1. `converts relative URL to ws:// on http pages`
2. `converts relative URL to wss:// on https pages`
3. `dispatchEvent sends JSON-encoded event over the socket`
4. `namespaced sockets work`
5. `on message as JSON handler decodes JSON payload`
6. `on message as JSON throws on non-JSON payload`
7. `on message handler fires on incoming text message`
8. `parses socket with absolute ws:// URL`
9. `rpc proxy blacklists then/catch/length/toJSON`
10. `rpc proxy default timeout rejects the promise`
11. `rpc proxy noTimeout avoids timeout rejection`
12. `rpc proxy reply with throw rejects the promise`
13. `rpc proxy sends a message and resolves the reply`
14. `rpc proxy timeout(n) rejects after a custom window`
15. `rpc reconnects after the underlying socket closes`
16. `with timeout parses and uses the configured timeout`

## 2. Upstream semantics (stock _hyperscript)

Verified against the JSON fixture and `hyperscript.org/features/socket/`. The prompt's "`with proxy { send, receive }`" wording is **not** part of the upstream socket feature — there is no `with proxy` clause. The shape actually under test is:

```
socket <Name[.dotted.path]> <url> [with timeout <ms>]
  [on message [as JSON]
     <cmd-list>]
end
```

**URL scheme normalisation.** Bare paths get a scheme prepended from `location.protocol`: `http:` → `ws://host[:port]<path>`, `https:` → `wss://…`. Absolute `ws://` / `wss://` URLs pass through unchanged.

**Named binding.** `socket Foo url end` assigns `window.Foo = socketObj`. A dotted name `socket MyApp.chat url end` walks/creates `window.MyApp` then sets `window.MyApp.chat = socketObj`.

**Socket object shape** (what the tests assert against):
```
{ raw: WebSocket,           // underlying ws
  rpc: <Proxy>,              // see below
  dispatchEvent(evt),        // JSON-encodes the event and sends over ws
  // 'on message' handler registered internally via ws.onmessage
}
```

**`dispatchEvent` protocol.** On `send NAME(a:1) to Foo` or `Foo.dispatchEvent(evt)`:
- Outgoing payload: `JSON.stringify({ type: evt.type, ...evt.detail })` with `sender` and `_namedArgList_` stripped.

**`on message [as JSON]`.** `ws.onmessage({data})` fires the cmd-list. `as JSON` calls `JSON.parse(data)` first and the parsed object becomes the event; non-JSON throws `"Received non-JSON message"`.

**RPC proxy (`socket.rpc`).** ES6 `Proxy` over an object. Accessing `.anyName` normally returns a callable; four property names are blacklisted and return `null`: `then`, `catch`, `length`, `toJSON` (so the proxy isn't mistaken for a thenable and JSON-serialises cleanly). Two reserved chainable properties:
- `.noTimeout` — returns a proxy that will never time out.
- `.timeout(n)` — returns a proxy that times out after `n` ms.

Calling `proxy.fnName(arg1, arg2)` performs:
1. Allocate a unique `iid` (string).
2. Send `JSON.stringify({iid, function: "fnName", args: [arg1, arg2]})` over ws.
3. Return a Promise. The promise is tracked by `iid`.
4. Incoming messages with matching `iid` are routed: `{iid, return: v}` resolves with `v`; `{iid, throw: e}` rejects with `e`.
5. Default/custom timeout (from `with timeout N`, default per stock is ~0 i.e. "Timed out" reject; override via `.timeout(n)` or `.noTimeout`).

**Reconnect.** When the underlying socket fires `close`, the socket wrapper marks itself dead. The *next* RPC (or `dispatchEvent`) call lazily reconstructs a fresh `new WebSocket(url)`. Test 15 asserts `created.length === 2` after one close + one RPC.

## 3. Proposed user-facing shape

### 3a. Tokens

No new tokens needed. `socket` is a bare identifier that becomes a keyword in `parse-feat`. `end` is already known. `with`, `timeout`, `on`, `message`, `as`, `JSON` are already existing keyword/identifier tokens. Dotted names (`MyApp.chat`) already tokenise as `local` + `dot` + `local`.

### 3b. Parse tree

A new feature AST node, peer to `(def ...)` / `(on ...)`:

```
(socket <name-path> <url-expr> :timeout <expr-or-nil> :on-message <handler-or-nil>)
```

- `<name-path>` is a list of symbol strings: `["Foo"]` or `["MyApp" "chat"]`.
- `<url-expr>` is any parseable expression (usually a string literal).
- `:timeout` is the parsed ms expression, or `nil` if absent.
- `:on-message` is `nil`, `(on-message :json? false <body>)`, or `(on-message :json? true <body>)`.

Parser function: `parse-socket-feat` dispatched from `parse-feat` on `val == "socket"`. Reuses `parse-cmd-list` for handler bodies. `as JSON` is detected inline before collecting the body.

### 3c. Compile output

The compiler emits a call to a runtime registration primitive that carries everything the runtime needs:

```
(hs-socket-register!
  (list "MyApp" "chat")            ; name path
  <url-compiled>                   ; usually a string
  <timeout-ms-or-nil>              ; number or nil
  (fn (event) <on-message-body>)   ; handler closure or nil
  :json-message? true|false)
```

The runtime owns all WebSocket construction, URL normalisation, window binding, promise/timeout bookkeeping, and the RPC proxy — keeping the compiler output small and re-usable.

### 3d. Worked example

Source (from fixture test 13, simplified):
```
socket RpcSocket ws://localhost/ws end
```

Tokens: `socket` `RpcSocket` `ws` `:` `//localhost/ws` `end` (URL tokenising is already handled by the string-like path reader; see how `fetch` URLs parse today).

AST:
```
(socket ("RpcSocket") "ws://localhost/ws" :timeout nil :on-message nil)
```

Compiled SX (emitted into the feature body):
```
(hs-socket-register!
  (list "RpcSocket")
  "ws://localhost/ws"
  nil
  nil
  :json-message? false)
```

Fuller example:
```
socket MyApp.chat /ws with timeout 1500
  on message as JSON
    set window.got to the event
end
```
→
```
(socket ("MyApp" "chat") "/ws"
        :timeout 1500
        :on-message (on-message :json? true
                                ((set! window.got event))))
```
→
```
(hs-socket-register!
  (list "MyApp" "chat")
  "/ws"
  1500
  (fn (event) (host-set! (host-global "window") "got" event))
  :json-message? true)
```

## 4. Runtime architecture

All in `lib/hyperscript/runtime.sx`. One big primitive plus small helpers.

### 4a. `hs-socket-register!` — lifecycle

1. **Normalise URL**: if starts with `ws://`/`wss://` use as-is; else derive scheme from `(host-get (host-global "location") "protocol")` and prepend `ws://host[:port]` or `wss://…`. Preserves path.
2. **Construct** `(host-new "WebSocket" url)`.
3. **Build wrapper object** (SX dict): `{:raw ws :url url :timeout <ms> :pending <dict-of-iid->resolver> :handler <closure> :json? <bool> :closed? false}`.
4. **Install `ws.onmessage`** via `host-set!` to a native callback (`host-callback`) that:
   - Reads `data = event.data`.
   - If JSON-looking (starts with `{` or `[`) tries `json-parse`; if parse OK and has `iid` field → RPC reply dispatch (see 4c).
   - Otherwise → fire user's `on message` handler with `event` (optionally JSON-parsing first if `:json? true`; non-JSON there → `(error "Received non-JSON message")`).
5. **Install `ws.addEventListener("close", …)`** (via `host-call`) that sets `:closed? true` so next RPC reconnects.
6. **Install `.dispatchEvent`** method on wrapper via `host-set!`: takes an `Event`, builds `{type, ...detail minus sender,_namedArgList_}`, `JSON.stringify`s, `ws.send`s.
7. **Install `.rpc`** — the proxy (see 4b).
8. **Bind** on `window`: walk `name-path` creating intermediate `{}` dicts as needed; final segment assigned the wrapper.

### 4b. RPC proxy

Implemented as a native JS object constructed via a new FFI primitive we already have: `host-call` to a global helper we register **inside the test mock and inside the real runtime**. Concretely the runtime calls `(host-call (host-global "_hs_make_rpc_proxy") "call" nil wrapper)` → returns a `Proxy`. In prod `_hs_make_rpc_proxy` is shipped in `shared/static/scripts/sx-platform-*.js`; in the test mock it's inlined into `tests/hs-run-filtered.js`.

Pseudo-JS for the proxy factory (this is what lives in the platform/mock, not in `.sx`):

```js
function makeRpcProxy(wrapper, overrides = {}){
  return new Proxy({}, {
    get(_, name) {
      if (['then','catch','length','toJSON'].includes(name)) return null;
      if (name === 'noTimeout') return makeRpcProxy(wrapper, {...overrides, timeout: Infinity});
      if (name === 'timeout')   return (n) => makeRpcProxy(wrapper, {...overrides, timeout: n});
      return (...args) => hsRpcCall(wrapper, name, args, overrides.timeout);
    }
  });
}
```

`hsRpcCall` returns a native `Promise` whose resolver is stored in `wrapper.pending[iid]`. A `setTimeout` (skipped when `Infinity`) rejects with `"Timed out"`. The sx runtime never calls `hsRpcCall` directly — the proxy does, in JS, because tests read `.then(...)` on the returned promise.

**Why this split**: the test fixtures all go through `.then`/`.catch` in JS, so the proxy and its promise plumbing *must* be real JS. The SX runtime only wires up message routing.

### 4c. Inbound dispatch pipeline

`onmessage` in the runtime (SX):

```
(fn (event)
  (let ((data (host-get event "data")))
    (let ((parsed (hs-try-json-parse data)))
      (cond
        ;; RPC reply: has :iid
        ((and parsed (host-get parsed "iid"))
          (hs-socket-resolve-rpc! wrapper parsed))
        ;; User 'on message as JSON'
        (user-handler-json?
          (if parsed
              (user-handler parsed)
              (error "Received non-JSON message")))
        ;; User 'on message' plain
        (user-handler
          (user-handler event))
        (true nil)))))
```

`hs-socket-resolve-rpc!` pulls the resolver/rejecter pair out of `:pending` by iid and calls the JS-side `resolve(ret)` or `reject(thrown)`.

### 4d. `send NAME(a:1) to Foo` integration

Today `emit-send` in compiler.sx always emits `dom-dispatch`. We widen it to `hs-dispatch-to` (new runtime helper) which dispatches:
- If target exposes `_hs_socket? === true` → JSON-encode + `ws.send`.
- Else → falls through to `dom-dispatch` (existing behaviour).

This keeps backwards compat — all non-socket send targets continue to use DOM events.

### 4e. `call Foo.rpc.greet("world")` integration

No compiler changes needed. `call` already compiles property+method chains via `method-call`. `Foo.rpc.greet` resolves to the proxy method which returns a Promise. Hyperscript's existing await semantics (via `io-*` suspension or `then` absorption in `result`) then wait on it. For the non-Playwright test harness, the promise is resolved synchronously inside the mock's `_driveAsync` loop — see §5.

### 4f. IO suspension

Sockets do **not** need a new `io-*` operation for the basic tests: `onmessage` is a callback that fires user cmd-lists directly via `host-callback` → `K.callFn` → `_driveAsync` (this already works for `hs-on-every`). RPC promise resolution similarly runs inside a native callback. The only place we might want IO suspension is if someone writes `wait for message in Foo` — **not in this cluster**. Keep `hs-socket-register!` synchronous and callback-driven; defer any `(perform …)` integration to a later cluster.

## 5. Test mock strategy

Minimum surface in `tests/hs-run-filtered.js`:

### 5a. `WebSocket` mock (test-keyed)

Each test self-describes via its test name. We key behaviour off `_current-test-name`. A single `globalThis.WebSocket` constructor looks up the script-driven scenario. But the fixture's tests all re-mock `window.WebSocket` in-test — they are self-contained. That means **the mock runner merely needs a default that records constructions and lets test code override it**.

```js
globalThis.WebSocket = function(url) {
  const sock = { url, onmessage: null, _listeners: {},
                 send(msg){ sock._sent = sock._sent || []; sock._sent.push(msg); },
                 addEventListener(t, h){ (sock._listeners[t]=sock._listeners[t]||[]).push(h); },
                 close(){} };
  globalThis.__hs_ws_created = globalThis.__hs_ws_created || [];
  globalThis.__hs_ws_created.push(sock);
  return sock;
};
```

### 5b. `_hs_make_rpc_proxy` in the mock

Ship the factory from §4b as a top-level function in `hs-run-filtered.js`. It must use the test-runtime `Promise` (real JS Promise), and `setTimeout` (real host — Node's, already global). Timeout rejection must fire inside `_driveAsync`'s execution window; a short `setImmediate`/microtask tick suffices because the tests all `await evaluate(() => new Promise(resolve => …))`.

### 5c. Generator patterns (`tests/playwright/generate-sx-tests.py`)

The socket fixtures are unusual: they directly evaluate browser-side JS that attaches a `<script type="text/hyperscript">` then calls `_hyperscript.processNode`. We do **not** try to translate that path verbatim. Instead the generator gets a `_socket_setup_ops` recogniser (peer to `_hs_config_setup_ops`) that:

1. Recognises the pattern `var OrigWS = window.WebSocket; … window.WebSocket = function() {...}; … script.textContent = 'socket …'; …`.
2. Extracts the inner hyperscript source from the `script.textContent` string.
3. Emits an SX test body that:
   - Pre-installs a matching WS mock (`host-set! window "WebSocket" ...`) — parameterised by which behaviour the test needs: (a) record URL, (b) record sent messages, (c) record close handlers, (d) provide an `onmessage` hook.
   - Activates the hyperscript on a `div` (so the socket feature runs through the existing `hs-activate!` pipeline).
   - Invokes whatever the test asserted against: reads `urls[0]`, calls `.dispatchEvent`, drives `mockSocket.onmessage(…)`, etc.
   - Asserts the final value (translated from `expect(…).toBe(…)` via the existing `pw_assertion_to_sx`).

A single shared **socket test mini-DSL** — 5 behaviour codes — covers all 16 tests:

| Code | Records | Injects | Asserts |
|------|---------|---------|---------|
| `url-capture` | constructor URL | — | URL string |
| `send-capture` | outgoing frames | — | JSON payload fields |
| `inbound-text` | — | `mockSocket.onmessage({data:"…"})` | window flag set |
| `inbound-json` | — | `mockSocket.onmessage({data:JSON…})` | window flag set |
| `rpc-roundtrip` | outgoing frames | onmessage with `{iid,return:v}` or `{iid,throw:e}` | promise resolves/rejects |
| `rpc-timeout` | — | timer advance | promise rejects with `"Timed out"` |
| `close-reconnect` | close handlers + construction count | fire all close handlers then call rpc | `created.length === 2` |

The generator dispatches tests 1,2,8 → `url-capture`; 3 → `send-capture`; 5,6,7 → inbound; 10,11,12,13,14 → rpc; 15 → close-reconnect; 4,9,16 → small probes of wrapper shape (`.rpc typeof === "object"`, namespaced binding exists).

## 6. Test delta estimate

- **Best case:** 16/16. All tests are self-contained scenarios; with the WebSocket mock, proxy factory, and generator recogniser they all translate.
- **Realistic:** **12–14**. Timeout/reconnect tests (10, 11, 14, 15) need real-time behaviour from the Node mock; if `setTimeout` behaviour is flaky under the step-limited driver, they may go partial.
- **Worst case:** **8**. Only the synchronous URL/shape/inbound tests land (1, 2, 4, 5, 6, 7, 8, 9, 16); the five full RPC round-trip tests stay SKIP and get handled in a follow-up.

Target commit should aim for **+12 tests, partial on RPC timeout/reconnect**. Mark any stragglers `partial` per rule 7.

## 7. Risks / open questions (human decisions)

1. **Reconnect scope.** The fixture only tests lazy reconnect on next RPC. Do we also reconnect on `dispatchEvent` after close? (Stock yes; tests don't cover it. Decision: yes, keep parity.)
2. **Buffering while disconnected.** Between `close` and first reconnecting call, does `dispatchEvent` drop the frame or queue it? Stock *drops*. Recommendation: drop, log once.
3. **Binary frames.** None of the 16 tests exercise `Blob`/`ArrayBuffer`. Explicitly out of scope — document as `not yet supported` in runtime.
4. **Close-code handling.** `close(code, reason)` / CloseEvent — no test exercises it. Recommendation: ignore code/reason, just mark closed.
5. **Interaction with IO suspension.** RPC promises live in JS, not SX; a hyperscript `call Foo.rpc.x()` under the current runtime returns the JS Promise and hyperscript's `then` absorption takes over. Verify that the CEK/VM `hs-safe-call` path doesn't eagerly `(host-await)`.
6. **Proxy implementation language.** The proxy *must* be JS (ES6 `Proxy`) — SX has no equivalent and the tests read `.then` etc. as literal JS property accesses. Means the proxy factory lives **both** in the test mock and in a shipped platform JS file. Do we want to factor out a shared `platform/sockets.js`? (Recommendation: yes, land it in `shared/static/scripts/sx-platform-sockets.js` in a follow-up commit after the mock lands; the mock inline version stays as the reference.)
7. **Namespace dotted binding collisions.** If `window.MyApp` already exists as a non-dict, the runtime should refuse overwriting it (tests only use fresh names). Recommendation: silently overwrite with a dict — matches stock.
8. **`emit-send` widening.** Changing send to branch on `_hs_socket?` touches a code path used by every existing send test. Needs a smoke-regression check against `hs-upstream-send` before landing.

## 8. Implementation checklist (one commit per step; must land +N)

Each step is scoped to be independently testable. `sync-wasm` means `cp lib/hyperscript/X.sx shared/static/wasm/sx/hs-X.sx`.

1. **Mock groundwork** *(0 tests)*. Add `WebSocket` constructor + `_hs_make_rpc_proxy` factory to `tests/hs-run-filtered.js`. No runtime/compiler changes. Verify no regressions on smoke 0–195. Commit: `HS-prep: WebSocket + RPC proxy mock`.
2. **Parser: `socket` feature** *(0 tests)*. Add `parse-socket-feat` in `parser.sx` dispatched from `parse-feat`. Parses name-path, URL, `with timeout N`, `on message [as JSON] … end`, trailing `end`. Emits `(socket …)` AST. Unit-verify via `hs-compile` eval trace. sync-wasm. Commit: `HS: parse socket feature`.
3. **Compiler + runtime: basic socket registration** *(+3 tests: 1, 2, 8)*. Implement `hs-socket-register!` runtime primitive (URL normalise, construct WS, window bind — **no** RPC/on-message yet). Compiler emits `hs-socket-register!` call from `(socket …)`. sync-wasm. Land tests: `converts relative URL to ws://`, `…to wss://`, `parses socket with absolute ws:// URL`. Commit: `HS: socket URL binding (+3)`.
4. **Namespaced binding + wrapper shape** *(+2 tests: 4, 16)*. Walk name-path assigning intermediate dicts. Ensure wrapper exposes `.raw` and `.rpc` objects (rpc can be the bare proxy for now). sync-wasm. Land: `namespaced sockets work`, `with timeout parses and uses the configured timeout`. Commit: `HS: socket namespaced names + timeout plumbing (+2)`.
5. **Inbound `on message` dispatch** *(+3 tests: 5, 6, 7)*. Wire `ws.onmessage` → SX handler. Support `as JSON` parse branch + non-JSON error. Generator: `inbound-text` + `inbound-json` dispatch codes. sync-wasm. Land: `on message handler fires…`, `on message as JSON … decodes…`, `…throws on non-JSON`. Commit: `HS: socket on-message + as JSON (+3)`.
6. **RPC proxy — happy path** *(+2 tests: 9, 13)*. Wire `.rpc` proxy with blacklist, `iid` allocation, send JSON, resolve on matching `iid.return`. Uses real JS `Promise` from the mock. Generator `rpc-roundtrip` code. sync-wasm. Land: `rpc proxy blacklists …`, `rpc proxy sends a message and resolves…`. Commit: `HS: socket RPC proxy (+2)`.
7. **RPC error + reconnect** *(+3 tests: 12, 15, plus `dispatchEvent`=3)*. Add `{iid, throw: e}` path. Add close-handler tracking + lazy reconnect on next RPC/dispatch. Add `dispatchEvent` JSON-encode (strips `sender`/`_namedArgList_`). Generator `close-reconnect` + `send-capture`. sync-wasm. Land: `rpc proxy reply with throw rejects`, `rpc reconnects after … closes`, `dispatchEvent sends JSON-encoded event`. Commit: `HS: socket dispatchEvent + reconnect + throw (+3)`.
8. **RPC timeouts** *(+3 tests: 10, 11, 14)*. `.timeout(n)` / `.noTimeout` chainable; default timeout from `with timeout N`. `setTimeout` reject with `"Timed out"`. Generator `rpc-timeout` code — may need a virtual-clock helper in the mock if real `setTimeout` is flaky under the step limiter. sync-wasm. Land: `…default timeout rejects`, `…noTimeout avoids timeout rejection`, `…timeout(n) rejects after a custom window`. Commit: `HS: socket RPC timeouts (+3)`.
9. **Polish + scoreboard** *(0 tests)*. Update `plans/hs-conformance-to-100.md` row 36 to `done (+N)`, bump scoreboard. Smoke 0–195 + `hs-upstream-send` regression check. Commit: `HS-plan: E36 complete (+N)`.

Total runway: **8 feature commits** to move cluster 36 from `blocked` → `done (+12–16)`. No commit may land a regression in smoke 0–195 or in `hs-upstream-send`.
