rose-ash/plans/designs/e36-websocket.md

E36 — WebSocket + socket feature + RPC proxy

Design for cluster 36 of plans/hs-conformance-to-100.md. Scope: +16 tests in hs-upstream-socket, all currently SKIP (untranslated).

1. Failing tests (all SKIP'd — untranslated)

Exact names from spec/tests/test-hyperscript-behavioral.sx:11354:

1. converts relative URL to ws:// on http pages
2. converts relative URL to wss:// on https pages
3. dispatchEvent sends JSON-encoded event over the socket
4. namespaced sockets work
5. on message as JSON handler decodes JSON payload
6. on message as JSON throws on non-JSON payload
7. on message handler fires on incoming text message
8. parses socket with absolute ws:// URL
9. rpc proxy blacklists then/catch/length/toJSON
10. rpc proxy default timeout rejects the promise
11. rpc proxy noTimeout avoids timeout rejection
12. rpc proxy reply with throw rejects the promise
13. rpc proxy sends a message and resolves the reply
14. rpc proxy timeout(n) rejects after a custom window
15. rpc reconnects after the underlying socket closes
16. with timeout parses and uses the configured timeout

2. Upstream semantics (stock _hyperscript)

Verified against the JSON fixture and hyperscript.org/features/socket/. The prompt's "with proxy { send, receive }" wording is not part of the upstream socket feature — there is no with proxy clause. The shape actually under test is:

socket <Name[.dotted.path]> <url> [with timeout <ms>]
  [on message [as JSON]
     <cmd-list>]
end

URL scheme normalisation. Bare paths get a scheme prepended from location.protocol: http: → ws://host[:port]<path>, https: → wss://…. Absolute ws:// / wss:// URLs pass through unchanged.

Named binding. socket Foo url end assigns window.Foo = socketObj. A dotted name socket MyApp.chat url end walks/creates window.MyApp then sets window.MyApp.chat = socketObj.

Socket object shape (what the tests assert against):

{ raw: WebSocket,           // underlying ws
  rpc: <Proxy>,              // see below
  dispatchEvent(evt),        // JSON-encodes the event and sends over ws
  // 'on message' handler registered internally via ws.onmessage
}

dispatchEvent protocol. On send NAME(a:1) to Foo or Foo.dispatchEvent(evt):

• Outgoing payload: JSON.stringify({ type: evt.type, ...evt.detail }) with sender and _namedArgList_ stripped.

on message [as JSON]. ws.onmessage({data}) fires the cmd-list. as JSON calls JSON.parse(data) first and the parsed object becomes the event; non-JSON throws "Received non-JSON message".

RPC proxy (socket.rpc). ES6 Proxy over an object. Accessing .anyName normally returns a callable; four property names are blacklisted and return null: then, catch, length, toJSON (so the proxy isn't mistaken for a thenable and JSON-serialises cleanly). Two reserved chainable properties:

• .noTimeout — returns a proxy that will never time out.
• .timeout(n) — returns a proxy that times out after n ms.

Calling proxy.fnName(arg1, arg2) performs:

1. Allocate a unique iid (string).
2. Send JSON.stringify({iid, function: "fnName", args: [arg1, arg2]}) over ws.
3. Return a Promise. The promise is tracked by iid.
4. Incoming messages with matching iid are routed: {iid, return: v} resolves with v; {iid, throw: e} rejects with e.
5. Default/custom timeout (from with timeout N, default per stock is ~0 i.e. "Timed out" reject; override via .timeout(n) or .noTimeout).

Reconnect. When the underlying socket fires close, the socket wrapper marks itself dead. The next RPC (or dispatchEvent) call lazily reconstructs a fresh new WebSocket(url). Test 15 asserts created.length === 2 after one close + one RPC.

3. Proposed user-facing shape

3a. Tokens

No new tokens needed. socket is a bare identifier that becomes a keyword in parse-feat. end is already known. with, timeout, on, message, as, JSON are already existing keyword/identifier tokens. Dotted names (MyApp.chat) already tokenise as local + dot + local.

3b. Parse tree

A new feature AST node, peer to (def ...) / (on ...):

(socket <name-path> <url-expr> :timeout <expr-or-nil> :on-message <handler-or-nil>)

• <name-path> is a list of symbol strings: ["Foo"] or ["MyApp" "chat"].
• <url-expr> is any parseable expression (usually a string literal).
• :timeout is the parsed ms expression, or nil if absent.
• :on-message is nil, (on-message :json? false <body>), or (on-message :json? true <body>).

Parser function: parse-socket-feat dispatched from parse-feat on val == "socket". Reuses parse-cmd-list for handler bodies. as JSON is detected inline before collecting the body.

3c. Compile output

The compiler emits a call to a runtime registration primitive that carries everything the runtime needs:

(hs-socket-register!
  (list "MyApp" "chat")            ; name path
  <url-compiled>                   ; usually a string
  <timeout-ms-or-nil>              ; number or nil
  (fn (event) <on-message-body>)   ; handler closure or nil
  :json-message? true|false)

The runtime owns all WebSocket construction, URL normalisation, window binding, promise/timeout bookkeeping, and the RPC proxy — keeping the compiler output small and re-usable.

3d. Worked example

Source (from fixture test 13, simplified):

socket RpcSocket ws://localhost/ws end

Tokens: socket RpcSocket ws : //localhost/ws end (URL tokenising is already handled by the string-like path reader; see how fetch URLs parse today).

AST:

(socket ("RpcSocket") "ws://localhost/ws" :timeout nil :on-message nil)

Compiled SX (emitted into the feature body):

(hs-socket-register!
  (list "RpcSocket")
  "ws://localhost/ws"
  nil
  nil
  :json-message? false)

Fuller example:

socket MyApp.chat /ws with timeout 1500
  on message as JSON
    set window.got to the event
end

→

(socket ("MyApp" "chat") "/ws"
        :timeout 1500
        :on-message (on-message :json? true
                                ((set! window.got event))))

→

(hs-socket-register!
  (list "MyApp" "chat")
  "/ws"
  1500
  (fn (event) (host-set! (host-global "window") "got" event))
  :json-message? true)

4. Runtime architecture

All in lib/hyperscript/runtime.sx. One big primitive plus small helpers.

4a. hs-socket-register! — lifecycle

1. Normalise URL: if starts with ws:///wss:// use as-is; else derive scheme from (host-get (host-global "location") "protocol") and prepend ws://host[:port] or wss://…. Preserves path.
2. Construct (host-new "WebSocket" url).
3. Build wrapper object (SX dict): {:raw ws :url url :timeout <ms> :pending <dict-of-iid->resolver> :handler <closure> :json? <bool> :closed? false}.
4. Install ws.onmessage via host-set! to a native callback (host-callback) that:
   • Reads data = event.data.
   • If JSON-looking (starts with { or [) tries json-parse; if parse OK and has iid field → RPC reply dispatch (see 4c).
   • Otherwise → fire user's on message handler with event (optionally JSON-parsing first if :json? true; non-JSON there → (error "Received non-JSON message")).
5. Install ws.addEventListener("close", …) (via host-call) that sets :closed? true so next RPC reconnects.
6. Install .dispatchEvent method on wrapper via host-set!: takes an Event, builds {type, ...detail minus sender,_namedArgList_}, JSON.stringifys, ws.sends.
7. Install .rpc — the proxy (see 4b).
8. Bind on window: walk name-path creating intermediate {} dicts as needed; final segment assigned the wrapper.

4b. RPC proxy

Implemented as a native JS object constructed via a new FFI primitive we already have: host-call to a global helper we register inside the test mock and inside the real runtime. Concretely the runtime calls (host-call (host-global "_hs_make_rpc_proxy") "call" nil wrapper) → returns a Proxy. In prod _hs_make_rpc_proxy is shipped in shared/static/scripts/sx-platform-*.js; in the test mock it's inlined into tests/hs-run-filtered.js.

Pseudo-JS for the proxy factory (this is what lives in the platform/mock, not in .sx):

function makeRpcProxy(wrapper, overrides = {}){
  return new Proxy({}, {
    get(_, name) {
      if (['then','catch','length','toJSON'].includes(name)) return null;
      if (name === 'noTimeout') return makeRpcProxy(wrapper, {...overrides, timeout: Infinity});
      if (name === 'timeout')   return (n) => makeRpcProxy(wrapper, {...overrides, timeout: n});
      return (...args) => hsRpcCall(wrapper, name, args, overrides.timeout);
    }
  });
}

hsRpcCall returns a native Promise whose resolver is stored in wrapper.pending[iid]. A setTimeout (skipped when Infinity) rejects with "Timed out". The sx runtime never calls hsRpcCall directly — the proxy does, in JS, because tests read .then(...) on the returned promise.

Why this split: the test fixtures all go through .then/.catch in JS, so the proxy and its promise plumbing must be real JS. The SX runtime only wires up message routing.

4c. Inbound dispatch pipeline

onmessage in the runtime (SX):

(fn (event)
  (let ((data (host-get event "data")))
    (let ((parsed (hs-try-json-parse data)))
      (cond
        ;; RPC reply: has :iid
        ((and parsed (host-get parsed "iid"))
          (hs-socket-resolve-rpc! wrapper parsed))
        ;; User 'on message as JSON'
        (user-handler-json?
          (if parsed
              (user-handler parsed)
              (error "Received non-JSON message")))
        ;; User 'on message' plain
        (user-handler
          (user-handler event))
        (true nil)))))

hs-socket-resolve-rpc! pulls the resolver/rejecter pair out of :pending by iid and calls the JS-side resolve(ret) or reject(thrown).

4d. send NAME(a:1) to Foo integration

Today emit-send in compiler.sx always emits dom-dispatch. We widen it to hs-dispatch-to (new runtime helper) which dispatches:

• If target exposes _hs_socket? === true → JSON-encode + ws.send.
• Else → falls through to dom-dispatch (existing behaviour).

This keeps backwards compat — all non-socket send targets continue to use DOM events.

4e. call Foo.rpc.greet("world") integration

No compiler changes needed. call already compiles property+method chains via method-call. Foo.rpc.greet resolves to the proxy method which returns a Promise. Hyperscript's existing await semantics (via io-* suspension or then absorption in result) then wait on it. For the non-Playwright test harness, the promise is resolved synchronously inside the mock's _driveAsync loop — see §5.

4f. IO suspension

Sockets do not need a new io-* operation for the basic tests: onmessage is a callback that fires user cmd-lists directly via host-callback → K.callFn → _driveAsync (this already works for hs-on-every). RPC promise resolution similarly runs inside a native callback. The only place we might want IO suspension is if someone writes wait for message in Foo — not in this cluster. Keep hs-socket-register! synchronous and callback-driven; defer any (perform …) integration to a later cluster.

5. Test mock strategy

Minimum surface in tests/hs-run-filtered.js:

5a. WebSocket mock (test-keyed)

Each test self-describes via its test name. We key behaviour off _current-test-name. A single globalThis.WebSocket constructor looks up the script-driven scenario. But the fixture's tests all re-mock window.WebSocket in-test — they are self-contained. That means the mock runner merely needs a default that records constructions and lets test code override it.

globalThis.WebSocket = function(url) {
  const sock = { url, onmessage: null, _listeners: {},
                 send(msg){ sock._sent = sock._sent || []; sock._sent.push(msg); },
                 addEventListener(t, h){ (sock._listeners[t]=sock._listeners[t]||[]).push(h); },
                 close(){} };
  globalThis.__hs_ws_created = globalThis.__hs_ws_created || [];
  globalThis.__hs_ws_created.push(sock);
  return sock;
};

5b. _hs_make_rpc_proxy in the mock

Ship the factory from §4b as a top-level function in hs-run-filtered.js. It must use the test-runtime Promise (real JS Promise), and setTimeout (real host — Node's, already global). Timeout rejection must fire inside _driveAsync's execution window; a short setImmediate/microtask tick suffices because the tests all await evaluate(() => new Promise(resolve => …)).

5c. Generator patterns (tests/playwright/generate-sx-tests.py)

The socket fixtures are unusual: they directly evaluate browser-side JS that attaches a <script type="text/hyperscript"> then calls _hyperscript.processNode. We do not try to translate that path verbatim. Instead the generator gets a _socket_setup_ops recogniser (peer to _hs_config_setup_ops) that:

1. Recognises the pattern var OrigWS = window.WebSocket; … window.WebSocket = function() {...}; … script.textContent = 'socket …'; ….
2. Extracts the inner hyperscript source from the script.textContent string.
3. Emits an SX test body that:
   • Pre-installs a matching WS mock (host-set! window "WebSocket" ...) — parameterised by which behaviour the test needs: (a) record URL, (b) record sent messages, (c) record close handlers, (d) provide an onmessage hook.
   • Activates the hyperscript on a div (so the socket feature runs through the existing hs-activate! pipeline).
   • Invokes whatever the test asserted against: reads urls[0], calls .dispatchEvent, drives mockSocket.onmessage(…), etc.
   • Asserts the final value (translated from expect(…).toBe(…) via the existing pw_assertion_to_sx).

A single shared socket test mini-DSL — 5 behaviour codes — covers all 16 tests:

Code	Records	Injects	Asserts
url-capture	constructor URL	—	URL string
send-capture	outgoing frames	—	JSON payload fields
inbound-text	—	mockSocket.onmessage({data:"…"})	window flag set
inbound-json	—	mockSocket.onmessage({data:JSON…})	window flag set
rpc-roundtrip	outgoing frames	onmessage with {iid,return:v} or {iid,throw:e}	promise resolves/rejects
rpc-timeout	—	timer advance	promise rejects with "Timed out"
close-reconnect	close handlers + construction count	fire all close handlers then call rpc	created.length === 2

The generator dispatches tests 1,2,8 → url-capture; 3 → send-capture; 5,6,7 → inbound; 10,11,12,13,14 → rpc; 15 → close-reconnect; 4,9,16 → small probes of wrapper shape (.rpc typeof === "object", namespaced binding exists).

6. Test delta estimate

• Best case: 16/16. All tests are self-contained scenarios; with the WebSocket mock, proxy factory, and generator recogniser they all translate.
• Realistic: 12–14. Timeout/reconnect tests (10, 11, 14, 15) need real-time behaviour from the Node mock; if setTimeout behaviour is flaky under the step-limited driver, they may go partial.
• Worst case: 8. Only the synchronous URL/shape/inbound tests land (1, 2, 4, 5, 6, 7, 8, 9, 16); the five full RPC round-trip tests stay SKIP and get handled in a follow-up.

Target commit should aim for +12 tests, partial on RPC timeout/reconnect. Mark any stragglers partial per rule 7.

7. Risks / open questions (human decisions)

1. Reconnect scope. The fixture only tests lazy reconnect on next RPC. Do we also reconnect on dispatchEvent after close? (Stock yes; tests don't cover it. Decision: yes, keep parity.)
2. Buffering while disconnected. Between close and first reconnecting call, does dispatchEvent drop the frame or queue it? Stock drops. Recommendation: drop, log once.
3. Binary frames. None of the 16 tests exercise Blob/ArrayBuffer. Explicitly out of scope — document as not yet supported in runtime.
4. Close-code handling. close(code, reason) / CloseEvent — no test exercises it. Recommendation: ignore code/reason, just mark closed.
5. Interaction with IO suspension. RPC promises live in JS, not SX; a hyperscript call Foo.rpc.x() under the current runtime returns the JS Promise and hyperscript's then absorption takes over. Verify that the CEK/VM hs-safe-call path doesn't eagerly (host-await).
6. Proxy implementation language. The proxy must be JS (ES6 Proxy) — SX has no equivalent and the tests read .then etc. as literal JS property accesses. Means the proxy factory lives both in the test mock and in a shipped platform JS file. Do we want to factor out a shared platform/sockets.js? (Recommendation: yes, land it in shared/static/scripts/sx-platform-sockets.js in a follow-up commit after the mock lands; the mock inline version stays as the reference.)
7. Namespace dotted binding collisions. If window.MyApp already exists as a non-dict, the runtime should refuse overwriting it (tests only use fresh names). Recommendation: silently overwrite with a dict — matches stock.
8. emit-send widening. Changing send to branch on _hs_socket? touches a code path used by every existing send test. Needs a smoke-regression check against hs-upstream-send before landing.

8. Implementation checklist (one commit per step; must land +N)

Each step is scoped to be independently testable. sync-wasm means cp lib/hyperscript/X.sx shared/static/wasm/sx/hs-X.sx.

1. Mock groundwork (0 tests). Add WebSocket constructor + _hs_make_rpc_proxy factory to tests/hs-run-filtered.js. No runtime/compiler changes. Verify no regressions on smoke 0–195. Commit: HS-prep: WebSocket + RPC proxy mock.
2. Parser: socket feature (0 tests). Add parse-socket-feat in parser.sx dispatched from parse-feat. Parses name-path, URL, with timeout N, on message [as JSON] … end, trailing end. Emits (socket …) AST. Unit-verify via hs-compile eval trace. sync-wasm. Commit: HS: parse socket feature.
3. Compiler + runtime: basic socket registration (+3 tests: 1, 2, 8). Implement hs-socket-register! runtime primitive (URL normalise, construct WS, window bind — no RPC/on-message yet). Compiler emits hs-socket-register! call from (socket …). sync-wasm. Land tests: converts relative URL to ws://, …to wss://, parses socket with absolute ws:// URL. Commit: HS: socket URL binding (+3).
4. Namespaced binding + wrapper shape (+2 tests: 4, 16). Walk name-path assigning intermediate dicts. Ensure wrapper exposes .raw and .rpc objects (rpc can be the bare proxy for now). sync-wasm. Land: namespaced sockets work, with timeout parses and uses the configured timeout. Commit: HS: socket namespaced names + timeout plumbing (+2).
5. Inbound on message dispatch (+3 tests: 5, 6, 7). Wire ws.onmessage → SX handler. Support as JSON parse branch + non-JSON error. Generator: inbound-text + inbound-json dispatch codes. sync-wasm. Land: on message handler fires…, on message as JSON … decodes…, …throws on non-JSON. Commit: HS: socket on-message + as JSON (+3).
6. RPC proxy — happy path (+2 tests: 9, 13). Wire .rpc proxy with blacklist, iid allocation, send JSON, resolve on matching iid.return. Uses real JS Promise from the mock. Generator rpc-roundtrip code. sync-wasm. Land: rpc proxy blacklists …, rpc proxy sends a message and resolves…. Commit: HS: socket RPC proxy (+2).
7. RPC error + reconnect (+3 tests: 12, 15, plus dispatchEvent=3). Add {iid, throw: e} path. Add close-handler tracking + lazy reconnect on next RPC/dispatch. Add dispatchEvent JSON-encode (strips sender/_namedArgList_). Generator close-reconnect + send-capture. sync-wasm. Land: rpc proxy reply with throw rejects, rpc reconnects after … closes, dispatchEvent sends JSON-encoded event. Commit: HS: socket dispatchEvent + reconnect + throw (+3).
8. RPC timeouts (+3 tests: 10, 11, 14). .timeout(n) / .noTimeout chainable; default timeout from with timeout N. setTimeout reject with "Timed out". Generator rpc-timeout code — may need a virtual-clock helper in the mock if real setTimeout is flaky under the step limiter. sync-wasm. Land: …default timeout rejects, …noTimeout avoids timeout rejection, …timeout(n) rejects after a custom window. Commit: HS: socket RPC timeouts (+3).
9. Polish + scoreboard (0 tests). Update plans/hs-conformance-to-100.md row 36 to done (+N), bump scoreboard. Smoke 0–195 + hs-upstream-send regression check. Commit: HS-plan: E36 complete (+N).

Total runway: 8 feature commits to move cluster 36 from blocked → done (+12–16). No commit may land a regression in smoke 0–195 or in hs-upstream-send.