rose-ash/plans/designs/e40-real-fetch.md

E40 — Fetch non-2xx + hyperscript:beforeFetch + real Response

Cluster 40 of plans/hs-conformance-to-100.md. Target: +7 tests.

1. Failing tests

All seven currently emit SKIP (skip-list) from the generator (SKIP_TEST_NAMES in tests/playwright/generate-sx-tests.py lines 151-158). Upstream bodies — stripped to assertions:

#	Name	Mock	Script	Assertion
1	Response can be converted to JSON via as JSON	/test → 200, JSON {name:"Joe"}	fetch /test as Response then put (it as JSON).name into me	text = "Joe"
2	as response does not throw on 404	/test → 404, body not found	fetch /test as response then put it.status into me	text = "404"
3	can catch an error that occurs when using fetch	/test → route.abort() (network error)	fetch /test catch e log e put "yay" into me	text = "yay"
4	do not throw passes through 404 response	/test → 404, body the body	fetch /test do not throw then put it into me	text = "the body"
5	don't throw passes through 404 response	idem, apostrophe form	fetch /test don't throw then put it into me	text = "the body"
6	throws on non-2xx response by default	/test → 404, body not found	fetch /test catch e put "caught" into me	text = "caught"
7	triggers an event just before fetching	/test → 200, yay; listener on hyperscript:beforeFetch sets target class to foo-set	on click fetch "/test" then put it into my.innerHTML end	initial: no foo-set; after click: has foo-set AND text yay

2. Upstream semantics

hyperscript:beforeFetch — dispatched on the element running the feature, just before window.fetch() is called. Upstream CustomEvent detail: { resolve, reject, url, options, response? } — handlers may mutate options, call resolve(value) to short-circuit, or reject(err) to fail. The test only checks that the event fires on the element; detail fields are not asserted. We implement the event but only cancelable + mutable options matter for conformance.

Non-2xx handling — upstream default is: when response.ok === false, the command throws FetchError carrying the response. Two opt-outs:

• as response / as Response — yields the Response wrapper; no throw.
• do not throw / don't throw — after the conversion, if there was an HTTP error, still return the converted body (whichever format was chosen).

Network errors (route abort, DNS fail) — always thrown regardless of do not throw. catch e handles them.

Response API tests rely on: .ok, .status, .url, and chained conversion — (it as JSON) when it is a Response parses the body as JSON. Upstream fetch /test as Response then … (it as JSON) reuses the Response's body; our wrapper must remember the raw body.

3. Proposed runtime shape

3a. Response wrapper (SX dict, not host Response)

{ :_hs-response true
  :ok       <bool>
  :status   <int>
  :url      <string>
  :headers  <dict>
  :_body    <string>      ;; raw text, always kept
  :_json    <string|nil>  ;; raw JSON string (for re-parse on `as JSON`)
  :_html    <string|nil> }

A single dict representation is enough; the kernel already passes dicts through as hs values. No new FFI type.

3b. hs-fetch restructured

Pseudocode (design only — no emission here):

(define hs-fetch
  (fn (url format opts do-not-throw target)
    ;; 1. Dispatch beforeFetch event on `target` element.
    ;; 2. If event.defaultPrevented → return nil.
    ;; 3. Call (perform (io-fetch url "response" opts)) — always fetch
    ;;    the full wrapper, regardless of `format`.
    ;; 4. If the host op returned a :_network-error key → raise a
    ;;    FetchError (catchable by hyperscript `catch`).
    ;; 5. If (not :ok) and (not do-not-throw) and format ≠ "response" →
    ;;    raise FetchError carrying the wrapper.
    ;; 6. Convert by format:
    ;;       "response" → return the wrapper as-is
    ;;       "json"     → (host-call JSON "parse" (:_json wrapper)) → hs-host-to-sx
    ;;       "html"     → io-parse-html on (:_html wrapper) → fragment
    ;;       "text"/other → (:_body wrapper)
    ;;    `do not throw` only matters if (not :ok); the value is still
    ;;    whatever the conversion yields for the real body.))

3c. Chained as JSON on a Response (test 1)

(it as Response) as JSON — hyperscript's as operator needs a Response-aware branch: if (dict? x) and (get x :_hs-response) and target type is JSON, call host JSON.parse on :_json. Add a clause to the existing as / conversion dispatcher in runtime.sx.

3d. Error propagation

Introduce hs-fetch-error — a dict {:_hs-error :FetchError :response <w> :message "..."}. Hyperscript catch already binds the raised value to the catch symbol; tests can catch … and throws on non-2xx … both just put "yay" into me / "caught" so the error's shape is uninspected.

The compiler / parser need two keyword paths:

• do not throw / don't throw — tokenizer already tokenizes don't as don 't (two tokens) or as a single don't; verify and emit a :do-not-throw flag into the fetch AST.
• as Response / as response — already handled (runtime line 763).

3e. beforeFetch event

Use the existing DOM event plumbing. Add a primitive call: (host-new "CustomEvent" "hyperscript:beforeFetch" {:detail {...} :bubbles true :cancelable true}), then dispatchEvent on the fetch target. Listen for defaultPrevented before proceeding.

4. Test mock strategy

Stay in-process. Extend tests/hs-run-filtered.js:

4a. Test-name-keyed route table

Add a _fetchScripts map keyed by test name. The runner already tracks _current-test-name (used for the ask/answer cluster, cluster 28). Each entry describes the /test route for that test:

const _fetchScripts = {
  "as response does not throw on 404":
      { "/test": { status: 404, body: "not found" } },
  "do not throw passes through 404 response":
      { "/test": { status: 404, body: "the body" } },
  "don't throw passes through 404 response":
      { "/test": { status: 404, body: "the body" } },
  "throws on non-2xx response by default":
      { "/test": { status: 404, body: "not found" } },
  "Response can be converted to JSON via as JSON":
      { "/test": { status: 200, body: '{"name":"Joe"}',
                   contentType: "application/json" } },
  "can catch an error that occurs when using fetch":
      { "/test": { networkError: true } },
  "triggers an event just before fetching":
      { "/test": { status: 200, body: "yay",
                   contentType: "text/html" } },
};

_mockFetch(url) / the io-fetch branch of _driveAsync reads _fetchScripts[_currentTestName]?.[url] first, falls back to _fetchRoutes[url]. Network-error entries set networkError: true → the io-fetch handler resumes with { _network_error: true, message: "aborted" } so hs-fetch can raise.

4b. window.addEventListener('hyperscript:beforeFetch', …) support

The JSDOM-lite El in the runner already implements addEventListener/dispatchEvent on elements. window in the runner is a plain object — we need a minimal window.addEventListener that records listeners and fires when hs-fetch dispatches the event. Either:

• Option A: make window an El instance (cheap, uniform).
• Option B: add a tiny event-target shim on window with the usual methods.

Option A is smaller. The test-7 listener is registered on window, but since bubbles: true, we can also dispatch on document.body if window routing is awkward; upstream dispatches on the element and bubbles.

4c. No real HTTP

All fetches remain synchronous-resume via _driveAsync. Zero network.

5. Test delta estimate (per-test)

All seven are currently SKIP and count zero. Converting them means (a) removing from SKIP_TEST_NAMES, (b) teaching the generator to emit them, (c) runtime + mock changes. Each is mechanically independent:

#	Test	Risk	Est
1	Response → JSON chain	Compiler: chained as; runtime: :_json re-parse	+1
2	as response on 404	Runtime: don't throw when format=response	+1
3	catch abort	Runtime: raise on network error; mock: networkError flag	+1
4	do not throw + 404	Tokenizer/parser: multi-word modifier; runtime: flag	+1
5	don't throw + 404	Tokenizer: apostrophe contraction accepted	+1
6	non-2xx throws default	Runtime: throw when !ok && !format=response && !do-not-throw	+1
7	beforeFetch event	Runtime: dispatch event on target; runner: window listener	+1

Total: +7. Plus possible knock-on fixes in neighbouring non-skipped tests — the existing "can do a simple fetch with a response object" test (line 7177) currently passes with a thin mock; the restructure must keep it green.

6. Risks

• Cluster-1 JSON unwrap collision. hs-fetch today unwraps JSON via hs-host-to-sx raw when fmt = "json". The new wrapper keeps :_json as a raw string; the existing as json path must now parse :_json through host JSON before calling hs-host-to-sx. Failure mode: doubled parse or raw-string leak into tests can do a simple fetch w/ json* (currently green — regression risk).
• Generator coverage. Five of the seven upstream bodies use page.route/route.fulfill/route.abort — the generator currently gives up on these. Need a new translator step that recognises page.route('**/test', …) and emits a (set-fetch-script! "...") preamble so _driveAsync picks up the right response. This is a small generator change, not a deep rewrite.
• Request interception for the beforeFetch test requires window.addEventListener on a JSDOM-lite host — new surface; keep the shim minimal (addEventListener/removeEventListener/dispatchEvent only).
• Tokenizer for don't. Apostrophe handling in the HS tokenizer is fragile (see hs-tokenizer.sx). May require a targeted fix so don't tokenises as one word in the fetch-modifier position.
• Catchable errors. Hyperscript catch flow in our runtime currently catches host exceptions; confirm raising a dict via (raise {...}) propagates into the catch handler and binds as e.
• hyperscript:beforeFetch bubble path. Upstream dispatches on the element; the test listener is on window. Element→document→window bubbling in our JSDOM-lite is partial; simplest fix is to dispatch on the element and invoke any window listeners for that event type.

7. Implementation checklist (commit-sized)

1. Design merge (this doc). One commit, no code.
2. Runner: _fetchScripts + _currentTestName plumbing. Extend tests/hs-run-filtered.js only; add a self-test row in the mock scripts. No generator / runtime changes yet.
3. Runner: window.addEventListener shim + dispatch relay from element events. Add a smoke assertion.
4. Generator: recognise page.route / route.fulfill / route.abort. Emit (set-fetch-script! …) SX form; remove matched tests from SKIP_TEST_NAMES incrementally (one per commit below).
5. Runtime: response wrapper + reshape hs-fetch. Keep existing as json/as text/naked paths green; un-skip test 2 (as response on 404) first — smallest surface. Commit: HS: fetch response wrapper preserves status on 404 (+1).
6. Runtime: chained as JSON on Response. Un-skip test 1. Commit: HS: as JSON on Response wrapper (+1).
7. Tokenizer / parser: do not throw modifier. Un-skip test 4. Commit: HS: fetch do-not-throw modifier (+1).
8. Tokenizer: don't throw contraction. Un-skip test 5. Commit: HS: fetch dont-throw contraction (+1).
9. Runtime: throw on non-2xx default. Un-skip test 6. Risk: the default-behaviour change could break unrelated fetch /test tests if any current routes are non-2xx. Audit _fetchRoutes. Commit: HS: fetch throws on non-2xx default (+1).
10. Runtime: raise on network error + runner networkError flag. Un-skip test 3. Commit: HS: fetch catch network error (+1).
11. Runtime: dispatch hyperscript:beforeFetch. Un-skip test 7. Commit: HS: hyperscript:beforeFetch event (+1).
12. Sync WASM staging after each runtime change: cp lib/hyperscript/runtime.sx shared/static/wasm/sx/hs-runtime.sx (per plan rule 4).
13. Scoreboard + plan log in the same commit as each code commit (plan rule 11).

Total commits: 1 design + 2 infrastructure + 7 test-unlocks + 1 wrap-up scoreboard summary = ~10-11 commits. Each unlock is independently committable and safely revertible.