rose-ash/plans/dream-on-sx.md

# Dream-on-SX: OCaml's Dream web framework on the SX CEK

`[activated — target user confirmed; gated only on ocaml-on-sx]`

Carved out of `plans/ocaml-on-sx.md`. The OCaml-on-SX plan was scoped down to **substrate validation + HM + reference oracle** (Phases 1–5 + minimal stdlib slice). Dream is the practical alternative-stack story — the opposite framing — and is now the **chosen framework layer for the rose-ash host**: the decision is to move off Quart and adopt Dream (not Quart) as the ergonomic HTTP front door over the native SX server. `plans/host-on-sx.md` Phase 4 is the concrete consumer that pulls Dream.

**Target user — CONFIRMED.** The earlier "needs a concrete target user" condition is met: rose-ash itself is the user — the subsystems (feed/acl/mod/commerce/identity/…) need an ergonomic HTTP front door, and the project owner has chosen Dream over Quart for it. This plan is no longer cold; it is *gated*, not deferred.

**Do not start without (the one remaining gate):**
1. OCaml-on-SX Phases 1–5 + Phase 6 minimal stdlib green. (As of writing ocaml-on-sx is at 480/480 and advancing — verify its scoreboard covers Phases 1–5 + the stdlib slice before starting.)

Until that gate is green, the native server (host-on-sx Phases 1–3) carries the host; do not block host migration on Dream.

## Why this might be worth doing (when the time comes)

Dream is the cleanest middleware-shaped HTTP framework in any language:
- `handler = request -> response promise`
- `middleware = handler -> handler`
- `m1 @@ m2 @@ handler` — left-fold composition

It maps onto SX with almost no impedance — `@@` is function composition, `request → response promise` is `(perform (:http-respond ...))`, middleware chain is plain SX function composition. So the integration cost is low *if* the OCaml-on-SX foundation is in place.

The user-facing story: rose-ash users who'd never touch s-expressions might write Dream/OCaml apps that integrate with the same federation, auth, and storage primitives. Demo: a Dream app serving sx.rose-ash.com — the framework that describes the runtime it runs on.

## Dream semantic mappings

| Dream construct | SX mapping |
|----------------|-----------|
| `handler = request -> response promise` | `(fn (req) (perform (:http-respond ...)))` |
| `middleware = handler -> handler` | `(fn (next) (fn (req) ...))` |
| `Dream.router [routes]` | `(ocaml-dream-router routes)` — dispatch on method+path |
| `Dream.get "/path" h` | route record `{:method "GET" :path "/path" :handler h}` |
| `Dream.scope "/p" [ms] [rs]` | prefix mount with middleware chain |
| `Dream.param req "name"` | path param extracted during routing |
| `m1 @@ m2 @@ handler` | `(m1 (m2 handler))` — left-fold composition |
| `Dream.session_field req "k"` | `(perform (:session-get req "k"))` |
| `Dream.set_session_field req "k" v` | `(perform (:session-set req "k" v))` |
| `Dream.flash req` | `(perform (:flash-get req))` |
| `Dream.form req` | `(perform (:form-parse req))` — returns Ok/Error ADT |
| `Dream.websocket handler` | `(perform (:websocket handler))` |
| `Dream.run handler` | starts SX HTTP server with handler as root |

## Roadmap

The five types: `request`, `response`, `handler = request -> response`, `middleware = handler -> handler`, `route`. Everything else is a function over these.

- [x] **Core types** in `lib/dream/types.sx`: request/response records, route record.
- [x] **Router** in `lib/dream/router.sx`:
      - `dream-get path handler`, `dream-post path handler`, etc. for all HTTP methods.
      - `dream-scope prefix middlewares routes` — prefix mount with middleware chain.
      - `dream-router routes` — dispatch tree, returns handler; no match → 404.
      - Path param extraction: `:name` segments, `**` wildcard.
      - `dream-param req name` — retrieve matched path param.
- [x] **Middleware** in `lib/dream/middleware.sx`:
      - `dream-pipeline middlewares handler` — compose middleware left-to-right.
      - `dream-no-middleware` — identity.
      - Logger: `(dream-logger next req)` — logs method, path, status, timing.
      - Content-type sniffer.
- [x] **Sessions** in `lib/dream/session.sx`:
      - Cookie-backed session middleware.
      - `dream-session-field req key`, `dream-set-session-field req key val`.
      - `dream-invalidate-session req`.
- [x] **Flash messages** in `lib/dream/flash.sx`:
      - `dream-flash-middleware` — single-request cookie store.
      - `dream-add-flash-message req category msg`.
      - `dream-flash-messages req` — returns list of `(category, msg)`.
- [x] **Forms + CSRF** in `lib/dream/form.sx`:
      - [x] `dream-form req` — returns `(Ok fields)` or `(Err :csrf-token-invalid)`.
      - [x] `dream-multipart req` — multipart form data (in-memory, not yet streaming).
      - [x] CSRF middleware: stateless signed tokens, session-scoped.
      - [x] `dream-csrf-tag req` — returns hidden input fragment for SX templates.
- [x] **WebSockets** in `lib/dream/websocket.sx`:
      - `dream-websocket handler` — upgrades request; handler `(fn (ws) ...)`.
      - `dream-send ws msg`, `dream-receive ws`, `dream-close ws`.
- [x] **Static files:** `dream-static root-path` — serves files, ETags, range requests.
- [x] **`dream-run`**: wires root handler into SX's `perform (:http-listen ...)`.
- [x] **Demos** in `lib/dream/demos/`:
      - [x] `hello.ml` → `lib/dream/demos/hello.sx`: "Hello, World!" route.
      - [x] `counter.ml` → `lib/dream/demos/counter.sx`: in-memory counter with sessions.
      - [x] `chat.ml` → `lib/dream/demos/chat.sx`: multi-room WebSocket chat.
      - [x] `todo.ml` → `lib/dream/demos/todo.sx`: CRUD list with forms + CSRF.
- [x] Tests in `lib/dream/tests/`: routing dispatch, middleware composition, session round-trip, CSRF accept/reject, flash read-after-write — **258 tests across 10 suites** (well past the 60+ target). Runner: `lib/dream/conformance.sh`.

**Roadmap complete (2026-06-07): all boxes ticked, 258/258 green.** Loop continues
with extensions + hardening below.
- **2026-06-07 — Ext: router HTTP correctness** (router suite 27→36, 267 total).
  Dispatch now tracks which routes' *paths* matched: path matched + method didn't →
  `405 Method Not Allowed` with an `Allow` header listing the path's methods (was a
  blanket 404); genuinely-absent paths stay 404. `HEAD` falls back to the matching
  `GET` handler with the body blanked but headers kept. `dr/route-params` (path-only
  match) + `dr/method-accepts?` (ANY / HEAD→GET) + `dream-method-not-allowed`. NOTE:
  in this worktree every `sx-tree` *edit* tool (`sx_replace_node`,
  `sx_replace_by_pattern`, `sx_insert_near`) raises a yojson `Expected string, got
  null` error — only `sx_write_file` works, so edits rewrite the whole file.
- **2026-06-07 — Ext: error handling + status phrases** (`lib/dream/error.sx`, 15
  tests, 282 total). `dream-status-text` / `dream-status-line` reason-phrase map (string
  keys); `dream-status-page` renders a status page. `dream-catch` is a `guard`-based
  middleware that turns a raised error into a 500 (`dream-catch-with on-error` for a
  custom page receiving `(req e)`); normal responses pass through untouched, composes
  around a router. (`guard` catches explicit `(error …)` raises; `e` stringifies to the
  message.)
- **2026-06-07 — Ext: CORS** (`lib/dream/cors.sx`, 12 tests, 294 total). `dream-cors`
  decorates responses with `Access-Control-Allow-Origin` (+ credentials), and
  short-circuits preflight `OPTIONS` with a 204 carrying Allow-Methods/Headers/Max-Age.
  `dream-cors-origin` for a specific origin, `dream-cors-with opts` for full control
  (origin/methods/headers/credentials/max-age). Composes around a router.
- **2026-06-07 — Ext: JSON** (`lib/dream/json.sx`, 35 tests, 329 total). Host JSON
  primitives live in the ocaml-on-sx runtime (not the base env), so Dream ships its own
  pure-SX `dream-json-encode` (scalars/list/dict, string escaping) + `dream-json-parse`
  (recursive-descent over chars, objects/arrays/strings/numbers/true/false/null,
  whitespace-tolerant). `dream-json-value` (encode → application/json response) and
  `dream-json-body` (parse request body). GOTCHA: `number?` is unreliable in this env —
  used `(= (type-of v) "number")`; `parse-float` handles decimals. Multi-key dict
  encode order follows `keys` (non-deterministic) so tests assert via parse round-trip.
- **2026-06-07 — Ext: signed session cookies** (`lib/dream/session.sx`, session suite
  30→41, 340 total). The default store uses guessable sids (`s1`, `s2`), so
  `dream-sessions-signed backend secret` signs the cookie value (`sid.signature`) and
  rejects any cookie whose signature doesn't verify — a forged plaintext `s1` or a
  wrong-secret cookie yields a fresh session instead of a hijack. `dream-cookie-sign` /
  `dream-cookie-unsign` (keyed hash; same not-cryptographic caveat — inject a host HMAC
  in production). Plain `dream-sessions` unchanged for the no-secret case.
- **2026-06-07 — Ext: query/header convenience** (`lib/dream/types.sx`, types suite
  41→59, 358 total). `dream-queries`, `dream-query-param-or` / `dream-header-or` /
  `dream-param-or` (defaults), `dream-has-query?` / `-header?` / `-param?`,
  `dream-content-type-of`, `dream-method-is?`, `dream-accepts?` / `dream-wants-json?`
  (Accept-header content negotiation).
- **2026-06-07 — Ext: api.sx facade + README** (`lib/dream/api.sx`, 9 tests, 367 total).
  `dream-version`, `dream-defaults` (pure stack: error-catch + content-type; logger is
  opt-in since it performs IO), `dream-make-app routes`, `dream-make-app-with`,
  `dream-serve`/`dream-serve-port`. `lib/dream/README.md` documents the full public
  surface, quickstart, the dependency-injection testing story, and caveats. **All
  planned extensions complete — 367/367 across 14 suites.**
- **2026-06-07 — Ext: auth** (`lib/dream/auth.sx`, 23 tests, 390 total). Pure-SX base64
  codec (`dream-base64-encode`/`-decode`, arithmetic via `quotient`/`mod` — no bitwise),
  verified against RFC vectors (Man/Ma/M padding). `dream-basic-auth realm check` →
  401 + `WWW-Authenticate: Basic realm=…`, attaches `:dream-user` on success;
  `dream-basic-credentials` / `dream-authorization` accessors. `dream-require-bearer
  check` → attaches `:dream-principal` or 401; `dream-bearer-token` accessor.
- **2026-06-07 — Ext: HTML escaping** (`lib/dream/html.sx`, 11 tests, 401 total).
  `dream-escape` (&/</>/"/' entities, ampersand first to avoid double-escape),
  `dream-attr`, `dream-escape-join`. Fixed a real **XSS hole** in the todo demo, which
  interpolated user text into `<li>` unescaped — now `(dream-escape (get it :text))`;
  regression test asserts `<script>` renders as `&lt;script&gt;`. 16 suites, 401/401.
- **2026-06-07 — Ext: security headers + cache-control** (`lib/dream/headers.sx`, 12
  tests, 413 total). `dream-security-headers` middleware (X-Content-Type-Options
  nosniff, X-Frame-Options DENY, Referrer-Policy no-referrer; opt-in HSTS via
  `dream-security-headers-with`). Cache helpers `dream-cache`/`dream-private-cache`/
  `dream-no-store`/`dream-no-cache` + `dream-cache-for` middleware. **dream-on-sx is
  feature-complete: roadmap + 10 extensions, 413/413 across 17 suites. SATURATED —
  remaining work is host-on-sx's job to consume `dream-run` (don't edit hosts/).**

## Extensions (post-roadmap)

The five-types core is complete; these harden it toward a production HTTP front door.

- [x] **Router HTTP correctness**: 405 Method Not Allowed + `Allow` header; automatic
      HEAD (serve the GET handler with an empty body).
- [x] **Status reason phrases** + `dream-status-text` (`lib/dream/error.sx`).
- [x] **CORS middleware** (`dream-cors`).
- [x] **Error-handling middleware** (`dream-catch` / custom 500 templates; `guard`-based).
- [x] **Signed session cookies** (`dream-sessions-signed` — tamper-evident sid).
- [x] **JSON helpers** (encode + recursive-descent parse, pure SX).
- [x] **Query/header convenience** (`dream-queries`, `*-or` defaults, `dream-accepts?`).
- [x] **`api.sx` facade + README** — `dream-make-app` / `dream-serve` + `README.md`.
- [x] **Auth** — base64 (pure SX), HTTP Basic auth + Bearer-token middleware.
- [x] **HTML escaping** (`dream-escape`/`dream-attr`) — fixed an XSS hole in the todo demo.
- [x] **Security headers + cache-control** (`dream-security-headers`, `dream-cache`/`-no-store`).

## Stdlib additions Dream will need

Dream pushes beyond OCaml-on-SX's Phase 6 minimal stdlib slice. When this plan activates, OCaml-on-SX gets a follow-on phase that adds at minimum:

- `Bytes` (binary buffers — request bodies, websocket frames)
- `Buffer` (mutable string building)
- `Format` (full pretty-printer, not just `Printf.sprintf`)
- More `String` (`index_opt`, `contains`, `starts_with`, `ends_with`, `replace_all`)
- `Sys` (`argv`, `getenv_opt`, `getcwd`)
- `Hashtbl` extensions (`iter`, `fold`, `length`, `remove`)
- `Map.Make` / `Set.Make` functors

Confirm scope before starting; some of these may be addable as Dream-internal helpers rather than full stdlib modules.

## Ground rules

- **Scope:** only `lib/dream/**` and `plans/dream-on-sx.md`. Plus the stdlib additions listed above which land in `lib/ocaml/runtime.sx`.
- **Hard prerequisite:** OCaml-on-SX Phases 1–5 + Phase 6 minimal stdlib. Verify scoreboard before starting.
- **SX files:** `sx-tree` MCP tools only.
- **Don't reinvent the SX HTTP server.** Dream wraps the existing `perform (:http-listen ...)` — it does not implement its own listener loop.

## Progress log

- **2026-06-07 — Core types** (`lib/dream/types.sx`, 41 tests). OCaml gate verified
  green (scoreboard 480/480, Phases 1–5 + Phase 6 stdlib). Dream is implemented in
  plain SX over the CEK — keywords are strings, so headers are dicts with lowercased
  string keys (`:content-type` == `"content-type"`). request (method/target/path/
  query/headers/body/params), response (status/headers/body), route records with
  constructors + accessors; smart response constructors (html/text/json/empty/
  not-found/redirect); `dream-coerce-response` wraps bare strings; query-string
  parsing. Conformance runner `lib/dream/conformance.sh` modelled on flow's.
- **2026-06-07 — Router** (`lib/dream/router.sx`, 27 tests). `dream-get/post/put/
  delete/patch/head/options/any` route constructors; `dream-router` flattens routes
  (incl. nested scopes) and dispatches by method+path, first-match-wins, 404 on no
  match. Path matching is recursive over `/`-split segments: literal, `:name` binds
  a param, `**` catch-all binds remaining path under key `"**"`. Trailing slashes and
  query strings are ignored for routing. `dream-scope prefix mws routes` prepends the
  prefix and folds the middleware chain (`m1 @@ m2 @@ h`, first = outermost) onto each
  route's handler; nests correctly (inner mw innermost). Shared `dr/apply-middlewares`
  fold will back `dream-pipeline`.
- **2026-06-07 — Middleware** (`lib/dream/middleware.sx`, 20 tests). `dream-pipeline`
  (reuses `dr/apply-middlewares`), `dream-no-middleware` identity. `dream-logger-with
  clock sink` is the testable core (records `{:method :path :status :elapsed}`);
  `dream-logger` wires it to `(perform (:dream-clock))` / `(perform (:dream-log …))`;
  `dream-log-line` formats one line. `dream-content-type` sniffs body (`<`→html,
  `{`/`[`→json, else text) only when the handler left Content-Type unset. Bonus
  `dream-set-header` and `dream-tap-request` combinators.
- **2026-06-07 — Sessions** (`lib/dream/session.sx`, 30 tests). Solved the
  request→response mutation-visibility problem the way Dream does: the cookie carries
  only a session id; fields live in an injectable back-end store (the mapping table's
  `(perform (:session-get …))`). `dream-memory-sessions` is an in-memory store built
  on a `set!`-mutated captured `let` binding (no `ref`/`atom` in base env);
  `dream-perform-sessions` is the production back-end. `dream-sessions backend`
  middleware reads/creates the id, attaches `{:sid :io}` to the request, and emits a
  `Set-Cookie` (HttpOnly, SameSite=Lax) only for new sessions. Handler API:
  `dream-session-field` / `dream-set-session-field` / `dream-session-all` /
  `dream-invalidate-session` / `dream-session-id`. Also added shared cookie infra
  (`dr/parse-cookies`, `dream-cookie(s)`, `dr/build-cookie`, `dream-set-cookie`,
  `dream-resp-cookies`, `dream-drop-cookie`) — outgoing cookies accumulate in a
  `:set-cookies` list on the response so multiple Set-Cookie headers don't collide;
  reused by flash + CSRF. Full counter round-trip verified across three requests.
- **2026-06-07 — Flash** (`lib/dream/flash.sx`, 14 tests). `dream-flash` middleware:
  decodes the incoming `dream.flash` cookie into the request, gives the handler a
  mutable outbox cell (`dr/flash-box`, the same `set!`-captured-`let` trick), then on
  response writes the outbox as a fresh flash cookie, or drops the cookie (Max-Age=0)
  when there were incoming messages but no new ones — so messages show exactly once.
  Handler API: `dream-add-flash-message` / `dream-flash-messages` (returns the
  PREVIOUS request's messages) / `dream-flash-of` (by category) / accessors. Cookie
  codec percent-escapes the `|`/`~`/`%` separators so categories/messages round-trip.
  Read-after-write verified across request boundaries incl. multi-category.
- **2026-06-07 — Forms + CSRF (urlencoded)** (`lib/dream/form.sx`, 26 tests). Ok/Err
  result values (`dream-ok`/`dream-err` + predicates/accessors). `dream-form-fields`
  parses `application/x-www-form-urlencoded` with a full percent-decoder
  (`%XX` via `char-from-code`, `+`→space). CSRF is stateless + signed + session-
  scoped: token = `sid.signature`, verified by recomputing the signature and checking
  the session id — no server storage. Signing is **injectable** (`dream-csrf-with`);
  the default `dream-csrf-sign-default` is a pure-SX dual-base polynomial keyed hash
  (NOT cryptographic — production should inject a host HMAC). `dream-csrf` attaches
  context (needs the session middleware upstream for the sid); `dream-csrf-token` /
  `dream-csrf-tag` (hidden input for templates); `dream-form` returns `Ok fields` or
  `Err :csrf-token-invalid`; `dream-csrf-protect` auto-rejects unsafe methods (403)
  lacking a valid token. Full session→csrf→form stack verified accept + reject.
  Multipart deferred to the next commit.
- **2026-06-07 — Multipart** (`lib/dream/form.sx` +9 tests, 35 total). `dream-multipart
  req` parses `multipart/form-data` into parts `{:name :filename :content-type
  :content}`, returns `Ok parts | Err :not-multipart`. Needed a substring splitter
  `dr/split-on` because the `split` primitive is **character-class** based (multi-char
  separators split on every char) — important gotcha. Boundary from the Content-Type
  (handles quoted form); segments filtered to those starting with CRLF; each split on
  the first `\r\n\r\n` into headers/content with one edge CRLF stripped (inner CRLFs
  in file content preserved). `dream-multipart-field` / `dream-multipart-file`
  accessors. In-memory, not streaming (noted for future). `\r`/`\n` string escapes
  work in SX literals.
- **2026-06-07 — WebSockets** (`lib/dream/websocket.sx`, 16 tests). `dream-websocket
  handler` wraps a `(fn (ws) …)` into an ordinary handler returning a 101 upgrade
  response carrying the ws handler (`dream-websocket?` / `dream-ws-handler` for the
  host to detect + dispatch). `dream-send` / `dream-receive` / `dream-close` /
  `dream-ws-open?` / `dream-ws-broadcast` operate over an injectable io; production io
  is `(perform op)`, tests use `dream-mock-ws` (in-memory inbox/outbox/closed via the
  cell pattern) with `dream-ws-sent` / `dream-ws-closed?` introspection and
  `dream-ws-run` to drive a handler. Echo loop + room broadcast verified.
- **2026-06-07 — Static files** (`lib/dream/static.sx`, 28 tests). `dream-static root`
  mounts at a `**` route and serves files: content-type by extension (mime map),
  weak ETag (`"hash-length"`) with `If-None-Match` → 304 (incl. `*`), and `Range:
  bytes=` requests → 206 with `Content-Range` (open-ended `bytes=N-` supported,
  unsatisfiable → 416). `..`/absolute path traversal → 403; missing → 404; full
  responses advertise `Accept-Ranges`. Filesystem is injectable —
  `dream-static-perform-fs` (host) vs `dream-memory-fs` (in-memory map for tests).
- **2026-06-07 — dream-run** (`lib/dream/run.sx`, 20 tests). `dream-run handler`
  installs the root handler via `(perform (:http/listen {:port :host :app …}))` — no
  socket code, it wraps the existing server. `dream-app handler` is the adapter the
  host invokes per request: raw `{:method :target :headers :body}` → `dream-request`
  → handler → serialised `{:status :headers :body :set-cookies}`, or a `{:status 101
  :websocket …}` upgrade. Bare-string handlers coerced; method defaults to GET;
  set-cookies (from session/flash) flow through. Listen transport injectable
  (`dream-run-with`) so the full wiring is tested with a mock that captures the op and
  re-runs the captured app. `dream-run-port` / `dream-run-opts` variants.
- **2026-06-07 — Demos: hello + counter** (`lib/dream/demos/`, 10 tests). `hello.sx`
  is the canonical router with a `:name` param route. `counter.sx` is a per-session
  visit counter on the session middleware (+ a `/reset` POST that redirects),
  demonstrating session isolation across browsers. End-to-end tests drive both apps
  as the host would. chat (ws) + todo (forms+CSRF) next.
- **2026-06-07 — Demos: chat + todo** (`lib/dream/demos/`, demos suite now 27 tests).
  `chat.sx` is a multi-room WebSocket chat over a room registry (join/leave/members/
  broadcast on the cell pattern); verified three clients see each other's broadcasts
  and a disconnect leaves the room. `todo.sx` is a CRUD list wiring session→csrf→
  router: add/toggle/delete go through `dream-form` (CSRF-guarded), an in-memory store
  holds items, pages render the list + `dream-csrf-tag`; verified the full
  add→render→toggle→delete cycle plus a 403 on a token-less POST. ws object equality
  is by reference, so the `:leave` filter removes exactly the right connection.

## Blockers

_(none — gate green, loop active)_