giles
04b44401fb
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 34s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
196 lines
14 KiB
Markdown
196 lines
14 KiB
Markdown
# Dream-on-SX: OCaml's Dream web framework on the SX CEK
|
||
|
||
`[activated — target user confirmed; gated only on ocaml-on-sx]`
|
||
|
||
Carved out of `plans/ocaml-on-sx.md`. The OCaml-on-SX plan was scoped down to **substrate validation + HM + reference oracle** (Phases 1–5 + minimal stdlib slice). Dream is the practical alternative-stack story — the opposite framing — and is now the **chosen framework layer for the rose-ash host**: the decision is to move off Quart and adopt Dream (not Quart) as the ergonomic HTTP front door over the native SX server. `plans/host-on-sx.md` Phase 4 is the concrete consumer that pulls Dream.
|
||
|
||
**Target user — CONFIRMED.** The earlier "needs a concrete target user" condition is met: rose-ash itself is the user — the subsystems (feed/acl/mod/commerce/identity/…) need an ergonomic HTTP front door, and the project owner has chosen Dream over Quart for it. This plan is no longer cold; it is *gated*, not deferred.
|
||
|
||
**Do not start without (the one remaining gate):**
|
||
1. OCaml-on-SX Phases 1–5 + Phase 6 minimal stdlib green. (As of writing ocaml-on-sx is at 480/480 and advancing — verify its scoreboard covers Phases 1–5 + the stdlib slice before starting.)
|
||
|
||
Until that gate is green, the native server (host-on-sx Phases 1–3) carries the host; do not block host migration on Dream.
|
||
|
||
## Why this might be worth doing (when the time comes)
|
||
|
||
Dream is the cleanest middleware-shaped HTTP framework in any language:
|
||
- `handler = request -> response promise`
|
||
- `middleware = handler -> handler`
|
||
- `m1 @@ m2 @@ handler` — left-fold composition
|
||
|
||
It maps onto SX with almost no impedance — `@@` is function composition, `request → response promise` is `(perform (:http-respond ...))`, middleware chain is plain SX function composition. So the integration cost is low *if* the OCaml-on-SX foundation is in place.
|
||
|
||
The user-facing story: rose-ash users who'd never touch s-expressions might write Dream/OCaml apps that integrate with the same federation, auth, and storage primitives. Demo: a Dream app serving sx.rose-ash.com — the framework that describes the runtime it runs on.
|
||
|
||
## Dream semantic mappings
|
||
|
||
| Dream construct | SX mapping |
|
||
|----------------|-----------|
|
||
| `handler = request -> response promise` | `(fn (req) (perform (:http-respond ...)))` |
|
||
| `middleware = handler -> handler` | `(fn (next) (fn (req) ...))` |
|
||
| `Dream.router [routes]` | `(ocaml-dream-router routes)` — dispatch on method+path |
|
||
| `Dream.get "/path" h` | route record `{:method "GET" :path "/path" :handler h}` |
|
||
| `Dream.scope "/p" [ms] [rs]` | prefix mount with middleware chain |
|
||
| `Dream.param req "name"` | path param extracted during routing |
|
||
| `m1 @@ m2 @@ handler` | `(m1 (m2 handler))` — left-fold composition |
|
||
| `Dream.session_field req "k"` | `(perform (:session-get req "k"))` |
|
||
| `Dream.set_session_field req "k" v` | `(perform (:session-set req "k" v))` |
|
||
| `Dream.flash req` | `(perform (:flash-get req))` |
|
||
| `Dream.form req` | `(perform (:form-parse req))` — returns Ok/Error ADT |
|
||
| `Dream.websocket handler` | `(perform (:websocket handler))` |
|
||
| `Dream.run handler` | starts SX HTTP server with handler as root |
|
||
|
||
## Roadmap
|
||
|
||
The five types: `request`, `response`, `handler = request -> response`, `middleware = handler -> handler`, `route`. Everything else is a function over these.
|
||
|
||
- [x] **Core types** in `lib/dream/types.sx`: request/response records, route record.
|
||
- [x] **Router** in `lib/dream/router.sx`:
|
||
- `dream-get path handler`, `dream-post path handler`, etc. for all HTTP methods.
|
||
- `dream-scope prefix middlewares routes` — prefix mount with middleware chain.
|
||
- `dream-router routes` — dispatch tree, returns handler; no match → 404.
|
||
- Path param extraction: `:name` segments, `**` wildcard.
|
||
- `dream-param req name` — retrieve matched path param.
|
||
- [x] **Middleware** in `lib/dream/middleware.sx`:
|
||
- `dream-pipeline middlewares handler` — compose middleware left-to-right.
|
||
- `dream-no-middleware` — identity.
|
||
- Logger: `(dream-logger next req)` — logs method, path, status, timing.
|
||
- Content-type sniffer.
|
||
- [x] **Sessions** in `lib/dream/session.sx`:
|
||
- Cookie-backed session middleware.
|
||
- `dream-session-field req key`, `dream-set-session-field req key val`.
|
||
- `dream-invalidate-session req`.
|
||
- [x] **Flash messages** in `lib/dream/flash.sx`:
|
||
- `dream-flash-middleware` — single-request cookie store.
|
||
- `dream-add-flash-message req category msg`.
|
||
- `dream-flash-messages req` — returns list of `(category, msg)`.
|
||
- [x] **Forms + CSRF** in `lib/dream/form.sx`:
|
||
- [x] `dream-form req` — returns `(Ok fields)` or `(Err :csrf-token-invalid)`.
|
||
- [x] `dream-multipart req` — multipart form data (in-memory, not yet streaming).
|
||
- [x] CSRF middleware: stateless signed tokens, session-scoped.
|
||
- [x] `dream-csrf-tag req` — returns hidden input fragment for SX templates.
|
||
- [x] **WebSockets** in `lib/dream/websocket.sx`:
|
||
- `dream-websocket handler` — upgrades request; handler `(fn (ws) ...)`.
|
||
- `dream-send ws msg`, `dream-receive ws`, `dream-close ws`.
|
||
- [x] **Static files:** `dream-static root-path` — serves files, ETags, range requests.
|
||
- [ ] **`dream-run`**: wires root handler into SX's `perform (:http-listen ...)`.
|
||
- [ ] **Demos** in `lib/dream/demos/`:
|
||
- `hello.ml` → `lib/dream/demos/hello.sx`: "Hello, World!" route.
|
||
- `counter.ml` → `lib/dream/demos/counter.sx`: in-memory counter with sessions.
|
||
- `chat.ml` → `lib/dream/demos/chat.sx`: multi-room WebSocket chat.
|
||
- `todo.ml` → `lib/dream/demos/todo.sx`: CRUD list with forms + CSRF.
|
||
- [ ] Tests in `lib/dream/tests/`: routing dispatch, middleware composition, session round-trip, CSRF accept/reject, flash read-after-write — 60+ tests.
|
||
|
||
## Stdlib additions Dream will need
|
||
|
||
Dream pushes beyond OCaml-on-SX's Phase 6 minimal stdlib slice. When this plan activates, OCaml-on-SX gets a follow-on phase that adds at minimum:
|
||
|
||
- `Bytes` (binary buffers — request bodies, websocket frames)
|
||
- `Buffer` (mutable string building)
|
||
- `Format` (full pretty-printer, not just `Printf.sprintf`)
|
||
- More `String` (`index_opt`, `contains`, `starts_with`, `ends_with`, `replace_all`)
|
||
- `Sys` (`argv`, `getenv_opt`, `getcwd`)
|
||
- `Hashtbl` extensions (`iter`, `fold`, `length`, `remove`)
|
||
- `Map.Make` / `Set.Make` functors
|
||
|
||
Confirm scope before starting; some of these may be addable as Dream-internal helpers rather than full stdlib modules.
|
||
|
||
## Ground rules
|
||
|
||
- **Scope:** only `lib/dream/**` and `plans/dream-on-sx.md`. Plus the stdlib additions listed above which land in `lib/ocaml/runtime.sx`.
|
||
- **Hard prerequisite:** OCaml-on-SX Phases 1–5 + Phase 6 minimal stdlib. Verify scoreboard before starting.
|
||
- **SX files:** `sx-tree` MCP tools only.
|
||
- **Don't reinvent the SX HTTP server.** Dream wraps the existing `perform (:http-listen ...)` — it does not implement its own listener loop.
|
||
|
||
## Progress log
|
||
|
||
- **2026-06-07 — Core types** (`lib/dream/types.sx`, 41 tests). OCaml gate verified
|
||
green (scoreboard 480/480, Phases 1–5 + Phase 6 stdlib). Dream is implemented in
|
||
plain SX over the CEK — keywords are strings, so headers are dicts with lowercased
|
||
string keys (`:content-type` == `"content-type"`). request (method/target/path/
|
||
query/headers/body/params), response (status/headers/body), route records with
|
||
constructors + accessors; smart response constructors (html/text/json/empty/
|
||
not-found/redirect); `dream-coerce-response` wraps bare strings; query-string
|
||
parsing. Conformance runner `lib/dream/conformance.sh` modelled on flow's.
|
||
- **2026-06-07 — Router** (`lib/dream/router.sx`, 27 tests). `dream-get/post/put/
|
||
delete/patch/head/options/any` route constructors; `dream-router` flattens routes
|
||
(incl. nested scopes) and dispatches by method+path, first-match-wins, 404 on no
|
||
match. Path matching is recursive over `/`-split segments: literal, `:name` binds
|
||
a param, `**` catch-all binds remaining path under key `"**"`. Trailing slashes and
|
||
query strings are ignored for routing. `dream-scope prefix mws routes` prepends the
|
||
prefix and folds the middleware chain (`m1 @@ m2 @@ h`, first = outermost) onto each
|
||
route's handler; nests correctly (inner mw innermost). Shared `dr/apply-middlewares`
|
||
fold will back `dream-pipeline`.
|
||
- **2026-06-07 — Middleware** (`lib/dream/middleware.sx`, 20 tests). `dream-pipeline`
|
||
(reuses `dr/apply-middlewares`), `dream-no-middleware` identity. `dream-logger-with
|
||
clock sink` is the testable core (records `{:method :path :status :elapsed}`);
|
||
`dream-logger` wires it to `(perform (:dream-clock))` / `(perform (:dream-log …))`;
|
||
`dream-log-line` formats one line. `dream-content-type` sniffs body (`<`→html,
|
||
`{`/`[`→json, else text) only when the handler left Content-Type unset. Bonus
|
||
`dream-set-header` and `dream-tap-request` combinators.
|
||
- **2026-06-07 — Sessions** (`lib/dream/session.sx`, 30 tests). Solved the
|
||
request→response mutation-visibility problem the way Dream does: the cookie carries
|
||
only a session id; fields live in an injectable back-end store (the mapping table's
|
||
`(perform (:session-get …))`). `dream-memory-sessions` is an in-memory store built
|
||
on a `set!`-mutated captured `let` binding (no `ref`/`atom` in base env);
|
||
`dream-perform-sessions` is the production back-end. `dream-sessions backend`
|
||
middleware reads/creates the id, attaches `{:sid :io}` to the request, and emits a
|
||
`Set-Cookie` (HttpOnly, SameSite=Lax) only for new sessions. Handler API:
|
||
`dream-session-field` / `dream-set-session-field` / `dream-session-all` /
|
||
`dream-invalidate-session` / `dream-session-id`. Also added shared cookie infra
|
||
(`dr/parse-cookies`, `dream-cookie(s)`, `dr/build-cookie`, `dream-set-cookie`,
|
||
`dream-resp-cookies`, `dream-drop-cookie`) — outgoing cookies accumulate in a
|
||
`:set-cookies` list on the response so multiple Set-Cookie headers don't collide;
|
||
reused by flash + CSRF. Full counter round-trip verified across three requests.
|
||
- **2026-06-07 — Flash** (`lib/dream/flash.sx`, 14 tests). `dream-flash` middleware:
|
||
decodes the incoming `dream.flash` cookie into the request, gives the handler a
|
||
mutable outbox cell (`dr/flash-box`, the same `set!`-captured-`let` trick), then on
|
||
response writes the outbox as a fresh flash cookie, or drops the cookie (Max-Age=0)
|
||
when there were incoming messages but no new ones — so messages show exactly once.
|
||
Handler API: `dream-add-flash-message` / `dream-flash-messages` (returns the
|
||
PREVIOUS request's messages) / `dream-flash-of` (by category) / accessors. Cookie
|
||
codec percent-escapes the `|`/`~`/`%` separators so categories/messages round-trip.
|
||
Read-after-write verified across request boundaries incl. multi-category.
|
||
- **2026-06-07 — Forms + CSRF (urlencoded)** (`lib/dream/form.sx`, 26 tests). Ok/Err
|
||
result values (`dream-ok`/`dream-err` + predicates/accessors). `dream-form-fields`
|
||
parses `application/x-www-form-urlencoded` with a full percent-decoder
|
||
(`%XX` via `char-from-code`, `+`→space). CSRF is stateless + signed + session-
|
||
scoped: token = `sid.signature`, verified by recomputing the signature and checking
|
||
the session id — no server storage. Signing is **injectable** (`dream-csrf-with`);
|
||
the default `dream-csrf-sign-default` is a pure-SX dual-base polynomial keyed hash
|
||
(NOT cryptographic — production should inject a host HMAC). `dream-csrf` attaches
|
||
context (needs the session middleware upstream for the sid); `dream-csrf-token` /
|
||
`dream-csrf-tag` (hidden input for templates); `dream-form` returns `Ok fields` or
|
||
`Err :csrf-token-invalid`; `dream-csrf-protect` auto-rejects unsafe methods (403)
|
||
lacking a valid token. Full session→csrf→form stack verified accept + reject.
|
||
Multipart deferred to the next commit.
|
||
- **2026-06-07 — Multipart** (`lib/dream/form.sx` +9 tests, 35 total). `dream-multipart
|
||
req` parses `multipart/form-data` into parts `{:name :filename :content-type
|
||
:content}`, returns `Ok parts | Err :not-multipart`. Needed a substring splitter
|
||
`dr/split-on` because the `split` primitive is **character-class** based (multi-char
|
||
separators split on every char) — important gotcha. Boundary from the Content-Type
|
||
(handles quoted form); segments filtered to those starting with CRLF; each split on
|
||
the first `\r\n\r\n` into headers/content with one edge CRLF stripped (inner CRLFs
|
||
in file content preserved). `dream-multipart-field` / `dream-multipart-file`
|
||
accessors. In-memory, not streaming (noted for future). `\r`/`\n` string escapes
|
||
work in SX literals.
|
||
- **2026-06-07 — WebSockets** (`lib/dream/websocket.sx`, 16 tests). `dream-websocket
|
||
handler` wraps a `(fn (ws) …)` into an ordinary handler returning a 101 upgrade
|
||
response carrying the ws handler (`dream-websocket?` / `dream-ws-handler` for the
|
||
host to detect + dispatch). `dream-send` / `dream-receive` / `dream-close` /
|
||
`dream-ws-open?` / `dream-ws-broadcast` operate over an injectable io; production io
|
||
is `(perform op)`, tests use `dream-mock-ws` (in-memory inbox/outbox/closed via the
|
||
cell pattern) with `dream-ws-sent` / `dream-ws-closed?` introspection and
|
||
`dream-ws-run` to drive a handler. Echo loop + room broadcast verified.
|
||
- **2026-06-07 — Static files** (`lib/dream/static.sx`, 28 tests). `dream-static root`
|
||
mounts at a `**` route and serves files: content-type by extension (mime map),
|
||
weak ETag (`"hash-length"`) with `If-None-Match` → 304 (incl. `*`), and `Range:
|
||
bytes=` requests → 206 with `Content-Range` (open-ended `bytes=N-` supported,
|
||
unsatisfiable → 416). `..`/absolute path traversal → 403; missing → 404; full
|
||
responses advertise `Accept-Ranges`. Filesystem is injectable —
|
||
`dream-static-perform-fs` (host) vs `dream-memory-fs` (in-memory map for tests).
|
||
|
||
## Blockers
|
||
|
||
_(none — gate green, loop active)_
|