rose-ash/plans/ocaml-on-sx.md

# OCaml-on-SX: OCaml + ReasonML + Dream on the CEK/VM

The meta-circular demo: SX's native evaluator is OCaml, so implementing OCaml on top of
SX closes the loop — the source language of the host is running inside the host it
compiles to. Beyond the elegance, it's practically useful: once OCaml expressions run on
the SX CEK/VM you get Dream (a clean OCaml web framework) almost for free, and ReasonML
is a syntax variant that shares the same transpiler output.

End-state goal: **OCaml programs running on the SX CEK/VM**, with enough of the standard
library to support Dream's middleware model. Dream-on-SX is the integration target —
a `handler`/`middleware`/`router` API that feels idiomatic while running purely in SX.
ReasonML (Phase 8) adds an alternative syntax frontend that targets the same transpiler.

## What this covers that nothing else in the set does

- **Strict ML semantics** — unlike Haskell, OCaml is call-by-value with explicit `Lazy.t`
  for laziness. Pattern match is exhaustive. Polymorphic variants. Structural equality.
- **First-class modules and functors** — modules as values (phase 4); functors as SX
  higher-order functions over module records. Unlike Haskell typeclasses, OCaml's module
  system is explicit and compositional.
- **Mutable state without monads** — `ref`, `:=`, `!` are primitives. Arrays. `Hashtbl`.
  The IO model is direct; `Lwt`/Dream map to `perform`/`cek-resume` for async.
- **Dream's composable HTTP model** — `handler = request -> response promise`,
  `middleware = handler -> handler`. Algebraically clean; `@@` composition maps to SX
  function composition trivially.
- **ReasonML** — same semantics, JS-friendly surface syntax. JSX variant pairs with SX
  component rendering.

## Ground rules

- **Scope:** only touch `lib/ocaml/**`, `lib/dream/**`, `lib/reasonml/**`, and
  `plans/ocaml-on-sx.md`. Do **not** edit `spec/`, `hosts/`, `shared/`, or other
  `lib/<lang>/`.
- **Shared-file issues** go under "Blockers" below with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** OCaml source → AST → SX AST → CEK. No standalone OCaml evaluator.
  The OCaml AST is walked by an `ocaml-eval` function in SX that produces SX values.
- **Type system:** deferred until Phase 5. Phases 1–4 are intentionally untyped —
  get the evaluator right first, then layer HM inference on top.
- **Dream:** implemented as a library in Phase 7; no separate build step. `Dream.run`
  wraps SX's existing HTTP server machinery via `perform`/`cek-resume`.
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick boxes.

## Architecture sketch

```
OCaml source text
    │
    ▼
lib/ocaml/tokenizer.sx   — keywords, operators, string/char literals, comments
    │
    ▼
lib/ocaml/parser.sx      — OCaml AST: let/let rec, fun, match, if, begin/end,
    │                      module/struct/functor, type decls, expressions
    ▼
lib/ocaml/desugar.sx     — surface → core: tuple patterns, or-patterns,
    │                      sequence (;) → (do), when guards, field punning
    ▼
lib/ocaml/transpile.sx   — OCaml AST → SX AST
    │
    ▼
lib/ocaml/runtime.sx     — ADT constructors, module primitives, ref/array ops,
    │                      Stdlib shims, Dream server (phase 7)
    ▼
SX CEK evaluator (both JS and OCaml hosts)
```

## Semantic mappings

| OCaml construct | SX mapping |
|----------------|-----------|
| `let x = e` (top-level) | `(define x e)` |
| `let f x y = e` | `(define (f x y) e)` |
| `let rec f x = e` | `(define (f x) e)` — SX define is already recursive |
| `fun x -> e` | `(fn (x) e)` |
| `e1 \|> f` | `(f e1)` — pipe desugars to reverse application |
| `e1; e2` | `(do e1 e2)` |
| `begin e1; e2; e3 end` | `(do e1 e2 e3)` |
| `if c then e1 else e2` | `(if c e1 e2)` |
| `match x with \| P -> e` | `(match x (P e) ...)` via Phase 6 ADT primitive |
| `type t = A \| B of int` | `(define-type t (A) (B v))` |
| `module M = struct ... end` | SX dict `{:let-bindings ...}` — module as record |
| `functor (M : S) -> ...` | `(fn (M) ...)` — functor as SX lambda over module record |
| `open M` | inject M's bindings into scope via `env-merge` |
| `M.field` | `(get M :field)` |
| `{ r with f = v }` | `(dict-set r :f v)` |
| `ref x` | `(make-ref x)` — mutable cell |
| `!r` | `(deref-ref r)` |
| `r := v` | `(set-ref! r v)` |
| `(a, b, c)` | tagged list `(:tuple a b c)` |
| `[1; 2; 3]` | `(list 1 2 3)` |
| `[| 1; 2; 3 |]` | `(make-array 1 2 3)` (Phase 6) |
| `try e with \| Ex -> h` | `(guard (fn (ex) h) e)` via SX exception system |
| `raise Ex` | `(perform (:raise Ex))` |
| `Printf.printf "%d" x` | `(perform (:print (format "%d" x)))` |

## Dream semantic mappings (Phase 7)

| Dream construct | SX mapping |
|----------------|-----------|
| `handler = request -> response promise` | `(fn (req) (perform (:http-respond ...)))` |
| `middleware = handler -> handler` | `(fn (next) (fn (req) ...))` |
| `Dream.router [routes]` | `(ocaml-dream-router routes)` — dispatch on method+path |
| `Dream.get "/path" h` | route record `{:method "GET" :path "/path" :handler h}` |
| `Dream.scope "/p" [ms] [rs]` | prefix mount with middleware chain |
| `Dream.param req "name"` | path param extracted during routing |
| `m1 @@ m2 @@ handler` | `(m1 (m2 handler))` — left-fold composition |
| `Dream.session_field req "k"` | `(perform (:session-get req "k"))` |
| `Dream.set_session_field req "k" v` | `(perform (:session-set req "k" v))` |
| `Dream.flash req` | `(perform (:flash-get req))` |
| `Dream.form req` | `(perform (:form-parse req))` — returns Ok/Error ADT |
| `Dream.websocket handler` | `(perform (:websocket handler))` |
| `Dream.run handler` | starts SX HTTP server with handler as root |

## Roadmap

### Phase 1 — Tokenizer + parser

- [ ] **Tokenizer:** keywords (`let`, `rec`, `in`, `fun`, `function`, `match`, `with`,
      `type`, `of`, `module`, `struct`, `end`, `functor`, `sig`, `open`, `include`,
      `if`, `then`, `else`, `begin`, `try`, `exception`, `raise`, `mutable`,
      `for`, `while`, `do`, `done`, `and`, `as`, `when`), operators (`->`, `|>`,
      `<|`, `@@`, `@`, `:=`, `!`, `::`, `**`, `:`, `;`, `;;`), identifiers (lower,
      upper/ctor, labels `~label:`, optional `?label:`), char literals `'c'`,
      string literals (escaped + heredoc `{|...|}`), int/float literals,
      line comments `(*` nested block comments `*)`.
- [ ] **Parser:** top-level `let`/`let rec`/`type`/`module`/`exception`/`open`/`include`
      declarations; expressions: literals, identifiers, constructor application,
      lambda, application (left-assoc), binary ops with precedence table,
      `if`/`then`/`else`, `match`/`with`, `try`/`with`, `let`/`in`, `begin`/`end`,
      `fun`/`function`, tuples, list literals, record literals/updates, field access,
      sequences `;`, unit `()`.
- [ ] **Patterns:** constructor, literal, variable, wildcard `_`, tuple, list cons `::`,
      list literal, record, `as`, or-pattern `P1 | P2`, `when` guard.
- [ ] OCaml is **not** indentation-sensitive — no layout algorithm needed.
- [ ] Tests in `lib/ocaml/tests/parse.sx` — 50+ round-trip parse tests.

### Phase 2 — Core evaluator (untyped)

- [ ] `ocaml-eval` entry: walks OCaml AST, produces SX values.
- [ ] `let`/`let rec`/`let ... in` (mutually recursive with `and`).
- [ ] Lambda + application (curried by default — auto-curry multi-param defs).
- [ ] `fun`/`function` (single-arg lambda with immediate match on arg).
- [ ] `if`/`then`/`else`, `begin`/`end`, sequence `;`.
- [ ] Arithmetic, comparison, boolean ops, string `^`, `mod`.
- [ ] Unit `()` value; `ignore`.
- [ ] References: `ref`, `!`, `:=`.
- [ ] Mutable record fields.
- [ ] `for i = lo to hi do ... done` loop; `while cond do ... done`.
- [ ] `try`/`with` — maps to SX `guard`; `raise` via perform.
- [ ] Tests in `lib/ocaml/tests/eval.sx` — 50+ tests, pure + imperative.

### Phase 3 — ADTs + pattern matching

- [ ] `type` declarations: `type t = A | B of t1 * t2 | C of { x: int }`.
- [ ] Constructors as tagged lists: `A` → `(:A)`, `B(1, "x")` → `(:B 1 "x")`.
- [ ] `match`/`with`: constructor, literal, variable, wildcard, tuple, list cons/nil,
      `as` binding, or-patterns, nested patterns, `when` guard.
- [ ] Exhaustiveness: runtime error on incomplete match (no compile-time check yet).
- [ ] Built-in types: `option` (`None`/`Some`), `result` (`Ok`/`Error`),
      `list` (nil/cons), `bool`, `unit`, `exn`.
- [ ] `exception` declarations; built-in: `Not_found`, `Invalid_argument`,
      `Failure`, `Match_failure`.
- [ ] Polymorphic variants (surface syntax `\`Tag value`; runtime same tagged list).
- [ ] Tests in `lib/ocaml/tests/adt.sx` — 40+ tests: ADTs, match, option/result.

### Phase 4 — Modules + functors

- [ ] `module M = struct let x = 1 let f y = x + y end` → SX dict `{:x 1 :f <fn>}`.
- [ ] `module type S = sig val x : int val f : int -> int end` → interface record
      (runtime stub; typed checking in Phase 5).
- [ ] `module M : S = struct ... end` — coercive sealing (runtime: pass-through).
- [ ] `functor (M : S) -> struct ... end` → SX `(fn (M) ...)`.
- [ ] `module F = Functor(Base)` — functor application.
- [ ] `open M` — merge M's dict into current env (`env-merge`).
- [ ] `include M` — same as open at structure level.
- [ ] `M.name` — dict get via `:name` key.
- [ ] First-class modules (pack/unpack) — deferred to Phase 5.
- [ ] Standard module hierarchy: `List`, `Option`, `Result`, `String`, `Char`,
      `Int`, `Float`, `Bool`, `Unit`, `Printf`, `Format` (stubs, filled in Phase 6).
- [ ] Tests in `lib/ocaml/tests/modules.sx` — 30+ tests.

### Phase 5 — Hindley-Milner type inference

- [ ] Algorithm W: `gen`/`inst`, `unify`, `infer-expr`, `infer-decl`.
- [ ] Type variables: `'a`, `'b`; unification with occur-check.
- [ ] Let-polymorphism: generalise at let-bindings.
- [ ] ADT types: `type 'a option = None | Some of 'a`.
- [ ] Function types, tuple types, record types.
- [ ] Type signatures: `val f : int -> int` — verify against inferred type.
- [ ] Module type checking: seal against `sig` (Phase 4 stubs become real checks).
- [ ] Error reporting: position-tagged errors with expected vs actual types.
- [ ] First-class modules: `(module M : S)` pack; `(val m : (module S))` unpack.
- [ ] No rank-2 polymorphism, no GADTs (out of scope).
- [ ] Tests in `lib/ocaml/tests/types.sx` — 60+ inference tests.

### Phase 6 — Standard library

- [ ] `List`: `map`, `filter`, `fold_left`, `fold_right`, `length`, `rev`, `append`,
      `concat`, `flatten`, `iter`, `iteri`, `mapi`, `for_all`, `exists`, `find`,
      `find_opt`, `mem`, `assoc`, `assq`, `sort`, `stable_sort`, `nth`, `hd`, `tl`,
      `init`, `combine`, `split`, `partition`.
- [ ] `Option`: `map`, `bind`, `fold`, `get`, `value`, `join`, `iter`, `to_list`,
      `to_result`, `is_none`, `is_some`.
- [ ] `Result`: `map`, `bind`, `fold`, `get_ok`, `get_error`, `map_error`,
      `to_option`, `is_ok`, `is_error`.
- [ ] `String`: `length`, `get`, `sub`, `concat`, `split_on_char`, `trim`,
      `uppercase_ascii`, `lowercase_ascii`, `contains`, `starts_with`, `ends_with`,
      `index_opt`, `replace_all` (non-stdlib but needed).
- [ ] `Char`: `code`, `chr`, `escaped`, `lowercase_ascii`, `uppercase_ascii`.
- [ ] `Int`/`Float`: arithmetic, `to_string`, `of_string_opt`, `min_int`, `max_int`.
- [ ] `Hashtbl`: `create`, `add`, `replace`, `find`, `find_opt`, `remove`, `mem`,
      `iter`, `fold`, `length` — backed by SX mutable dict.
- [ ] `Map.Make` functor — balanced BST backed by SX sorted dict.
- [ ] `Set.Make` functor.
- [ ] `Printf`: `sprintf`, `printf`, `eprintf` — format strings via `(format ...)`.
- [ ] `Sys`: `argv`, `getenv_opt`, `getcwd` — via `perform` IO.
- [ ] Scoreboard runner: `lib/ocaml/conformance.sh` + `scoreboard.json`.
- [ ] Target: 150+ tests across all stdlib modules.

### Phase 7 — Dream web framework (`lib/dream/`)

The five types: `request`, `response`, `handler = request -> response`,
`middleware = handler -> handler`, `route`. Everything else is a function over these.

- [ ] **Core types** in `lib/dream/types.sx`: request/response records, route record.
- [ ] **Router** in `lib/dream/router.sx`:
      - `dream-get path handler`, `dream-post path handler`, etc. for all HTTP methods.
      - `dream-scope prefix middlewares routes` — prefix mount with middleware chain.
      - `dream-router routes` — dispatch tree, returns handler; no match → 404.
      - Path param extraction: `:name` segments, `**` wildcard.
      - `dream-param req name` — retrieve matched path param.
- [ ] **Middleware** in `lib/dream/middleware.sx`:
      - `dream-pipeline middlewares handler` — compose middleware left-to-right.
      - `dream-no-middleware` — identity.
      - Logger: `(dream-logger next req)` — logs method, path, status, timing.
      - Content-type sniffer.
- [ ] **Sessions** in `lib/dream/session.sx`:
      - Cookie-backed session middleware.
      - `dream-session-field req key`, `dream-set-session-field req key val`.
      - `dream-invalidate-session req`.
- [ ] **Flash messages** in `lib/dream/flash.sx`:
      - `dream-flash-middleware` — single-request cookie store.
      - `dream-add-flash-message req category msg`.
      - `dream-flash-messages req` — returns list of `(category, msg)`.
- [ ] **Forms + CSRF** in `lib/dream/form.sx`:
      - `dream-form req` — returns `(Ok fields)` or `(Err :csrf-token-invalid)`.
      - `dream-multipart req` — streaming multipart form data.
      - CSRF middleware: stateless signed tokens, session-scoped.
      - `dream-csrf-tag req` — returns hidden input fragment for SX templates.
- [ ] **WebSockets** in `lib/dream/websocket.sx`:
      - `dream-websocket handler` — upgrades request; handler `(fn (ws) ...)`.
      - `dream-send ws msg`, `dream-receive ws`, `dream-close ws`.
- [ ] **Static files:** `dream-static root-path` — serves files, ETags, range requests.
- [ ] **`dream-run`**: wires root handler into SX's `perform (:http-listen ...)`.
- [ ] **Demos** in `lib/dream/demos/`:
      - `hello.ml` → `lib/dream/demos/hello.sx`: "Hello, World!" route.
      - `counter.ml` → `lib/dream/demos/counter.sx`: in-memory counter with sessions.
      - `chat.ml` → `lib/dream/demos/chat.sx`: multi-room WebSocket chat.
      - `todo.ml` → `lib/dream/demos/todo.sx`: CRUD list with forms + CSRF.
- [ ] Tests in `lib/dream/tests/`: routing dispatch, middleware composition,
      session round-trip, CSRF accept/reject, flash read-after-write — 60+ tests.

### Phase 8 — ReasonML syntax variant (`lib/reasonml/`)

ReasonML is OCaml with a JS-friendly surface: semicolons, `let` with `=` everywhere,
`=>` for lambdas, `switch` for match, `{j|...|j}` string interpolation. Same semantics —
different tokenizer + parser, same `lib/ocaml/transpile.sx` output.

- [ ] **Tokenizer** in `lib/reasonml/tokenizer.sx`:
      - `let x = e;` binding syntax (semicolons required).
      - `(x, y) => e` arrow function syntax.
      - `switch (x) { | Pat => e | ... }` for match.
      - JSX: `<Comp prop=val />`, `<div>children</div>`.
      - String interpolation: `{j|hello $(name)|j}`.
      - Type annotations: `x : int`, `let f : int => int = x => x + 1`.
- [ ] **Parser** in `lib/reasonml/parser.sx`:
      - Produce same OCaml AST nodes as `lib/ocaml/parser.sx`.
      - JSX → SX component calls: `<Comp x=1 />` → `(~comp :x 1)`.
      - Multi-arg functions: `(x, y) => e` → auto-curried pair.
- [ ] Shared transpiler: `lib/reasonml/transpile.sx` delegates to
      `lib/ocaml/transpile.sx` (parse → ReasonML AST → OCaml AST → SX AST).
- [ ] Tests in `lib/reasonml/tests/`: tokenizer, parser, eval, JSX — 40+ tests.
- [ ] ReasonML Dream demos: translate Phase 7 demos to ReasonML syntax.

## The meta-circular angle

SX is bootstrapped to OCaml (`hosts/ocaml/`). Running OCaml inside SX running on OCaml is
the "mother tongue" closure: OCaml → SX → OCaml. This means:

- The OCaml host's native pattern matching and ADTs are exact reference semantics for
  the SX-level implementation — any mismatch is a bug.
- The SX `match` / `define-type` primitives (Phase 6 of the primitives roadmap) were
  built knowing OCaml was the intended target.
- When debugging the transpiler, the OCaml REPL is always available as oracle.
- Dream running in SX can serve the sx.rose-ash.com docs site — the framework that
  describes the runtime it runs on.

## Key dependencies

- **Phase 6 ADT primitive** (`define-type`/`match`) — required before Phase 3.
- **`perform`/`cek-resume`** IO suspension — required before Phase 7 (Dream async).
- **HO forms** and first-class lambdas — already in spec, no blocker.
- **Module system** (Phase 4) is independent of type inference (Phase 5) — can overlap.
- **ReasonML** (Phase 8) can start once OCaml parser is stable (after Phase 2).

## Progress log

_Newest first._

_(awaiting phase 1)_

## Blockers

_(none yet)_