rose-ash/plans/host-on-sx.md

# host-on-sx: The SX web host — off Quart, onto the kernel (Dream-bound)

> **DRAFT outline.** The integration boundary that turns the subsystem libraries
> into running services, and the strangler path off Python/Quart. This is the
> dependency hub — it imports every subsystem. Decision recorded below: native
> server + SXTP **now**, `dream-on-sx` framework layer **next**, Python only at the
> external-integration edges.

The subsystems (`feed`, `search`, `acl`, `mod`, `flow`, `commerce`, `identity`,
`content`, `events`) are libraries. Something has to receive an HTTP request, route
it, call the right subsystem, and serialize the response. Today that's Python/Quart
— the one large non-SX component in the stack: separate runtime, deploy, and
failure mode. The goal is to move the web/host/domain layer onto the SX substrate
and retire Quart, **incrementally (strangler-fig), never big-bang.**

This is already underway: a native OCaml HTTP server is live in prod on
`sx.rose-ash.com` (~3ms cached, ~323 req/s, ~2MB RSS), `defhandler`/`defpage`
exist, and a partial **SXTP** protocol is specced. That is the unblocked near-term
host — no `ocaml-on-sx` dependency.

## Two layers, two timelines

1. **Now (unblocked): native server + SXTP adapter + SX handlers.** Route rose-ash
   endpoints onto the SX host one at a time. Each migrated endpoint is an SX
   handler dispatching to a subsystem; Quart proxies the rest until cut over.
2. **Next: `dream-on-sx` as the framework layer.** Dream gives Quart-grade
   ergonomics — typed routing, middleware stacks, sessions, CSRF. It is gated on
   `ocaml-on-sx` Phases 1–5 + minimal stdlib. **This plan is the concrete target
   user that un-parks `dream-on-sx`** (see `plans/dream-on-sx.md`): "the subsystems
   need an HTTP front door" is the real feature pulling Dream. Until then, do not
   block migration on Dream — the native server is sufficient.
3. **Always: Python only at the edges.** External integrations — SumUp payments,
   Ghost CMS, ActivityPub crypto, IPFS/Kubo — ride Python libraries today. They
   stay as thin injected adapters (Python/FFI) behind subsystem interfaces until
   native replacements exist. "Drop Quart" ≠ "drop every line of Python."

## Status (rolling)

`bash lib/host/conformance.sh` → **82/82** (5 suites: handler, middleware, sxtp,
router, feed). Phases 1 & 2 DONE; Phase 3 (strangler ledger) next.

## Ground rules

- **Scope:** `lib/host/**` and `plans/host-on-sx.md`. May **import** every subsystem
  + the kernel's server/SXTP surface. Do **not** edit `spec/`, `hosts/`, `shared/`,
  or subsystem internals — wire to their public APIs only. Host-primitive / server
  changes belong in `hosts/` (out of scope) → Blockers.
- **Architecture:** a route maps (method, path) → handler; a handler is an SX fn
  `request -> response` that calls subsystem APIs; middleware is composed handlers
  (auth via `identity`, permission via `acl`, mute via subsystem prefs). SXTP is the
  wire format between host and subsystem-as-service.
- **Migration discipline:** each endpoint moved must be behavior-equivalent to its
  Quart original (golden-response test before flip). Keep a migration ledger.
- **Commits:** one feature per commit. Progress log + tick boxes.

## Architecture sketch

```
HTTP request                            HTTP response
        │                                       ▲
        ▼                                       │
native OCaml http server (prod)  ──────►  lib/host/router.sx
   (hosts/ — out of scope)                  — (method,path) → handler
        │                                       ▲
        ▼                                       │
lib/host/middleware.sx                  lib/host/handler.sx
  — auth(identity) ∘ acl ∘ mute ∘ ...     — request → subsystem call → response
        │                                       ▲
        ▼                                       │
lib/host/sxtp.sx                        subsystem APIs (feed/search/commerce/…)
  — wire format, host↔service             — called via public interfaces
        │
        └── external edges: SumUp / Ghost / AP / IPFS  → injected Python/FFI adapters
```

## Phase 1 — Router + handler + one real endpoint
- [x] `router.sx` — `host/make-app` assembles per-domain route groups + a built-in
      `/health` probe into one Dream router (reuses Dream's `dr/flatten-routes`)
- [x] `handler.sx` — JSON envelope (`host/ok`/`host/ok-status`/`host/error`),
      status-carrying `host/json-status` (Dream's `dream-json` is 200-only), and
      `host/query-int`. A host handler IS a Dream handler (request -> response).
- [x] migrate ONE read endpoint: `GET /feed` (`lib/host/feed.sx`) reads
      `feed/all` + stream combinators, serialises recent-first; `?actor=` filter,
      `?limit=` cap. Golden test asserts body == subsystem recent stream + envelope.
- [x] `conformance.sh` (mirrors `lib/dream`'s runner) — 28/28

## Phase 2 — Middleware + SXTP
- [x] `middleware.sx` — composable layers as `handler->handler`: `host/wrap-errors`
      (JSON 500), `host/require-auth` (bearer -> principal, JSON 401, INJECTED token
      resolver), `host/require-permission` (ACL `acl/permit?` gate, JSON 403,
      INJECTED resource extractor), `host/pipeline` (first = outermost). Reuses
      Dream's `dream-bearer-token` + `dream-catch-with`; calls lib/acl public API.
      Mute/prefs layer deferred (no blocker, add when a domain needs it).
- [x] `sxtp.sx` — host↔subsystem wire format (per `applications/sxtp/spec.sx`).
      Message algebra (`sxtp/request`/`response`/`condition`/`event` + status
      helpers `sxtp/ok`/`created`/`not-found`/`forbidden`/`invalid`/`fail`) as
      string-keyed dicts; verb/status/type as symbols (ride the wire bare). Codec:
      `sxtp/serialize` (dict → `text/sx` list form, deterministic field order,
      nested messages in their own list form, no `:msg` leak) and `sxtp/parse`
      (`text/sx` → dict, deep keyword-token→string normaliser). Dream bridge:
      `sxtp/from-dream` (HTTP req → SXTP req, method→verb, query→params) and
      `sxtp/to-dream` (SXTP resp → HTTP resp, status→code, body→`text/sx`).
- [x] migrate a write endpoint (auth + permission + action): `POST /feed`
      (`host/feed-write-routes resolve`) — auth ∘ ACL("post","feed") ∘ wrap-errors
      over `host/feed-create`, which parses the JSON body and `feed/post`s it (201);
      non-object body -> 400. Created activity is readable back via `GET /feed`.

## Phase 3 — Strangler migration ledger
- [ ] enumerate Quart endpoints; track migrated vs proxied
- [ ] golden-response harness vs the live Quart responses
- [ ] cut over a whole domain (smallest: `likes` or `relations`) as proof

## Phase 4 — Dream framework layer (gated)
- [ ] gate: `ocaml-on-sx` Phases 1–5 + minimal stdlib green
- [ ] adopt `dream-on-sx` routing/middleware/session ergonomics over the same handlers
- [ ] re-home external adapters as native where replacements land

## Progress log

- **Phase 1 (DONE, 28/28).** `lib/host/{handler,router,feed}.sx` + three test
  suites + `conformance.sh`. The host is a thin wiring layer: a host handler is a
  Dream handler that calls a subsystem public API and serialises the result via a
  shared JSON envelope. First migrated endpoint: `GET /feed`.
  - **Decision — build on Dream from Phase 1, not a throwaway native model.** The
    plan front-matter gated Dream to Phase 4, but `dream-on-sx` is merged
    (commit fe958bda) and its gate (`ocaml-on-sx` P1–5+P6) is green (480/480), so
    reinventing request/response + routing would be pure duplication. Host reuses
    Dream's `types.sx` (request/response dicts), `json.sx` (encode), and
    `router.sx` (`dream-router`/`dream-get`/`dr/flatten-routes`). Phase 4's
    "adopt Dream ergonomics" is therefore largely already satisfied; what remains
    for Phase 4 is the live wiring against the real OCaml HTTP server + session.
  - The OCaml server handing a `dream-request`-shaped dict to SX handlers is a
    `hosts/` change (out of scope) — tracked under Blockers as the eventual
    live-wiring step. For now the host layer is exercised purely via conformance.

- **Phase 2 (middleware + write endpoint DONE, 43/43).** `lib/host/middleware.sx`
  + a guarded `POST /feed`. Middleware is plain function composition over Dream's
  primitives; auth/permission *policy* is injected (token resolver, resource
  extractor) so the layer is policy-free and testable. ACL authorisation runs
  against lib/acl's public `acl/permit?` (string atoms work — no symbol coercion
  needed). The write path proves the auth ∘ permission ∘ action stack end-to-end:
  401 unauth, 403 unpermitted, 201 + readback on success, 400 on bad body.
- **Phase 2 COMPLETE (82/82).** `lib/host/sxtp.sx` adds the SXTP codec + Dream
  bridge (39-test suite). Key representation calls, learned by probing the runtime:
  keywords are strings at eval time but the `serialize` primitive renders
  string-keyed dicts back as `{:k v}` and symbols bare — so messages are
  string-keyed dicts with verb/status/type as symbols, and a small str-based
  emitter produces wire-faithful list form. `parse` needs a deep normaliser
  because parsed keyword tokens are a distinct type (not `=` to string literals).
  `unquote-splicing` is unreliable here, so the serializer is str-based, not
  quasiquote-based.
  - **Next: Phase 3 — strangler migration ledger.** Enumerate the Quart endpoints
    (use the `rose-ash-services` `svc_routes` MCP tool), track migrated vs proxied,
    and stand up a golden-response harness against the live Quart responses. Then
    cut over the smallest whole domain (`likes` or `relations`) as proof.

## Blockers

- **Live wiring to the native OCaml HTTP server** (Phase 3/4): the prod server in
  `hosts/` must hand SX handlers a `dream-request` dict and serialise the returned
  `dream-response`. That is a `hosts/` change (out of scope for this loop, which is
  `lib/host/**` only). Until then, endpoints are verified via `conformance.sh`, not
  HTTP. Not blocking Phase 2 (middleware + SXTP + a write endpoint).
- **Worktree tooling:** in this `loops/host` worktree every sx-tree *write* tool
  (`sx_write_file`, `sx_replace_node`, …) raises `yojson "Expected string, got
  null"` at the MCP layer — same class as the `loops/dream` worktree gotcha, but
  here even `sx_write_file` fails. Read-side sx-tree tools work. New `.sx` files
  were created with the `Write` tool (the .sx hook is inactive in this worktree)
  and each validated afterwards with `sx_validate` to keep the parse guarantee.