# host-on-sx: The SX web host — off Quart, onto the kernel (Dream-bound) > **DRAFT outline.** The integration boundary that turns the subsystem libraries > into running services, and the strangler path off Python/Quart. This is the > dependency hub — it imports every subsystem. Decision recorded below: native > server + SXTP **now**, `dream-on-sx` framework layer **next**, Python only at the > external-integration edges. The subsystems (`feed`, `search`, `acl`, `mod`, `flow`, `commerce`, `identity`, `content`, `events`) are libraries. Something has to receive an HTTP request, route it, call the right subsystem, and serialize the response. Today that's Python/Quart — the one large non-SX component in the stack: separate runtime, deploy, and failure mode. The goal is to move the web/host/domain layer onto the SX substrate and retire Quart, **incrementally (strangler-fig), never big-bang.** This is already underway: a native OCaml HTTP server is live in prod on `sx.rose-ash.com` (~3ms cached, ~323 req/s, ~2MB RSS), `defhandler`/`defpage` exist, and a partial **SXTP** protocol is specced. That is the unblocked near-term host — no `ocaml-on-sx` dependency. ## Two layers, two timelines 1. **Now (unblocked): native server + SXTP adapter + SX handlers.** Route rose-ash endpoints onto the SX host one at a time. Each migrated endpoint is an SX handler dispatching to a subsystem; Quart proxies the rest until cut over. 2. **Next: `dream-on-sx` as the framework layer.** Dream gives Quart-grade ergonomics — typed routing, middleware stacks, sessions, CSRF. It is gated on `ocaml-on-sx` Phases 1–5 + minimal stdlib. **This plan is the concrete target user that un-parks `dream-on-sx`** (see `plans/dream-on-sx.md`): "the subsystems need an HTTP front door" is the real feature pulling Dream. Until then, do not block migration on Dream — the native server is sufficient. 3. **Always: Python only at the edges.** External integrations — SumUp payments, Ghost CMS, ActivityPub crypto, IPFS/Kubo — ride Python libraries today. They stay as thin injected adapters (Python/FFI) behind subsystem interfaces until native replacements exist. "Drop Quart" ≠ "drop every line of Python." ## Status (rolling) `bash lib/host/conformance.sh` → **171/171** (9 suites: handler, middleware, sxtp, router, feed, relations, blog, server, ledger). **Blog now runs on the EDITOR's content model** (`sx_content` = SX element markup, what `blog/sx/editor.sx` emits), NOT content-on-sx CtDoc: a post is a `{slug,title,sx_content,status}` record in the durable persist **KV**, and a post page is `render-to-html (parse sx_content)`. Full CRUD + an editor form-ingest endpoint (`POST /new`, form-urlencoded) + JSON API, writes auth+ACL guarded. **`render-to-html` is fast (~0ms)** — it doesn't hit the JIT-miscompiled Smalltalk path, so blog rendering is no longer the 2s problem (that was content-on-sx's `asHTML`). > **Per-request IO (kernel) — FIXED.** `http-listen` handlers used to run via > `Sx_runtime.sx_call` (bare CEK, no IO resolution), so a handler doing a durable > `persist/read` returned an unresolved suspension. Fixed in `sx_server.ml`: the > handler now runs through `cek_run_with_io` (`Sx_ref.continue_with_call` → > `cek_run_with_io`), the same IO-driving runner the REPL uses — it resolves > persist ops via `Sx_persist_store.handle_op` between CEK steps. Verified: > handlers do per-request durable reads + writes (incl. 10 concurrent, 15 events > on disk, no corruption); handler errors don't crash the server. NOTE: this is > the per-request *IO* fix; it does NOT speed up the interpreted Smalltalk render > (`/welcome/` still ~2s) — that's a separate concern, addressed by caching the > rendered HTML at boot. (Pre-existing: an erroring handler closes the connection > with no response instead of a 500 — worth improving later.) > > **Render speed (separate from IO) — NOT precompiled.** `/welcome/` is ~2s because > the interpreted Smalltalk-on-SX render runs on the tree-walking CEK: the JIT hook > (`register_jit_hook`) is installed only in `--http` page mode, not the epoch/ > http-listen serving mode (`make_server_env`), so zero `[jit]` activity. Enabling > it in that mode breaks correctness (router 3/6, feed 4/11, … — the known JIT- > bytecode bug on complex nested ASTs, which the Smalltalk evaluator is). So the > render is slow until the JIT compiler is fixed (big win, broad payoff — its own > loop) or the Smalltalk interpreter is optimised. Blog is FULLY DYNAMIC (reads > store + renders per request, no cache) — slowness is honest, not hidden. Phases 1 & 2 DONE; Phase 3 cut-over landed (50% off Quart). **The host now serves live HTTP** — `lib/host/server.sx` bridges the native `http-listen` server to the Dream app and `lib/host/serve.sh` boots it (verified: GET /health, /feed, /feed?actor=, relations get-children/ get-parents all serve real JSON on a host port; unknown→404). Remaining: golden harness vs live Quart, internal-HMAC middleware, docker stack + Caddy subdomain. ## Ground rules - **Scope:** `lib/host/**` and `plans/host-on-sx.md`. May **import** every subsystem + the kernel's server/SXTP surface. Do **not** edit `spec/`, `hosts/`, `shared/`, or subsystem internals — wire to their public APIs only. Host-primitive / server changes belong in `hosts/` (out of scope) → Blockers. - **Architecture:** a route maps (method, path) → handler; a handler is an SX fn `request -> response` that calls subsystem APIs; middleware is composed handlers (auth via `identity`, permission via `acl`, mute via subsystem prefs). SXTP is the wire format between host and subsystem-as-service. - **Migration discipline:** each endpoint moved must be behavior-equivalent to its Quart original (golden-response test before flip). Keep a migration ledger. - **Commits:** one feature per commit. Progress log + tick boxes. ## Architecture sketch ``` HTTP request HTTP response │ ▲ ▼ │ native OCaml http server (prod) ──────► lib/host/router.sx (hosts/ — out of scope) — (method,path) → handler │ ▲ ▼ │ lib/host/middleware.sx lib/host/handler.sx — auth(identity) ∘ acl ∘ mute ∘ ... — request → subsystem call → response │ ▲ ▼ │ lib/host/sxtp.sx subsystem APIs (feed/search/commerce/…) — wire format, host↔service — called via public interfaces │ └── external edges: SumUp / Ghost / AP / IPFS → injected Python/FFI adapters ``` ## Phase 1 — Router + handler + one real endpoint - [x] `router.sx` — `host/make-app` assembles per-domain route groups + a built-in `/health` probe into one Dream router (reuses Dream's `dr/flatten-routes`) - [x] `handler.sx` — JSON envelope (`host/ok`/`host/ok-status`/`host/error`), status-carrying `host/json-status` (Dream's `dream-json` is 200-only), and `host/query-int`. A host handler IS a Dream handler (request -> response). - [x] migrate ONE read endpoint: `GET /feed` (`lib/host/feed.sx`) reads `feed/all` + stream combinators, serialises recent-first; `?actor=` filter, `?limit=` cap. Golden test asserts body == subsystem recent stream + envelope. - [x] `conformance.sh` (mirrors `lib/dream`'s runner) — 28/28 ## Phase 2 — Middleware + SXTP - [x] `middleware.sx` — composable layers as `handler->handler`: `host/wrap-errors` (JSON 500), `host/require-auth` (bearer -> principal, JSON 401, INJECTED token resolver), `host/require-permission` (ACL `acl/permit?` gate, JSON 403, INJECTED resource extractor), `host/pipeline` (first = outermost). Reuses Dream's `dream-bearer-token` + `dream-catch-with`; calls lib/acl public API. Mute/prefs layer deferred (no blocker, add when a domain needs it). - [x] `sxtp.sx` — host↔subsystem wire format (per `applications/sxtp/spec.sx`). Message algebra (`sxtp/request`/`response`/`condition`/`event` + status helpers `sxtp/ok`/`created`/`not-found`/`forbidden`/`invalid`/`fail`) as string-keyed dicts; verb/status/type as symbols (ride the wire bare). Codec: `sxtp/serialize` (dict → `text/sx` list form, deterministic field order, nested messages in their own list form, no `:msg` leak) and `sxtp/parse` (`text/sx` → dict, deep keyword-token→string normaliser). Dream bridge: `sxtp/from-dream` (HTTP req → SXTP req, method→verb, query→params) and `sxtp/to-dream` (SXTP resp → HTTP resp, status→code, body→`text/sx`). - [x] migrate a write endpoint (auth + permission + action): `POST /feed` (`host/feed-write-routes resolve`) — auth ∘ ACL("post","feed") ∘ wrap-errors over `host/feed-create`, which parses the JSON body and `feed/post`s it (201); non-object body -> 400. Created activity is readable back via `GET /feed`. ## Phase 3 — Strangler migration ledger - [x] enumerate Quart endpoints; track migrated vs proxied — `ledger.sx`: a catalogue of every endpoint (domain, method, path, Quart original, status `:native`/`:migrated`/`:proxied`, SX handler) + queries (by-status/by-domain, `host/ledger-find`, `host/ledger-served?`, distinct domains) and `host/ledger-coverage` (off-Quart % = (migrated+native)/total). Seeded with the live state: feed reads+writes migrated, `/health` native, the internal-only `relations`/`likes` data+action endpoints proxied. - [ ] golden-response harness vs the live Quart responses - [x] cut over a whole domain (`relations`) as proof — the CONTAINER relations are fully on the host (`lib/host/relations.sx`): reads `GET .../get-children` + `/get-parents` → `relations/children`/`parents`; writes `POST .../attach-child` + `/detach-child` → `relations/relate`/`unrelate`, behind the auth+ACL pipeline (mirrors POST /feed). Node model: graph atom = symbol `"type:id"`, edge = relation-type; `child`/`parent-type` params filter by `"type:"` prefix. Closed-loop test: attach → visible via get-children → detach → gone. The TYPED actions (`relate`/`unrelate`/`can-relate`) stay proxied by design — registry + cardinality validation lib/relations lacks. ## Phase 4 — Live wiring + Dream framework layer - [x] native `http-listen` ↔ Dream-app bridge (`lib/host/server.sx`: `host/native-handler`/`host/serve`) + `lib/host/serve.sh` launcher. Serves real HTTP on a host port — verified live (health/feed/relations reads + 404). - [x] promote into the docker stack + a Caddy subdomain — **LIVE at `https://blog.rose-ash.com`** (reusing a down Quart subdomain). New compose service `sx_host` (`docker-compose.dev-sx-host.yml`, container `sx-dev-sx_host-1`) runs `serve.sh` on `externalnet`; Caddy reverse-proxies `blog.rose-ash.com` → `sx-dev-sx_host-1:8000`. Required a `hosts/` fix: `http-listen` bound `inet_addr_loopback` only — added `SX_HTTP_HOST` env (default loopback; stack sets `0.0.0.0`) in `sx_server.ml`, rebuilt this worktree's binary. Verified: `/health`, `/feed`, relations reads serve real JSON through Cloudflare→Caddy; `/` 404 (no root route yet). `rose-ash.com` untouched. (Inode-pinned bind-mount gotcha: editing `/root/caddy/Caddyfile` via a tool swaps its inode so the container kept the old content — loaded live via reload-from-non-bind-path, then RECONCILED by restarting Caddy so the bind re-points to the corrected file. Verified post-restart: blog serves, and `sx.rose-ash.com`/`rose-ash.com` survived.) - [x] blog published-post read endpoint — `lib/host/blog.sx`: `GET //` renders a content-on-sx `CtDoc` to HTML via `content/html` (anonymous, world-visible). In-memory slug→doc registry now (swap `host/blog-lookup` for a persist-backed content stream later, handler/route unchanged). `:slug` catch-all mounted LAST so domain routes win. **LIVE**: `blog.rose-ash.com/ welcome/` renders real HTML through Caddy. Needs Smalltalk+persist+content preloads + `(st-bootstrap-classes!)`+`(content/bootstrap!)` (self-bootstraps at load). - [ ] proxy-to-Quart fallback for un-migrated paths (strangler requirement before a real subdomain fronts users). - [ ] internal-HMAC middleware on `/internal/*` (service-to-service auth; protocol checks native, signature check needs an HMAC-SHA256 kernel prim — absent today). - [ ] (gated) adopt `dream-on-sx` session/CSRF ergonomics; re-home external adapters as native where replacements land. ## Phase 5 — Generic interactive SX-page serving (host SSR) **The generic gap.** A host serves three classes: (1) JSON/data endpoints — DONE; (2) static content pages — DONE (`render-to-html` on *parsed* markup, e.g. blog post `sx_content`); (3) **interactive UI pages** — component/island trees with attributes + client behaviour — **the host cannot do this at all.** The "editor problem" is one instance; dashboards, account, market-browse, any admin screen are the same gap. The capability — not the editor — is the deliverable. **Why `render-to-html` alone is insufficient (proven).** `render-to-html` on parsed markup handles attributes (`
`); but an *evaluated* component tree mangles them (`(form :id ..)` → `
idpost-new-form…`) because in the host preload tags don't collect keyword args as attrs. The `--http` docs server already does this correctly via its component-render + shell pipeline. So: reuse that pipeline, don't reinvent or patch per-component. **Reuse, don't rebuild.** The kernel already has: `~shared:shell/sx-page-shell` (emits `` + inlined component/island defs in `