rose-ash/plans/host-on-sx.md

host-on-sx: The SX web host — off Quart, onto the kernel (Dream-bound)

DRAFT outline. The integration boundary that turns the subsystem libraries into running services, and the strangler path off Python/Quart. This is the dependency hub — it imports every subsystem. Decision recorded below: native server + SXTP now, dream-on-sx framework layer next, Python only at the external-integration edges.

The subsystems (feed, search, acl, mod, flow, commerce, identity, content, events) are libraries. Something has to receive an HTTP request, route it, call the right subsystem, and serialize the response. Today that's Python/Quart — the one large non-SX component in the stack: separate runtime, deploy, and failure mode. The goal is to move the web/host/domain layer onto the SX substrate and retire Quart, incrementally (strangler-fig), never big-bang.

This is already underway: a native OCaml HTTP server is live in prod on sx.rose-ash.com (~3ms cached, ~323 req/s, ~2MB RSS), defhandler/defpage exist, and a partial SXTP protocol is specced. That is the unblocked near-term host — no ocaml-on-sx dependency.

Two layers, two timelines

1. Now (unblocked): native server + SXTP adapter + SX handlers. Route rose-ash endpoints onto the SX host one at a time. Each migrated endpoint is an SX handler dispatching to a subsystem; Quart proxies the rest until cut over.
2. Next: dream-on-sx as the framework layer. Dream gives Quart-grade ergonomics — typed routing, middleware stacks, sessions, CSRF. It is gated on ocaml-on-sx Phases 1–5 + minimal stdlib. This plan is the concrete target user that un-parks dream-on-sx (see plans/dream-on-sx.md): "the subsystems need an HTTP front door" is the real feature pulling Dream. Until then, do not block migration on Dream — the native server is sufficient.
3. Always: Python only at the edges. External integrations — SumUp payments, Ghost CMS, ActivityPub crypto, IPFS/Kubo — ride Python libraries today. They stay as thin injected adapters (Python/FFI) behind subsystem interfaces until native replacements exist. "Drop Quart" ≠ "drop every line of Python."

Status (rolling)

bash lib/host/conformance.sh → 43/43 (4 suites: handler, middleware, router, feed). Phase 1 DONE; Phase 2 in progress (middleware + write endpoint DONE, SXTP next).

Ground rules

• Scope: lib/host/** and plans/host-on-sx.md. May import every subsystem
  • the kernel's server/SXTP surface. Do not edit spec/, hosts/, shared/, or subsystem internals — wire to their public APIs only. Host-primitive / server changes belong in hosts/ (out of scope) → Blockers.
• Architecture: a route maps (method, path) → handler; a handler is an SX fn request -> response that calls subsystem APIs; middleware is composed handlers (auth via identity, permission via acl, mute via subsystem prefs). SXTP is the wire format between host and subsystem-as-service.
• Migration discipline: each endpoint moved must be behavior-equivalent to its Quart original (golden-response test before flip). Keep a migration ledger.
• Commits: one feature per commit. Progress log + tick boxes.

Architecture sketch

HTTP request                            HTTP response
        │                                       ▲
        ▼                                       │
native OCaml http server (prod)  ──────►  lib/host/router.sx
   (hosts/ — out of scope)                  — (method,path) → handler
        │                                       ▲
        ▼                                       │
lib/host/middleware.sx                  lib/host/handler.sx
  — auth(identity) ∘ acl ∘ mute ∘ ...     — request → subsystem call → response
        │                                       ▲
        ▼                                       │
lib/host/sxtp.sx                        subsystem APIs (feed/search/commerce/…)
  — wire format, host↔service             — called via public interfaces
        │
        └── external edges: SumUp / Ghost / AP / IPFS  → injected Python/FFI adapters

Phase 1 — Router + handler + one real endpoint

• router.sx — host/make-app assembles per-domain route groups + a built-in /health probe into one Dream router (reuses Dream's dr/flatten-routes)
• handler.sx — JSON envelope (host/ok/host/ok-status/host/error), status-carrying host/json-status (Dream's dream-json is 200-only), and host/query-int. A host handler IS a Dream handler (request -> response).
• migrate ONE read endpoint: GET /feed (lib/host/feed.sx) reads feed/all + stream combinators, serialises recent-first; ?actor= filter, ?limit= cap. Golden test asserts body == subsystem recent stream + envelope.
• conformance.sh (mirrors lib/dream's runner) — 28/28

Phase 2 — Middleware + SXTP

• middleware.sx — composable layers as handler->handler: host/wrap-errors (JSON 500), host/require-auth (bearer -> principal, JSON 401, INJECTED token resolver), host/require-permission (ACL acl/permit? gate, JSON 403, INJECTED resource extractor), host/pipeline (first = outermost). Reuses Dream's dream-bearer-token + dream-catch-with; calls lib/acl public API. Mute/prefs layer deferred (no blocker, add when a domain needs it).
• sxtp.sx — host↔subsystem wire format (align with existing spec at applications/sxtp/spec.sx)
• migrate a write endpoint (auth + permission + action): POST /feed (host/feed-write-routes resolve) — auth ∘ ACL("post","feed") ∘ wrap-errors over host/feed-create, which parses the JSON body and feed/posts it (201); non-object body -> 400. Created activity is readable back via GET /feed.

Phase 3 — Strangler migration ledger

• enumerate Quart endpoints; track migrated vs proxied
• golden-response harness vs the live Quart responses
• cut over a whole domain (smallest: likes or relations) as proof

Phase 4 — Dream framework layer (gated)

• gate: ocaml-on-sx Phases 1–5 + minimal stdlib green
• adopt dream-on-sx routing/middleware/session ergonomics over the same handlers
• re-home external adapters as native where replacements land

Progress log

• Phase 1 (DONE, 28/28). lib/host/{handler,router,feed}.sx + three test suites + conformance.sh. The host is a thin wiring layer: a host handler is a Dream handler that calls a subsystem public API and serialises the result via a shared JSON envelope. First migrated endpoint: GET /feed.

  • Decision — build on Dream from Phase 1, not a throwaway native model. The plan front-matter gated Dream to Phase 4, but dream-on-sx is merged (commit fe958bda) and its gate (ocaml-on-sx P1–5+P6) is green (480/480), so reinventing request/response + routing would be pure duplication. Host reuses Dream's types.sx (request/response dicts), json.sx (encode), and router.sx (dream-router/dream-get/dr/flatten-routes). Phase 4's "adopt Dream ergonomics" is therefore largely already satisfied; what remains for Phase 4 is the live wiring against the real OCaml HTTP server + session.
  • The OCaml server handing a dream-request-shaped dict to SX handlers is a hosts/ change (out of scope) — tracked under Blockers as the eventual live-wiring step. For now the host layer is exercised purely via conformance.

• Phase 2 (middleware + write endpoint DONE, 43/43). lib/host/middleware.sx

  • a guarded POST /feed. Middleware is plain function composition over Dream's primitives; auth/permission policy is injected (token resolver, resource extractor) so the layer is policy-free and testable. ACL authorisation runs against lib/acl's public acl/permit? (string atoms work — no symbol coercion needed). The write path proves the auth ∘ permission ∘ action stack end-to-end: 401 unauth, 403 unpermitted, 201 + readback on success, 400 on bad body.
  • Remaining for Phase 2: sxtp.sx — the host↔subsystem wire format. Align with the existing spec at applications/sxtp/spec.sx. This is the next tick.

Blockers

• Live wiring to the native OCaml HTTP server (Phase 3/4): the prod server in hosts/ must hand SX handlers a dream-request dict and serialise the returned dream-response. That is a hosts/ change (out of scope for this loop, which is lib/host/** only). Until then, endpoints are verified via conformance.sh, not HTTP. Not blocking Phase 2 (middleware + SXTP + a write endpoint).
• Worktree tooling: in this loops/host worktree every sx-tree write tool (sx_write_file, sx_replace_node, …) raises yojson "Expected string, got null" at the MCP layer — same class as the loops/dream worktree gotcha, but here even sx_write_file fails. Read-side sx-tree tools work. New .sx files were created with the Write tool (the .sx hook is inactive in this worktree) and each validated afterwards with sx_validate to keep the parse guarantee.