rose-ash/plans/host-on-sx.md

host-on-sx: The SX web host — off Quart, onto the kernel (Dream-bound)

DRAFT outline. The integration boundary that turns the subsystem libraries into running services, and the strangler path off Python/Quart. This is the dependency hub — it imports every subsystem. Decision recorded below: native server + SXTP now, dream-on-sx framework layer next, Python only at the external-integration edges.

The subsystems (feed, search, acl, mod, flow, commerce, identity, content, events) are libraries. Something has to receive an HTTP request, route it, call the right subsystem, and serialize the response. Today that's Python/Quart — the one large non-SX component in the stack: separate runtime, deploy, and failure mode. The goal is to move the web/host/domain layer onto the SX substrate and retire Quart, incrementally (strangler-fig), never big-bang.

This is already underway: a native OCaml HTTP server is live in prod on sx.rose-ash.com (~3ms cached, ~323 req/s, ~2MB RSS), defhandler/defpage exist, and a partial SXTP protocol is specced. That is the unblocked near-term host — no ocaml-on-sx dependency.

Two layers, two timelines

1. Now (unblocked): native server + SXTP adapter + SX handlers. Route rose-ash endpoints onto the SX host one at a time. Each migrated endpoint is an SX handler dispatching to a subsystem; Quart proxies the rest until cut over.
2. Next: dream-on-sx as the framework layer. Dream gives Quart-grade ergonomics — typed routing, middleware stacks, sessions, CSRF. It is gated on ocaml-on-sx Phases 1–5 + minimal stdlib. This plan is the concrete target user that un-parks dream-on-sx (see plans/dream-on-sx.md): "the subsystems need an HTTP front door" is the real feature pulling Dream. Until then, do not block migration on Dream — the native server is sufficient.
3. Always: Python only at the edges. External integrations — SumUp payments, Ghost CMS, ActivityPub crypto, IPFS/Kubo — ride Python libraries today. They stay as thin injected adapters (Python/FFI) behind subsystem interfaces until native replacements exist. "Drop Quart" ≠ "drop every line of Python."

Status (rolling)

bash lib/host/conformance.sh → 107/107 (6 suites: handler, middleware, sxtp, router, feed, ledger). Phases 1 & 2 DONE; Phase 3 (strangler ledger) underway — ledger module landed; relations cut-over next.

Ground rules

• Scope: lib/host/** and plans/host-on-sx.md. May import every subsystem
  • the kernel's server/SXTP surface. Do not edit spec/, hosts/, shared/, or subsystem internals — wire to their public APIs only. Host-primitive / server changes belong in hosts/ (out of scope) → Blockers.
• Architecture: a route maps (method, path) → handler; a handler is an SX fn request -> response that calls subsystem APIs; middleware is composed handlers (auth via identity, permission via acl, mute via subsystem prefs). SXTP is the wire format between host and subsystem-as-service.
• Migration discipline: each endpoint moved must be behavior-equivalent to its Quart original (golden-response test before flip). Keep a migration ledger.
• Commits: one feature per commit. Progress log + tick boxes.

Architecture sketch

HTTP request                            HTTP response
        │                                       ▲
        ▼                                       │
native OCaml http server (prod)  ──────►  lib/host/router.sx
   (hosts/ — out of scope)                  — (method,path) → handler
        │                                       ▲
        ▼                                       │
lib/host/middleware.sx                  lib/host/handler.sx
  — auth(identity) ∘ acl ∘ mute ∘ ...     — request → subsystem call → response
        │                                       ▲
        ▼                                       │
lib/host/sxtp.sx                        subsystem APIs (feed/search/commerce/…)
  — wire format, host↔service             — called via public interfaces
        │
        └── external edges: SumUp / Ghost / AP / IPFS  → injected Python/FFI adapters

Phase 1 — Router + handler + one real endpoint

• router.sx — host/make-app assembles per-domain route groups + a built-in /health probe into one Dream router (reuses Dream's dr/flatten-routes)
• handler.sx — JSON envelope (host/ok/host/ok-status/host/error), status-carrying host/json-status (Dream's dream-json is 200-only), and host/query-int. A host handler IS a Dream handler (request -> response).
• migrate ONE read endpoint: GET /feed (lib/host/feed.sx) reads feed/all + stream combinators, serialises recent-first; ?actor= filter, ?limit= cap. Golden test asserts body == subsystem recent stream + envelope.
• conformance.sh (mirrors lib/dream's runner) — 28/28

Phase 2 — Middleware + SXTP

• middleware.sx — composable layers as handler->handler: host/wrap-errors (JSON 500), host/require-auth (bearer -> principal, JSON 401, INJECTED token resolver), host/require-permission (ACL acl/permit? gate, JSON 403, INJECTED resource extractor), host/pipeline (first = outermost). Reuses Dream's dream-bearer-token + dream-catch-with; calls lib/acl public API. Mute/prefs layer deferred (no blocker, add when a domain needs it).
• sxtp.sx — host↔subsystem wire format (per applications/sxtp/spec.sx). Message algebra (sxtp/request/response/condition/event + status helpers sxtp/ok/created/not-found/forbidden/invalid/fail) as string-keyed dicts; verb/status/type as symbols (ride the wire bare). Codec: sxtp/serialize (dict → text/sx list form, deterministic field order, nested messages in their own list form, no :msg leak) and sxtp/parse (text/sx → dict, deep keyword-token→string normaliser). Dream bridge: sxtp/from-dream (HTTP req → SXTP req, method→verb, query→params) and sxtp/to-dream (SXTP resp → HTTP resp, status→code, body→text/sx).
• migrate a write endpoint (auth + permission + action): POST /feed (host/feed-write-routes resolve) — auth ∘ ACL("post","feed") ∘ wrap-errors over host/feed-create, which parses the JSON body and feed/posts it (201); non-object body -> 400. Created activity is readable back via GET /feed.

Phase 3 — Strangler migration ledger

• enumerate Quart endpoints; track migrated vs proxied — ledger.sx: a catalogue of every endpoint (domain, method, path, Quart original, status :native/:migrated/:proxied, SX handler) + queries (by-status/by-domain, host/ledger-find, host/ledger-served?, distinct domains) and host/ledger-coverage (off-Quart % = (migrated+native)/total). Seeded with the live state: feed reads+writes migrated, /health native, the internal-only relations/likes data+action endpoints proxied.
• golden-response harness vs the live Quart responses
• cut over a whole domain (relations — it has a real SX subsystem lib/relations; likes has none so it stays proxied) as proof

Phase 4 — Dream framework layer (gated)

• gate: ocaml-on-sx Phases 1–5 + minimal stdlib green
• adopt dream-on-sx routing/middleware/session ergonomics over the same handlers
• re-home external adapters as native where replacements land

Progress log

• Phase 1 (DONE, 28/28). lib/host/{handler,router,feed}.sx + three test suites + conformance.sh. The host is a thin wiring layer: a host handler is a Dream handler that calls a subsystem public API and serialises the result via a shared JSON envelope. First migrated endpoint: GET /feed.

  • Decision — build on Dream from Phase 1, not a throwaway native model. The plan front-matter gated Dream to Phase 4, but dream-on-sx is merged (commit fe958bda) and its gate (ocaml-on-sx P1–5+P6) is green (480/480), so reinventing request/response + routing would be pure duplication. Host reuses Dream's types.sx (request/response dicts), json.sx (encode), and router.sx (dream-router/dream-get/dr/flatten-routes). Phase 4's "adopt Dream ergonomics" is therefore largely already satisfied; what remains for Phase 4 is the live wiring against the real OCaml HTTP server + session.
  • The OCaml server handing a dream-request-shaped dict to SX handlers is a hosts/ change (out of scope) — tracked under Blockers as the eventual live-wiring step. For now the host layer is exercised purely via conformance.

• Phase 2 (middleware + write endpoint DONE, 43/43). lib/host/middleware.sx

  • a guarded POST /feed. Middleware is plain function composition over Dream's primitives; auth/permission policy is injected (token resolver, resource extractor) so the layer is policy-free and testable. ACL authorisation runs against lib/acl's public acl/permit? (string atoms work — no symbol coercion needed). The write path proves the auth ∘ permission ∘ action stack end-to-end: 401 unauth, 403 unpermitted, 201 + readback on success, 400 on bad body.

• Phase 2 COMPLETE (82/82). lib/host/sxtp.sx adds the SXTP codec + Dream bridge (39-test suite). Key representation calls, learned by probing the runtime: keywords are strings at eval time but the serialize primitive renders string-keyed dicts back as {:k v} and symbols bare — so messages are string-keyed dicts with verb/status/type as symbols, and a small str-based emitter produces wire-faithful list form. parse needs a deep normaliser because parsed keyword tokens are a distinct type (not = to string literals). unquote-splicing is unreliable here, so the serializer is str-based, not quasiquote-based.

  • Next: Phase 3 — strangler migration ledger. Enumerate the Quart endpoints (use the rose-ash-services svc_routes MCP tool), track migrated vs proxied, and stand up a golden-response harness against the live Quart responses. Then cut over the smallest whole domain (likes or relations) as proof.

• Phase 3 — ledger module (DONE, 107/107). lib/host/ledger.sx + a 25-test suite. Enumerated the endpoint surface via the rose-ash-services MCP (svc_routes/svc_queries/svc_actions): likes and relations have no public blueprint routes — they're internal-only, exposed as /internal/data/{query} + /internal/actions/{action} (HMAC-signed). The ledger is a pure-data catalogue keyed by (domain, method, path) carrying each endpoint's Quart original, status, and serving SX handler; coverage reports the off-Quart percentage. Cut-over target chosen: relations (already has a real SX subsystem lib/relations — children/parents reads + relate/unrelate writes map straight onto its public API); likes stays proxied (no SX lib to dispatch to). NEXT: migrate the relations read endpoints onto host handlers (flip their ledger status to :migrated) with golden tests.

Blockers

• Live wiring to the native OCaml HTTP server (Phase 3/4): the prod server in hosts/ must hand SX handlers a dream-request dict and serialise the returned dream-response. That is a hosts/ change (out of scope for this loop, which is lib/host/** only). Until then, endpoints are verified via conformance.sh, not HTTP. Not blocking Phase 2 (middleware + SXTP + a write endpoint).
• Worktree tooling: in this loops/host worktree every sx-tree write tool (sx_write_file, sx_replace_node, …) raises yojson "Expected string, got null" at the MCP layer — same class as the loops/dream worktree gotcha, but here even sx_write_file fails. Read-side sx-tree tools work. New .sx files were created with the Write tool (the .sx hook is inactive in this worktree) and each validated afterwards with sx_validate to keep the parse guarantee.