rose-ash/plans/host-on-sx.md

host-on-sx: The SX web host — off Quart, onto the kernel (Dream-bound)

DRAFT outline. The integration boundary that turns the subsystem libraries into running services, and the strangler path off Python/Quart. This is the dependency hub — it imports every subsystem. Decision recorded below: native server + SXTP now, dream-on-sx framework layer next, Python only at the external-integration edges.

The subsystems (feed, search, acl, mod, flow, commerce, identity, content, events) are libraries. Something has to receive an HTTP request, route it, call the right subsystem, and serialize the response. Today that's Python/Quart — the one large non-SX component in the stack: separate runtime, deploy, and failure mode. The goal is to move the web/host/domain layer onto the SX substrate and retire Quart, incrementally (strangler-fig), never big-bang.

This is already underway: a native OCaml HTTP server is live in prod on sx.rose-ash.com (~3ms cached, ~323 req/s, ~2MB RSS), defhandler/defpage exist, and a partial SXTP protocol is specced. That is the unblocked near-term host — no ocaml-on-sx dependency.

Two layers, two timelines

1. Now (unblocked): native server + SXTP adapter + SX handlers. Route rose-ash endpoints onto the SX host one at a time. Each migrated endpoint is an SX handler dispatching to a subsystem; Quart proxies the rest until cut over.
2. Next: dream-on-sx as the framework layer. Dream gives Quart-grade ergonomics — typed routing, middleware stacks, sessions, CSRF. It is gated on ocaml-on-sx Phases 1–5 + minimal stdlib. This plan is the concrete target user that un-parks dream-on-sx (see plans/dream-on-sx.md): "the subsystems need an HTTP front door" is the real feature pulling Dream. Until then, do not block migration on Dream — the native server is sufficient.
3. Always: Python only at the edges. External integrations — SumUp payments, Ghost CMS, ActivityPub crypto, IPFS/Kubo — ride Python libraries today. They stay as thin injected adapters (Python/FFI) behind subsystem interfaces until native replacements exist. "Drop Quart" ≠ "drop every line of Python."

Status (rolling)

bash lib/host/conformance.sh → 158/158 (9 suites: handler, middleware, sxtp, router, feed, relations, blog, server, ledger). Blog posts now persist in the durable SX store (persist/durable-backend, on-disk under $SX_PERSIST_DIR), materialised into an in-memory view at boot and served from there.

Per-request IO (kernel) — FIXED. http-listen handlers used to run via Sx_runtime.sx_call (bare CEK, no IO resolution), so a handler doing a durable persist/read returned an unresolved suspension. Fixed in sx_server.ml: the handler now runs through cek_run_with_io (Sx_ref.continue_with_call → cek_run_with_io), the same IO-driving runner the REPL uses — it resolves persist ops via Sx_persist_store.handle_op between CEK steps. Verified: handlers do per-request durable reads + writes (incl. 10 concurrent, 15 events on disk, no corruption); handler errors don't crash the server. NOTE: this is the per-request IO fix; it does NOT speed up the interpreted Smalltalk render (/welcome/ still ~2s) — that's a separate concern, addressed by caching the rendered HTML at boot. (Pre-existing: an erroring handler closes the connection with no response instead of a 500 — worth improving later.)

Render speed (separate from IO) — NOT precompiled. /welcome/ is ~2s because the interpreted Smalltalk-on-SX render runs on the tree-walking CEK: the JIT hook (register_jit_hook) is installed only in --http page mode, not the epoch/ http-listen serving mode (make_server_env), so zero [jit] activity. Enabling it in that mode breaks correctness (router 3/6, feed 4/11, … — the known JIT- bytecode bug on complex nested ASTs, which the Smalltalk evaluator is). So the render is slow until the JIT compiler is fixed (big win, broad payoff — its own loop) or the Smalltalk interpreter is optimised. Blog is FULLY DYNAMIC (reads store + renders per request, no cache) — slowness is honest, not hidden. Phases 1 & 2 DONE; Phase 3 cut-over landed (50% off Quart). The host now serves live HTTP — lib/host/server.sx bridges the native http-listen server to the Dream app and lib/host/serve.sh boots it (verified: GET /health, /feed, /feed?actor=, relations get-children/ get-parents all serve real JSON on a host port; unknown→404). Remaining: golden harness vs live Quart, internal-HMAC middleware, docker stack + Caddy subdomain.

Ground rules

• Scope: lib/host/** and plans/host-on-sx.md. May import every subsystem
  • the kernel's server/SXTP surface. Do not edit spec/, hosts/, shared/, or subsystem internals — wire to their public APIs only. Host-primitive / server changes belong in hosts/ (out of scope) → Blockers.
• Architecture: a route maps (method, path) → handler; a handler is an SX fn request -> response that calls subsystem APIs; middleware is composed handlers (auth via identity, permission via acl, mute via subsystem prefs). SXTP is the wire format between host and subsystem-as-service.
• Migration discipline: each endpoint moved must be behavior-equivalent to its Quart original (golden-response test before flip). Keep a migration ledger.
• Commits: one feature per commit. Progress log + tick boxes.

Architecture sketch

HTTP request                            HTTP response
        │                                       ▲
        ▼                                       │
native OCaml http server (prod)  ──────►  lib/host/router.sx
   (hosts/ — out of scope)                  — (method,path) → handler
        │                                       ▲
        ▼                                       │
lib/host/middleware.sx                  lib/host/handler.sx
  — auth(identity) ∘ acl ∘ mute ∘ ...     — request → subsystem call → response
        │                                       ▲
        ▼                                       │
lib/host/sxtp.sx                        subsystem APIs (feed/search/commerce/…)
  — wire format, host↔service             — called via public interfaces
        │
        └── external edges: SumUp / Ghost / AP / IPFS  → injected Python/FFI adapters

Phase 1 — Router + handler + one real endpoint

• router.sx — host/make-app assembles per-domain route groups + a built-in /health probe into one Dream router (reuses Dream's dr/flatten-routes)
• handler.sx — JSON envelope (host/ok/host/ok-status/host/error), status-carrying host/json-status (Dream's dream-json is 200-only), and host/query-int. A host handler IS a Dream handler (request -> response).
• migrate ONE read endpoint: GET /feed (lib/host/feed.sx) reads feed/all + stream combinators, serialises recent-first; ?actor= filter, ?limit= cap. Golden test asserts body == subsystem recent stream + envelope.
• conformance.sh (mirrors lib/dream's runner) — 28/28

Phase 2 — Middleware + SXTP

• middleware.sx — composable layers as handler->handler: host/wrap-errors (JSON 500), host/require-auth (bearer -> principal, JSON 401, INJECTED token resolver), host/require-permission (ACL acl/permit? gate, JSON 403, INJECTED resource extractor), host/pipeline (first = outermost). Reuses Dream's dream-bearer-token + dream-catch-with; calls lib/acl public API. Mute/prefs layer deferred (no blocker, add when a domain needs it).
• sxtp.sx — host↔subsystem wire format (per applications/sxtp/spec.sx). Message algebra (sxtp/request/response/condition/event + status helpers sxtp/ok/created/not-found/forbidden/invalid/fail) as string-keyed dicts; verb/status/type as symbols (ride the wire bare). Codec: sxtp/serialize (dict → text/sx list form, deterministic field order, nested messages in their own list form, no :msg leak) and sxtp/parse (text/sx → dict, deep keyword-token→string normaliser). Dream bridge: sxtp/from-dream (HTTP req → SXTP req, method→verb, query→params) and sxtp/to-dream (SXTP resp → HTTP resp, status→code, body→text/sx).
• migrate a write endpoint (auth + permission + action): POST /feed (host/feed-write-routes resolve) — auth ∘ ACL("post","feed") ∘ wrap-errors over host/feed-create, which parses the JSON body and feed/posts it (201); non-object body -> 400. Created activity is readable back via GET /feed.

Phase 3 — Strangler migration ledger

• enumerate Quart endpoints; track migrated vs proxied — ledger.sx: a catalogue of every endpoint (domain, method, path, Quart original, status :native/:migrated/:proxied, SX handler) + queries (by-status/by-domain, host/ledger-find, host/ledger-served?, distinct domains) and host/ledger-coverage (off-Quart % = (migrated+native)/total). Seeded with the live state: feed reads+writes migrated, /health native, the internal-only relations/likes data+action endpoints proxied.
• golden-response harness vs the live Quart responses
• cut over a whole domain (relations) as proof — the CONTAINER relations are fully on the host (lib/host/relations.sx): reads GET .../get-children + /get-parents → relations/children/parents; writes POST .../attach-child + /detach-child → relations/relate/unrelate, behind the auth+ACL pipeline (mirrors POST /feed). Node model: graph atom = symbol "type:id", edge = relation-type; child/parent-type params filter by "type:" prefix. Closed-loop test: attach → visible via get-children → detach → gone. The TYPED actions (relate/unrelate/can-relate) stay proxied by design — registry + cardinality validation lib/relations lacks.

Phase 4 — Live wiring + Dream framework layer

• native http-listen ↔ Dream-app bridge (lib/host/server.sx: host/native-handler/host/serve) + lib/host/serve.sh launcher. Serves real HTTP on a host port — verified live (health/feed/relations reads + 404).
• promote into the docker stack + a Caddy subdomain — LIVE at https://blog.rose-ash.com (reusing a down Quart subdomain). New compose service sx_host (docker-compose.dev-sx-host.yml, container sx-dev-sx_host-1) runs serve.sh on externalnet; Caddy reverse-proxies blog.rose-ash.com → sx-dev-sx_host-1:8000. Required a hosts/ fix: http-listen bound inet_addr_loopback only — added SX_HTTP_HOST env (default loopback; stack sets 0.0.0.0) in sx_server.ml, rebuilt this worktree's binary. Verified: /health, /feed, relations reads serve real JSON through Cloudflare→Caddy; / 404 (no root route yet). rose-ash.com untouched. (Inode-pinned bind-mount gotcha: editing /root/caddy/Caddyfile via a tool swaps its inode so the container kept the old content — loaded live via reload-from-non-bind-path, then RECONCILED by restarting Caddy so the bind re-points to the corrected file. Verified post-restart: blog serves, and sx.rose-ash.com/rose-ash.com survived.)
• blog published-post read endpoint — lib/host/blog.sx: GET /<slug>/ renders a content-on-sx CtDoc to HTML via content/html (anonymous, world-visible). In-memory slug→doc registry now (swap host/blog-lookup for a persist-backed content stream later, handler/route unchanged). :slug catch-all mounted LAST so domain routes win. LIVE: blog.rose-ash.com/ welcome/ renders real HTML through Caddy. Needs Smalltalk+persist+content preloads + (st-bootstrap-classes!)+(content/bootstrap!) (self-bootstraps at load).
• proxy-to-Quart fallback for un-migrated paths (strangler requirement before a real subdomain fronts users).
• internal-HMAC middleware on /internal/* (service-to-service auth; protocol checks native, signature check needs an HMAC-SHA256 kernel prim — absent today).
• (gated) adopt dream-on-sx session/CSRF ergonomics; re-home external adapters as native where replacements land.

Progress log

• Phase 1 (DONE, 28/28). lib/host/{handler,router,feed}.sx + three test suites + conformance.sh. The host is a thin wiring layer: a host handler is a Dream handler that calls a subsystem public API and serialises the result via a shared JSON envelope. First migrated endpoint: GET /feed.

  • Decision — build on Dream from Phase 1, not a throwaway native model. The plan front-matter gated Dream to Phase 4, but dream-on-sx is merged (commit fe958bda) and its gate (ocaml-on-sx P1–5+P6) is green (480/480), so reinventing request/response + routing would be pure duplication. Host reuses Dream's types.sx (request/response dicts), json.sx (encode), and router.sx (dream-router/dream-get/dr/flatten-routes). Phase 4's "adopt Dream ergonomics" is therefore largely already satisfied; what remains for Phase 4 is the live wiring against the real OCaml HTTP server + session.
  • The OCaml server handing a dream-request-shaped dict to SX handlers is a hosts/ change (out of scope) — tracked under Blockers as the eventual live-wiring step. For now the host layer is exercised purely via conformance.

• Phase 2 (middleware + write endpoint DONE, 43/43). lib/host/middleware.sx

  • a guarded POST /feed. Middleware is plain function composition over Dream's primitives; auth/permission policy is injected (token resolver, resource extractor) so the layer is policy-free and testable. ACL authorisation runs against lib/acl's public acl/permit? (string atoms work — no symbol coercion needed). The write path proves the auth ∘ permission ∘ action stack end-to-end: 401 unauth, 403 unpermitted, 201 + readback on success, 400 on bad body.

• Phase 2 COMPLETE (82/82). lib/host/sxtp.sx adds the SXTP codec + Dream bridge (39-test suite). Key representation calls, learned by probing the runtime: keywords are strings at eval time but the serialize primitive renders string-keyed dicts back as {:k v} and symbols bare — so messages are string-keyed dicts with verb/status/type as symbols, and a small str-based emitter produces wire-faithful list form. parse needs a deep normaliser because parsed keyword tokens are a distinct type (not = to string literals). unquote-splicing is unreliable here, so the serializer is str-based, not quasiquote-based.

  • Next: Phase 3 — strangler migration ledger. Enumerate the Quart endpoints (use the rose-ash-services svc_routes MCP tool), track migrated vs proxied, and stand up a golden-response harness against the live Quart responses. Then cut over the smallest whole domain (likes or relations) as proof.

• Phase 3 — ledger module (DONE, 107/107). lib/host/ledger.sx + a 25-test suite. Enumerated the endpoint surface via the rose-ash-services MCP (svc_routes/svc_queries/svc_actions): likes and relations have no public blueprint routes — they're internal-only, exposed as /internal/data/{query} + /internal/actions/{action} (HMAC-signed). The ledger is a pure-data catalogue keyed by (domain, method, path) carrying each endpoint's Quart original, status, and serving SX handler; coverage reports the off-Quart percentage. Cut-over target chosen: relations (already has a real SX subsystem lib/relations — children/parents reads + relate/unrelate writes map straight onto its public API); likes stays proxied (no SX lib to dispatch to). NEXT: migrate the relations read endpoints onto host handlers (flip their ledger status to :migrated) with golden tests.

• Phase 3 — relations READ cut-over (DONE, 121/121). lib/host/relations.sx

  • a 13-test golden suite; ledger flipped (off-Quart coverage 27% → 45%). The two internal read queries (get-children, get-parents) now dispatch to the lib/relations Datalog graph. Bridge: the Quart (type, id) node key maps to a graph atom (string->symbol "type:id") with relation-type as the edge kind; optional child-type/parent-type params filter the result list by "type:" prefix (verified live: composite-string nodes round-trip through relations/relate → relations/children). Golden discipline: relations is internal-only (no public Quart route — confirmed via svc_routes), so the golden is a pinned fixture (a known graph loaded in-test, asserted as subsystem-call + envelope) rather than a live Quart capture. Reads are unguarded for now — the signed-internal-auth gate is a separate middleware layer, same as the feed reads. NEXT: relations WRITE actions (relate/unrelate) behind the auth+ACL pipeline (mirroring POST /feed).

• Phase 3 — relations WRITE cut-over (DONE, 132/132). lib/host/relations.sx gains host/relations-attach/-detach (POST .../attach-child + /detach-child) and host/relations-write-routes — the write side of the container reads, dispatching to relations/relate/unrelate over the same "type:id" node model so an attach is immediately visible through get-children. Each runs behind the host pipeline wrap-errors ∘ require-auth ∘ require-permission ("relate"/"unrelate" on "relations") — exactly the POST /feed stack. The relations test suite proves the closed loop end-to-end: 401 unauth, 403 authed- but-unpermitted (graph unchanged), 201 attach → child visible via the migrated read → 200 detach → child gone; 400 on bad/short payloads. The ledger now models the full relations surface (7 endpoints): container reads+writes :migrated, typed relate/unrelate/can-relate :proxied (registry/cardinality validation not in lib/relations). Off-Quart coverage 45% → 50% (7/14). relations is the first whole coherent feature (container relations) fully off Quart. NEXT: golden-response harness vs live Quart, then survey the next domain (blog/likes proxied — likes needs an SX subsystem first).

• Phase 4 — live wiring bridge (DONE, 145/145). lib/host/server.sx adapts the native http-listen contract (string-keyed req {"method" "path" "query" "headers" "body"} → {:status :headers :body}) to the Dream app: host/-native ->dream reassembles path+query into a target dream-request parses; host/-dream->native is near-identity (dream-response is already {:body :headers :status}). host/serve port groups = http-listen over host/native-handler (host/make-app groups). lib/host/serve.sh boots the full module set (mirrors conformance) and serves in the foreground (container-entry shaped). Verified live on a host port: /health 200 JSON, /feed recent- first seeded activities, /feed?actor= filtered, relations get-children/get- parents real JSON, unknown→404. Demo run was a standalone sx_server.exe process (NOT the docker stack) — killed by its own PID, never pkill (siblings share the binary). The standing "live wiring is a hosts/ change" Blocker is resolved for the SX side: the bridge is pure SX in lib/host; only the launch (docker stack + Caddy) remains. NEXT: golden harness, internal-HMAC, then promote into the stack behind a fresh subdomain.

SX gotchas + how this loop guards against them

The SX dev experience has real footguns. Most are statically detectable; the tools exist (sx_validate, deps-check, sx_format_check) but must be gated. Hit/relevant here:

• Reserved-name shadowing — guard/bind/conj/disj are special forms or host primitives; a local binding of that name is silently shadowed by the form. ((let ((guard ...))) made (guard handler) invoke the R7RS guard special form → first: expected list.) Fix: namespace-prefix every helper (host/blog--protect, never guard).
• Silent test truncation — a test file that errors mid-load returns only the tests that ran before the error, reporting a FALSE GREEN ("blog 13 passed, 0 failed" while 16 CRUD tests never ran). GUARDED: conformance.sh now greps the run output for Undefined symbol / Unhandled exception / expected list, got / [load] … error and aborts loudly before the tally can hide it.
• let is parallel (bindings can't see each other), bodies need (do …) (only the last expr evaluates), append! no-ops on map/rest-derived lists, parsed keyword tokens ≠ string literals. These produce wrong results, so test coverage catches them as red (not silent) — provided the runner is honest, which the truncation guard now ensures.

Prevention ladder: parse (sx_validate after every edit) → unresolved/shadowed symbols (deps-check, candidate pre-commit gate) → fail-loud runner (done) → behavioural tests. A deps-check-style "binding shadows a special form" lint would catch the reserved-name class before runtime — a worthwhile follow-up.

Blockers

• Live wiring to the native OCaml HTTP server (Phase 3/4): the prod server in hosts/ must hand SX handlers a dream-request dict and serialise the returned dream-response. That is a hosts/ change (out of scope for this loop, which is lib/host/** only). Until then, endpoints are verified via conformance.sh, not HTTP. Not blocking Phase 2 (middleware + SXTP + a write endpoint).
• Worktree tooling: in this loops/host worktree every sx-tree write tool (sx_write_file, sx_replace_node, …) raises yojson "Expected string, got null" at the MCP layer — same class as the loops/dream worktree gotcha, but here even sx_write_file fails. Read-side sx-tree tools work. New .sx files were created with the Write tool (the .sx hook is inactive in this worktree) and each validated afterwards with sx_validate to keep the parse guarantee.