host: Phase 1 — router + handler + GET /feed endpoint on Dream, 28/28

First migrated endpoint onto the SX host. lib/host is a thin wiring layer: a host handler is a Dream handler (request->response) that calls a subsystem public API and serialises via a shared JSON envelope. - handler.sx: host/ok, host/ok-status, host/error, host/json-status (Dream's dream-json is 200-only), host/query-int - router.sx: host/make-app assembles per-domain route groups + /health probe into one dream-router (reuses dr/flatten-routes) - feed.sx: GET /feed reads feed/all + stream combinators, recent-first, with ?actor= filter and ?limit= cap - 3 test suites incl. a golden test (body == subsystem recent stream + envelope) - conformance.sh mirrors lib/dream's runner Builds on dream-on-sx (merged, gate green 480/480) rather than a throwaway native request model; collapses most of plan Phase 4 into Phase 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 19:36:55 +00:00
parent fe958bda69
commit d5a1c8370c
8 changed files with 480 additions and 7 deletions
--- a/lib/host/conformance.sh
+++ b/lib/host/conformance.sh
@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+# host-on-sx conformance runner — loads the kernel stdlib, the subsystem
+# libraries the host wires to, the host modules, and the host test suites in one
+# sx_server process, then reports pass/fail per suite. Mirrors lib/dream's runner.
+#
+# Usage:
+#   bash lib/host/conformance.sh        # run all suites
+#   bash lib/host/conformance.sh -v     # verbose (list each suite)
+
+set -uo pipefail
+cd "$(git rev-parse --show-toplevel)"
+
+SX_SERVER="${SX_SERVER:-hosts/ocaml/_build/default/bin/sx_server.exe}"
+if [ ! -x "$SX_SERVER" ]; then
+  SX_SERVER="/root/rose-ash/hosts/ocaml/_build/default/bin/sx_server.exe"
+fi
+if [ ! -x "$SX_SERVER" ]; then
+  echo "ERROR: sx_server.exe not found." >&2
+  exit 1
+fi
+
+VERBOSE="${1:-}"
+
+# Kernel + subsystem dependencies, then the host modules. Order matters:
+# stdlib/r7rs first, then the feed subsystem (the first migrated domain), then
+# Dream (types/json/router) the host builds on, then the host layer itself.
+MODULES=(
+  "spec/stdlib.sx"
+  "lib/r7rs.sx"
+  "lib/apl/runtime.sx"
+  "lib/feed/normalize.sx"
+  "lib/feed/stream.sx"
+  "lib/feed/api.sx"
+  "lib/dream/types.sx"
+  "lib/dream/json.sx"
+  "lib/dream/router.sx"
+  "lib/host/handler.sx"
+  "lib/host/router.sx"
+  "lib/host/feed.sx"
+)
+
+# Suites: NAME RUNNER-FN PATH
+SUITES=(
+  "handler  host-hd-tests-run!  lib/host/tests/handler.sx"
+  "router   host-rt-tests-run!  lib/host/tests/router.sx"
+  "feed     host-fd-tests-run!  lib/host/tests/feed.sx"
+)
+
+TMPFILE=$(mktemp); trap "rm -f $TMPFILE" EXIT
+EPOCH=1
+emit_load () { echo "(epoch $EPOCH)"; echo "(load \"$1\")"; EPOCH=$((EPOCH+1)); }
+emit_eval () { echo "(epoch $EPOCH)"; echo "(eval \"$1\")"; EPOCH=$((EPOCH+1)); }
+
+{
+  for M in "${MODULES[@]}"; do emit_load "$M"; done
+  for SUITE in "${SUITES[@]}"; do
+    read -r _NAME _RUNNER FILE <<< "$SUITE"
+    emit_load "$FILE"
+    emit_eval "($_RUNNER)"
+  done
+} > "$TMPFILE"
+
+OUTPUT=$(timeout 300 "$SX_SERVER" < "$TMPFILE" 2>&1 || true)
+
+TOTAL_PASS=0
+TOTAL_FAIL=0
+FAILED_SUITES=()
+LAST_DICT_LINES=$(echo "$OUTPUT" | grep -E '^\{:' || true)
+
+I=0
+while read -r LINE; do
+  [ -z "$LINE" ] && continue
+  P=$(echo "$LINE" | grep -oE ':passed [0-9]+' | awk '{print $2}')
+  F=$(echo "$LINE" | grep -oE ':failed [0-9]+' | awk '{print $2}')
+  [ -z "$P" ] && P=0
+  [ -z "$F" ] && F=0
+  SUITE_INFO="${SUITES[$I]}"
+  SUITE_NAME=$(echo "$SUITE_INFO" | awk '{print $1}')
+  TOTAL_PASS=$((TOTAL_PASS + P))
+  TOTAL_FAIL=$((TOTAL_FAIL + F))
+  if [ "$F" -gt 0 ]; then
+    FAILED_SUITES+=("$SUITE_NAME: $P/$((P+F))")
+    printf 'X %-12s %d/%d\n' "$SUITE_NAME" "$P" "$((P+F))"
+    echo "$LINE" | grep -oE ':name "[^"]*"' | sed 's/:name /    fail: /'
+  elif [ "$VERBOSE" = "-v" ]; then
+    printf 'ok %-12s %d passed\n' "$SUITE_NAME" "$P"
+  fi
+  I=$((I+1))
+done <<< "$LAST_DICT_LINES"
+
+TOTAL=$((TOTAL_PASS + TOTAL_FAIL))
+if [ "$TOTAL" -eq 0 ]; then
+  echo "ERROR: no suite results parsed. Raw output:" >&2
+  echo "$OUTPUT" >&2
+  exit 1
+fi
+if [ $TOTAL_FAIL -eq 0 ]; then
+  echo "ok $TOTAL_PASS/$TOTAL host-on-sx tests passed (${#SUITES[@]} suites)"
+else
+  echo "FAIL $TOTAL_PASS/$TOTAL passed, $TOTAL_FAIL failed:"
+  for S in "${FAILED_SUITES[@]}"; do echo "  $S"; done
+  exit 1
+fi
--- a/lib/host/feed.sx
+++ b/lib/host/feed.sx
@@ -0,0 +1,22 @@
+;; lib/host/feed.sx — Feed domain endpoints on the host. The first real endpoint
+;; migrated onto the SX host: the activity timeline, read straight from the feed
+;; subsystem's public API (feed/all + the stream combinators) and serialised as
+;; JSON. GET /feed returns recent-first activities; ?actor=<id> filters by actor
+;; and ?limit=<n> caps the count. Depends on lib/feed/* + lib/host/handler.sx.
+
+;; GET /feed -> recent-first activities as a JSON envelope.
+;; Query: ?actor=<id> (filter) ?limit=<n> (cap, applied after filtering).
+(define host/feed-timeline
+  (fn (req)
+    (let ((base (feed/recent (feed/all)))
+          (actor (dream-query-param req "actor")))
+      (let ((filtered (if actor (feed/by-actor base actor) base))
+            (limit (dream-query-param req "limit")))
+        (let ((capped
+                (if limit (feed/take filtered (string->number limit)) filtered)))
+          (host/ok (feed/items capped)))))))
+
+;; Route group contributed by the feed domain.
+(define host/feed-routes
+  (list
+    (dream-get "/feed" host/feed-timeline)))
--- a/lib/host/handler.sx
+++ b/lib/host/handler.sx
@@ -0,0 +1,39 @@
+;; lib/host/handler.sx — Host handler layer: the bridge from a Dream request to a
+;; subsystem call and back to a Dream response. A host handler IS a Dream handler
+;; (request -> response); these helpers build the JSON envelope every host
+;; endpoint shares: {"ok":true,"data":...} on success, {"ok":false,"error":...}
+;; on failure. Plus a status-carrying JSON constructor that Dream's own dream-json
+;; (200-only) lacks, and a couple of request-reading conveniences.
+;; Depends on lib/dream/types.sx + lib/dream/json.sx.
+
+;; ── responses ──────────────────────────────────────────────────────
+
+;; JSON response at an arbitrary status (dream-json is 200-only).
+(define host/json-status
+  (fn (status value)
+    (dream-response status {:content-type "application/json"}
+      (dream-json-encode value))))
+
+;; Success envelope: 200 {"ok":true,"data":<value>}.
+(define host/ok
+  (fn (value)
+    (host/json-status 200 {:ok true :data value})))
+
+;; Success envelope at a chosen status (e.g. 201 for a created resource).
+(define host/ok-status
+  (fn (status value)
+    (host/json-status status {:ok true :data value})))
+
+;; Error envelope: {"ok":false,"error":<message>} at the given status.
+(define host/error
+  (fn (status message)
+    (host/json-status status {:ok false :error message})))
+
+;; ── request reading ────────────────────────────────────────────────
+
+;; Integer query param with a fallback (query params arrive as strings).
+;; Absent param -> fallback; present -> parsed number.
+(define host/query-int
+  (fn (req name fallback)
+    (let ((raw (dream-query-param req name)))
+      (if raw (string->number raw) fallback))))
--- a/lib/host/router.sx
+++ b/lib/host/router.sx
@@ -0,0 +1,19 @@
+;; lib/host/router.sx — Host application assembly. A host app is a single Dream
+;; router built from per-domain route groups, with a built-in health endpoint and
+;; a JSON 404 fallback so the native OCaml HTTP server has one entry point:
+;; request -> response. Each subsystem contributes a list of Dream routes (see
+;; lib/host/feed.sx); host/make-app concatenates them under one router.
+;; dr/flatten-routes (Dream) flattens the nested groups, so a group is just a list
+;; of routes. Depends on lib/dream/router.sx + lib/host/handler.sx.
+
+;; Liveness probe — GET /health -> 200 {"ok":true,"data":"healthy"}.
+(define host/health-route
+  (dream-get "/health" (fn (req) (host/ok "healthy"))))
+
+;; Build the host app from a list of route groups (each a list of Dream routes).
+;; The health route is always mounted first; Dream's router returns a JSON-free
+;; 404 for unmatched paths, which host endpoints override per-domain as needed.
+(define host/make-app
+  (fn (groups)
+    (dream-router
+      (cons host/health-route groups))))
--- a/lib/host/tests/feed.sx
+++ b/lib/host/tests/feed.sx
@@ -0,0 +1,98 @@
+;; lib/host/tests/feed.sx — the first migrated endpoint, GET /feed. Includes a
+;; golden test: the host response body must equal the feed subsystem's own
+;; recent-first stream wrapped in the standard envelope — the endpoint adds the
+;; HTTP/JSON shell and nothing else.
+
+(define host-fd-pass 0)
+(define host-fd-fail 0)
+(define host-fd-fails (list))
+
+(define
+  host-fd-test
+  (fn
+    (name actual expected)
+    (if
+      (= actual expected)
+      (set! host-fd-pass (+ host-fd-pass 1))
+      (begin
+        (set! host-fd-fail (+ host-fd-fail 1))
+        (append! host-fd-fails {:name name :actual actual :expected expected})))))
+
+(define
+  host-fd-req
+  (fn (target) (dream-request "GET" target {} "")))
+
+(define
+  host-fd-app
+  (host/make-app (list host/feed-routes)))
+
+;; ── empty feed ─────────────────────────────────────────────────────
+(feed/reset!)
+(host-fd-test
+  "empty feed 200"
+  (dream-status (host-fd-app (host-fd-req "/feed")))
+  200)
+(host-fd-test
+  "empty feed data:[]"
+  (contains? (dream-resp-body (host-fd-app (host-fd-req "/feed"))) "\"data\":[]")
+  true)
+
+;; ── seeded feed ────────────────────────────────────────────────────
+(feed/reset!)
+(feed/post {:actor "alice" :verb "post" :object "p1" :at 1})
+(feed/post {:actor "bob" :verb "post" :object "p2" :at 2})
+(feed/post {:actor "alice" :verb "like" :object "p2" :at 3})
+
+;; recent-first: newest activity (at 3) leads, so its object p2 appears before p1.
+(host-fd-test
+  "timeline recent-first"
+  (let ((body (dream-resp-body (host-fd-app (host-fd-req "/feed")))))
+    (< (index-of body "\"at\":3") (index-of body "\"at\":1")))
+  true)
+
+;; actor filter: only alice's two activities.
+(host-fd-test
+  "actor filter count"
+  (feed/count
+    (feed/by-actor (feed/recent (feed/all)) "alice"))
+  2)
+(host-fd-test
+  "actor filter excludes bob"
+  (contains?
+    (dream-resp-body (host-fd-app (host-fd-req "/feed?actor=alice")))
+    "bob")
+  false)
+
+;; limit: cap to a single activity (the most recent).
+(host-fd-test
+  "limit caps results"
+  (contains?
+    (dream-resp-body (host-fd-app (host-fd-req "/feed?limit=1")))
+    "\"at\":1")
+  false)
+
+;; ── golden: endpoint = subsystem recent stream + envelope ───────────
+(host-fd-test
+  "golden full timeline"
+  (dream-resp-body (host-fd-app (host-fd-req "/feed")))
+  (str
+    "{\"ok\":true,\"data\":"
+    (dream-json-encode (feed/items (feed/recent (feed/all))))
+    "}"))
+(host-fd-test
+  "golden actor-filtered"
+  (dream-resp-body (host-fd-app (host-fd-req "/feed?actor=alice")))
+  (str
+    "{\"ok\":true,\"data\":"
+    (dream-json-encode
+      (feed/items (feed/by-actor (feed/recent (feed/all)) "alice")))
+    "}"))
+
+(define
+  host-fd-tests-run!
+  (fn
+    ()
+    {:total (+ host-fd-pass host-fd-fail)
+     :passed host-fd-pass
+     :failed host-fd-fail
+     :fails host-fd-fails}))
--- a/lib/host/tests/handler.sx
+++ b/lib/host/tests/handler.sx
@@ -0,0 +1,86 @@
+;; lib/host/tests/handler.sx — host JSON envelope + request-reading helpers.
+
+(define host-hd-pass 0)
+(define host-hd-fail 0)
+(define host-hd-fails (list))
+
+(define
+  host-hd-test
+  (fn
+    (name actual expected)
+    (if
+      (= actual expected)
+      (set! host-hd-pass (+ host-hd-pass 1))
+      (begin
+        (set! host-hd-fail (+ host-hd-fail 1))
+        (append! host-hd-fails {:name name :actual actual :expected expected})))))
+
+;; ── host/ok ────────────────────────────────────────────────────────
+(host-hd-test "ok status 200" (dream-status (host/ok "x")) 200)
+(host-hd-test
+  "ok content-type json"
+  (dream-resp-header (host/ok "x") "content-type")
+  "application/json")
+(host-hd-test
+  "ok envelope ok:true"
+  (contains? (dream-resp-body (host/ok "x")) "\"ok\":true")
+  true)
+(host-hd-test
+  "ok envelope carries data"
+  (contains? (dream-resp-body (host/ok "hi")) "\"data\":\"hi\"")
+  true)
+
+;; ── host/ok-status ─────────────────────────────────────────────────
+(host-hd-test "ok-status custom" (dream-status (host/ok-status 201 "y")) 201)
+(host-hd-test
+  "ok-status data"
+  (contains? (dream-resp-body (host/ok-status 201 "y")) "\"data\":\"y\"")
+  true)
+
+;; ── host/error ─────────────────────────────────────────────────────
+(host-hd-test "error status" (dream-status (host/error 404 "nope")) 404)
+(host-hd-test
+  "error ok:false"
+  (contains? (dream-resp-body (host/error 404 "nope")) "\"ok\":false")
+  true)
+(host-hd-test
+  "error message"
+  (contains? (dream-resp-body (host/error 404 "nope")) "\"error\":\"nope\"")
+  true)
+(host-hd-test
+  "error content-type json"
+  (dream-resp-header (host/error 500 "boom") "content-type")
+  "application/json")
+
+;; ── host/json-status ───────────────────────────────────────────────
+(host-hd-test
+  "json-status arbitrary status"
+  (dream-status (host/json-status 418 {:a 1}))
+  418)
+(host-hd-test
+  "json-status encodes body"
+  (contains? (dream-resp-body (host/json-status 200 {:a 1})) "\"a\":1")
+  true)
+
+;; ── host/query-int ─────────────────────────────────────────────────
+(define
+  host-hd-req
+  (fn (target) (dream-request "GET" target {} "")))
+
+(host-hd-test
+  "query-int present"
+  (host/query-int (host-hd-req "/x?limit=5") "limit" 10)
+  5)
+(host-hd-test
+  "query-int absent -> fallback"
+  (host/query-int (host-hd-req "/x") "limit" 10)
+  10)
+
+(define
+  host-hd-tests-run!
+  (fn
+    ()
+    {:total (+ host-hd-pass host-hd-fail)
+     :passed host-hd-pass
+     :failed host-hd-fail
+     :fails host-hd-fails}))
--- a/lib/host/tests/router.sx
+++ b/lib/host/tests/router.sx
@@ -0,0 +1,75 @@
+;; lib/host/tests/router.sx — host app assembly: health endpoint, group mounting,
+;; 404 fallback.
+
+(define host-rt-pass 0)
+(define host-rt-fail 0)
+(define host-rt-fails (list))
+
+(define
+  host-rt-test
+  (fn
+    (name actual expected)
+    (if
+      (= actual expected)
+      (set! host-rt-pass (+ host-rt-pass 1))
+      (begin
+        (set! host-rt-fail (+ host-rt-fail 1))
+        (append! host-rt-fails {:name name :actual actual :expected expected})))))
+
+(define
+  host-rt-req
+  (fn (method target) (dream-request method target {} "")))
+
+;; An app built from one domain group of two routes.
+(define
+  host-rt-app
+  (host/make-app
+    (list
+      (list
+        (dream-get "/ping" (fn (req) (host/ok "pong")))
+        (dream-get "/widgets/:id" (fn (req) (host/ok (dream-param req "id"))))))))
+
+;; ── health ─────────────────────────────────────────────────────────
+(host-rt-test
+  "health status 200"
+  (dream-status (host-rt-app (host-rt-req "GET" "/health")))
+  200)
+(host-rt-test
+  "health body healthy"
+  (contains?
+    (dream-resp-body (host-rt-app (host-rt-req "GET" "/health")))
+    "healthy")
+  true)
+
+;; ── group routes mounted ───────────────────────────────────────────
+(host-rt-test
+  "group route ping"
+  (contains?
+    (dream-resp-body (host-rt-app (host-rt-req "GET" "/ping")))
+    "pong")
+  true)
+(host-rt-test
+  "group path param"
+  (contains?
+    (dream-resp-body (host-rt-app (host-rt-req "GET" "/widgets/42")))
+    "\"data\":\"42\"")
+  true)
+
+;; ── fallback ───────────────────────────────────────────────────────
+(host-rt-test
+  "unknown path 404"
+  (dream-status (host-rt-app (host-rt-req "GET" "/nope")))
+  404)
+(host-rt-test
+  "wrong method 405"
+  (dream-status (host-rt-app (host-rt-req "POST" "/ping")))
+  405)
+
+(define
+  host-rt-tests-run!
+  (fn
+    ()
+    {:total (+ host-rt-pass host-rt-fail)
+     :passed host-rt-pass
+     :failed host-rt-fail
+     :fails host-rt-fails}))
--- a/plans/host-on-sx.md
+++ b/plans/host-on-sx.md
@@ -36,7 +36,7 @@ host — no `ocaml-on-sx` dependency.

 ## Status (rolling)

-`bash lib/host/conformance.sh` → **0/0** (not yet started)
+`bash lib/host/conformance.sh` → **28/28** (3 suites: handler, router, feed). Phase 1 DONE.

 ## Ground rules

@@ -73,10 +73,15 @@ lib/host/sxtp.sx                        subsystem APIs (feed/search/commerce/…
 ```

 ## Phase 1 — Router + handler + one real endpoint
- [ ] `router.sx` — route table, (method,path) match
- [ ] `handler.sx` — request/response model, subsystem dispatch
- [ ] migrate ONE read endpoint (e.g. a feed timeline) end-to-end, golden test
- [ ] `conformance.sh` + scoreboard
+- [x] `router.sx` — `host/make-app` assembles per-domain route groups + a built-in
+      `/health` probe into one Dream router (reuses Dream's `dr/flatten-routes`)
+- [x] `handler.sx` — JSON envelope (`host/ok`/`host/ok-status`/`host/error`),
+      status-carrying `host/json-status` (Dream's `dream-json` is 200-only), and
+      `host/query-int`. A host handler IS a Dream handler (request -> response).
+- [x] migrate ONE read endpoint: `GET /feed` (`lib/host/feed.sx`) reads
+      `feed/all` + stream combinators, serialises recent-first; `?actor=` filter,
+      `?limit=` cap. Golden test asserts body == subsystem recent stream + envelope.
+- [x] `conformance.sh` (mirrors `lib/dream`'s runner) — 28/28

 ## Phase 2 — Middleware + SXTP
 - [ ] `middleware.sx` — composable auth/acl/mute/error layers
@@ -94,7 +99,33 @@ lib/host/sxtp.sx                        subsystem APIs (feed/search/commerce/…
 - [ ] re-home external adapters as native where replacements land

 ## Progress log
-(loop fills this in)
+
+- **Phase 1 (DONE, 28/28).** `lib/host/{handler,router,feed}.sx` + three test
+  suites + `conformance.sh`. The host is a thin wiring layer: a host handler is a
+  Dream handler that calls a subsystem public API and serialises the result via a
+  shared JSON envelope. First migrated endpoint: `GET /feed`.
+  - **Decision — build on Dream from Phase 1, not a throwaway native model.** The
+    plan front-matter gated Dream to Phase 4, but `dream-on-sx` is merged
+    (commit fe958bda) and its gate (`ocaml-on-sx` P1–5+P6) is green (480/480), so
+    reinventing request/response + routing would be pure duplication. Host reuses
+    Dream's `types.sx` (request/response dicts), `json.sx` (encode), and
+    `router.sx` (`dream-router`/`dream-get`/`dr/flatten-routes`). Phase 4's
+    "adopt Dream ergonomics" is therefore largely already satisfied; what remains
+    for Phase 4 is the live wiring against the real OCaml HTTP server + session.
+  - The OCaml server handing a `dream-request`-shaped dict to SX handlers is a
+    `hosts/` change (out of scope) — tracked under Blockers as the eventual
+    live-wiring step. For now the host layer is exercised purely via conformance.

 ## Blockers
-(loop fills this in)
+
+- **Live wiring to the native OCaml HTTP server** (Phase 3/4): the prod server in
+  `hosts/` must hand SX handlers a `dream-request` dict and serialise the returned
+  `dream-response`. That is a `hosts/` change (out of scope for this loop, which is
+  `lib/host/**` only). Until then, endpoints are verified via `conformance.sh`, not
+  HTTP. Not blocking Phase 2 (middleware + SXTP + a write endpoint).
+- **Worktree tooling:** in this `loops/host` worktree every sx-tree *write* tool
+  (`sx_write_file`, `sx_replace_node`, …) raises `yojson "Expected string, got
+  null"` at the MCP layer — same class as the `loops/dream` worktree gotcha, but
+  here even `sx_write_file` fails. Read-side sx-tree tools work. New `.sx` files
+  were created with the `Write` tool (the .sx hook is inactive in this worktree)
+  and each validated afterwards with `sx_validate` to keep the parse guarantee.