diff --git a/plans/agent-briefings/host-fed-sx-adapter-loop.md b/plans/agent-briefings/host-fed-sx-adapter-loop.md new file mode 100644 index 00000000..1e4fa1c6 --- /dev/null +++ b/plans/agent-briefings/host-fed-sx-adapter-loop.md @@ -0,0 +1,211 @@ +; -*- mode: markdown -*- +# loops/host — fed-sx adapter slice (host side of host-type federation) + +Scoped briefing for the follow-up that wires `loops/host`'s SX/dream +front door to the fed-sx kernel substrate landed by +`loops/fed-sx-types`. Companion to `plans/fed-sx-host-types.md` (the +substrate design + public surface). This is the build sheet for the +host-side adapters the substrate loop deliberately deferred. + +``` +description: loops/host — fed-sx adapter (publish/serve/ingest typed posts) +subagent_type: general-purpose +run_in_background: true +isolation: worktree # worktree at /root/rose-ash-loops/host +``` + +## Why this is small now + +The substrate is done and tested (`origin/loops/fed-sx-types`, 4 +phases). And the host already has *most of a type system*: +`lib/host/blog.sx` models a **type as a post** with a content-address +`:cid`, a `:schema` (`{:required [...]}`), `:fields`, `:template`, and a +**`subtype-of` graph over lib/relations**. So this loop is not building +a type model — it is **projecting the host's existing one onto fed-sx** +and ingesting peers' types back. The pieces line up almost 1:1: + +| host (lib/host/blog.sx) | fed-sx (next/kernel) | +|--------------------------------------------|------------------------------------------| +| a type-post + `:schema {:required [...]}` | `DefineType` activity + refinement `{required, [...]}` | +| `subtype-of` edge (lib/relations) | `SubtypeOf` activity | +| `host/blog-by-cid` / `host/blog-type-defs` | `peer_types` cache + `GET /types/` | +| `host/blog-type-issues` (local validate) | `pipeline:apply_object_schema/2` (inbound) | + +The host's `{:required [...]}` schema maps **directly** onto the +term_codec-safe `{required, [Field,...]}` refinement form the substrate +already validates — so schema translation is nearly trivial. Derive +both validators from the *same* schema data to avoid drift. + +## Scope + +**In scope** — `lib/host/**` only (the mirror of fed-sx-types' +`next/**` only). New: `lib/host/fed_sx_outbox.sx`, +`lib/host/fed_sx_inbox.sx` (and a small shared `lib/host/fed_sx.sx` +bridge + a `host/types-routes`), plus `serve.sh` bring-up wiring and +`lib/host/tests/`. + +**Out of scope** — `next/**`. The substrate is frozen public surface; +do **not** edit `next/kernel/**` or the genesis verbs from here. If a +gap is found, file it against fed-sx-types, don't patch across the +boundary. **Hard line: do not edit `next/`.** + +## Branch base + +Start from `loops/host`, then bring in the substrate (clean, additive — +disjoint paths, no conflicts): + +```bash +cd /root/rose-ash-loops/host +git fetch origin +git merge origin/loops/fed-sx-types # adds next/kernel + genesis + plan doc +``` + +After the merge the worktree has both `lib/host/**` (host) and +`next/kernel/**` (fed-sx substrate) on one branch. + +## Phase 0 — settle the runtime boundary (DECIDE FIRST, blocks all else) + +`lib/host` is **pure same-runtime SX** (one `sx_server.exe`: stdlib → +R7RS → APL → Datalog → ACL → Relations → Feed → Persist → Dream → +Host, per `serve.sh`). The fed-sx kernel is **Erlang-on-SX** on the +er-scheduler (`erlang-load-module` + gen_servers: `peer_types`, +`nx_kernel`). The host's dream handlers run on the native `http-listen` +accept loop — **outside** the er-scheduler. Calling a kernel +`gen_server:call` synchronously from a native-thread handler hits the +known scheduler-context deadlock (see +`plans/fed-sx-design.md` §, and the fed-prims http-listen note: a +handler on `Thread.create` outside er-sched can't complete a +`gen_server:call → receive`). + +Two architectures; **the loop's first deliverable is choosing one with +a tiny spike**: + +- **Option A — in-process Erlang bridge.** Host's sx_server also + `erlang-load-module`s the kernel and calls it directly. Pro: one + process, no serialization. Con: the deadlock above — kernel calls + must be marshalled onto the er-scheduler or restricted to pure + (non-gen_server) functions. Fragile; not recommended. + +- **Option B — HTTP boundary (RECOMMENDED).** Run the fed-sx kernel + with its own `http_server`/`http-listen` loop (it already has the + whole route surface, and the m2 two-instance smoke test proves + HTTP federation between fed-sx nodes). The host talks to its local + fed-sx node **over localhost HTTP using the wire it already speaks** + (term_codec / activity+json / type-doc). This is literally what + federation is — the host is just another peer to its own node. An + `httpc`/localhost call from a native-thread host handler does **not** + touch the er-scheduler, so the deadlock never arises; the kernel's + own listen handler runs the gen_server calls within er-sched context. + Works whether the kernel is a sidecar process or spawned on an + er-scheduler process inside the host's sx_server (two ports, one + process). Pro: clean, reuses the fully-tested surface, no deadlock. + Con: serialization + lifecycle coordination. + +**Recommendation: Option B.** Spike: host handler → `httpc` POST to the +local kernel's `/activity` → 200/cid back, with no hang. Lock the +decision before Phase 1. + +## Phase 1 — outbox: project host types → DefineType / SubtypeOf + +`lib/host/fed_sx_outbox.sx`. When a host type-post is created/updated +(`host/blog-put!` path), project it and publish to the local fed-sx +node: + +- type-post → `DefineType` activity: `:object` = `{name: slug, + fields: (host/blog-fields-of slug), refinement-schema: + (host/blog-schema-of slug), instance-type: }`. The host + `{:required [...]}` becomes the substrate `{required, [...]}` form + verbatim. +- each `subtype-of` edge (`relations/parents` over `"subtype-of"`) → + a `SubtypeOf` activity `{child-type-cid, parent-type-cid}`. +- publish via Phase 0's transport (POST `/activity` to the local node, + authed with the node's publish token). + +**Key open decision — the type CID.** The host computes `:cid` via +`host/blog--cid-of` (double-hash over the canonical record); fed-sx +keys `peer_types` by a `TypeCid`. Either: + (a) **adopt the host `:cid` as the fed-sx TypeCid** — one identity, + no reconciliation, but peers can't content-verify it from the + wire bytes; or + (b) **let the kernel content-address the TypeRecord** — verifiable, + but the host must keep a `slug → fed-sx-cid` map (and + `SubtypeOf` edges must reference fed-sx CIDs, not host CIDs). +Pick one and document it in `plans/fed-sx-host-types.md`. (a) is +simpler and probably right for v1; revisit when cross-node verification +matters. + +## Phase 2 — inbox: ingest peers' types + validate typed objects + +`lib/host/fed_sx_inbox.sx`. Inbound from the local node's inbox: + +- inbound `DefineType` → `peer_types:put` (cache it). Decide whether to + also **materialize** it as a host post (`host/blog-put!` + + `host/blog--set-schema!`) or keep federation-only types out of the + local blog (recommended for v1: cache-only, materialize on demand). +- inbound `SubtypeOf` → record the edge (peer_types hierarchy and/or + `host/blog-relate! child parent "subtype-of"` if materialized). +- inbound typed `Create` (a post that `is-a` some refinement type) → + the kernel inbound pipeline runs `pipeline:apply_object_schema/2` + (configured with a `type_index` + `{peer_types, peer_types}` + + `type_fetch_fn`), so a typed object is validated against its declared + type **before** the host sees it. Choose `strict_object_schema` + per-node (default false = open-world). + +**Avoid double-validation drift:** the host already has +`host/blog-type-issues`. Let the **kernel validate federation inbound** +and the **host validate local writes**, both deriving from the same +`{:required [...]}` schema data — don't fork the rules. + +## Phase 3 — serve + bring-up wiring + +- **Serve `GET /types/`** on the host front door. Either proxy to + the kernel's `/types/` (Option B keeps one source of truth), or + serve directly from `host/blog-by-cid` + the projected TypeRecord. + Hook as `(dream-get "/types/:cid" host/types-by-cid)` and add + `host/types-routes` to the `host/serve` list (per `router.sx` / + `serve.sh` pattern). +- **Bring-up** in `serve.sh`: start the fed-sx node (Phase 0 transport), + start `peer_types`, configure `type_fetch_fn = + discovery_type_fetch:make_fetch_fn()` + a `type_url` resolver, and on + startup project existing host types (Phase 1) so the node is + type-aware from boot. Gate writes behind the existing + `host/require-auth` / `host/require-permission` middleware, same as + the relations write routes. + +## Phase 4 — end-to-end round-trip test + +Two nodes (host A + host B, or host + a sidecar fed-sx node): A defines +a refinement type → B fetches the type-doc via `GET /types/` → B +ingests an inbound typed object and `apply_object_schema` accepts the +valid one / rejects a refinement-failing one. Mirror the m2 +two-instance smoke test style. Plus per-phase suites in +`lib/host/tests/` (the host runs its own `conformance.sh`). + +## Tests discipline + +- The host's `lib/host/conformance.sh` green before AND after every + commit. `lib/host` is **LIVE at blog.rose-ash.com** — pushing + `loops/host` reloads dev, so treat pushes as deliberate. +- Commits scoped to `lib/host/**` (+ `plans/fed-sx-host-types.md` as + decisions ratify). Do **not** edit `next/**`. +- One commit per phase; smaller intermediate commits fine if each + leaves the gate green. The Phase-0 spike can be its own commit. + +## Done when + +- A host type round-trips: defined locally → published as + `DefineType`/`SubtypeOf` → fetchable at `GET /types/` → a peer + validates an inbound typed object against it. +- `peer_types` is populated from inbound `DefineType`, and the inbound + pipeline rejects refinement-failing typed objects (strict node). +- The runtime-boundary decision and the type-CID decision are recorded + in `plans/fed-sx-host-types.md`. +- Host `conformance.sh` + the new fed-sx adapter suites green. + +## Parallel-safety with loops/fed-sx-types + +That loop owns `next/**` and is feature-complete; this loop owns +`lib/host/**`. Disjoint surfaces — they meet only at the merge that +brings the substrate in (Branch base, above). If this loop needs a +substrate change, file it against fed-sx-types rather than editing +`next/` here.