From 4c0a48834ee144de4b5316d1bb4450ec73f1a978 Mon Sep 17 00:00:00 2001 From: giles Date: Thu, 2 Jul 2026 12:25:50 +0000 Subject: [PATCH] preserve fed-sx-m1 loop briefings before pruning its worktree 4 untracked agent-briefing docs from the fed-sx-m1 worktree (merged branch loops/fed-sx-m2), saved here so they survive the worktree cleanup. --- plans/agent-briefings/erlang-send-after.md | 151 +++++++ plans/agent-briefings/fed-sx-m1-loop.md | 170 ++++++++ plans/agent-briefings/fed-sx-triggers-loop.md | 388 ++++++++++++++++++ plans/agent-briefings/fed-sx-types-loop.md | 218 ++++++++++ 4 files changed, 927 insertions(+) create mode 100644 plans/agent-briefings/erlang-send-after.md create mode 100644 plans/agent-briefings/fed-sx-m1-loop.md create mode 100644 plans/agent-briefings/fed-sx-triggers-loop.md create mode 100644 plans/agent-briefings/fed-sx-types-loop.md diff --git a/plans/agent-briefings/erlang-send-after.md b/plans/agent-briefings/erlang-send-after.md new file mode 100644 index 00000000..b3562ce2 --- /dev/null +++ b/plans/agent-briefings/erlang-send-after.md @@ -0,0 +1,151 @@ +; -*- mode: markdown -*- +# loops/erlang — `erlang:send_after` substrate primitive + +Scoped briefing for a single focused iteration loop on `loops/erlang`. +Not a replacement for the general `erlang-loop.md`; this is the +load-bearing-blocker task that, when done, unblocks +`plans/fed-sx-milestone-2.md` Step **8b-timer** (delivery retry loop) +and the standard `gen_server` `timeout` return. + +``` +description: loops/erlang — send_after primitive +subagent_type: general-purpose +run_in_background: true +isolation: worktree # worktree at /root/rose-ash-loops/erlang +``` + +## The goal + +Implement the standard Erlang timer primitives so a process can +schedule a message-to-self (or to another pid) after N milliseconds: + +```erlang +Ref = erlang:send_after(Time, Dest, Msg). %% Time: int millis. Dest: pid or atom. +ok | TimeLeft = erlang:cancel_timer(Ref). %% returns remaining ms, or false if fired/expired. +``` + +These are the same primitives `gen_server` uses internally to emit +`{noreply, S, Timeout}` returns — when the gen_server's `handle_*` +callback returns `{noreply, NewState, T}`, the gen_server schedules +`{timeout}` to itself after T ms via `send_after`, and the next +`handle_info({timeout}, S)` fires when no other message arrives first. + +Without `send_after`, anything that wants a delayed self-cast has to +busy-loop or block — neither acceptable for the kernel. + +## Acceptance — single SX file `lib/erlang/tests/send_after.sx` + +The standard test suite plus this one. Run the conformance gate after +each commit (`bash lib/erlang/conformance.sh`). + +- `T1` — `erlang:send_after(50, self(), hello)` returns a Ref; after + 60ms a receive picks up `hello`. Round-trip latency under 100ms in + steady state. +- `T2` — `Ref = erlang:send_after(1000, self(), late), erlang:cancel_timer(Ref)` + returns an integer ~1000 (remaining ms) AND a subsequent + `receive late -> got after 50 -> none end` returns `none` — the + cancelled message never arrives. +- `T3` — multiple in-flight timers fire in deadline order, not + schedule order: `send_after(80, self(), b)`, then + `send_after(20, self(), a)` — selective receive on `a` first. +- `T4` — cancel_timer on an already-fired timer returns `false`. +- `T5` — `send_after` to a registered atom: register a process, queue + a delayed message to its name, it lands in that process's mailbox. +- `T6` — `gen_server` `{noreply, S, 100}` → `handle_info({timeout}, S)` + fires after 100ms when no other message arrives. (Sanity check that + the gen_server library, currently shipped, hooks up correctly.) + +Conformance gate stays green (currently 725/725 on loops/erlang per +`lib/erlang/scoreboard.json`, 761/761 after the m2 BIFs are in — +both numbers will move when this lands). + +## Implementation shape (suggested, not prescribed) + +The OCaml host has no event loop today — `bin/sx_server.ml` is a +single-thread epoch protocol. But the Erlang scheduler in +`lib/erlang/runtime.sx` IS a SX-level event loop already: `er-sched` +holds the global dict; `er-sched-step-alive!` advances one runnable +process; `er-sched-run-all!` drains the runnable queue until all +processes are waiting / exiting / dead. Two options: + +**Option A — pure-SX timer wheel.** Extend the scheduler dict with +`:timers (sorted-list-of {:deadline-ms _ :pid _ :msg _ :ref _ :alive +true})`. `er-sched-run-all!` already loops; when it finds the runnable +queue empty but processes still alive in `receive` blocks, walk the +timer list and check whether any deadline has passed (need a +monotonic clock — see below). For deadlines in the future, sleep +until the next one or until the runnable queue gets work. Cancel = set +`:alive false` on the timer entry; cull on next scan. + +**Option B — OCaml-side `Unix.setitimer` / `Unix.select` wakeup.** +Heavier, lets the SX scheduler park properly on `Unix.select` against +a self-pipe. Right shape for a long-running kernel that may sit idle +for minutes between events. Probably the production answer, but +significantly more work than A. + +Recommend A first for the unblock; B as a follow-up if the kernel +needs to idle gracefully. + +### Clock primitive + +`erlang:monotonic_time/0,1` and `erlang:system_time/0,1` ought to +exist for completeness — `send_after` needs at least a monotonic ms +clock. If the substrate doesn't have one, add `erlang:monotonic_time(millisecond)` +as a small native BIF in `bin/sx_server.ml` (Unix.gettimeofday) and +expose it via `er-bif-erlang-monotonic-time`. Time travel safety: +DO NOT use wall-clock for timer deadlines. + +## Files in play (loops/erlang scope) + +- `lib/erlang/runtime.sx` — scheduler state, BIFs, primitive registration. + Scheduler dict at `er-scheduler` (single-element list). + Step function: `er-sched-step-alive!` (line ~708 on m2; check + line numbers freshly — recent commits moved code). +- `lib/erlang/tests/send_after.sx` — new file, the test suite above. +- `lib/erlang/scoreboard.json` + `scoreboard.md` — bump counts. +- (If Option B is taken) `bin/sx_server.ml` — out of `loops/erlang` + scope per the loop's standing rules. Surface a separate + `loops/fed-prims` task for the native bits and keep `send_after`'s + SX wrapper minimal. + +## Base branch / rebase note + +The substrate change for the kernel mutex (Blockers #4 in +fed-sx-milestone-2.md) lands on architecture via the m2 merge. The +key change is in `lib/erlang/runtime.sx`: + +- `er-sched-step-alive!` reads `:pending-args` from the process record + when first invoked (line 736 on m2; was hardcoded to `(list)`). +- `er-bif-http-listen` spawns the user's handler as a real er-process + with `:pending-args (list req-pl)` instead of `er-apply-fun`-inline. + +If `loops/erlang` rebases onto `origin/architecture` and that merge +hasn't landed yet (still sitting on `local /root/rose-ash` at +`089ed88f`-or-later), fetch `origin/loops/fed-sx-m2` and cherry-pick +or rebase against `29e4234b` to pick up the `:pending-args` field +shape. The work area (`er-sched-step-alive!`) is the same surface +this loop will touch — without the m2 change, the timer wakeup path +won't dovetail cleanly with the spawn-from-BIF pattern. Conflict +risk: medium. Conflict surface: small (~5 lines around the +`:pending-args` read). + +## Test discipline + +- `bash lib/erlang/conformance.sh` green before and after every commit. +- Commits scoped to `lib/erlang/**`. No edits to `next/**`, + `bin/sx_server.ml`, `spec/**`, or other `lib//**` from this + loop. +- One commit per acceptance test cluster — `T1+T2`, `T3+T4`, `T5+T6` + is a reasonable cadence. +- Push to `origin/loops/erlang` after each commit. The fed-sx-m2 loop + is dormant waiting on this. + +## Done when + +- 6/6 new `send_after` tests green. +- conformance gate green at the new total. +- single line ticked in `plans/erlang-on-sx.md` Progress log + a tick + on whatever roadmap entry covers timers. +- Then update the m2 plan (`plans/fed-sx-milestone-2.md` Blockers #3) + to **RESOLVED** and unblock Step 8b-timer; that update belongs to + the fed-sx-m2 loop, not this one. diff --git a/plans/agent-briefings/fed-sx-m1-loop.md b/plans/agent-briefings/fed-sx-m1-loop.md new file mode 100644 index 00000000..921cd06c --- /dev/null +++ b/plans/agent-briefings/fed-sx-m1-loop.md @@ -0,0 +1,170 @@ +# fed-sx Milestone 1 loop agent (single agent, step-ordered) + +Role: iterates `plans/fed-sx-milestone-1.md` forever. Builds the smallest fed-sx +kernel that proves the architecture works end-to-end. One feature per commit. + +``` +description: fed-sx Milestone 1 kernel loop +subagent_type: general-purpose +run_in_background: true +isolation: worktree +``` + +## Prompt + +You are the sole background agent working `/root/rose-ash/plans/fed-sx-milestone-1.md`. +You run in an isolated git worktree on branch `loops/fed-sx-m1` at +`/root/rose-ash-loops/fed-sx-m1`. You work the plan's Steps in dependency order +(1→9), forever, one commit per feature. Push to `origin/loops/fed-sx-m1` after +every commit. Never `main`, never `architecture`. + +## Restart baseline — check before iterating + +1. Read `plans/fed-sx-milestone-1.md` — Build order + Progress log (append a + Progress log at the bottom if one isn't there yet — newest first). +2. `ls next/` — pick up from the most advanced step that exists. If `next/` + doesn't exist, you're at Step 1. +3. Erlang substrate must be green: + `cd lib/erlang && bash conformance.sh 2>&1 | tail -2` → expect `729 / 729`. + If broken and not by your edits, Blockers entry + stop. +4. If `next/tests/*.sh` exist, run them. They must be green before new work. +5. Read the Erlang Phase 8 status in `plans/erlang-on-sx.md` — `httpc:request` + and `sqlite:*` remain BLOCKED. Milestone 1 does not need them per the + non-goals section. If you find yourself needing them, you're off-plan. + +## The build queue + +Each Step has concrete deliverables + tests + acceptance check in the plan. +Within a Step, pick the smallest unchecked sub-deliverable. Don't batch Steps. + +- **Step 1** — Repo skeleton (`next/`) + canonical CID computation +- **Step 2** — Activity envelope shape + signature verify +- **Step 3** — JSONL log + sequence numbers (per-actor outbox) +- **Step 4** — Genesis bundle: SX sources + bundler + CID verification +- **Step 5** — Registry mechanism + bootstrap-projection dispatch +- **Step 6** — Validation pipeline driver + `POST /activity` +- **Step 7** — Projection scheduler (gen_server per projection) +- **Step 8** — HTTP server + AP endpoints + projection queries +- **Step 9** — Smoke tests: Pin verb extensibility + reactive application + +The iteration: +implement → run step's tests → run no-regression gate (Erlang conformance) → +commit → tick the `[ ]` in the plan → append one dated line to the Progress +log → push → stop. + +## How fed-sx code lives in this repo + +The kernel is **Erlang-on-SX**. Three concrete patterns: + +1. **Kernel modules as `.erl` source files** at `next/kernel/*.erl`. Loaded at + boot via `code:load_binary(Mod, Filename, SourceString)` (Phase 7 BIF). + Example: `next/kernel/cid.erl` with `-module(cid). -export([from_sx/1, ...]).` +2. **Genesis bundle entries as `.sx` files** at `next/genesis/**/*.sx`. + These ARE small SX expressions per design §12.2 (DefineActivity{}, + DefineProjection{}, etc.). +3. **Test scripts as bash** at `next/tests/*.sh`. Each one feeds an epoch + protocol script to `hosts/ocaml/_build/default/bin/sx_server.exe` that + loads kernel modules, drives them, and asserts on output. + +The `epoch` protocol pattern: +```bash +printf '(epoch 1)\n(load "lib/erlang/runtime.sx")\n(epoch 2)\n\n' \ + | hosts/ocaml/_build/default/bin/sx_server.exe +``` + +## Phase 8 BIFs available to you + +These are wired and tested in `lib/erlang/runtime.sx` (Erlang conformance 729/729): + +- `crypto:hash/2` — `sha256` / `sha512` / `sha3_256`, returns raw binary +- `cid:from_bytes/1` — bytes → CIDv1 (raw codec, sha2-256 multihash) as binary +- `cid:to_string/1` — any term → canonical CIDv1 string via `er-format-value` +- `file:read_file/1`, `file:write_file/2`, `file:delete/1`, `file:list_dir/1` +- `code:load_binary/3` — hot-load an Erlang module from source string + +Native HTTP server (registered in `bin/sx_server.ml`, not yet wrapped as an +Erlang BIF): + +- SX primitive `http-listen` — exposed at the SX layer, native-only + +For Step 8 you will probably need to add an Erlang BIF wrapper `http:listen/2` +that delegates to the SX `http-listen` primitive. That single wrapper is an +allowed exception to the scope rules (see below) — flag it explicitly in the +commit message. + +**Blocked primitives** (do NOT use, Milestone 1 doesn't need them): +- `httpc:request/4` — HTTP client (Milestone 2 federation) +- `sqlite:*` — SQLite (deferred storage backend) + +## Ground rules (hard) + +- **Scope:** only `next/**` and `plans/fed-sx-milestone-1.md`. Single allowed + exception: a `http:listen/2` BIF wrapper in `lib/erlang/runtime.sx` for Step 8 + (one commit, clearly flagged). Do **not** touch `lib/erlang/` otherwise, + `hosts/ocaml/`, `spec/`, `shared/`, or other `lib//`. +- **Erlang-on-SX is the substrate.** Kernel modules are `.erl` source loaded + via `code:load_binary/3`. Don't reach for pure SX or Python — the substrate + for this work is Erlang. +- **No new opam deps.** No new host primitives. If you find yourself wanting a + new primitive, that's a Blockers entry — the `loops/fed-prims` loop owns + primitives, not this loop. +- **No-regression gate:** after every commit, `bash lib/erlang/conformance.sh` + must report `729 / 729`. Your new tests are additive: `bash next/tests/*.sh` + also pass. Test the gate before pushing. +- **Builds are slow.** `dune build` (if you ever need it — you shouldn't) gets + `timeout: 600000`. Conformance gate: `timeout: 400000`. If a build genuinely + hangs > 10min, Blockers entry + stop. +- **Commit granularity:** one feature per commit. Short factual messages: + `fed-sx-m1: Step 1 — canonical CID + 10 round-trip tests`. Update plan + checkboxes + Progress log in the SAME commit as the feature. +- **`.erl`/`.sh`/`.md` files:** ordinary `Read`/`Edit`/`Write`. The hook only + blocks `.sx`/`.sxc`. For `.sx` files (genesis bundle, Step 4 onwards) use + `sx-tree` MCP tools and `sx_write_file` exclusively. +- **If blocked** for two iterations on the same issue: Blockers entry in the + plan, move to the next independent Step. Step dependencies in the plan's + build order table. + +## Specific gotchas + +- **The plan's `.erl` snippets are illustrative.** This Erlang port doesn't + necessarily support every BEAM feature. Verify what works by smoke-testing + one expression at a time via the epoch protocol before writing 200-line + modules. The Erlang port's binary syntax in particular: `<<"abc">>` produces + an **empty** binary in this port (string-literal segments unsupported) — + use integer segments `<<97,98,99>>` instead. This bit the erlang loop on + Phase 8; don't get burned again. +- **`cid:to_string/1` on compound terms uses `er-format-value`**, not + `er-to-sx`. `cbor-encode` rejects marshalled symbols, so the canonical + string form goes via the deterministic textual form. Mirror this when you + implement `from_sx/1`. +- **Genesis SX files use `defactivity`, `defobject`, `defprojection`, ...** — + these aren't real special forms; they're data shapes. The bundler reads the + `.sx` file as a list of `(DefineActivity ...)` etc. expressions and serialises + the parsed AST to dag-cbor. The genesis bundle CID is a fixed constant + baked into the kernel binary (or, for v1, into a sibling `.cidhash` file). +- **`replay/3` (Step 3)** is the hot path for projection cold-start. Make + sure it streams (read line, fold, repeat) — don't load all activities into + memory. +- **Validation pipeline stages (Step 6)** halt on first failure. Each stage + returns `ok | {error, Reason}`. The pipeline driver is one fold over a list + of stage functions. Resist the urge to make it a gen_server. +- **Projection sandbox mode (Step 7)** must be **pure**: no IO platform, no + clock, no random except CID-seeded. Use `sandbox:eval_pure/2` for fold + bodies. This is load-bearing for determinism — every host must reproduce + the same projection state from the same log prefix. +- **HTTP endpoint shapes (Step 8)** follow design §16.1. Content negotiation + on `Accept`: default `application/activity+json`, plus `cbor`/`json`/`sx`. + Auth on `POST /activity` is bearer token from env `NEXT_PUBLISH_TOKEN`. + +## Style + +- No comments in `.erl` unless non-obvious. Cite design §-numbers when a + decision is non-obvious to a reader. +- No new planning docs — update `plans/fed-sx-milestone-1.md` inline. Add a + "Progress log" section at the bottom on first iteration. +- One Step (or sub-deliverable for the big Steps 5/6/7/8) per iteration. + Implement. Test. Gate. Commit. Log. Push. Next. + +Go. Read the plan. Run the restart baseline. Find the first unchecked deliverable +in Step 1. Implement it. Remember: no commit without the step's acceptance +tests passing AND Erlang conformance 729/729 unaffected. diff --git a/plans/agent-briefings/fed-sx-triggers-loop.md b/plans/agent-briefings/fed-sx-triggers-loop.md new file mode 100644 index 00000000..0d401b2b --- /dev/null +++ b/plans/agent-briefings/fed-sx-triggers-loop.md @@ -0,0 +1,388 @@ +; -*- mode: markdown -*- +# loops/fed-sx-triggers — activity-driven flow triggers + +Scoped briefing for a focused iteration loop that wires fed-sx +activities to durable business flows on top of `lib/flow/`. Every +arriving activity (federation OR local publish) can fan out into +one or more registered flows; the flow engine handles multi-step +workflows, branches, side effects, retries, and human gates. + +Companion to `plans/fed-sx-design.md` §13 (delivery / projection +model) and `plans/flow-on-sx.md` (flow engine surface). This is +the build sheet for the substrate side of "an activity fires a +business flow." + +``` +description: loops/fed-sx-triggers — activity → flow dispatch (substrate) +subagent_type: general-purpose +run_in_background: true +isolation: worktree # /root/rose-ash-loops/fed-sx-triggers +``` + +## The mental model + +``` +activity arrives ─→ pipeline validates ─→ kernel appends ─→ fan-out + │ + trigger_registry lookup + │ + ┌───────────────┬───────────────────┴────────┐ + ↓ ↓ ↓ + flow:start(F1) flow:start(F2) flow:start(F3) + │ │ │ + (suspend, (sync side (multi-step: + resume on effect: payment → ship + timer) send email) → notify, with + branches) +``` + +Activities are the unit of "something happened." Flows are the +unit of "what we DO about it" — composable, durable, branching. + +## Concrete examples (motivating) + +- **Blog publish** (`Create` for an `Article`): index in search, + send digest to followers (per-actor preference dictates batching), + push to federation peers, invalidate caches. Branch: if the + article references unpublished drafts, hold publication until + they land. +- **Order placed** (`Create` for an `Order`): charge payment → + on fail, cancel + notify; on success, allocate inventory → + on out-of-stock, refund + notify; on success, schedule shipping + + send confirmation + start a follow-up flow for "ask for review + in 14 days." +- **Comment posted**: run moderation pipeline → on fail, hold for + human review (suspend until a moderator activity arrives); on + pass, notify the post author + bump comment count + check for + follow-up triggers. +- **Membership renews**: bill the card → on fail, retry 3 days + later; on terminal fail, downgrade access + notify. Suspend + for 364 days, then fire again. + +All of these are 5-20 step flows with branches, retries, +suspensions on external events, and human gates. They are +`lib/flow/` territory — m2's `delivery_worker` retry is much +simpler (one HTTP call, fixed backoff) and lives in +`next/kernel/`, not here. + +## Scope + +**In scope** — `next/**` only. Four pieces: + +1. `DefineTrigger` genesis activity-type + `trigger_registry.erl` +2. Trigger fan-out stage in `pipeline.erl` (post-append) +3. `flow_dispatch.erl` — bridges activity → `lib/flow/` start call +4. End-to-end integration test exercising a multi-step flow with + one branch and one suspension + +**In scope to IMPORT from** — `lib/flow/**` is a public dependency. +This loop calls `flow:start/2`, `flow:resume/2`, etc. as a +consumer; it does not modify `lib/flow/`. + +**Out of scope** — `lib/host/**`, `lib/erlang/**`, `bin/sx_server.ml`, +any `lib//` directory. Touching them would conflict with +parallel loops. + +**Parallel-safety with `loops/fed-sx-types`** — that loop also +edits `pipeline.erl` (adding `apply_object_schema/2` between +activity-type validation and append). This loop adds a fan-out +stage AFTER append. Different positions in the pipeline. Resolve +by inserting both stages in a single commit OR by branching this +loop from fed-sx-types' tip once fed-sx-types completes. +**Default: branch from fed-sx-types' tip.** If fed-sx-types is +mid-flight, wait for it to land Phase 4 before starting this +loop's Phase 2. + +## Branch base + +`origin/loops/fed-sx-types` if it exists and has Phase 4 landed. +Else `origin/loops/fed-sx-m2` (has all m2 substrate + send_after ++ http-listen + peer_actors pattern). Either base has flow-on-sx +available via `lib/flow/**`. + +## Phase 1 — `DefineTrigger` + `trigger_registry.erl` + +### `next/genesis/activity-types/define_trigger.sx` + +Verb declaration. The `:object` payload binds an activity-type +(possibly per-actor, possibly with a guard predicate) to a flow +name: + +```sx +(DefineTrigger + :name "Trigger" + :doc "Bind an activity-type to a flow. When a matching activity + is appended, fan out to flow:start with the activity as + input." + :schema (fn (act) + (and (string? (-> act :object :activity-type)) + (string? (-> act :object :flow-name)) + ;; guard is optional — a (fn (activity) ...) returning bool + (or (nil? (-> act :object :guard)) + (callable? (-> act :object :guard))))) + :semantics (fn (state act) + ;; Folds into the actor's :triggers projection — registry on + ;; restart hydrates from this. + (trigger-state-add state act))) +``` + +The `:guard` lets one type bind to multiple flows with +discriminators ("only fire `EmailDigest` for Articles with +`:category "newsletter"`"). Optional; default true. + +### `next/kernel/trigger_registry.erl` + +In-memory registry, hydrated from the actor's trigger projection +on start. Mirrors `peer_actors.erl` / `peer_types.erl` shape: + +```erlang +trigger_registry:start_link(_) -> Pid +trigger_registry:add(ActivityType, Spec) -> ok +trigger_registry:remove(Trigger Cid) -> ok +trigger_registry:lookup(ActivityType) -> [Spec] +trigger_registry:all_triggers() -> [{ActivityType, [Spec]}] +``` + +Where `Spec` is `{trigger_cid, flow_name, guard_fn|undefined, +actor_scope|any}`. Multiple triggers per activity-type are +supported and fire independently. + +### Tests — `next/tests/define_trigger.sh` + `next/tests/trigger_registry.sh` + +- Verb round-trips through term_codec +- Schema accepts well-formed, rejects missing fields +- Registry: add/lookup/remove, lookup with no matches → [], multi-bind + one activity-type → list of specs +- Restart hydration from a fold over Trigger activities + +Phase 1 acceptance: 12-14 tests. + +## Phase 2 — Pipeline fan-out stage + +In `next/kernel/pipeline.erl`, add a stage **after the kernel +append** (not before — fan-out fires only on successfully accepted +activities; rejected activities don't trigger flows): + +```erlang +%% Pipeline (revised): +%% 1. envelope shape +%% 2. signature against peer-AS +%% 3. activity-type :schema +%% 4. object-schema validation (fed-sx-types Phase 4) +%% 5. kernel append +%% 6. trigger fan-out (THIS PHASE) + +apply_triggers(Activity, ActorState, Cfg) -> + Type = activity_type_atom(Activity), + Specs = trigger_registry:lookup(Type), + Matching = [S || S <- Specs, guard_passes(S, Activity, ActorState)], + [flow_dispatch:start(Spec, Activity, ActorState, Cfg) + || Spec <- Matching], + ok. +``` + +`apply_triggers` is fire-and-forget from the pipeline's POV — +the kernel response doesn't wait for flows to complete. Each +spawned flow is its own erlang process; failures are isolated +to that one flow. + +### Idempotency + +Federation deduplication is a known weak spot: the same activity +can arrive twice via different peers. Fan-out must therefore +de-duplicate per `{activity-cid, trigger-cid}` pair before +calling flow_dispatch. The `:triggers_fired` actor-state field +holds `[{activity_cid, trigger_cid}]` of historical fires; before +calling flow_dispatch, check membership; on dispatch, add. + +### Tests — `next/tests/pipeline_triggers.sh` + +- Activity append → trigger lookup → flow_dispatch invoked +- Activity with no matching trigger → no dispatch +- Guard returns false → no dispatch +- Same activity appended twice → flow_dispatch invoked once +- Failed flow_dispatch (raises) does not block the kernel append +- Multiple triggers for same activity-type → each dispatched + +Phase 2 acceptance: 10-12 tests. + +## Phase 3 — `flow_dispatch.erl` + +The bridge from "an activity matched a trigger" to "a flow +started with that activity as input." Calls into `lib/flow/`'s +public API. + +```erlang +%% flow_dispatch:start/4 starts the flow named in Spec with +%% the activity bound as :activity in the flow's input env. The +%% started flow's id is returned (or {error, _} on failure). + +flow_dispatch:start(Spec, Activity, ActorState, Cfg) -> + FlowName = field(flow_name, Spec), + Input = [{activity, Activity}, + {actor, actor_state:actor_id(ActorState)}, + {trigger_cid, field(trigger_cid, Spec)}], + case flow:start(FlowName, Input) of + {ok, FlowId} -> {ok, FlowId}; + {error, Reason} -> log_flow_start_failure(Spec, Activity, Reason), + {error, Reason} + end. +``` + +The `Input` proplist becomes the flow's initial environment. +Flow steps can read the activity, the actor id, and the trigger +cid (for audit chain). + +### Audit chain + +Every flow started from an activity records its `{activity_cid, +trigger_cid, flow_id}` triple to the actor's projection. Replaying +the actor log + the trigger registry reconstructs which flows +fired for what reasons. Same content-addressing discipline as the +rest of fed-sx. + +### Tests — `next/tests/flow_dispatch.sh` + +- Successful flow start records audit triple +- Flow names that don't resolve in the flow registry → log + return + `{error, no_such_flow}`; don't crash +- Flow's first step runs synchronously before flow_dispatch returns + (proves the activity payload landed in the flow's env) +- Multi-step flow: trigger + first step + suspend on + `flow:wait_for_message`; resume on synthetic message → second + step completes +- Branch test: flow has if/else gate on activity field; both + branches exercised + +Phase 3 acceptance: 10-14 tests. + +## Phase 4 — End-to-end integration: blog-publish digest + +A motivating, multi-step business flow that ties everything +together. Lives at `next/genesis/flows/blog-publish-digest.sx` +as a flow definition consumable by `lib/flow/`. Demonstrates: + +- Multi-step (3+ named steps) +- Branch on activity field (article :category) +- Side effect (mock email sender via `:dispatch_fn` Cfg hook) +- Suspension on a timer (e.g. "send digest at 9am the next day") +- Completion records a `DigestSent` activity (closing the loop — + the flow's own output is itself an activity, which can in turn + trigger downstream flows) + +### `next/genesis/flows/blog-publish-digest.sx` + +```sx +(defflow blog-publish-digest + (step :validate + (let ((art (-> $input :activity :object))) + (if (= (-> art :type) "Article") + (continue :decide-batch art) + (done :skipped)))) + (step :decide-batch + (fn (art) + (case (-> art :category) + :newsletter (continue :batch-until-morning art) + :urgent (continue :send-now art) + :else (done :skipped)))) + (step :batch-until-morning + (fn (art) + (wait-until (next-morning-utc)) + (continue :send-now art))) + (step :send-now + (fn (art) + (let ((followers (fetch-followers (-> $input :actor)))) + (for-each follower followers + (dispatch-email follower art)) + (continue :emit-digest-sent art)))) + (step :emit-digest-sent + (fn (art) + (publish-activity + (Create :object {:type "DigestSent" + :for (-> art :id) + :follower-count (-> art :follower-count)})) + (done :ok)))) +``` + +The exact `defflow` syntax is what `lib/flow/` already provides — +mirror its existing test fixtures (`lib/flow/tests/programs/*.sx`) +for the canonical surface. + +### Integration test — `next/tests/triggers_e2e.sh` + +End-to-end: +1. Bootstrap an actor with a `Trigger` activity binding + `Article` → `blog-publish-digest`. +2. Mock `dispatch_email` and `fetch_followers` via Cfg hooks. +3. Publish a Create-Article activity with `:category :urgent`. +4. Assert: flow ran to completion within one append cycle, three + emails dispatched, `DigestSent` activity appended. +5. Publish a Create-Article with `:category :newsletter`. +6. Assert: flow suspended after `:decide-batch`; advance the + logical clock 16 hours; flow resumes; emails sent; + `DigestSent` appended. +7. Publish a Create-Article with `:category :draft`. +8. Assert: flow took the `:else` branch, no emails, no + `DigestSent`. + +Phase 4 acceptance: 8-10 e2e tests. + +## Out-of-scope deliberately + +- **In-process local subscriber path on delivery_worker.** Right + now `delivery_worker` dispatches over HTTP; an "internal sink" + abstraction would let activities fan out via the delivery_set + to a local consumer. That's a separate, smaller change in + `delivery_worker.erl` + `outbox.erl`. Keep it out of this loop + to keep the scope tight. +- **Per-flow auth / capability gates.** Triggers fire flows with + the activity's actor identity. Authorisation of what flows can + publish what activities is a separate concern handled by the + existing pipeline. +- **Flow versioning / hot-reload.** When a flow definition + changes, in-flight instances continue with the version they + started on. New instances pick up the new version. flow-on-sx + handles this; nothing to do here. +- **Cross-actor triggers.** A trigger lives on the actor it's + defined for. "When ANY actor publishes an Article, fire this + flow" is a fan-in pattern that needs a different design. + +## Tests discipline + +- `bash lib/erlang/conformance.sh` green before AND after every + commit (substrate not touched, should stay 771/771). +- `bash lib/flow/conformance.sh` green before AND after (the flow + engine is a dependency we use as-is — if our changes break it, + we did something wrong). +- Commits scoped to `next/**` and `plans/`. No edits to + `lib/flow/**`, `lib/host/**`, `lib/erlang/**`, `bin/sx_server.ml`, + or other `lib//**`. +- One commit per phase as outlined. +- Push to `origin/loops/fed-sx-triggers` after each commit. + +## Done when + +- Phases 1-4 land with their acceptance tests green. +- The blog-publish-digest e2e test demonstrates a multi-step + flow with a branch, a suspension, a side effect, and a + follow-up activity emit, all driven by one trigger arriving + in the pipeline. +- `plans/fed-sx-design.md` updated (small note in §13) describing + the trigger fan-out stage as a kernel-level convention. + +When this loop closes, the substrate supports declaring business +flows as data, binding them to activity-types as Define\* +activities, and seeing them fire automatically whenever the +matching activity lands — locally OR via federation. The whole +"event-driven CQRS" pattern that the user asked about works +end-to-end with `lib/flow/` doing the actual workflow lifting. + +## Parallel safety with other loops + +- `loops/host` — no overlap (next/ only). +- `loops/fed-sx-types` — pipeline.erl overlap. Branch base on + fed-sx-types' tip OR coordinate via a single combined + pipeline-stage insertion commit. +- `loops/erlang` — no overlap (lib/erlang/ untouched). +- `loops/flow` — no overlap (lib/flow/ consumed as a dependency, + never edited). diff --git a/plans/agent-briefings/fed-sx-types-loop.md b/plans/agent-briefings/fed-sx-types-loop.md new file mode 100644 index 00000000..94beb79a --- /dev/null +++ b/plans/agent-briefings/fed-sx-types-loop.md @@ -0,0 +1,218 @@ +; -*- mode: markdown -*- +# loops/fed-sx-types — fed-sx-side prep for host-type federation + +Scoped briefing for a focused iteration loop that lands the +substrate-side pieces needed for host's typed-post graph to flow +across fed-sx nodes. Companion to `plans/fed-sx-host-types.md` (full +design + context); this is the build sheet. + +``` +description: loops/fed-sx-types — federation of typed posts (fed-sx side) +subagent_type: general-purpose +run_in_background: true +isolation: worktree # worktree at /root/rose-ash-loops/fed-sx-types +``` + +## Scope + +**In scope** — `next/**` only. Four pieces, in dependency order: + +1. `DefineType` + `SubtypeOf` genesis activity-types +2. `peer_types.erl` receiver-side cache (mirror of `peer_actors.erl`) +3. `http_server.erl` route for serving type-docs over the wire +4. Object-schema validation stage in `pipeline.erl` + +**Out of scope** — `lib/host/**`. The host-side adapters +(`lib/host/fed_sx_outbox.sx`, `lib/host/fed_sx_inbox.sx`) are a +follow-up that needs `loops/host`'s metamodel to settle first. +Touching `lib/host/**` here would conflict with `loops/host`'s +in-flight slices. **Hard line: do not edit `lib/host/`.** + +**Substrate** — the fed-sx kernel modules (`nx_kernel.erl`, +`http_server.erl`, `pipeline.erl`, `outbox.erl`, etc.) are fine to +edit — they're `next/kernel/` scope, this loop owns them. + +## Branch base + +Start from `origin/architecture` (currently has m2 fully merged +locally as `ef3d2df4`; once that's pushed it'll be available on +origin too). Tip has: + +- All of fed-sx-m2 (multi-actor, signature pipeline, peer_actors, + delivery_worker with 8b-timer, two-instance smoke test, etc.) +- Native `http-listen` and `http-request` primitives +- `erlang:send_after` / `cancel_timer` / `monotonic_time` substrate +- The Erlang substrate `:pending-args` scheduler fix that makes + kernel-aware routes work over real HTTP + +If `origin/architecture` is still missing m2 at start time (pending +push), rebase against `origin/loops/fed-sx-m2` instead. + +## Phase 1 — `DefineType` + `SubtypeOf` genesis activity-types + +New files in `next/genesis/activity-types/`: + +- `define_type.sx` — the verb declaration. Schema accepts + `{name, fields, refinement-schema, instance-type}`. Semantics + registers the type into a kernel-side type registry. +- `subtype_of.sx` — relation verb. `{child-type-cid, parent-type-cid}`. + Schema validates both are previously-defined types. Semantics adds + the edge to the registry's hierarchy index. + +These are data-shaped (`DefineActivity` form), parsed by +`bootstrap.erl` at startup. No kernel code change at this phase — +the registry surface needed to consume them lands in Phase 2. + +Sketch: + +```sx +(DefineType + :name "Type" ;; the verb-name (will be the activity :type) + :doc "..." + :schema (fn (act) + (and (string? (-> act :object :name)) + (or (nil? (-> act :object :fields)) + (and (list? (-> act :object :fields)) + (every? type-field-shape? (-> act :object :fields)))))) + :semantics (fn (state act) state)) +``` + +Tests in `next/tests/define_type.sh` and `next/tests/subtype_of.sh` +— round-trip through `term_codec`, schema accept/reject cases. + +Phase 1 acceptance: 6 tests (3 per verb) + envelope shape round-trip. + +## Phase 2 — `peer_types.erl` receiver-side cache + +Mirror `next/kernel/peer_actors.erl`. State shape: +`[{TypeCidBytes, TypeRecord}]` where `TypeRecord` is the parsed +`DefineType` envelope's `:object` payload — name, fields, +refinement-schema, instance-type-binding. + +Public API (mirrors `peer_actors`): + +```erlang +peer_types:start_link(_) -> Pid +peer_types:put(TypeCid, TypeRecord) -> ok +peer_types:lookup(TypeCid) -> {ok, TR} | not_found +peer_types:lookup_or_fetch(TypeCid, Cfg) -> {ok, TR} | {error, _} +peer_types:state_for(TypeCid) -> {ok, TR} | not_found +peer_types:known_types() -> [TypeCid] +``` + +The `lookup_or_fetch` path delegates to a `Cfg`-supplied +`type_fetch_fn :: fun((TypeCid, Cfg) -> {ok, Bytes} | {error, _})` +to keep transport pluggable (same pattern as +`discovery_fetch:make_fetch_fn`). + +Tests in `next/tests/peer_types.sh` — put/lookup, miss → fetch via +mocked Cfg.fn, cache populated after fetch, lookup_or_fetch with +no Cfg.fn → `{error, no_fetch_fn}`. + +Phase 2 acceptance: 8-10 tests covering the cache surface. + +## Phase 3 — `http_server.erl` type-doc route + +Add a route mirroring `actor_doc_response_for/3`: + +``` +GET /types/ with Accept: application/vnd.fed-sx.type-doc +``` + +Encodes the cached TypeRecord via `term_codec:encode/1` and serves +the bytes. 404 if not in cache. Wire format matches the actor-doc +shape (per `plans/fed-sx-design.md` §10). + +Also add a small `discovery_type_fetch.erl` (sibling to +`discovery_fetch.erl`) holding the closure that `peer_types`'s +`lookup_or_fetch` calls — keeps the live HTTP path testable +end-to-end. + +Tests: +- `next/tests/peer_types_route.sh` (server-side, port + actor-style + curl) — known type-cid → 200 + decoded body, unknown → 404 +- Extend `next/tests/discovery_type_fetch.sh` to cover the + client-side closure round-trip with a python http.server stub + +Phase 3 acceptance: 8 server-side tests + 6 closure tests. + +## Phase 4 — Object-schema validation stage in `pipeline.erl` + +The pipeline currently validates: envelope shape, signature against +peer-AS, activity-type schema (the activity's outer `:schema` fn). +**It does not validate the inner object against its declared type +schema.** This phase plugs that in. + +When an activity arrives whose `{type, _}` is a known refinement- +typed activity AND whose `:object` carries a `{type, TypeName}` +field, look up the TypeRecord (local Define-name index → TypeCid → +`peer_types:lookup_or_fetch`) and apply its refinement-schema to +the object's `:field-values`. Reject the activity on schema-fail. + +New `apply_object_schema/2` between activity-type validation and +the kernel append. Default behaviour when the type isn't in the +local Define-name index: try `peer_types:lookup_or_fetch`; if that +errors and Cfg has `{strict_object_schema, true}` reject the +activity, otherwise let it through with a `validation_skipped` +log entry. Default `strict_object_schema = false` — opt-in for +nodes that want airtight validation. + +Tests in `next/tests/object_schema.sh`: +- Activity with type that matches local registry + valid object + → accepted, appended +- Same activity with refinement-failing object → rejected with + `{validation_failed, object_schema}` +- Activity with type unknown to local registry, peer_types fetch + succeeds → validates against fetched TypeRecord +- Activity with type unknown, fetch fails, strict_object_schema + not set → accepted with skip log +- Activity with type unknown, fetch fails, strict set → rejected +- Activity without inner `{type, _}` → skipped (no schema applies) + +Phase 4 acceptance: 10-12 tests covering the matrix. + +## Out-of-scope deliberately + +- **host-side adapters** (`lib/host/fed_sx_outbox.sx`, + `lib/host/fed_sx_inbox.sx`). Belongs to whatever loop wires + `loops/host` to fed-sx. That loop can read this branch and + consume `peer_types`/`DefineType` as public surface. +- **Type evolution / version migration**. Refinement schemas are + immutable per CID; an updated TypeRecord is a new CID. The + "name → currently-valid CID" routing layer is a separate + problem. +- **Subtype-of unification across nodes**. The graph data lands + via `SubtypeOf` activities; rendering / display / dedup is a + consumer concern. + +## Tests discipline + +- `bash lib/erlang/conformance.sh` green before AND after every + commit (currently 771/771 on `origin/architecture` after the m2 + merge). +- Commits scoped to `next/**` and `plans/fed-sx-host-types.md` + (the design plan; update it as you ratify decisions). No edits + to `lib/erlang/**`, `lib/host/**`, `bin/sx_server.ml`, or other + `lib//**`. +- One commit per phase as outlined; smaller intermediate commits + fine as long as each leaves both gates green. +- Push to `origin/loops/fed-sx-types` after each commit. + +## Done when + +- Phases 1-4 land with their acceptance tests green. +- `plans/fed-sx-host-types.md` updated to mark steps 1-4 done and + describe what the host-side adapter loop now has to work with. +- Conformance + every `next/tests/*.sh` green at finish. + +When this loop closes, the substrate side of host-type federation +is complete. The remaining `loops/host`-side work (outbox/inbox +adapters) becomes a small, well-scoped follow-up. + +## Parallel-safety with loops/host + +`loops/host` is iterating heavily on `lib/host/blog.sx` and the +metamodel slices. This loop **does not touch any `lib/host/` file** +— scope is `next/**` only. No conflict surface. Both loops run +their own worktrees on separate branches; they meet only at the +eventual architecture merge.