From 689d4bd363b020af0f76350ee13468400eef2d4a Mon Sep 17 00:00:00 2001 From: giles Date: Thu, 2 Jul 2026 14:00:02 +0000 Subject: [PATCH] =?UTF-8?q?plan:=20whole-plan=20coherence=20review=20?= =?UTF-8?q?=E2=80=94=20align=20the=20middle=20with=20the=20artdag+seam=20r?= =?UTF-8?q?eframe?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reframe updated the vision but left the P0 section stale + contradictory: P0.2/P0.3 still described the Erlang-bridge-first path the reframe deferred; P0.1's activity ({:type :object-dict}) didn't match the seam's canonical activity ({:verb :object-cid}); the seam-contract section predated the 2 enrichment passes (no status/dedup/pump). Coherence fixes: - P0 rewritten around the seam + SX op-table runner (all-SX publish-DAG, local-SX registry, in-process transport, host driver) — no Erlang/fed-sx. - Erlang/fed-sx DEMOTED to explicit adapter phases: RA (durable Erlang runner wrapping next/ flow_dispatch) + TA (fed-sx transport wrapping next/ delivery). P3-federation folded into TA. - canonical seam activity shape defined; P0.4 reconciles P0.1's next/-shaped activity + a marshaller. - seam contract refreshed to behavior.sx (result {:status :effects :resume :error}, dedup per-invocation, pump/async-completion, behavior 10/10); stray fragments + 9/9→10/10 cleared. Doc-only. Co-Authored-By: Claude Opus 4.8 --- plans/business-logic-fed-flows.md | 192 +++++++++++++++--------------- 1 file changed, 96 insertions(+), 96 deletions(-) diff --git a/plans/business-logic-fed-flows.md b/plans/business-logic-fed-flows.md index e15e6a96..33a2a47f 100644 --- a/plans/business-logic-fed-flows.md +++ b/plans/business-logic-fed-flows.md @@ -34,43 +34,32 @@ render-vs-execute-vs-deps, applied to execution/communication/deployment: So: the TYPE carries content-grammar + allowed-relations + a **behavior DAG (+ triggers)**; the object's state changes emit activities; the platform picks runner/transport/placement per context. -**Prior narrower framing (kept below as the concrete first slice):** wire the live host's publish -onto next/'s Erlang trigger→flow machinery. That's now understood as *one adapter choice* — a good -concrete spike, but P0 should keep the DAG + the state-change event substrate-CLEAN so runners and -transports swap trivially. - **Design (decided 2026-07-02; corrected after review):** -- **Activity log = every OBSERVABLE object-level state change** — the federated event source. - NOT just CID deltas: verified that relations write `edge:*` rows, NOT the record, so a relation - change does NOT shift the CID. So the log has TWO event classes (ActivityPub-faithful): - content/status change → a CID-carrying `Create`/`Update` (the record's canon incl. :status → the - CID); relation change (relate/unrelate/tag) → an `Add`/`Remove` activity referencing the edge. - (CID delta is one class, not the whole log — this is the fix to the original "every CID delta".) -- **Verbs are TRANSITIONS, not raw deltas.** `on-publish` = the draft→published transition - (fire-once), not every CID delta of a published post. The emitter picks the verb: `Create` on - first publish, `Update` on subsequent content edits, `Add`/`Remove` on relations, `Delete` on - unpublish/delete. Triggers are scoped to the transition, so re-editing doesn't re-fire on-publish. -- **Triggers = declared subscriptions** — a type declares named triggers (on-publish, on-relate, - …); flows fire only on matching ones. (fed-sx `DefineTrigger`: activity-type → flow-name + guard - + actor-scope.) Log complete, execution precise. -- **Flows = hybrid, split by DURABILITY not "complexity":** SYNCHRONOUS declarative logic authored - as **SX composition** (the execute-fold: effect/alt/each — eager, one-pass, NO suspend; live in - /workflow-demo). Anything needing a timer / suspend-resume / human-in-the-loop is a named - **Erlang flow** (next/flow/*.erl — flow_spec:sequence/branch/wait, effect-as-data, deterministic - replay). The execute-fold canNOT express `wait`; that's the escape-hatch boundary. -- **Effects are DATA; a DRIVER dispatches them.** Flows return effect descriptions (digest_sent, - a DigestSent activity) — they perform no IO (a blocking call deadlocks the er-scheduler). For P0 - the HOST is the driver (dispatch the effect → a durable record / append the follow-up activity / - show it). The driver closing the loop back to object state is P4. -- **Federated execution = Erlang** (`next/` fed-sx Milestone-1 kernel: trigger_registry + - flow_dispatch + pipeline post-append fan-out). Authoring stays SX; the fed-sx activity is the - bridge. flow-on-sx (Scheme, lib/flow) remains for purely-local durable logic. -- **The type carries its whole contract:** fields+grammar (content) · allowed relations - (external) · triggers+flows (behavior). All composition, all editable in the type-def editor. +- **Activity log = every OBSERVABLE object-level state change** — the event source. NOT just CID + deltas: relations write `edge:*` rows, NOT the record, so a relation change does NOT shift the + CID. Two ActivityPub-faithful event classes: content/status change → a CID-carrying + `Create`/`Update`; relation change → an `Add`/`Remove` referencing the edge. +- **Verbs are TRANSITIONS, not raw deltas.** `on-publish` = draft→published (fire-once), not every + CID delta of a published post. Emitter picks the verb (Create/Update/Add/Remove/Delete). Global + fire-once is the emitter's job (+ a durable inbox for federation); the seam only dedups in-call. +- **Triggers = declared subscriptions** — a type declares named triggers; flows fire only on + matching ones. Log complete, execution precise. +- **Flows split by DURABILITY not "complexity":** SYNCHRONOUS logic = an SX artdag DAG run by the + op-table runner (eager, one-pass, no suspend). Anything needing timer/suspend/human-gate = a + DURABLE runner (Erlang flow_dispatch, celery-sx). Same DAG; the RUNNER supplies durability. +- **Effects are DATA; a DRIVER dispatches them** (perform no IO — a blocking call deadlocks the + er-scheduler). The host is the driver for P0. +- **The type carries its whole contract:** fields+grammar (content) · allowed relations (external) + · **behavior bindings** (triggers → DAG). All composition, all editable in the type-def editor. +- **CANONICAL ACTIVITY = the seam shape** `{:verb :actor :object :object-type :delta :prev + :ts :id}`; each runner adapter MARSHALS it to its substrate (e.g. the Erlang runner → next/'s + `[{type,…},{object,[…]}]` proplist). NOTE: P0.1's host/blog--publish-activity currently emits the + next/-Erlang shape ({:type :object-dict}); P0.4 reconciles it to the canonical seam shape + a + marshaller. -**Verified baseline:** `next/tests/triggers_e2e.sh` = 10/10 — publish activity → trigger → -blog_publish_digest flow (urgent/newsletter-suspend/draft-skip/guard-reject/dedup). This is the -reference P0 wires the live host onto. +**Reference (one durable-runner substrate, verified):** `next/tests/triggers_e2e.sh` = 10/10 — +next/'s trigger_registry + flow_dispatch + blog_publish_digest (suspend/resume/guard/dedup). This +is what the ERLANG RUNNER ADAPTER wraps (Phase RA), not what P0 wires directly. ## The ADAPTER SEAM (design first — the contract every substrate plugs into) The invariant is an **activity** (state-change event) + a **behavior DAG**. Everything between is @@ -82,79 +71,81 @@ a swappable adapter — each a dict-of-functions (SX-native, like the fold domai }`. The type carries content-grammar + allowed-relations + these bindings. 3. **Trigger-registry adapter** — `{:register! (fn spec dag hint) :match (fn activity -> [binding])}`. Impls: a local SX matcher / next/'s Erlang trigger_registry. -4. **Runner adapter** — `{:run (fn dag env -> {:status "done"|"suspended"|"failed" :results :effects - :resume})}`; env = `{:activity :actor :ctx :effects}`. Impls: sync op-table (artdag/run) / Erlang - durable (flow_dispatch, may suspend) / celery-sx (queue+workers). Durability = the runner's, not - the DAG's. -5. **Transport adapter** — `{:emit (fn activity) :deliver (fn -> [activity])}`. Impls: in-process / - fed-sx (next/) / internal HTTP / IPFS. Content-ids are global → results move by id. +4. **Runner adapter** — `{:run (fn dag env -> {:status "done"|"suspended"|"failed" :effects :resume + :error})}`; env = `{:activity :actor :ctx :effects :binding}` (:effects = injected external-read + interfaces, deterministic for replay). `:results` is runner-INTERNAL (artdag's memoized outputs). + Durability = the runner's, not the DAG's. A durable runner that SUSPENDS is wired at construction + to the transport's INBOUND channel and injects its out-of-band completion there (pump drains it). +5. **Transport adapter** — `{:emit (fn activity) :deliver (fn -> [activity])}`. `:emit` = outbound + (log/publish), `:deliver` = inbound (peers + async runner completions). Impls: in-process / + fed-sx (next/) / HTTP / IPFS. Content-ids global → results move by id. 6. **Effect driver** — `{:dispatch (fn effect -> [activity])}`. Perform the effect-as-data; may emit NEW activities (loop closure). -**Engine** (orchestrator): `behavior/make-engine {:triggers :runner :transport :driver}` → -`behavior/process(engine, activity)`: +**Engine** (`behavior/make-engine {:triggers :runner :transport :driver :effects? :ctx-of?}`) → +`behavior/process(engine, activity)` / `behavior/pump(engine)`: ``` -emit(transport, activity) -for b in match(triggers, activity): - r = run(runner, b.dag, {:activity activity …}) - for eff in r.effects: - for a in dispatch(driver, eff): process(engine, a) ; loop closes +step(activity): if activity.id already :seen → skip (dedup) ; in-call cycle guard + emit(transport, activity) ; log + for b in match(triggers, activity): + r = run(runner, b.dag, {:activity :actor :ctx :effects :binding}) + case r.status: + "done" → for eff in r.effects: for a in dispatch(driver, eff): step(a) ; loop closes + "suspended" → record (+ r.resume); a durable runner completes out-of-band → inbound → pump + "failed" → record (+ r.error) for flow-level retry / dead-letter +pump(): for a in transport.deliver(): step(a) ; inbound drain, shared :seen ``` -Every stage injected → swap runner (sync→Erlang→celery-sx), transport (in-proc→fed-sx), trigger -registry, driver — DAG + engine unchanged. +Every stage injected → swap runner (sync→Erlang→celery-sx), transport (in-proc→fed-sx), registry, +driver — DAG + engine unchanged. **Global** idempotency = emitter fire-once + a durable inbox. -**Existing pieces implement the contracts:** activity ← host/blog--publish-activity (P0.1 ✓) · -trigger-registry ← next/ trigger_registry OR a local SX matcher · runner ← artdag/run+op-table -(sync) / next/ flow_dispatch (durable) / celery-sx · transport ← artdag/federation (transport -injected) / next/ delivery / in-process · driver ← host writes / email / append-activity. +**Build order:** (a) **DONE** — seam as SX contracts + reference engine, tested substrate-agnostic: +:status branch (done/suspended/failed), injected env (:ctx + :effects), dedup by activity :id, +behavior/pump for inbound, async-completion loop (suspend → out-of-band inject → pump). **behavior +10/10.** (b) P0 supplies the REAL SYNCHRONOUS adapters; durable/federated runners+transports are +later adapter phases (RA/TA below). -**Build order:** (a) DONE — the seam as SX contracts + reference engine, tested. Substrate-agnostic after review: :status branch (done/suspended/failed), injected env (:ctx + :effects), dedup by activity :id, behavior/pump for inbound (peer + async completions). behavior 9/9. -(process a mock activity → effect flows → loop closes). (b) P0 then supplies the REAL adapters -(publish activity, local-SX trigger, sync op-table runner over a publish-DAG, host driver). +## P0 — publish workflow on the seam (SYNCHRONOUS, all-SX) +Prove: live host publish → the seam engine (in-process transport · local-SX trigger registry · +sync op-table runner over an SX publish-DAG · host driver) → the effect surfaces. NO Erlang, NO +fed-sx yet — those are adapter phases (RA/TA). Every piece swaps later; the DAG + engine don't. -## P0 — publish workflow, end-to-end (spike) — on the seam -Prove: live host publishes a post → fed-sx activity → on-publish trigger → blog_publish_digest. +- [x] **P0.1 — publish-activity contract (SX side).** host/blog--publish-activity + post-category. + blog 200/200. NOTE: emits the next/-Erlang shape today; P0.4 reconciles to the canonical seam shape. +- [ ] **P0.2 — the publish-DAG + op-table runner.** Author the publish workflow as an SX artdag DAG + (validate → publish → notify/digest), + a runner (artdag/op-table-runner over an op-table of the + verbs, OR a thin op-table). Test: run the DAG with a publish env → the expected effect-as-data. + Synchronous — no wait/suspend (that's the durable runner, RA). +- [ ] **P0.3 — wire the seam on the live host.** A local-SX trigger registry (on-publish → + publish-DAG), an in-process transport (a KV-backed log), the host as effect driver. In edit-submit + detect the draft→published TRANSITION (prev≠published & new=published — fire-once), build the + activity, behavior/process it (in the handler BODY, not a render). ACCEPTANCE: publish a post on + the LIVE host → the effect surfaces (a /flows page + a durable record). +- [ ] **P0.4 — canonical activity + reconcile.** Move host/blog--publish-activity to the seam shape + {:verb :actor :object :object-type :delta :ts :id}; keep the category/slug fields the DAG reads. -- [x] **P0.1 — the publish-activity contract (SX side).** host/blog--publish-activity(slug): - a post record → the fed-sx activity {:type "create" :actor "site" :id :object {:type - "article" :category … :slug …}}. category from field-value "category", else first tag, else - "urgent". + host/blog--post-category. blog 200/200 (3 tests). DONE 2026-07-02. -- [ ] **P0.2 — the dispatch bridge.** Detect the draft→published TRANSITION (edit-submit /a publish - action where prev status ≠ "published" and new = "published") — fire-once, not on every edit — - and emit host/blog--publish-activity into the trigger machinery. RUNTIME CHOICE for P0: in-process - — serve.sh loads next/ kernel + registers the on-publish trigger; host emits via the erlang-on-sx - bridge (erlang-eval-ast pipeline:apply_triggers). RISK: run this in the handler BODY (never a - render/quasiquote — VmSuspended), and the flow must stay effect-as-data (no blocking, or the - er-scheduler deadlocks). SPIKE the SX→erlang-on-sx call in isolation first. (P3 swaps this for - real fed-sx delivery over next/kernel/http_server.erl.) -- [ ] **P0.3 — the effect is dispatched (host = driver).** pipeline:apply_triggers returns the - flow's effect-as-data ({digest_sent, Emails, DigestObject}); the HOST driver dispatches it — - P0: append a DigestSent record / activity + show it on a /flows page (or on the post). ACCEPTANCE: - publish a post on the LIVE host → the DigestSent surfaces, driven by the real flow. (Marshalling: - the SX activity dict ↔ the erlang proplist term is the fiddly part — factor host/blog--activity->erl.) +## P1 — types DECLARE behavior (generalize) +- [ ] The type carries :behavior = [{:on {:verb :object-type :guard} :dag :runner }] — + edited in the type-def editor beside grammar + relations. +- [ ] host/blog--engine-for(object) builds the seam engine from the object's type bindings + + registers the triggers. Simple DAGs authored as SX composition; complex ones name a durable runner. -## P1 — types declare behavior (generalize) -- [ ] A Composition field / the type carries a :triggers list (on-publish → flow-name + guard) — - edited in the type-def editor, like grammar + relations. -- [ ] A fold turns a type's declared behavior into DefineTrigger + flow registrations at boot. -- [ ] SYNCHRONOUS flows authored as SX composition (execute-fold: effect/alt/each — one-pass, no - suspend) → dispatched to the engine; anything needing wait/suspend/human-gate references a named - Erlang flow. (Optional bigger: extend the SX flow vocabulary with a `wait` that compiles to - flow_spec — only if authoring durable flows in SX proves worth it.) +## P2 — state-change → activity emission (ALL events, not just publish) +- [ ] Wire the host write path: put!/set-comp!/edit-submit → Create/Update; relate!/unrelate!/tag → + Add/Remove. Emit canonical activities into the transport log. Define the delta summary. -## P2 — state-change → activity emission (the CID-delta event source) -- [ ] Every content mutation (new CID) appends a state-change activity to the log. Define the - activity envelope (verb, actor, object CID, prev CID, delta summary). -- [ ] Wire the host's write path (put!/set-comp!/edit-submit) to the append. +## RA — the ERLANG (durable) RUNNER adapter ← the old "fed-sx spike", now an adapter +- [ ] A seam runner wrapping next/'s flow_dispatch + flow_store: marshal the canonical activity → + next/'s proplist, dispatch to a named flow (blog_publish_digest), map the result back to + {:status done|suspended :effects :resume}. On suspend, wire the resumed completion → the transport + inbound (pump). Baseline: next/tests/triggers_e2e.sh 10/10. RISK: er-scheduler context + (project_fed_prims_http_listen_scheduler); keep flows effect-as-data (no blocking). SPIKE the + SX→erlang-on-sx call in isolation first. -## P3 — federation proper -- [ ] Activities cross peers via next/ delivery (http_server / outbox / follower_graph). A remote - service's trigger_registry fires the flow. Everything works over fed-sx. - RISK: next/ delivery had M2 blockers (http-listen handler runs off the er-scheduler context → - gen_server:call can't complete; project_fed_prims_http_listen_scheduler). Confirm delivery is - green before depending on it. Also: the ACTOR MODEL (:actor "site" is a P0 placeholder) is - foundational here — peer_actors / follower_graph / per-author identity underpin who federates to - whom. Deferred through P0–P2, but P3 needs it real. +## TA — the FED-SX TRANSPORT adapter ← federation proper +- [ ] A seam transport over next/ delivery: :emit → outbox → peers; :deliver → inbox. A remote + peer's engine fires its own triggers. Everything works over fed-sx. RISK: next/ delivery M2 + blockers (er-scheduler context). Needs the ACTOR MODEL real (:actor was a P0 placeholder — + peer_actors / follower_graph / per-author identity). ## RX — celery-sx runner (DEMAND-DRIVEN, not scheduled) Build the distributed/durable runner adapter the moment a real DAG needs heavy compute / @@ -170,6 +161,15 @@ covers everything until a DAG's cost/latency/placement forces the substrate. activities), so business logic can change state, which federates, which triggers more flows. ## Progress log (newest first) +- 2026-07-02 — whole-plan coherence review. The reframe (artdag+seam) had left the middle stale: + P0.2/P0.3 still described the Erlang-bridge-first path; P0.1's activity didn't match the seam + contract; the seam section predated the enrichment. Fixed: P0 rewritten around the seam + SX + op-table runner (all-SX, no Erlang/fed-sx); the Erlang/fed-sx path DEMOTED to explicit adapter + phases RA (durable runner) + TA (fed-sx transport); canonical activity shape + P0.4 reconcile; + seam contract refreshed to behavior.sx (status/dedup/pump/async, behavior 10/10); stray bits cleared. +- 2026-07-02 — seam DONE + reviewed twice. lib/host/behavior.sx (engine + 4 adapters); enriched + substrate-agnostic (status/env/dedup/pump); 2nd review corrected the async-completion contract + (construction-wired inbound + pump, not env :emit) + proved it. behavior 10/10, conformance 580. - 2026-07-02 — P0.1 done. host/blog--publish-activity + host/blog--post-category; the publish contract in SX, 200/200. Verified next/ triggers e2e baseline 10/10. Roadmap anchored. NEXT: P0.2 the dispatch bridge (in-process: serve.sh loads next/ kernel + registers the on-publish