From f240c46fa88ca54d8bbfbd97565b45769615d2eb Mon Sep 17 00:00:00 2001 From: giles Date: Thu, 2 Jul 2026 13:35:44 +0000 Subject: [PATCH] plan: account for celery-sx as the distributed/durable runner adapter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit celery-sx = one more runner on artdag/op-table-runner, not a Celery port: broker=persist KV, workers=er-scheduler, result backend=content-addressed (dedup free), retries/replay=flow-on- erlang, fan-out=artdag/schedule. ~few hundred lines of glue, zero packages, 'Celery the way it should have been' on erlang-on-sx. DEMAND-DRIVEN (RX) — build when a DAG needs heavy compute / long-running-retryable / cross-machine fan-out; the synchronous op-table runner covers P0. Co-Authored-By: Claude Opus 4.8 --- plans/business-logic-fed-flows.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/plans/business-logic-fed-flows.md b/plans/business-logic-fed-flows.md index 4a019775..7e996904 100644 --- a/plans/business-logic-fed-flows.md +++ b/plans/business-logic-fed-flows.md @@ -8,9 +8,21 @@ render-vs-execute-vs-deps, applied to execution/communication/deployment: - **Behavior = an artdag DAG** — the invariant, content-addressed (`artdag/dag`, analyze/plan/ optimize/schedule). Business logic, art media pipelines, workflows — all the same abstraction. - **Execution = an injected RUNNER** (`artdag/run dag RUNNER cache`; `artdag/op-table-runner`). - Substrates are just runners: SX op-table (synchronous/local), Erlang (durable — suspend/resume/ - wait), Celery/JAX (heavy compute, artdag/l1), … **Durability is a runner capability, not a DAG - feature** — the same DAG runs eager or durable depending on the runner. + Substrates are just runners — a ladder by capability, same DAG throughout (**durability is a + runner capability, not a DAG feature**): + - **op-table / execute-fold runner** — synchronous, local, in-request. Covers P0. + - **Erlang runner** — durable: suspend/resume/`wait`, deterministic replay (flow-on-erlang). + - **celery-sx runner** — distributed/durable task executor, "Celery the way it should have + been" on erlang-on-sx, ZERO packages. It's a LEAN GLUE of parts we already have, not a + reimplementation: broker = lib/persist KV (durable enqueue/claim/ack/visibility-timeout) · + worker pool = the er-scheduler / Erlang processes · result backend = content-addressed results + (artdag keys by content-id → dedup/memoization FREE — Celery bolts this on badly) · retries/ + replay = flow-on-erlang · scheduling/fan-out/chords = artdag/schedule (minikanren CLP(FD)) + + the DAG's topo batches · the plug point = artdag/op-table-runner. The genuinely-new code is + small (~a few hundred lines): a durable queue + a worker loop (pull node → runner → write + content-addressed result) + retry/backoff. **BUILD WHEN A DAG DEMANDS IT** — heavy compute, + long-running/retryable tasks, or fan-out across machines — NOT for the synchronous P0. + - **real-Celery over artdag/L1** — the existing Python media pipeline (JAX/IPFS) as a runner. - **Communication = an injected TRANSPORT** (`artdag/federation`, transport injected). Substrates: fed-sx (ActivityPub/next/), internal HMAC HTTP (services), IPFS (content-addressed). Because content-ids are global, a result computed on one instance is reusable on another by id. @@ -104,6 +116,15 @@ Prove: live host publishes a post → fed-sx activity → on-publish trigger → foundational here — peer_actors / follower_graph / per-author identity underpin who federates to whom. Deferred through P0–P2, but P3 needs it real. +## RX — celery-sx runner (DEMAND-DRIVEN, not scheduled) +Build the distributed/durable runner adapter the moment a real DAG needs heavy compute / +long-running-retryable tasks / cross-machine fan-out (the artdag/JAX media case, or federated +flows that can't run in-request). New code is small — glue persist (durable queue: enqueue/claim/ +ack/visibility-timeout) + er-scheduler (worker loop: pull node → op-table-runner → content- +addressed result) + artdag/schedule (fan-out) + retry/backoff. Slots in at artdag/op-table-runner +alongside the synchronous + Erlang runners. Zero packages. Do NOT pre-build; the op-table runner +covers everything until a DAG's cost/latency/placement forces the substrate. + ## P4 — close the loop - [ ] Flow effects mutate objects back durably (a flow's DescribeEffects → host writes / new activities), so business logic can change state, which federates, which triggers more flows.