giles
c3a0727645
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 19s
Plans for acl-on-sx (Datalog), flow-on-sx (Scheme), feed-on-sx (APL), mod-on-sx (Prolog), search-on-sx (Haskell). Each is a 4-phase queue sitting on its respective guest language, targeting rose-ash needs: access control, durable workflows, activity feeds, moderation, search. Federation extension in Phase 4 of each (plugs into fed-sx). Briefings for the three loops we're kicking off now: acl-loop, flow-loop, feed-loop. mod-sx and search-sx briefings will follow once the first three have surfaced any shared infrastructure worth extracting to lib/guest/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
109 lines
4.9 KiB
Markdown
109 lines
4.9 KiB
Markdown
# flow-on-sx: Durable DAG Workflows on Scheme
|
|
|
|
rose-ash needs workflows that survive restarts: content pipelines (write → review →
|
|
publish → federate), scheduled jobs (digest emails), multi-step user flows (signup,
|
|
confirm, onboard). art-dag is the precedent — DAG-of-tasks with pause/resume at IO
|
|
boundaries.
|
|
|
|
Scheme's `call/cc` + delimited continuations make pause/resume natural: a `suspend`
|
|
captures the continuation, serializes it as part of the flow record, and `resume`
|
|
re-enters at exactly that point. No state-machine bookkeeping by hand. R7RS-small is
|
|
already at 2644/2644 (see kernel/architecture status).
|
|
|
|
End-state: a Scheme-on-SX layer over the existing scheme runtime, with combinators
|
|
for sequence/parallel/branch/retry/timeout/suspend, persistent flow store, and a
|
|
federation extension via fed-sx for remote-node execution.
|
|
|
|
## Status (rolling)
|
|
|
|
`bash lib/flow/conformance.sh` → **0/0** (not yet started)
|
|
|
|
## Ground rules
|
|
|
|
- **Scope:** only touch `lib/flow/**` and `plans/flow-on-sx.md`. Do **not** edit
|
|
`spec/`, `hosts/`, `shared/`, `lib/scheme/**`, or other `lib/<lang>/`. You may
|
|
**import** from `lib/scheme/` (public API via `lib/scheme/scheme.sx`); do **not**
|
|
modify Scheme.
|
|
- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here.
|
|
- **SX files:** use `sx-tree` MCP tools only.
|
|
- **Architecture:** flow combinators are Scheme macros + procedures. Runtime is a
|
|
driver loop that walks the flow graph and invokes `call/cc` at `suspend` points.
|
|
Persistence layer serializes the continuation + open file/socket placeholders are
|
|
forbidden (continuations must be resumable across process restart).
|
|
- **art-dag awareness:** read `plans/art-dag*` if it exists for design lineage; do not
|
|
import code.
|
|
- **Commits:** one feature per commit. Keep Progress log updated and tick boxes.
|
|
|
|
## Architecture sketch
|
|
|
|
```
|
|
(defflow publish
|
|
(sequence
|
|
(write-content)
|
|
(parallel
|
|
(review)
|
|
(spell-check))
|
|
(cond approved?
|
|
(sequence (publish) (federate))
|
|
(notify-author))))
|
|
│
|
|
▼
|
|
lib/flow/spec.sx lib/flow/runtime.sx lib/flow/store.sx
|
|
— defflow — driver loop — append-only flow log
|
|
— sequence/parallel — node dispatch — checkpoint serialize
|
|
— cond/retry/timeout — call/cc at suspend — restart loader
|
|
— suspend/resume │ │
|
|
▼ ▼
|
|
lib/flow/api.sx lib/flow/remote.sx
|
|
— (flow/start name args) — fed-sx adapter
|
|
— (flow/resume id value) — node-on-peer execution
|
|
— (flow/cancel id) — failure handling
|
|
```
|
|
|
|
## Phase 1 — Declarative DAG + sequential execution
|
|
|
|
- [ ] `lib/flow/spec.sx` — `defflow` macro, `sequence` combinator
|
|
- [ ] node = Scheme thunk; output threads to next node (data flow)
|
|
- [ ] `parallel` combinator (sequential semantics for now — TRUE parallelism in Phase 3)
|
|
- [ ] runtime executes a flow synchronously, returns final value
|
|
- [ ] `lib/flow/api.sx` — `(flow/start name args)` entry point
|
|
- [ ] `lib/flow/tests/basic.sx` — 15+ cases: linear sequence, nested sequences,
|
|
data flow between nodes, parallel-with-join
|
|
- [ ] `lib/flow/scoreboard.{json,md}`
|
|
- [ ] `lib/flow/conformance.sh`
|
|
|
|
## Phase 2 — Control flow + error handling
|
|
|
|
- [ ] `cond` combinator — predicate selects branch
|
|
- [ ] `retry n [backoff]` — re-runs node up to n times on exception
|
|
- [ ] `timeout ms` — bounds node execution
|
|
- [ ] `try-catch` — exception handler with reified error
|
|
- [ ] error model — exceptions vs explicit `(fail :reason ...)` results
|
|
- [ ] `lib/flow/tests/control.sx` — 25+ cases: each combinator + composition
|
|
|
|
## Phase 3 — Suspend / resume (the showcase)
|
|
|
|
- [ ] `(suspend reason)` — `call/cc` captures continuation, returns flow-id to caller
|
|
- [ ] `lib/flow/store.sx` — serialize flow state (continuation + open vars)
|
|
- [ ] `(flow/resume id value)` — load continuation, inject value, re-enter
|
|
- [ ] `(flow/cancel id)` — explicit termination
|
|
- [ ] crash recovery — on restart, scan store for paused flows, mark resumable
|
|
- [ ] `lib/flow/tests/suspend.sx` — pause-resume scenarios, cancellation, "restart"
|
|
scenarios (simulated by re-loading store)
|
|
|
|
## Phase 4 — Distributed nodes via fed-sx
|
|
|
|
- [ ] `(remote-node addr fn args)` — execute node on a federation peer
|
|
- [ ] failure semantics — retry on different peer, fall through to local
|
|
- [ ] persistence across instances — flow state replicates via fed-sx
|
|
- [ ] handoff — flow started here can resume on a peer if the local instance is down
|
|
- [ ] `lib/flow/tests/distributed.sx` — federated flow scenarios (mock fed-sx in tests)
|
|
|
|
## Progress log
|
|
|
|
(loop fills this in)
|
|
|
|
## Blockers
|
|
|
|
(loop fills this in)
|