rose-ash/plans/flow-on-sx.md

# flow-on-sx: Durable DAG Workflows on Scheme

rose-ash needs workflows that survive restarts: content pipelines (write → review →
publish → federate), scheduled jobs (digest emails), multi-step user flows (signup,
confirm, onboard). art-dag is the precedent — DAG-of-tasks with pause/resume at IO
boundaries.

Scheme's `call/cc` + delimited continuations make pause/resume natural: a `suspend`
captures the continuation, serializes it as part of the flow record, and `resume`
re-enters at exactly that point. No state-machine bookkeeping by hand. R7RS-small is
already at 2644/2644 (see kernel/architecture status).

End-state: a Scheme-on-SX layer over the existing scheme runtime, with combinators
for sequence/parallel/branch/retry/timeout/suspend, persistent flow store, and a
federation extension via fed-sx for remote-node execution.

## Status (rolling)

`bash lib/flow/conformance.sh` → **24/24** (Phase 1 done; Phase 2 in progress)

## Ground rules

- **Scope:** only touch `lib/flow/**` and `plans/flow-on-sx.md`. Do **not** edit
  `spec/`, `hosts/`, `shared/`, `lib/scheme/**`, or other `lib/<lang>/`. You may
  **import** from `lib/scheme/` (public API via `lib/scheme/scheme.sx`); do **not**
  modify Scheme.
- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** flow combinators are Scheme macros + procedures. Runtime is a
  driver loop that walks the flow graph and invokes `call/cc` at `suspend` points.
  Persistence layer serializes the continuation + open file/socket placeholders are
  forbidden (continuations must be resumable across process restart).
- **art-dag awareness:** read `plans/art-dag*` if it exists for design lineage; do not
  import code.
- **Commits:** one feature per commit. Keep Progress log updated and tick boxes.

## Architecture sketch

```
(defflow publish
  (sequence
    (write-content)
    (parallel
      (review)
      (spell-check))
    (cond approved?
      (sequence (publish) (federate))
      (notify-author))))
        │
        ▼
lib/flow/spec.sx           lib/flow/runtime.sx          lib/flow/store.sx
  — defflow                  — driver loop                 — append-only flow log
  — sequence/parallel        — node dispatch               — checkpoint serialize
  — cond/retry/timeout       — call/cc at suspend          — restart loader
  — suspend/resume                  │                            │
                                    ▼                            ▼
                            lib/flow/api.sx              lib/flow/remote.sx
                              — (flow/start name args)     — fed-sx adapter
                              — (flow/resume id value)     — node-on-peer execution
                              — (flow/cancel id)           — failure handling
```

## Phase 1 — Declarative DAG + sequential execution

- [x] `lib/flow/spec.sx` — `defflow` macro, `sequence` combinator
- [x] node = Scheme procedure of one arg (upstream value threaded in); output
  threads to next node (data flow). A node ignoring its arg is a thunk.
- [x] `parallel` combinator (sequential semantics for now — TRUE parallelism in Phase 3)
- [x] runtime executes a flow synchronously, returns final value
- [x] `lib/flow/api.sx` — `(flow/start flow input)` entry point
- [x] `lib/flow/tests/basic.sx` — 18 cases: single nodes, linear/nested sequence,
  data flow between nodes, parallel-with-join, publish-shaped flow
- [x] `lib/flow/scoreboard.{json,md}`
- [x] `lib/flow/conformance.sh`

## Phase 2 — Control flow + error handling

- [x] `cond` combinator — predicate selects branch (named `branch`; `cond` is a
  Scheme special form). `(branch pred then else)` — 6 tests.
- [ ] `retry n [backoff]` — re-runs node up to n times on exception
- [ ] `timeout ms` — bounds node execution
- [ ] `try-catch` — exception handler with reified error
- [ ] error model — exceptions vs explicit `(fail :reason ...)` results
- [ ] `lib/flow/tests/control.sx` — 25+ cases: each combinator + composition

## Phase 3 — Suspend / resume (the showcase)

- [ ] `(suspend reason)` — `call/cc` captures continuation, returns flow-id to caller
- [ ] `lib/flow/store.sx` — serialize flow state (continuation + open vars)
- [ ] `(flow/resume id value)` — load continuation, inject value, re-enter
- [ ] `(flow/cancel id)` — explicit termination
- [ ] crash recovery — on restart, scan store for paused flows, mark resumable
- [ ] `lib/flow/tests/suspend.sx` — pause-resume scenarios, cancellation, "restart"
  scenarios (simulated by re-loading store)

## Phase 4 — Distributed nodes via fed-sx

- [ ] `(remote-node addr fn args)` — execute node on a federation peer
- [ ] failure semantics — retry on different peer, fall through to local
- [ ] persistence across instances — flow state replicates via fed-sx
- [ ] handoff — flow started here can resume on a peer if the local instance is down
- [ ] `lib/flow/tests/distributed.sx` — federated flow scenarios (mock fed-sx in tests)

## Progress log

- **Phase 1 (combinators + sequential runtime).** Flow built as a Scheme prelude
  loaded onto `scheme-standard-env`: a flow is a Scheme procedure `input -> output`,
  so the whole flow runs inside the interpreter (sets up Phase 3 call/cc suspend).
  Combinators `flow-node`/`flow-id`/`flow-const`/`sequence`/`parallel`/`defflow` in
  `spec.sx`; `flow/start` + SX helpers (`flow-make-env`/`flow-run`) in `api.sx`.
  18/18 in `tests/basic.sx`. Substrate constraints found: dotted rest params
  `(a . rest)` and named `let` are unsupported in `lib/scheme/eval.sx`, so
  combinators use `(lambda args ...)` variadics + top-level recursion. Scheme
  strings come back boxed as `{:scm-string "..."}` — unwrap with `(get s :scm-string)`.

## Blockers

(none)