rose-ash/plans/flow-on-sx.md

flow-on-sx: Durable DAG Workflows on Scheme

rose-ash needs workflows that survive restarts: content pipelines (write → review → publish → federate), scheduled jobs (digest emails), multi-step user flows (signup, confirm, onboard). art-dag is the precedent — DAG-of-tasks with pause/resume at IO boundaries.

Scheme's call/cc + delimited continuations make pause/resume natural: a suspend captures the continuation, serializes it as part of the flow record, and resume re-enters at exactly that point. No state-machine bookkeeping by hand. R7RS-small is already at 2644/2644 (see kernel/architecture status).

End-state: a Scheme-on-SX layer over the existing scheme runtime, with combinators for sequence/parallel/branch/retry/timeout/suspend, persistent flow store, and a federation extension via fed-sx for remote-node execution.

Status (rolling)

bash lib/flow/conformance.sh → 24/24 (Phase 1 done; Phase 2 in progress)

Ground rules

• Scope: only touch lib/flow/** and plans/flow-on-sx.md. Do not edit spec/, hosts/, shared/, lib/scheme/**, or other lib/<lang>/. You may import from lib/scheme/ (public API via lib/scheme/scheme.sx); do not modify Scheme.
• Shared-file issues go under "Blockers" with a minimal repro; do not fix here.
• SX files: use sx-tree MCP tools only.
• Architecture: flow combinators are Scheme macros + procedures. Runtime is a driver loop that walks the flow graph and invokes call/cc at suspend points. Persistence layer serializes the continuation + open file/socket placeholders are forbidden (continuations must be resumable across process restart).
• art-dag awareness: read plans/art-dag* if it exists for design lineage; do not import code.
• Commits: one feature per commit. Keep Progress log updated and tick boxes.

Architecture sketch

(defflow publish
  (sequence
    (write-content)
    (parallel
      (review)
      (spell-check))
    (cond approved?
      (sequence (publish) (federate))
      (notify-author))))
        │
        ▼
lib/flow/spec.sx           lib/flow/runtime.sx          lib/flow/store.sx
  — defflow                  — driver loop                 — append-only flow log
  — sequence/parallel        — node dispatch               — checkpoint serialize
  — cond/retry/timeout       — call/cc at suspend          — restart loader
  — suspend/resume                  │                            │
                                    ▼                            ▼
                            lib/flow/api.sx              lib/flow/remote.sx
                              — (flow/start name args)     — fed-sx adapter
                              — (flow/resume id value)     — node-on-peer execution
                              — (flow/cancel id)           — failure handling

Phase 1 — Declarative DAG + sequential execution

• lib/flow/spec.sx — defflow macro, sequence combinator
• node = Scheme procedure of one arg (upstream value threaded in); output threads to next node (data flow). A node ignoring its arg is a thunk.
• parallel combinator (sequential semantics for now — TRUE parallelism in Phase 3)
• runtime executes a flow synchronously, returns final value
• lib/flow/api.sx — (flow/start flow input) entry point
• lib/flow/tests/basic.sx — 18 cases: single nodes, linear/nested sequence, data flow between nodes, parallel-with-join, publish-shaped flow
• lib/flow/scoreboard.{json,md}
• lib/flow/conformance.sh

Phase 2 — Control flow + error handling

• cond combinator — predicate selects branch (named branch; cond is a Scheme special form). (branch pred then else) — 6 tests.
• retry n [backoff] — re-runs node up to n times on exception
• timeout ms — bounds node execution
• try-catch — exception handler with reified error
• error model — exceptions vs explicit (fail :reason ...) results
• lib/flow/tests/control.sx — 25+ cases: each combinator + composition

Phase 3 — Suspend / resume (the showcase)

• (suspend reason) — call/cc captures continuation, returns flow-id to caller
• lib/flow/store.sx — serialize flow state (continuation + open vars)
• (flow/resume id value) — load continuation, inject value, re-enter
• (flow/cancel id) — explicit termination
• crash recovery — on restart, scan store for paused flows, mark resumable
• lib/flow/tests/suspend.sx — pause-resume scenarios, cancellation, "restart" scenarios (simulated by re-loading store)

Phase 4 — Distributed nodes via fed-sx

• (remote-node addr fn args) — execute node on a federation peer
• failure semantics — retry on different peer, fall through to local
• persistence across instances — flow state replicates via fed-sx
• handoff — flow started here can resume on a peer if the local instance is down
• lib/flow/tests/distributed.sx — federated flow scenarios (mock fed-sx in tests)

Progress log

• Phase 1 (combinators + sequential runtime). Flow built as a Scheme prelude loaded onto scheme-standard-env: a flow is a Scheme procedure input -> output, so the whole flow runs inside the interpreter (sets up Phase 3 call/cc suspend). Combinators flow-node/flow-id/flow-const/sequence/parallel/defflow in spec.sx; flow/start + SX helpers (flow-make-env/flow-run) in api.sx. 18/18 in tests/basic.sx. Substrate constraints found: dotted rest params (a . rest) and named let are unsupported in lib/scheme/eval.sx, so combinators use (lambda args ...) variadics + top-level recursion. Scheme strings come back boxed as {:scm-string "..."} — unwrap with (get s :scm-string).

Blockers

(none)