giles
8b3d92ed5f
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 49s
A native Erlang-on-SX durable workflow engine, so the fed-sx kernel can
fan activities out into business flows in its own runtime — no cross-
guest FFI to the Scheme lib/flow, no marshalling, no Scheme dependency.
The seed of a real engine (chosen over bridging Scheme flow) that can
later supersede it for substrate use.
- flow.erl — the deterministic-replay driver. Same durability model as
the Scheme engine (re-run from the top; effects go through suspend;
the replay log is plain [{Tag,Value}] data, restart-ready), but
adapted to three hard runtime constraints: no re-enterable
continuation, no process dictionary, and a blocking receive inside a
`try` deadlocks the cooperative scheduler. Resolution: thread the log
through a railway-style context and make suspend SHORT-CIRCUIT (like a
fail value) instead of throwing — purely functional, sidesteps all
three. Ctx = {flow_cont,V,Log} | {flow_susp,Tag,Log}.
- flow_spec.erl — combinator algebra mirrored from lib/flow/spec.sx:
leaves, sequence/parallel/map_flow, flow_while/flow_until, branch,
railway fail/recover/attempt, tap, try_catch/retry.
- flow_store.erl — durable gen_server: named-flow registry + instance
table + start/resume/status. Drives the pure flow from handle_call,
so no gen_server:call is ever inside the replay try-path.
Gate: next/flow/conformance.sh — 34/34. lib/erlang untouched (771/771).
See next/flow/README.md for the model + why railway threading.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
92 lines
4.0 KiB
Markdown
92 lines
4.0 KiB
Markdown
# flow-on-erlang — durable workflows in the fed-sx runtime
|
|
|
|
A native Erlang-on-SX port of the Scheme flow engine (`lib/flow`), so
|
|
the fed-sx kernel can fan arriving activities out into durable,
|
|
branching, multi-step business flows **in its own runtime** — no
|
|
cross-guest FFI, no marshalling, no Scheme dependency. The seed of a
|
|
real engine that can later supersede the Scheme one for substrate use.
|
|
|
|
Run the suite: `bash next/flow/conformance.sh` → engine conformance.
|
|
|
|
## Model
|
|
|
|
A **flow** is an Erlang `fun(Ctx) -> Ctx`. Combinators (`flow_spec`)
|
|
compose flows; user code stays value-level (the functions you hand to
|
|
`flow_node`/`branch`/… take and return plain values). A flow that
|
|
ignores its input is a thunk; composition *is* function composition.
|
|
|
|
```erlang
|
|
F = flow_spec:sequence([
|
|
flow_spec:flow_node(fun(Draft) -> Draft + 1 end),
|
|
flow_spec:branch(fun(P) -> P >= 3 end,
|
|
flow_spec:flow_const(ok),
|
|
flow_spec:flow_const(rejected))]),
|
|
flow:run(F, 2) %% => {flow_done, ok}
|
|
```
|
|
|
|
## Durability — deterministic replay
|
|
|
|
Same semantics as the Scheme engine: a flow re-runs from the top on
|
|
every resume; effects/non-determinism go through `flow:suspend/1`,
|
|
whose resolved values are logged; an already-resolved suspend replays
|
|
its logged value, and the first unresolved suspend short-circuits back
|
|
to the driver. The persisted state is the **replay log** — plain
|
|
`[{Tag, Value}]` data — so nothing live (no continuation, no process)
|
|
is ever serialized; an instance survives restart by re-driving its
|
|
named flow against its log.
|
|
|
|
```erlang
|
|
flow_store:register_flow(publish, F),
|
|
{ok, Id, R} = flow_store:start(publish, Draft), %% R = {flow_suspended, Tag} | {flow_done, V}
|
|
%% ... driver performs the effect for Tag, then:
|
|
flow_store:resume(Id, EffectResult) %% re-drives; completes or suspends again
|
|
```
|
|
|
|
## Why railway threading instead of call/cc + a global
|
|
|
|
The Scheme engine uses an escape-only `call/cc` plus a mutable global
|
|
replay log. This Erlang-on-SX runtime can't do either, and has a third
|
|
sharp edge:
|
|
|
|
- **No re-enterable continuation** — but suspend only needs to *escape*,
|
|
which Erlang `throw` could do …
|
|
- **… except a blocking `receive` / `gen_server:call` inside a `try`
|
|
deadlocks** the cooperative scheduler. So `suspend` must not consult
|
|
the log via a registry process while inside a `try`.
|
|
- **No process dictionary** — so there is no ambient per-process slot to
|
|
stash the replay log in.
|
|
|
|
The resolution: thread the replay log through a railway-style **context**
|
|
and make `suspend` *short-circuit* (like a `fail` value) rather than
|
|
throw. No ambient state, no throw, no gen_server in the hot path —
|
|
purely functional, which sidesteps all three constraints. The driver
|
|
(`flow_store`) is the only stateful part, and it calls the pure
|
|
`flow:drive/3` from inside `handle_call`, never wrapping a blocking
|
|
receive.
|
|
|
|
A `Ctx` is `{flow_cont, Value, Log}` (running) or `{flow_susp, Tag,
|
|
Log}` (short-circuited); every combinator passes a suspended context
|
|
straight through.
|
|
|
|
## Modules
|
|
|
|
| Module | Role |
|
|
|---|---|
|
|
| `flow.erl` | pure replay driver: `drive/3`, `run/2`, `suspend/1`, the `Ctx` constructors/accessors |
|
|
| `flow_spec.erl` | combinator algebra: leaves, `sequence`/`parallel`/`map_flow`, `flow_while`/`flow_until`, `branch`, railway `fail`/`recover`/`attempt`, `tap`, `try_catch`/`retry` |
|
|
| `flow_store.erl` | durable gen_server: named-flow registry + instance table + `start`/`resume`/`status` |
|
|
|
|
## Consumed by
|
|
|
|
The fed-sx kernel's trigger fan-out (`pipeline.erl` + `flow_dispatch`)
|
|
starts named flows from arriving activities; see
|
|
`plans/fed-sx-host-types.md` and the triggers phases.
|
|
|
|
## Not yet (later layers)
|
|
|
|
- Persisting instance logs to the kernel's durable on-disk log (the
|
|
data shape is already restart-ready; only the backing is in-memory).
|
|
- `parallel` with multiple independent suspends resolving concurrently
|
|
(current `parallel` is sequential under one shared log).
|
|
- Full parity with the Scheme engine's distributed/remote nodes.
|