# persist-on-sx: Durable state on the SX kernel

> **DRAFT outline.** Foundation subsystem — the durable substrate the other five
> currently fake with in-memory mutable lists. Build this first.
>
> **"persist" = persistence / data store, NOT the shop.** The shop/commerce vertical
> is `commerce-on-sx`.

rose-ash needs durable state: every subsystem (feed log, flow store, mod audit,
search index, acl grants, sessions) today hand-rolls an in-memory structure that
vanishes on restart. `persist-on-sx` is the one durable substrate they share. It
lives directly on the SX kernel's IO-suspension primitives (`perform`/`cek-resume`
— the third CEK phase) so a read/write `perform`s and the kernel persists at the
boundary. Concrete storage backends are injected.

## Does it cover ALL persistence? No — and on purpose.

Event-sourcing-everything is a known trap (replay cost, event schema evolution,
awkward ad-hoc queries, 5MB images in a log). So persist owns the **durable
source-of-truth substrate**, exposed as **two facets over one backend protocol**,
with two things explicitly delegated out:

| Shape | Owner | Notes |
|-------|-------|-------|
| **Event streams** (append-only, history matters) | persist — **log facet** | feed activities, mod audit, order ledger, flow state, content edits |
| **Current-state values** (KV / document, no history) | persist — **kv facet** | profiles, stock counts, config, session blobs; also where projections materialize |
| **Snapshots / read models** (derived, queryable) | persist — projections → kv/log | rebuildable from the log; persisted so you don't replay to answer a query |
| **Blobs / large objects** (images, media) | **delegated** → content-addressed store (artdag/IPFS already) | persist stores the *reference/CID*, never the bytes |
| **Cache** (ephemeral, evictable) | **out of scope** | not persistence — different lifecycle (Redis-shaped) |
| **Ad-hoc relational query** | the subsystem, over a projected read model | the log is bad at "all orders by X in March"; project into a queryable kv/SQL backend |

So: persist is the **single durable substrate** for state that's either a stream of
changes or a current value — but it does **not** force everything into an event
log, it does **not** hold blobs (only their content-addressed refs), and it does
**not** do caching. Those boundaries are the whole point of calling it a substrate
rather than "the database."

End-state: `log` (append/read streams) + `kv` (get/put/delete by key) facets, an
injectable backend protocol (mem → file → Postgres → IPFS-ref), pure projections
with incremental snapshots, optimistic concurrency, and a subscription hook so
read models (feeds, indices, audit logs) update incrementally.

## Status (rolling)

`bash lib/persist/conformance.sh` → **0/0** (not yet started)

## Ground rules

- **Scope:** only `lib/persist/**` and `plans/persist-on-sx.md`. May **import** the
  kernel's IO-suspension surface (`perform`, platform IO ops) — verify what's
  exported first. Do not add host primitives; a missing durable IO op is a Blockers
  entry (it belongs in `hosts/`, out of scope).
- **Architecture:** an event is `{:stream :seq :type :at :data}`; the log is an
  ordered append-only vector; a projection is `(fold step seed events)`; a kv value
  is `(get/put/delete key)`. Both facets sit on one injected backend
  `{:append :read :kv-get :kv-put :snapshot-read :snapshot-write}`. The in-memory
  backend is the test default; real backends wire in unchanged.
- **Determinism:** replay is pure — same log → same state, always. No clocks or
  randomness inside projections; time lives on the event.
- **Blobs:** store the content-address/CID and metadata; never the bytes. The blob
  backend is a separate injected dependency.
- **Commits:** one feature per commit. Progress log + tick boxes.

## Architecture sketch

```
Command / write                         Read model / value
  (append stream type data)               (project stream step seed)
  (kv-put key value)                       (kv-get key)
        │                                       ▲
        ▼                                       │
lib/persist/event.sx                    lib/persist/project.sx
  — {:stream :seq :type :at :data}        — fold step seed; incremental from snapshot
        │                                       ▲
        ▼                                       │
lib/persist/log.sx   lib/persist/kv.sx   lib/persist/snapshot.sx
  — append/read       — get/put/delete    — checkpoint; replay = snapshot + tail
  — optimistic seq    — current-state
        │                  │                    ▲
        └──────────────────┴── (perform → backend) ───┘
                            │
lib/persist/backend.sx              lib/persist/api.sx
  — injected protocol                 — (persist/append) (persist/project)
  — mem | file | pg | ipfs-ref        — (persist/kv-get/put) (persist/subscribe)
        │
        └── blobs → content-addressed store (artdag/IPFS), by reference only
```

## Phase 1 — Log + kv + in-memory backend
- [ ] `event.sx` — event record, stream/seq helpers
- [ ] `backend.sx` — injectable protocol + in-memory impl (log + kv)
- [ ] `log.sx` — `append` (optimistic seq), `read`, `read-from`
- [ ] `kv.sx` — `get`/`put`/`delete` current-state
- [ ] `api.sx` + tests + scoreboard + conformance.sh

## Phase 2 — Projections + subscriptions
- [ ] `project.sx` — `(project stream step seed)`, incremental fold
- [ ] subscription hook — projection / kv read model re-runs on append
- [ ] concurrency conflict surfaced as a real result, not a crash

## Phase 3 — Snapshots + replay
- [ ] `snapshot.sx` — checkpoint a projection; replay = snapshot + tail
- [ ] compaction policy; replay-determinism tests

## Phase 4 — Durable backends via kernel IO
- [ ] file/log backend driven through `perform` (IO-suspension boundary)
- [ ] blob backend interface (store ref/CID; bytes live in artdag/IPFS)
- [ ] crash/restart replay test (mock IO platform)
- [ ] migration notes for swapping mem → durable under a live subsystem

## Consumers (post-foundation, not in scope here)
feed/-log, flow store, mod/audit, search index, acl grants, identity sessions all
become `persist` log or kv. Track each migration in that subsystem's plan.

## Progress log
(loop fills this in)

## Blockers
(loop fills this in)