120 lines
6.4 KiB
Markdown
120 lines
6.4 KiB
Markdown
# persist-on-sx: Durable state on the SX kernel
|
|
|
|
> **DRAFT outline.** Foundation subsystem — the durable substrate the other five
|
|
> currently fake with in-memory mutable lists. Build this first.
|
|
>
|
|
> **"persist" = persistence / data store, NOT the shop.** The shop/commerce vertical
|
|
> is `commerce-on-sx`.
|
|
|
|
rose-ash needs durable state: every subsystem (feed log, flow store, mod audit,
|
|
search index, acl grants, sessions) today hand-rolls an in-memory structure that
|
|
vanishes on restart. `persist-on-sx` is the one durable substrate they share. It
|
|
lives directly on the SX kernel's IO-suspension primitives (`perform`/`cek-resume`
|
|
— the third CEK phase) so a read/write `perform`s and the kernel persists at the
|
|
boundary. Concrete storage backends are injected.
|
|
|
|
## Does it cover ALL persistence? No — and on purpose.
|
|
|
|
Event-sourcing-everything is a known trap (replay cost, event schema evolution,
|
|
awkward ad-hoc queries, 5MB images in a log). So persist owns the **durable
|
|
source-of-truth substrate**, exposed as **two facets over one backend protocol**,
|
|
with two things explicitly delegated out:
|
|
|
|
| Shape | Owner | Notes |
|
|
|-------|-------|-------|
|
|
| **Event streams** (append-only, history matters) | persist — **log facet** | feed activities, mod audit, order ledger, flow state, content edits |
|
|
| **Current-state values** (KV / document, no history) | persist — **kv facet** | profiles, stock counts, config, session blobs; also where projections materialize |
|
|
| **Snapshots / read models** (derived, queryable) | persist — projections → kv/log | rebuildable from the log; persisted so you don't replay to answer a query |
|
|
| **Blobs / large objects** (images, media) | **delegated** → content-addressed store (artdag/IPFS already) | persist stores the *reference/CID*, never the bytes |
|
|
| **Cache** (ephemeral, evictable) | **out of scope** | not persistence — different lifecycle (Redis-shaped) |
|
|
| **Ad-hoc relational query** | the subsystem, over a projected read model | the log is bad at "all orders by X in March"; project into a queryable kv/SQL backend |
|
|
|
|
So: persist is the **single durable substrate** for state that's either a stream of
|
|
changes or a current value — but it does **not** force everything into an event
|
|
log, it does **not** hold blobs (only their content-addressed refs), and it does
|
|
**not** do caching. Those boundaries are the whole point of calling it a substrate
|
|
rather than "the database."
|
|
|
|
End-state: `log` (append/read streams) + `kv` (get/put/delete by key) facets, an
|
|
injectable backend protocol (mem → file → Postgres → IPFS-ref), pure projections
|
|
with incremental snapshots, optimistic concurrency, and a subscription hook so
|
|
read models (feeds, indices, audit logs) update incrementally.
|
|
|
|
## Status (rolling)
|
|
|
|
`bash lib/persist/conformance.sh` → **0/0** (not yet started)
|
|
|
|
## Ground rules
|
|
|
|
- **Scope:** only `lib/persist/**` and `plans/persist-on-sx.md`. May **import** the
|
|
kernel's IO-suspension surface (`perform`, platform IO ops) — verify what's
|
|
exported first. Do not add host primitives; a missing durable IO op is a Blockers
|
|
entry (it belongs in `hosts/`, out of scope).
|
|
- **Architecture:** an event is `{:stream :seq :type :at :data}`; the log is an
|
|
ordered append-only vector; a projection is `(fold step seed events)`; a kv value
|
|
is `(get/put/delete key)`. Both facets sit on one injected backend
|
|
`{:append :read :kv-get :kv-put :snapshot-read :snapshot-write}`. The in-memory
|
|
backend is the test default; real backends wire in unchanged.
|
|
- **Determinism:** replay is pure — same log → same state, always. No clocks or
|
|
randomness inside projections; time lives on the event.
|
|
- **Blobs:** store the content-address/CID and metadata; never the bytes. The blob
|
|
backend is a separate injected dependency.
|
|
- **Commits:** one feature per commit. Progress log + tick boxes.
|
|
|
|
## Architecture sketch
|
|
|
|
```
|
|
Command / write Read model / value
|
|
(append stream type data) (project stream step seed)
|
|
(kv-put key value) (kv-get key)
|
|
│ ▲
|
|
▼ │
|
|
lib/persist/event.sx lib/persist/project.sx
|
|
— {:stream :seq :type :at :data} — fold step seed; incremental from snapshot
|
|
│ ▲
|
|
▼ │
|
|
lib/persist/log.sx lib/persist/kv.sx lib/persist/snapshot.sx
|
|
— append/read — get/put/delete — checkpoint; replay = snapshot + tail
|
|
— optimistic seq — current-state
|
|
│ │ ▲
|
|
└──────────────────┴── (perform → backend) ───┘
|
|
│
|
|
lib/persist/backend.sx lib/persist/api.sx
|
|
— injected protocol — (persist/append) (persist/project)
|
|
— mem | file | pg | ipfs-ref — (persist/kv-get/put) (persist/subscribe)
|
|
│
|
|
└── blobs → content-addressed store (artdag/IPFS), by reference only
|
|
```
|
|
|
|
## Phase 1 — Log + kv + in-memory backend
|
|
- [ ] `event.sx` — event record, stream/seq helpers
|
|
- [ ] `backend.sx` — injectable protocol + in-memory impl (log + kv)
|
|
- [ ] `log.sx` — `append` (optimistic seq), `read`, `read-from`
|
|
- [ ] `kv.sx` — `get`/`put`/`delete` current-state
|
|
- [ ] `api.sx` + tests + scoreboard + conformance.sh
|
|
|
|
## Phase 2 — Projections + subscriptions
|
|
- [ ] `project.sx` — `(project stream step seed)`, incremental fold
|
|
- [ ] subscription hook — projection / kv read model re-runs on append
|
|
- [ ] concurrency conflict surfaced as a real result, not a crash
|
|
|
|
## Phase 3 — Snapshots + replay
|
|
- [ ] `snapshot.sx` — checkpoint a projection; replay = snapshot + tail
|
|
- [ ] compaction policy; replay-determinism tests
|
|
|
|
## Phase 4 — Durable backends via kernel IO
|
|
- [ ] file/log backend driven through `perform` (IO-suspension boundary)
|
|
- [ ] blob backend interface (store ref/CID; bytes live in artdag/IPFS)
|
|
- [ ] crash/restart replay test (mock IO platform)
|
|
- [ ] migration notes for swapping mem → durable under a live subsystem
|
|
|
|
## Consumers (post-foundation, not in scope here)
|
|
feed/-log, flow store, mod/audit, search index, acl grants, identity sessions all
|
|
become `persist` log or kv. Track each migration in that subsystem's plan.
|
|
|
|
## Progress log
|
|
(loop fills this in)
|
|
|
|
## Blockers
|
|
(loop fills this in)
|