diff --git a/plans/commerce-on-sx.md b/plans/commerce-on-sx.md index fe7b7641..196a2dc7 100644 --- a/plans/commerce-on-sx.md +++ b/plans/commerce-on-sx.md @@ -1,8 +1,8 @@ # commerce-on-sx: Catalog, cart, pricing & orders on miniKanren -> **DRAFT outline.** The revenue vertical. Depends on `store-on-sx` (durable +> **DRAFT outline.** The revenue vertical. Depends on `persist-on-sx` (durable > orders) and `flow-on-sx` (checkout as a durable flow). Don't start before -> store-on-sx Phase 1 is green. +> persist-on-sx Phase 1 is green. rose-ash's revenue engine — market (catalog), cart (checkout), orders (SumUp payment, reconciliation) — has no SX subsystem. The hard part of commerce isn't @@ -13,7 +13,7 @@ yields this total?", "which line item triggered the discount?"). That's a miniKanren fit. Pricing/promotion rules are relational; cart and order *lifecycle* (reserve → pay → fulfil → reconcile) is a durable `flow`; the order -ledger is a `store` stream. Commerce is the first real **composition** subsystem. +ledger is a `persist` stream. Commerce is the first real **composition** subsystem. End-state: a catalog model, a relational pricing/promotion engine, a cart with deterministic totals, and an order lifecycle flow with payment-webhook @@ -26,7 +26,7 @@ reconciliation — all auditable via the event log. ## Ground rules - **Scope:** only `lib/commerce/**` and `plans/commerce-on-sx.md`. May **import** - from `lib/minikanren/`, and (once they exist) `lib/store/` + `lib/flow/`. Do not + from `lib/minikanren/`, and (once they exist) `lib/persist/` + `lib/flow/`. Do not edit substrates. - **Architecture:** prices/promotions are miniKanren relations over catalog facts; a cart total is a *deterministic* query result (first solution under a fixed rule @@ -68,7 +68,7 @@ lib/commerce/api.sx ── (commerce/add) (commerce/total) (commerce/checkout) ## Phase 3 — Order lifecycle (flow + store) - [ ] order flow: reserve stock → await payment → fulfil - [ ] payment webhook resumes the suspended flow -- [ ] order ledger as a `store` stream; idempotent reconciliation +- [ ] order ledger as a `persist` stream; idempotent reconciliation ## Phase 4 — Reconciliation + federation - [ ] mismatch detection (paid≠ordered) as queries over the ledger diff --git a/plans/content-on-sx.md b/plans/content-on-sx.md index 6096454c..03f606fa 100644 --- a/plans/content-on-sx.md +++ b/plans/content-on-sx.md @@ -1,7 +1,7 @@ # content-on-sx: Documents, blocks & collaborative editing on Smalltalk > **DRAFT outline.** The CMS vertical — blog, WYSIWYG editor, Ghost sync. Depends -> on `store-on-sx` (document history as an event log). Ghost/CMS sync stays a thin +> on `persist-on-sx` (document history as an event log). Ghost/CMS sync stays a thin > external adapter (Python/FFI) until a native replacement exists. rose-ash's `blog` domain is content management: a block-based WYSIWYG editor, @@ -24,11 +24,11 @@ injected adapter, not core. ## Ground rules - **Scope:** only `lib/content/**` and `plans/content-on-sx.md`. May **import** - from `lib/smalltalk/`, and (once it exists) `lib/store/`. Do not edit substrates. + from `lib/smalltalk/`, and (once it exists) `lib/persist/`. Do not edit substrates. - **Architecture:** a document is an ordered tree of blocks (objects); an edit is a message (`insert`/`update`/`move`/`delete`); concurrent edits merge via a commutative (CRDT/semilattice) operation so order doesn't matter. History is the - `store` event stream; any version is a replay. + `persist` event stream; any version is a replay. - **Determinism:** merge must be commutative + idempotent (test: apply ops in any order / twice → same document). - **Commits:** one feature per commit. Progress log + tick boxes. @@ -52,7 +52,7 @@ lib/content/doc.sx lib/content/merge.sx ▼ │ lib/content/api.sx ── (content/edit) (content/render) (content/history) ──┐ │ │ - ├── op log + versions → store │ + ├── op log + versions → persist │ └── Ghost/CMS sync → injected external adapter (thin, non-core) ──┘ ``` @@ -63,7 +63,7 @@ lib/content/api.sx ── (content/edit) (content/render) (content/history) ─ - [ ] `api.sx` + tests + scoreboard + conformance.sh ## Phase 2 — Op log + versioning -- [ ] edit ops as `store` events; replay to any version +- [ ] edit ops as `persist` events; replay to any version - [ ] `(content/history doc)`, diff between versions ## Phase 3 — Collaborative merge (CRDT) diff --git a/plans/events-on-sx.md b/plans/events-on-sx.md index f5d196a4..c5178db5 100644 --- a/plans/events-on-sx.md +++ b/plans/events-on-sx.md @@ -1,7 +1,7 @@ # events-on-sx: Calendar, ticketing & notification delivery on Datalog > **DRAFT outline.** The events vertical + the shared notification-delivery edge. -> Depends on `store-on-sx` (bookings ledger) and `flow-on-sx` (reminders, retrying +> Depends on `persist-on-sx` (bookings ledger) and `flow-on-sx` (reminders, retrying > delivery). Pairs with `commerce-on-sx` for paid tickets. rose-ash's `events` domain is calendar + ticketing: recurring events, availability, @@ -23,11 +23,11 @@ capacity rules, transactional booking, and a flow-driven notification dispatcher ## Ground rules - **Scope:** only `lib/events/**` and `plans/events-on-sx.md`. May **import** from - `lib/datalog/`, and (once they exist) `lib/store/` + `lib/flow/`. Do not edit + `lib/datalog/`, and (once they exist) `lib/persist/` + `lib/flow/`. Do not edit substrates. - **Architecture:** events/availability/capacity are Datalog facts + rules; recurrence expands to occurrence facts within a window; a booking checks rules - then appends a `store` event (idempotent, capacity-safe). Notifications are flows + then appends a `persist` event (idempotent, capacity-safe). Notifications are flows that suspend on transport IO and retry on failure. - **Determinism:** recurrence expansion + availability must be reproducible for a fixed window + ruleset; capacity checks must be race-safe (no overbooking). @@ -47,7 +47,7 @@ lib/events/calendar.sx lib/events/availability.sx ▼ │ lib/events/booking.sx lib/events/notify.sx (flow) — transactional, capacity-safe — reminders / digests, retry on fail - — bookings → store ledger — injected transport (email/push) + — bookings → persist ledger — injected transport (email/push) │ │ ▼ ▼ lib/events/api.sx ── (events/schedule) (events/book) (events/agenda) ──────┘ @@ -59,7 +59,7 @@ lib/events/api.sx ── (events/schedule) (events/book) (events/agenda) ── - [ ] `api.sx` + tests + scoreboard + conformance.sh ## Phase 2 — Ticketing + booking -- [ ] capacity rules; transactional booking → `store` (no overbooking) +- [ ] capacity rules; transactional booking → `persist` (no overbooking) - [ ] paid tickets compose with `commerce` order flow - [ ] tests: capacity edge, double-book guard, conflict detection diff --git a/plans/identity-on-sx.md b/plans/identity-on-sx.md index 3dd86dcb..f28f7095 100644 --- a/plans/identity-on-sx.md +++ b/plans/identity-on-sx.md @@ -2,7 +2,7 @@ > **DRAFT outline.** The identity core `acl-on-sx` assumes already exists. `acl` > answers "may X do Y"; identity answers "who is X, and how did they prove it." -> Depends on `store-on-sx` (grant/audit ledger). Pairs with `acl-on-sx`. +> Depends on `persist-on-sx` (grant/audit ledger). Pairs with `acl-on-sx`. rose-ash's `account` domain is the OAuth2 authorization server every other app is a client of: silent SSO, per-app first-party cookies, grant verification, @@ -24,7 +24,7 @@ through the event log, all authorization questions delegated to `acl-on-sx`. ## Ground rules - **Scope:** only `lib/identity/**` and `plans/identity-on-sx.md`. May **import** - from `lib/erlang/`, and (once they exist) `lib/store/` + `lib/acl/`. Do not edit + from `lib/erlang/`, and (once they exist) `lib/persist/` + `lib/acl/`. Do not edit substrates. - **Architecture:** a session/grant is a process holding its own state; the registry routes messages by subject/client id. Tokens are opaque + introspected, @@ -53,7 +53,7 @@ lib/identity/session.sx lib/identity/registry.sx ▼ ▼ lib/identity/api.sx ── (identity/login) (identity/grant?) (identity/revoke) ──┐ │ │ - └──────── grant + audit events → store ; permission? → acl ──────────┘ + └──────── grant + audit events → persist ; permission? → acl ──────────┘ ``` ## Phase 1 — Sessions + tokens @@ -73,7 +73,7 @@ lib/identity/api.sx ── (identity/login) (identity/grant?) (identity/revoke) - [ ] grant verification delegated cache (mirror Redis-cache pattern) ## Phase 4 — Audit + federation -- [ ] every issue/refresh/revoke is a `store` event; `(identity/audit subject)` +- [ ] every issue/refresh/revoke is a `persist` event; `(identity/audit subject)` - [ ] federated identity (peer-asserted subject) — advisory, trust-gated stub - [ ] tests: audit completeness, cross-instance subject mapping diff --git a/plans/persist-on-sx.md b/plans/persist-on-sx.md new file mode 100644 index 00000000..53c2b0dd --- /dev/null +++ b/plans/persist-on-sx.md @@ -0,0 +1,119 @@ +# persist-on-sx: Durable state on the SX kernel + +> **DRAFT outline.** Foundation subsystem — the durable substrate the other five +> currently fake with in-memory mutable lists. Build this first. +> +> **"persist" = persistence / data store, NOT the shop.** The shop/commerce vertical +> is `commerce-on-sx`. + +rose-ash needs durable state: every subsystem (feed log, flow store, mod audit, +search index, acl grants, sessions) today hand-rolls an in-memory structure that +vanishes on restart. `persist-on-sx` is the one durable substrate they share. It +lives directly on the SX kernel's IO-suspension primitives (`perform`/`cek-resume` +— the third CEK phase) so a read/write `perform`s and the kernel persists at the +boundary. Concrete storage backends are injected. + +## Does it cover ALL persistence? No — and on purpose. + +Event-sourcing-everything is a known trap (replay cost, event schema evolution, +awkward ad-hoc queries, 5MB images in a log). So persist owns the **durable +source-of-truth substrate**, exposed as **two facets over one backend protocol**, +with two things explicitly delegated out: + +| Shape | Owner | Notes | +|-------|-------|-------| +| **Event streams** (append-only, history matters) | persist — **log facet** | feed activities, mod audit, order ledger, flow state, content edits | +| **Current-state values** (KV / document, no history) | persist — **kv facet** | profiles, stock counts, config, session blobs; also where projections materialize | +| **Snapshots / read models** (derived, queryable) | persist — projections → kv/log | rebuildable from the log; persisted so you don't replay to answer a query | +| **Blobs / large objects** (images, media) | **delegated** → content-addressed store (artdag/IPFS already) | persist stores the *reference/CID*, never the bytes | +| **Cache** (ephemeral, evictable) | **out of scope** | not persistence — different lifecycle (Redis-shaped) | +| **Ad-hoc relational query** | the subsystem, over a projected read model | the log is bad at "all orders by X in March"; project into a queryable kv/SQL backend | + +So: persist is the **single durable substrate** for state that's either a stream of +changes or a current value — but it does **not** force everything into an event +log, it does **not** hold blobs (only their content-addressed refs), and it does +**not** do caching. Those boundaries are the whole point of calling it a substrate +rather than "the database." + +End-state: `log` (append/read streams) + `kv` (get/put/delete by key) facets, an +injectable backend protocol (mem → file → Postgres → IPFS-ref), pure projections +with incremental snapshots, optimistic concurrency, and a subscription hook so +read models (feeds, indices, audit logs) update incrementally. + +## Status (rolling) + +`bash lib/persist/conformance.sh` → **0/0** (not yet started) + +## Ground rules + +- **Scope:** only `lib/persist/**` and `plans/persist-on-sx.md`. May **import** the + kernel's IO-suspension surface (`perform`, platform IO ops) — verify what's + exported first. Do not add host primitives; a missing durable IO op is a Blockers + entry (it belongs in `hosts/`, out of scope). +- **Architecture:** an event is `{:stream :seq :type :at :data}`; the log is an + ordered append-only vector; a projection is `(fold step seed events)`; a kv value + is `(get/put/delete key)`. Both facets sit on one injected backend + `{:append :read :kv-get :kv-put :snapshot-read :snapshot-write}`. The in-memory + backend is the test default; real backends wire in unchanged. +- **Determinism:** replay is pure — same log → same state, always. No clocks or + randomness inside projections; time lives on the event. +- **Blobs:** store the content-address/CID and metadata; never the bytes. The blob + backend is a separate injected dependency. +- **Commits:** one feature per commit. Progress log + tick boxes. + +## Architecture sketch + +``` +Command / write Read model / value + (append stream type data) (project stream step seed) + (kv-put key value) (kv-get key) + │ ▲ + ▼ │ +lib/persist/event.sx lib/persist/project.sx + — {:stream :seq :type :at :data} — fold step seed; incremental from snapshot + │ ▲ + ▼ │ +lib/persist/log.sx lib/persist/kv.sx lib/persist/snapshot.sx + — append/read — get/put/delete — checkpoint; replay = snapshot + tail + — optimistic seq — current-state + │ │ ▲ + └──────────────────┴── (perform → backend) ───┘ + │ +lib/persist/backend.sx lib/persist/api.sx + — injected protocol — (persist/append) (persist/project) + — mem | file | pg | ipfs-ref — (persist/kv-get/put) (persist/subscribe) + │ + └── blobs → content-addressed store (artdag/IPFS), by reference only +``` + +## Phase 1 — Log + kv + in-memory backend +- [ ] `event.sx` — event record, stream/seq helpers +- [ ] `backend.sx` — injectable protocol + in-memory impl (log + kv) +- [ ] `log.sx` — `append` (optimistic seq), `read`, `read-from` +- [ ] `kv.sx` — `get`/`put`/`delete` current-state +- [ ] `api.sx` + tests + scoreboard + conformance.sh + +## Phase 2 — Projections + subscriptions +- [ ] `project.sx` — `(project stream step seed)`, incremental fold +- [ ] subscription hook — projection / kv read model re-runs on append +- [ ] concurrency conflict surfaced as a real result, not a crash + +## Phase 3 — Snapshots + replay +- [ ] `snapshot.sx` — checkpoint a projection; replay = snapshot + tail +- [ ] compaction policy; replay-determinism tests + +## Phase 4 — Durable backends via kernel IO +- [ ] file/log backend driven through `perform` (IO-suspension boundary) +- [ ] blob backend interface (store ref/CID; bytes live in artdag/IPFS) +- [ ] crash/restart replay test (mock IO platform) +- [ ] migration notes for swapping mem → durable under a live subsystem + +## Consumers (post-foundation, not in scope here) +feed/-log, flow store, mod/audit, search index, acl grants, identity sessions all +become `persist` log or kv. Track each migration in that subsystem's plan. + +## Progress log +(loop fills this in) + +## Blockers +(loop fills this in) diff --git a/plans/store-on-sx.md b/plans/store-on-sx.md deleted file mode 100644 index 2795b722..00000000 --- a/plans/store-on-sx.md +++ /dev/null @@ -1,92 +0,0 @@ -# store-on-sx: Event Sourcing on the SX kernel - -> **DRAFT outline.** Foundation subsystem — the durable substrate the other five -> currently fake with in-memory mutable lists. Build this first. - -rose-ash needs durable state: every subsystem (feed log, flow store, mod audit, -search index, acl grants) today hand-rolls an in-memory list that vanishes on -restart. They all secretly want the same thing — an append-only event log with -pure projections over it. Event sourcing makes that one substrate: the log is the -source of truth, state is a fold, durability is an IO boundary. - -This is **substrate-level**, not a guest language. It lives directly on the SX -kernel's IO-suspension primitives (`perform`/`cek-resume` — the third CEK phase) -so a projection can `perform` a read/append and the kernel persists at the -boundary. Storage backends (in-memory, file, later Postgres/IPFS) are injected. - -End-state: an `append`/`fold`/`project`/`snapshot` API with an injectable backend, -optimistic-concurrency on streams, replay + snapshotting, and a subscription hook -so projections (feeds, indices, audit logs) update incrementally. The other -subsystems swap their mutable list for a `store/stream`. - -## Status (rolling) - -`bash lib/store/conformance.sh` → **0/0** (not yet started) - -## Ground rules - -- **Scope:** only `lib/store/**` and `plans/store-on-sx.md`. Do **not** edit - `spec/`, `hosts/`, `shared/`, or `lib//`. May **import** the kernel's - IO-suspension surface (`perform`, the platform IO ops) — verify what's exported - before relying on it. Do not add host primitives; if a needed durable IO op is - missing, file it under Blockers (it belongs in `hosts/` / fed-prims, out of scope). -- **Architecture:** an event is `{:stream :seq :type :at :data}`. A log is an - ordered, append-only vector of events. A projection is `(fold step seed events)`. - Persistence is an injected backend `{:append :read :snapshot-read :snapshot-write}`; - the in-memory backend is the test default, real backends wire in unchanged. -- **Determinism:** replay must be pure — same log → same state, always. No clocks - or randomness inside projections; timestamps live on the event, not the fold. -- **Commits:** one feature per commit. Keep Progress log + tick boxes. - -## Architecture sketch - -``` -Command Read model - (append stream type data) (project stream step seed) - │ ▲ - ▼ │ -lib/store/event.sx lib/store/project.sx - — {:stream :seq :type :at :data} — fold step seed over a stream - — stream ids, seq ordering — incremental: resume from snapshot - │ ▲ - ▼ │ -lib/store/log.sx lib/store/snapshot.sx - — append-only, optimistic concurrency — periodic fold checkpoint - — read range / read-from-seq — replay = snapshot + tail - │ ▲ - ▼ (perform → backend) │ -lib/store/backend.sx lib/store/api.sx - — injected {:append :read ...} — (store/append ...) (store/project ...) - — mem backend (tests) | file | pg — (store/subscribe stream fn) -``` - -## Phase 1 — Log + in-memory backend -- [ ] `lib/store/event.sx` — event record, stream/seq helpers -- [ ] `lib/store/backend.sx` — injectable backend protocol + in-memory impl -- [ ] `lib/store/log.sx` — `append` (optimistic seq), `read`, `read-from` -- [ ] `lib/store/api.sx` — `(store/append ...)`, `(store/events stream)` -- [ ] `lib/store/tests/log.sx` + scoreboard + conformance.sh - -## Phase 2 — Projections + subscriptions -- [ ] `lib/store/project.sx` — `(project stream step seed)`, incremental fold -- [ ] subscription hook — projection re-runs on append -- [ ] concurrency conflict surfaced as a real result, not a crash - -## Phase 3 — Snapshots + replay -- [ ] `lib/store/snapshot.sx` — checkpoint a projection, replay = snapshot + tail -- [ ] compaction policy; replay determinism tests - -## Phase 4 — Durable backend via kernel IO -- [ ] file/log backend driven through `perform` (IO-suspension boundary) -- [ ] crash/restart replay test (mock IO platform) -- [ ] migration notes for swapping mem → durable under a live subsystem - -## Consumers (post-foundation, not in scope here) -feed/-log, flow store, mod/audit, search index, acl grant set all become -`store/stream`s. Track the migration in each subsystem's plan, not this one. - -## Progress log -(loop fills this in) - -## Blockers -(loop fills this in)