rose-ash/plans/store-on-sx.md

store-on-sx: Event Sourcing on the SX kernel

DRAFT outline. Foundation subsystem — the durable substrate the other five currently fake with in-memory mutable lists. Build this first.

rose-ash needs durable state: every subsystem (feed log, flow store, mod audit, search index, acl grants) today hand-rolls an in-memory list that vanishes on restart. They all secretly want the same thing — an append-only event log with pure projections over it. Event sourcing makes that one substrate: the log is the source of truth, state is a fold, durability is an IO boundary.

This is substrate-level, not a guest language. It lives directly on the SX kernel's IO-suspension primitives (perform/cek-resume — the third CEK phase) so a projection can perform a read/append and the kernel persists at the boundary. Storage backends (in-memory, file, later Postgres/IPFS) are injected.

End-state: an append/fold/project/snapshot API with an injectable backend, optimistic-concurrency on streams, replay + snapshotting, and a subscription hook so projections (feeds, indices, audit logs) update incrementally. The other subsystems swap their mutable list for a store/stream.

Status (rolling)

bash lib/store/conformance.sh → 0/0 (not yet started)

Ground rules

• Scope: only lib/store/** and plans/store-on-sx.md. Do not edit spec/, hosts/, shared/, or lib/<lang>/. May import the kernel's IO-suspension surface (perform, the platform IO ops) — verify what's exported before relying on it. Do not add host primitives; if a needed durable IO op is missing, file it under Blockers (it belongs in hosts/ / fed-prims, out of scope).
• Architecture: an event is {:stream :seq :type :at :data}. A log is an ordered, append-only vector of events. A projection is (fold step seed events). Persistence is an injected backend {:append :read :snapshot-read :snapshot-write}; the in-memory backend is the test default, real backends wire in unchanged.
• Determinism: replay must be pure — same log → same state, always. No clocks or randomness inside projections; timestamps live on the event, not the fold.
• Commits: one feature per commit. Keep Progress log + tick boxes.

Architecture sketch

Command                                 Read model
  (append stream type data)               (project stream step seed)
        │                                       ▲
        ▼                                       │
lib/store/event.sx                      lib/store/project.sx
  — {:stream :seq :type :at :data}        — fold step seed over a stream
  — stream ids, seq ordering              — incremental: resume from snapshot
        │                                       ▲
        ▼                                       │
lib/store/log.sx                        lib/store/snapshot.sx
  — append-only, optimistic concurrency   — periodic fold checkpoint
  — read range / read-from-seq            — replay = snapshot + tail
        │                                       ▲
        ▼  (perform → backend)                  │
lib/store/backend.sx                    lib/store/api.sx
  — injected {:append :read ...}          — (store/append ...) (store/project ...)
  — mem backend (tests) | file | pg       — (store/subscribe stream fn)

Phase 1 — Log + in-memory backend

• lib/store/event.sx — event record, stream/seq helpers
• lib/store/backend.sx — injectable backend protocol + in-memory impl
• lib/store/log.sx — append (optimistic seq), read, read-from
• lib/store/api.sx — (store/append ...), (store/events stream)
• lib/store/tests/log.sx + scoreboard + conformance.sh

Phase 2 — Projections + subscriptions

• lib/store/project.sx — (project stream step seed), incremental fold
• subscription hook — projection re-runs on append
• concurrency conflict surfaced as a real result, not a crash

Phase 3 — Snapshots + replay

• lib/store/snapshot.sx — checkpoint a projection, replay = snapshot + tail
• compaction policy; replay determinism tests

Phase 4 — Durable backend via kernel IO

• file/log backend driven through perform (IO-suspension boundary)
• crash/restart replay test (mock IO platform)
• migration notes for swapping mem → durable under a live subsystem

Consumers (post-foundation, not in scope here)

feed/-log, flow store, mod/audit, search index, acl grant set all become store/streams. Track the migration in each subsystem's plan, not this one.

Progress log

(loop fills this in)

Blockers

(loop fills this in)