Sx_persist_store services every persist/* IO op against on-disk storage
(append-only log + separate monotonic .seq high-water + per-key kv files,
SX-serialized). Wired into the (eval) suspension loop, cek_run_with_io
bridge, and in-process _cek_io_resolver. Data-loss repro now (3 3 3).
New persist_durable_test.sh: durable + monotonic-seq + streams + kv +
real process restart all green (5/5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Briefing for the loop that builds the host-side servicer for persist/* IO ops,
making lib/persist's durable backend actually durable. Points at the Blocker
spec in plans/persist-on-sx.md as the authoritative contract; hard rules on
build isolation (worktree _build only, never clobber the shared binary) and not
pkilling the shared sx_server.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the one gap to real durability: a hosts/ servicer for the persist/*
IO ops. Includes the silent-data-loss repro (durable-backend currently no-ops
under sx_server's default resolver), the full op contract table, hard
invariants (monotonic last-seq, etc.), the blob adapter shape, where to
register in sx_server.ml, and an acceptance test (swap transport, run durable +
recovery suites against real storage, survive a real restart).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
examples/acl.sx: a tested template migrating an ACL-grants store from a
hand-rolled ephemeral map to persist — grants/revokes as events, current set as
a projection, O(1) checks via a materialized view, audit via read-window.
Header carries the BEFORE->AFTER diff. Proves grants survive restart on the
durable backend (the capability the BEFORE version lacked). The pattern other
subsystem loops copy; does not touch the real lib/acl. 201/201.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
global.sx: persist/gappend records a pointer in a reserved $global index whose
seq is the global commit position; read-global/project-global replay every
event in commit order; global-from for incremental consumers. Opt-in (plain
append untouched); $-prefixed streams now reserved + hidden from the public
catalog (streams-all reveals them). Gives feed its unified timeline.
Deterministic across restart. 191/191.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
idempotency.sx: persist/append-once appends at most once per (stream,
idempotency key), returning the same event on a repeat. The marker lives in the
kv facet, so idempotency holds across a restart (verified on durable).
persist/seen? check. 180/180.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
upcast.sx: register a pure (event -> event) upcaster per type in an immutable
registry; read-upcast/project-upcast lift legacy events to the current shape on
read so projections see one shape (no version branching, no history rewrite).
upcast-data helper merges new :data fields. 171/171.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
batch.sx: persist/append-batch commits (type at data) specs as one contiguous
block; persist/append-batch-expect checks the stream is still at expected
before writing any event, so the batch is all-or-nothing under a concurrent
writer (conflict is a value, not a partial write). 162/162.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
query.sx: read-between (seq range), read-since/read-window (by :at),
read-by-type, read-where, count-where. Pure scans over persist/read for audit
windows, type filters, since-cursors. 152/152.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New backend op :streams (from seq high-water marks, so compacted streams still
list), threaded through mem-backend + durable serve/io-backend. catalog.sx:
persist/streams, stream-count, stream-exists?, total-events. 143/143.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kv.sx: persist/kv-cas sets a key only if its current value equals expected,
else returns {:conflict :expected :actual}; persist/kv-put-new is create-only.
The kv analogue of log append-expect — atomic current-state for sessions, acl
grants, stock counts. 133/133.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
view.sx: persist/view bundles stream + fold + snapshot name; view-attach
subscribes it to a hub so each publish refreshes the snapshot incrementally,
making view-peek an O(1) current read. view-value always folds the tail so it
is never stale. The consumer read-model abstraction (feed indices, audit
rollups, search counters). 122/122.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recovery.sx: 6-test end-to-end crash/restart of an order ledger (log +
subscription kv read model + snapshot + compaction + invoice blob ref) on the
durable backend; everything survives a restart over the same disk + content
store, seq continues, two restarts converge. Migration notes (mem → durable
under a live subsystem) added to the plan. Roadmap done, 111/111.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
blob.sx: a blob ref is {:cid :size :mime}; the blob store is a separate
injected dependency (perform in prod, mock content store in tests).
persist/blob-store puts bytes and returns only the ref; bytes live in a
content-addressed store (artdag/IPFS). Tests assert refs in log/kv never carry
the bytes + content-address dedup. 105/105.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
durable.sx: io-backend with an injectable transport — persist/durable-backend
performs each op as {:op "persist/..." :args (...)} (kernel suspends, host
resumes); persist/mock-durable services via persist/serve over an in-memory
disk. Identical request shapes mean the whole facet/projection/snapshot/
compaction stack runs unchanged on the durable backend. Crash/restart replay
recovers log+kv+snapshot. 91/91.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend now tracks last-seq as a monotonic high-water mark (survives
truncation) and exposes :truncate-through. compaction.sx: persist/compact
checkpoints then drops events with seq <= snapshot seq; should-compact?/
maybe-compact give an explicit every-N policy. Determinism: post-compaction
replay value == uncompacted full replay. Phase 3 complete, 76/76.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
snapshot.sx: snapshot is a projection state {:value :seq} stored in kv under
snapshot/<name>. persist/checkpoint replays and saves; persist/replay folds
only the tail after the snapshot. Tests assert snapshot+tail == full replay
both ways + determinism. 65/65.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
concurrency.sx: persist/append-expect refuses an append when the stream
advanced past the caller's expected seq, returning {:conflict :expected
:actual} instead of crashing or overwriting. persist/conflict? + accessors.
Phase 2 complete, 54/54.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
project.sx: projection state {:value :seq}; persist/project folds the whole
stream, persist/project-resume folds only the tail so read models update
incrementally. Pure step (value event)->value. 37/37.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gap analysis from the five-subsystem set (acl/feed/flow/mod/search):
- store-on-sx: event-sourcing foundation the others fake with in-memory lists (build first)
- commerce-on-sx: catalog/cart/pricing/orders on miniKanren (+ store + flow)
- identity-on-sx: OAuth2/sessions/membership on Erlang (the core acl assumes)
- content-on-sx: documents/blocks/CRDT on Smalltalk
- events-on-sx: calendar/ticketing on Datalog + flow-driven delivery
- host-on-sx: the web boundary — off Quart onto native server+SXTP now, dream-on-sx next
All DRAFT outlines; substrate choices proposed, not final.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plans for acl-on-sx (Datalog), flow-on-sx (Scheme), feed-on-sx (APL),
mod-on-sx (Prolog), search-on-sx (Haskell). Each is a 4-phase queue
sitting on its respective guest language, targeting rose-ash needs:
access control, durable workflows, activity feeds, moderation, search.
Federation extension in Phase 4 of each (plugs into fed-sx).
Briefings for the three loops we're kicking off now: acl-loop,
flow-loop, feed-loop. mod-sx and search-sx briefings will follow
once the first three have surfaced any shared infrastructure
worth extracting to lib/guest/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
plans/lib-guest-scheduler.md and plans/lib-guest-static-types-
bidirectional.md both have Phase 1 ticked complete from Go's side
with status blocks enumerating what landed.
Each sister diary received a consolidated chisel-summary entry:
the kit primitives the Go consumer chiselled out, the three
pluggable predicates / orthogonal first-class-tag axes, and the
v0 limitations the eventual kit must lift.
No new Go code — Phase 10 is doc-only per plan. Go-on-SX loop
fully landed: 11 phases, 7 test suites, 609/609 passing.
Two-consumer rule per sister plan now waits on TypeScript (Phase 2
of the bidirectional sister plan, owned outside this loop).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 canonical Go programs running through the full pipeline (lex +
parse + types + eval + sched + stdlib): sieve-of-Eratosthenes via
boolean slice (modulo-free), linear search, slice reverse, fib(10),
sum-of-squares via generic Map+Reduce, word-freq counter, channel
pipeline (gen→sq→sum), worker pool, bubble sort, sentence-reverse,
Filter+len, Ackermann, defer+recover on div-by-zero.
Each test threads ONE self-contained Go program through go-eval-
program. The v0 limitations chiselled in earlier phases (float
division, sync spawn, type erasure, nil-as-unbound) are now
durable as commit-trail artifacts; e2e variants written to avoid
them where possible. HTTP-ish ping-pong + WaitGroup deferred
(real preemption + sync package needed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New :go-package NAME ENTRIES value type with field lookup via
extended go-eval-select. New :go-builtin-fn callable for closure-
based stdlib functions. lib/go/std/strings.sx ships 12 functions
(Contains, HasPrefix, HasSuffix, Index, Count, Repeat, Join,
ToUpper, ToLower, TrimSpace, Split, Replace) + lib/go/std/strconv.sx
ships Itoa/Atoi.
Pre-existing bug fixed: parser was emitting (:literal V) for both
`42` and `"42"`, relying on first-char heuristic in eval/types.
Now emits :literal-string for string/rune literals so Atoi("42")
correctly receives the string. 3 parse tests + 2 in-composite-key
tests updated to new shape.
Total 597/597. Stdlib 41/41 — +40 acceptance bar cleared. Sister
diary documents the 11 value-type kinds (struct/slice/map/chan/
fn/method/builtin/builtin-fn/package/panic/defer) all sharing the
"(:KIND PAYLOAD...)" shape, alongside AST nodes and sentinel signals
as the kit's three orthogonal first-class-tag axes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Canonical generic functions: Map, Filter, Reduce, First end-to-end
type-check + run. Plus 20+ typer-only shape tests covering Apply,
Compose, ToMap, Swap, Box, Triple, ToSlice, Take, Send, Fill, Eq,
Values, Pair, Inspect, etc. Index synth (slice/array/map →
element type) added to typer.
v0 limitations stamped in tests: SX `/` is float (no int mod
emulation), `var r []T` indistinguishable from unbound, single-name
constraints opaque (no type-set arithmetic).
Shape locked in: "the parser recognizes shapes, the validator
recognizes roles." Same AST + different role-validators = different
guest semantics. Diary documents this as the lemma the kit should
extract — three deliverables (binding-groups, control-flow sentinels,
index synthesis) now all instantiate it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gp-parse-type-params consumes the optional [NAMES CONSTRAINT, ...]
clause after a func name. AST stays backward-compatible: 5-slot
func-decl when no [...] is present, 6-slot when it is.
Typer binds each type-param name as (:ty-param NAME CONSTRAINT) so
body's (:ty-name "T") references resolve. Eval is type-erasing —
ignores type info, dispatches by name + arity.
10 new tests: parse (3), types (5), eval (2). Total 527/527.
Shape: the field binding-group from the canonical kit now feeds
6 consumers (struct fields, var-decls, const-decls, params,
receivers, type-params). Confirms it as a TRUE cross-deliverable
shape — sister-plan diary documents the 5 roles binding-groups
take and why the kit should expose ONE parser + pluggable validators.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wired panic through :go stmt (v0 sync surfaces back to spawner —
matches real Go's "crash whole program" end-effect) and through
go-eval-for (was swallowing panic at the loop boundary).
8 tests added: goroutine-panic-surfaces, goroutine-recover-via-
spawner-defer, multi-defer-LIFO-with-recover, defer-fires-on-panic-
path, panic(nil), panic-in-loop, defer-still-runs-in-panicking-fn,
args-eager-on-panic-path. 20 Phase-6 tests total; +20 acceptance
bar cleared (eval/ 80 → 100).
Shape: 4 control-flow sites now repeat the same sentinel dispatch
arm (return-value, break, continue, eval-error, go-panic). The
scheduler kit should bake in a single propagates? helper rather
than have each guest evaluator list every sentinel inline — diary
documents the cross-cutting abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Panic/recover builtins + per-frame __go-panic-cell of shape
(STATE V). Body panic flips cell :none→:raised BEFORE defers drain
so recover() can find it. recover() walks env chain past shadowing
cells to the outermost :raised one — flips it :recovered, returns V.
Frame exit checks cell: :recovered → return clean; :raised →
propagate (:go-panic V).
6 tests: uncaught-from-program, panic-from-fn, defer-recover-swallow,
recover-captures-via-channel, propagation-through-no-defer-chain,
middle-frame-catches-deeper-panic.
Shape: panic cell is a frame-attached out-of-band channel that
survives function boundaries via env-chain walk. Same primitive
slots into the scheduler kit's termination-record + cleanup-with-
error-context hook. Maps cleanly to Erlang try/catch/after.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 6 first slice. New :defer stmt dispatch, go-eval-defer-stmt
captures (callee, eagerly-evaluated args) onto a frame-local
__go-defer-stack mutable list. go-eval-call installs the stack and
drains LIFO before returning; go-eval-program does the same for
the implicit main frame. New :quoted-value AST node lets defer
re-invoke calls with the frozen arg values.
6 eval tests: single defer, multi-LIFO, args-eager-at-defer-time,
fires-on-early-return, frame-local (no bleed to outer), defer-in-loop.
Shape: defer is a per-frame cleanup queue (LIFO on frame exit) that
the scheduler kit will reuse for panic-unwind + clean-exit + select-
case-rollback paths. Distinct from the scheduler's ready-queue —
diary updated to keep that distinction explicit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Acceptance bar hit (40 runtime, 497 total). Tests: timer ready,
select-with-timeout, fan-in (3 producers), worker queue, pipeline,
fan-out-then-fan-in, select source-order, fallback case, default,
producer-consumer, two-stage pipeline, channel-counter, after+default,
tick-collector.
Shape chiselled: timer collapses "after duration" into
"channel ready immediately" — select needs only ready? from each
case. Real time is when the flip happens, not what the protocol is.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 cont. New go-eval-range-for handles the parser's :range-for
AST shape. Dispatches on the collection's runtime type:
:go-slice → bind index + element, iterate by position
:go-map → bind key + value, walk entries assoc list
:go-chan → bind value, drain until buffer empty (v0 limitation)
Each loop carries:
- go-range-extend: handles 0/1/2-name binding patterns uniformly
- go-range-body: evaluates body whether it's a :block or other shape
- per-collection loop helper: threads env, catches :break/:continue/
:return-value/:eval-error sentinels
**Subtle break fix:** loops were previously returning the *pre-loop*
env when break fired, clobbering all assignments made in prior
iterations. Now returns the current iteration's input env (which
carries forward successful iterations' state). Patched for the three
range variants and for the regular for-loop where the same pattern
applied. The shape:
(= r :break) env ;; was: (= r :break) original-env
Tests:
range: slice — sum of 1..5 = 15
range: slice — key only (index)
range: map — sum values
range: channel — collect all buffered
range: slice with break exits early
range: slice with continue skips an element
range: empty slice — body never runs
range: chan + goroutine producer
runtime 26/26, total 483/483.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 cont. Adds `select` statement evaluation:
go-select-try-case env COMM →
:not-ready / extended-env / :eval-error
go-select-pick env CASES DEFAULT-OR-NIL →
body-result / blocked-error
go-eval-select-stmt env STMT — public entry
Walks cases in declared order:
* :send case — always ready in v0 (unbounded buffer). Sends value
via go-chan-send! and returns env unchanged.
* :short-decl / :assign case — RHS expected to be unary <- on a
channel. Ready iff go-chan-len > 0; on success, recv-into-var
binds the new value in env.
* Bare recv (:app (:var "<-") [CHAN]) — ready iff len > 0; consumes
the value (discarded).
* :default — deferred until end of walk. Runs if no other case
ready. Absence + no ready case → (:eval-error :select-blocked-
no-default).
New `go-chan-len` accessor on the channel closure-bundle so the
select can peek without consuming.
Subtle bug fix: the :select stmt branch in go-eval-stmt was returning
the old env instead of the env returned by the case body. Assignments
inside select cases (`select { case <-ch: x = 1 ; default: x = 99 }`)
now stick.
Tests (6):
default fires when no case ready
recv case fires when ready
recv-into-var binds the value
send case always ready
picks first ready case (deterministic order in v0)
no default + nothing ready → blocked error
combined with goroutine fan-in
runtime 18/18, total 475/475.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
