rose-ash/plans/abstractions.md

Abstraction Radar — backlog

Maintained by the read-only radar loop (see plans/agent-briefings/radar-loop.md). Detection only — implementation is a separate, coordinated step owned by the relevant subsystem loop, never by radar.

AHA gate to reach Proposed: ≥3 real consumers · all past Phase 2 & API-stable · structurally identical (file:line evidence) · a natural home (usually NOT lib/guest). Anything short → Watching (what's missing) or Rejected (why).

---

Last scan

• Date: 2026-06-07 (radar loop, pass 15)
• Pass 15: added the scanning-method note above after query.sx again proved to be merged-lib copies (lib/prolog + lib/persist in every worktree). Corrected census surfaced wire×2 (content+mod) → Rejected (shared role, divergent structure: generic SX serializer vs bespoke pipe-format under a Prolog-env string-prim constraint). events↔ commerce integration appeared (paid tickets); acl/mod/search quiescent ~7 passes (now API-stable). No new gate-clearer.
• Date: 2026-06-07 (radar loop, pass 14)
• Pass 14: filename census flagged snapshot×?? — but the */lib/persist/snapshot.sx copies are just the merged lib/persist in each worktree, NOT consumers (same artifact as lib/feed/rank.sx everywhere). The one distinct file, content/snapshot.sx, reimplements persist's projection-checkpoint on raw KV instead of using persist/snapshot → new W7 (persist-adoption nudge). audit×3 = the W4 fakes (acl/mod/identity), known.
• Date: 2026-06-07 (radar loop, pass 13)
• Pass 13 — honest re-test, no gate-clearer. Re-tested the two longest-waiting gates against the maturing app-domain loops: W2 (per-viewer visibility) still 2 consumers (feed, search) — commerce/content/events/identity add no per-viewer read filter; W3 (pagination) still 2 (feed, search) — content/page.sx is an HTML wrapper, not pagination (filename collision, noted in W3). Incremental churn only elsewhere.
• Date: 2026-06-07 (radar loop, pass 12)
• Pass 12: events shipped transactional booking on persist (3rd live persist consumer) using persist/append-expect (optimistic-concurrency CAS, lock-free capacity safety). W4 ledger now shows a persist feature-ladder append → append-once → append-expect that the hand-rolled fakes can't match. No new candidate; W4 reinforced.
• Date: 2026-06-07 (radar loop, pass 11)
• Pass 11 — W4 sharpened with a consumer ledger. commerce built an order ledger on persist (2nd live exemplar; uses persist/append-once for webhook idempotency) and identity a grant audit ledger (in-memory Erlang fake, gated on an Erlang↔persist bridge). The append-only monotonic-seq event-log pattern is now validated across 4 domains, 2 live on persist + 3 fakes flagged for adoption. See W4 table.
• Date: 2026-06-07 (radar loop, pass 10)
• Pass 10: commerce/content/events/identity advancing (content 238/238). Probed a shape outside the routing table — guarded lifecycle state machines (mod/lifecycle + identity/membership) → new W6: shared design principle, divergent structure (SX transition-table vs Erlang gen_server), NOT an extraction target. No gate-clearer.
• Date: 2026-06-07 (radar loop, pass 9)
• Pass 9: commerce + content reached Phase 2 (content 162/162). Key find: content built its op log directly on persist/log (backend-injected, append+replay- to-seq) — the live reference exemplar for W4 (see W4). events MONTHLY RRULE, identity OAuth2 auth-code + PKCE, search boolean-filtered ranked. A1 still 6 adopters.
• Date: 2026-06-06 (radar loop, pass 8)
• Pass 8 — fleet expanded by 4 app-domain loops (the briefing's anticipated commerce/identity arrivals, auto-picked up by dynamic discovery). All early-stage, pre-Phase-2 → moving targets, none count toward any gate yet:
  • commerce (Phase 1: api/cart/catalog/price). Its "per-line audit" is a cost breakdown view (api.sx:44), not an append-only decision log → NOT a W4 consumer.
  • events (Phase 1: calendar.sx, RRULE expansion).
  • identity (early: session/token). Defers authZ to acl (token.sx:15) — reinforces W2's "delegate permit? to acl-on-sx" routing; identity = authN, acl = authZ.
  • content (just-started: block.sx). These are the future consumers W2/W3 are waiting on — re-check their per-viewer filters / pagination once each clears Phase 2. No new gate-clearer this pass.
• Pass 7: A1 jumped 4→6 adopters — acl + mod migrated to the shared conformance driver (first app-domain adopters; proves it generalizes past substrates). host-persist closed its blob-adapter blocker (durable storage adapter now landing → W4 migration path opening). search shipped proximity/NEAR; flow + persist quiescent.
• Pass 6: new worktree host-persist (active — building persist's durable host adapter); feed went quiescent (left tmux). acl shipped hardening (+25), fed-sx-m1 at Step 6c. mod loop independently wrote a shared-plumbing note (mod-on-sx.md, 538b8a53) corroborating W4/W5 — folded its claims + home disagreements into W1/W4/W5. No new gate-clearer (audit log still 2 consumers), but consumers are now API-stable.
• Pass 5: search (+highlight/snippet) and fed-sx-m1 (+follower_graph) moved; rest unchanged. Filename census: api×6, fed×3, then schema/rank/query/page/explain/ engine/batch/audit×2. Examined the ×6 api.sx → Rejected (shared name, divergent structure incl. implicit-vs-explicit-state contract). rank/batch/engine all ≤2 + substrate/domain-divergent → no new gate-clearer.
• Pass 4: no churn vs pass 3 (same worktrees/tmux/HEADs/adopters). Swept audit+explain surfaces: acl/mod share an append-only-log shape (→ sharpened W4 with persist/log API evidence) and a proof-explain shape (→ new W5, substrate-bound). No new gate-clearer.
• Pass 3 (earlier today): subsystem set + tmux + A1 adopters (4) all unchanged vs pass 2. Loops advanced: acl shipped Phase 4 federation; search shipped Phase 4 + pagination; feed shipped pagination/threading; mod at Ext 19 (capstone); persist did a worked acl-grants migration (W4). New shape found: offset/limit pagination → folded into W3.
• Subsystem set discovered: loop worktrees acl, erlang, fed-prims, fed-sx-m1, feed, flow, go, kernel, mod, ocaml, persist, radar, ruby, search, sx-vm-extensions; main-repo lib/* incl. merged feed + substrates (apl, common-lisp, datalog, erlang, forth, go, haskell, hyperscript, js, lua, minikanren, ocaml, prolog, scheme, smalltalk, tcl) + lib/guest. Actively looping (tmux): acl, fed-sx-m1, feed, flow, mod, persist, search (+ radar).
• New since pass 1: worktrees kernel (empty/unset — not yet a repo) and ocaml (lib/ocaml/baseline only). Both early-stage, pre–Phase 2 → out of proposal scope.
• Re-enumerate every pass; new loops (e.g. a future commerce/identity) auto-join.

Scanning-method note (learned the hard way, passes 5/12/14/15): a filename census for cross-subsystem recurrence MUST restrict to each subsystem's OWN namespace — X/lib/X/*.sx — never X/lib/*/. The merged substrate libs (lib/prolog, lib/persist, lib/feed, lib/datalog, …) are checked out inside every worktree, so a naive census reports e.g. query.sx/snapshot.sx/rank.sx ×N as phantom recurrences that are really one merged file copied N times. Correct one-liner: for w in <subsystems>; do for f in $w/lib/$w/*.sx; do basename $f .sx; done; done | sort | uniq -c | sort -rn.

---

Proposed (cleared the gate)

A1 · Adopt the shared conformance driver across subsystems

• Pattern: every subsystem hand-rolls a near-identical conformance.sh (epoch-load → eval → scoreboard emit) and an inline <x>-test name got expected pass/fail counter.
• Consumers (≥3, overwhelming): 15 lib/*/conformance.sh — apl, feed, datalog, flow, mod, lua, erlang, forth, go, common-lisp, haskell, js, ocaml, prolog, smalltalk, tcl.
• Home: lib/guest — the one legitimate exception (the shared driver lib/guest/conformance.sh + lib/guest/conformance.sx already exist; modes dict and counters).
• Status: IN PROGRESS — 6 adopters (pass 7). prolog (dict), haskell (counters), apl (dict), datalog (dict), and acl (dict) + mod (dict), newly migrated this pass — all 3-line exec shims into lib/guest/conformance.sh with a conformance.conf. acl + mod are the first app-domain adopters (not language substrates) — strong evidence the driver generalizes beyond the substrate layer, which was the open question. The apl migration earlier surfaced a latent bug: the old awk extractor under-counted pipeline (40 vs the real 152 assertions); true apl total is 562, not 450 — evidence that adopting the driver also improves correctness.
• Not a target (different harness shape): lua/conformance.sh is a Python runner (lib/lua/conformance.py) that walks real *.lua source files via lua-eval-ast and classifies pass/fail/timeout — it does not run SX deftest suites with a counter/dict scoreboard, so the shared driver does not fit. Excluded, not pending.
• Remaining hand-rolled candidates (~120–220 lines each): common-lisp, erlang, feed, forth, go, js, ocaml, smalltalk, tcl — each its OWN loop's migration when quiescent. (search + lua excluded: different harness shapes — search assembles a Haskell source string, lua walks real *.lua files.)
• Action: each remaining subsystem's OWN loop migrates when quiescent — add a conformance.conf (+ a test-harness.sx preload defining its counters) and replace conformance.sh with the 1-line exec shim (exec bash …/guest/conformance.sh …/conformance.conf "$@"). Recipe template: lib/haskell/conformance.conf (counters) or lib/prolog/conformance.conf (dict). Keep the bash lib/X/conformance.sh entry point so no loop is disrupted.
• Priority: HIGH (15 consumers, low risk, interface-preserving, additive).

---

Watching (real but not yet through the gate)

W1 · Federation scaffold (merge / ingest / backfill / trust-gate)

• FAILS the structural-identity gate (deep-dived 2026-06-06, all 4 read). Consumer count is met (4) but they are superficially similar, not structurally identical — the federated unit and merge op differ fundamentally:

  Subsystem (file)	Federated unit	Merge op	Trust gate	Injected transport
  feed (fed.sx:14,18,40)	activity streams	dedupe by (actor verb object)	none (visibility via permit? separately)	send-fn, fetch-fn
  search (fed.sx:8)	inverted indices	relabel DocId peer*1000+local + union posting lists	none	none (pure merge fn)
  mod (fed.sx:11-14,99)	moderation decisions	advisory-list vs applied-list; bind iff mod/trusted?	yes — runtime list mod/trusted? peer scope	mock outbox / fed-send!
  acl (federation.sx:43,56)	Datalog delegate facts	pull facts, gate by trust/level_covers rule, re-saturate	yes — Datalog rule at query time	transport dict

• The ONLY real commonality is the injection seam, not extractable code: every one says "the real transport is fed-sx's job; inject send-fn/fetch-fn/transport/ outbox and mock it in tests." That is an architectural convention the fleet already follows, and the trust gate (where present) is implemented two incompatible ways (runtime list vs declarative rule). No shared merge, no shared trust mechanism.

• Disposition: do NOT extract a shared "federation lib." When fed-sx ships its real transport, these 4 become its consumers (wiring send-fn/fetch-fn/transport to it) — that work belongs to each subsystem's loop + the fed-sx loop, not a cross-cutting extraction. Stop re-proposing on the shared name. Home: fed-sx.

• Narrower sub-claim (mod note, pass 6): mod asserts the fed trust/outbox shape specifically shares between mod+acl. Radar evidence above shows the trust gate mechanism diverges (mod runtime-list vs acl Datalog-rule); the outbox/propagation envelope may share, but that's 2 consumers (mod, acl) on different substrates → Watching. Resolve at the architecture-merge point if it survives to a 3rd consumer.

W2 · Per-viewer visibility / permission filter

• 2 shipped consumers, same shape — filter <injected-permit> <ranked/candidate stream>:
  • feed/lib/feed/acl.sx:27 feed/visible = (feed/filter stream (fn (a) (permit? viewer a))), capstone at :34 (stream → ACL → rank → top-N). permit? injected, sig (viewer activity)→bool.
  • search/lib/search/fed.sx:16 aclFilter permit docs = filter permit docs; topNTfIdfAcl n permit ts idx = take n (aclFilter permit (rankTfIdf ts idx)). permit injected, sig DocId→Bool (viewer baked in by caller).
• NOT a consumer: mod/lib/mod/policy.sx is moderation policy (reviewer actions), no per-viewer read filter. So mod won't be the 3rd.
• Missing: (a) only 2 consumers, need ≥3; (b) the two interfaces diverge — feed passes (viewer, item), search bakes the viewer in — so any shared form must pick a convention; (c) both already inject the predicate, and the filter body is literally one line (filter permit xs). Leaning toward: the predicate's home is acl-on-sx (permit?), and the one-line filter is too thin to extract.
• Home when ripe: delegate permit? to acl-on-sx; do NOT extract the filter. Re-check if a 3rd genuine per-viewer read filter ships (e.g. events/commerce).

W3 · Collection helpers (group-by, dedupe-by-key, stable top-N, distinct-order, offset/limit page)

• feed built all of these on APL primitives. search/commerce/events will want group-by / top-N.
• NEW (2026-06-06): offset/limit pagination shipped in 2 subsystems, identical shape take limit (drop offset xs):
  • feed/lib/feed/page.sx:9 feed/page (offset/limit window over a stream).
  • search/lib/search/page.sx:9 paginate off lim docs = take lim (drop off docs).
  • NOT a 3rd: persist/lib/persist/query.sx:5 has a since-cursor for incremental log consumption — resumable-stream semantics, not result windowing. Different shape.
  • feed also has cursor-by-:at recency pagination (page.sx:21-44); search has no cursor. So only the plain offset/limit window is shared, and it is a literal 1-liner.
• Missing: ≥3 stable consumers; AND every item here is collection math that belongs in the substrate (APL/Haskell already expose grade/sort/unique/take/drop), not a shared lib. A 1-line take/drop window is far below the extraction threshold. Watch; revisit only if a non-substrate subsystem needs the same windowing without take/drop.
• Filename-collision caution (pass 13): content/lib/content/page.sx is an HTML page wrapper (full HTML5 doc), NOT pagination — do not count it as a 3rd pagination consumer. page.sx now means two unrelated things across the fleet. Re-tested pass 13: pagination still only feed + search (2).

W4 · In-memory store fakes → persist-on-sx

• Not an abstraction to extract — a migration target. Every subsystem fakes its store with a mutable list (feed/-log, flow store, mod audit, …).

• Owner: persist-on-sx (in progress). Tracked there, listed here for visibility.

• Concrete instance (file:line, found pass 4): the append-only decision/audit log. acl/lib/acl/audit.sx and mod/lib/mod/audit.sx are the SAME hand-rolled shape, and persist/lib/persist/log.sx (the persist log facet) already implements it durably:

  role	acl/audit.sx	mod/audit.sx	persist/log.sx (target)
  log var	acl-audit-log :9	mod/*audit-log* :10	backend stream
  monotonic seq	acl-audit-seq :10	mod/*audit-seq* :11	per-stream high-water :1
  append (auto-seq)	acl-audit-decide!	commit :32	persist/append :17
  count	acl-audit-count :51	mod/audit-count :44	persist/count :12
  read-all oldest-first	snapshot/tail :73	mod/audit-all :43	persist/read :29
  read seq≥from	—	by-seq	persist/read-from :31

  Both deliberately use a monotonic seq with no wall-clock (deterministic/testable) — identical to persist/log's design. Action when persist's host adapter lands: acl + mod loops swap their in-memory log for persist/log. 2 consumers today; not a new lib — the home already exists. Belongs to acl/mod loops × persist loop, not an extraction.

• Cross-loop corroboration (pass 6): the mod loop independently reached the same conclusion — mod/plans/mod-on-sx.md (commit 538b8a53): "mod-sx (Prolog) and acl-sx (Datalog) converged on the same module shape … only the audit log + fed trust/outbox shapes truly share; extract at the architecture-merge point, refactoring both consumers atomically, not unilaterally from a loop branch." Confirms the shape AND the do-not-extract-unilaterally stance.

• Home disagreement to resolve at merge: mod's note proposes lifting the audit-log primitives into lib/guest/. Radar routing disagrees: a durable append-only log is a persist-on-sx concern (the log facet already exists), not language-impl plumbing. Hold the line — lib/guest is lexer/parser/AST/HM/test-runner, not an event log.

• Migration is becoming concrete: new host-persist loop (worktree + tmux, pass 6) is building the durable-storage host adapter persist was blocked on — once it lands, acl/mod can actually swap to persist/log.

• LIVE REFERENCE EXEMPLAR (pass 9): content already does it right. content (Phase 2 complete, 162/162) built its op log directly on persist/log instead of faking it — content/lib/content/store.sx: backend injected via (persist/open) ("content knows nothing about which backend", :10); append op as event persist/append b (content/-stream doc-id) … (:20); read persist/read (:36); persist/last-seq (:47); version = replay op stream up to a seq (filter persist/event-seq ev <= seq, :61). "The op log is the source of truth … the materialised doc is a cache, never primary state." This proves the W4 target is real, not hypothetical: acl + mod's hand-rolled monotonic-seq logs should adopt exactly content's persist/log pattern.

• Consumer ledger of the append-only monotonic-seq event log (pass 11):

  consumer	what	backing	note
  content (store.sx)	doc op log	persist/log ✓ live	plain append + replay-to-seq
  commerce (ledger.sx)	order ledger	persist/log ✓ live	persist/append-once — idempotent, webhook-replay-safe :40,58
  events (booking.sx)	booking roster	persist/log ✓ live	persist/append-expect — optimistic-concurrency CAS, capacity-safe, lock-free
  acl (audit.sx)	decision log	in-memory fake (SX)	migrate directly when host adapter lands
  mod (audit.sx)	decision log	in-memory fake (SX)	migrate directly
  identity (audit.sx)	grant ledger	in-memory fake (Erlang)	{Seq,Subject,Action}; needs an Erlang↔persist bridge first — author scoped it out until persist lands ("queryable semantics identical")

• Two takeaways: (1) the pattern is validated across domains — CRDT doc ops, financial orders, event bookings, rule decisions, OAuth grants all reduce to the same append-only monotonic-seq stream; (2) migrating to persist/log is strictly better than the fakes — persist exposes a feature ladder the fakes don't have: append (content) → append-once/idempotency (commerce) → append-expect/optimistic- concurrency (events). Every fake would have to reinvent a weaker version of these. This is an adoption item (the home already exists), NOT a new extraction — owned by persist/host-persist × each consumer loop. The SX fakes (acl, mod) migrate directly; the Erlang fake (identity) is gated on an Erlang↔persist bridge.

W5 · Proof-tree explanation over a logic-program derivation

• acl/lib/acl/explain.sx (reconstructs a canonical proof by goal-directed search over a saturated Datalog db) and mod/lib/mod/explain.sx (renders a Prolog-style proof tree goal-by-goal with proved/unproved marks + unification bindings) are the same idea.
• Missing / disposition: only 2 consumers, and they sit on different substrates (acl→lib/datalog, mod→lib/prolog). Proof reconstruction/rendering is logic-engine machinery → it belongs in each substrate (datalog/prolog), not a shared app lib. Watch; revisit only if a 3rd logic-backed subsystem reimplements proof explanation.
• Cross-loop note (pass 6): mod's note calls mod/proof-goals (re-query-each-goal) generic and proposes lifting it into lib/guest/. Radar caveat: proof-tree reconstruction is engine-agnostic logic machinery, but lib/guest is for lexer/parser/AST/HM/match/test-runner — a logic-engine proof helper is a poor fit there. If genuinely shared by ≥3 engines, a lib/logic-style substrate helper is the better home than lib/guest. Still 2 consumers → stays Watching either way.

---

W7 · Snapshot/projection-checkpoint reimplemented vs persist/snapshot (delegate)

• persist/lib/persist/snapshot.sx already provides a generic projection checkpoint: store {:value :seq} in the kv facet under a namespaced key; the headline property is snapshot + tail == full replay (pure, clock-free).
• content/lib/content/snapshot.sx reimplements that same pattern on raw persist KV rather than delegating: persist/kv-put b (content/-snap-key doc-id) {:doc … :seq seq} (:20), persist/kv-has?/kv-get (:27-28), and its own tail-replay (:53-59). It never calls persist/snapshot-*. content's doc-materialisation is a projection fold over its op stream — exactly what persist/snapshot checkpoints generically.
• Disposition: persist-adoption nudge (like W4): content could delegate to persist/snapshot (its projection = "fold ops → doc"), dropping the duplicated KV+replay code. Home already exists → NOT an extraction; owned by content × persist loops. Only 1 reinventor today; watch whether commerce/events/identity also hand-roll a snapshot on raw KV instead of using the facet (would strengthen the nudge). NB timeline: unclear if persist/snapshot predated content's — flag, don't blame.

W6 · Guarded lifecycle state machine (illegal transition = explicit error)

• Recurs as a design principle, NOT a shared structure (found pass 10):
  • mod/lib/mod/lifecycle.sx — pure SX: immutable case {:state :error :history …}, explicit transition table mod/lc-transitions (:31), illegal transition returns the case unchanged with :error set. States open→triaged→decided→appealed→final.
  • identity/lib/identity/membership.sx — an Erlang gen_server fragment (identity runs on erlang-on-sx): a receive loop with case find(...) of … {error, St} guards. States none→pending→active→lapsed→revoked.
• Both share the guideline ("invalid transitions are explicit errors, never silent no-ops") but implement it substrate-idiomatically — SX transition-table over immutable values vs an Erlang process loop with per-message case guards. Same W1/api.sx trap: shared idea, divergent structure.
• Disposition: not an extraction target — the FSM mechanism is ~10 substrate-specific lines; the value is in each domain's state graph, not the plumbing. At most a design guideline ("model lifecycle as a guarded FSM with explicit-error transitions"). Watch whether commerce-checkout / events-booking add their own — if so it confirms the guideline, still not a lib. Do not propose extracting a shared state-machine lib.

Rejected (considered, declined — do not re-propose)

• "Continuous auto-implementing abstractor loop." Rejected at design time: an agent writing across lib/<x>/** breaks the worktree isolation that makes the fleet safe, and is rewarded for manufacturing premature/wrong abstractions. The radar is read-only by design. (This file is the alternative.)
• Shared api.sx "public boundary" module (×6). Rejected pass 4-5: every subsystem has an api.sx (acl, feed, flow, mod, persist, search — a 100% filename match), but it is a naming convention for the public entry point, not a shared structure. They disagree on the most basic contract: acl/feed use implicit module state (acl/api.sx "implicit current db", feed/api.sx "single mutable log") while persist/api.sx threads an explicit backend as every call's first arg; flow's api builds a Scheme env, search's api concatenates a Haskell source string, mod's is a lifecycle state-machine façade (17 defs vs persist's 1). Same role, no common shape — the W1 coincidental-resemblance trap. Do not re-propose on the filename.
• Shared wire.sx "serialization" module (×2). Rejected pass 15: content + mod both have a wire.sx, but content/wire.sx uses the generic SX serializer (serialize/parse, full-fidelity round-trip) while mod/wire.sx is a bespoke versioned pipe-delimited line (subset of fields, split hand-built over slice/len because mod's Prolog-loaded env strips string prims). Shared role (wire format), divergent structure + substrate constraint → not a candidate; the SX serializer is already the shared tool for SX-substrate subsystems, and mod can't use it. (Same family as the api.sx rejection above.)
• Dumping app-domain plumbing into lib/guest. Rejected: lib/guest is for language-implementation plumbing. App patterns route to acl/fed-sx/persist/ substrate/host instead (see the routing rule in the briefing).