rose-ash/plans/abstractions.md

Abstraction Radar — backlog

Maintained by the read-only radar loop (see plans/agent-briefings/radar-loop.md). Detection only — implementation is a separate, coordinated step owned by the relevant subsystem loop, never by radar.

AHA gate to reach Proposed: ≥3 real consumers · all past Phase 2 & API-stable · structurally identical (file:line evidence) · a natural home (usually NOT lib/guest). Anything short → Watching (what's missing) or Rejected (why).

---

Last scan

• Date: 2026-06-07 (radar loop, pass 10)
• Pass 10: commerce/content/events/identity advancing (content 238/238). Probed a shape outside the routing table — guarded lifecycle state machines (mod/lifecycle + identity/membership) → new W6: shared design principle, divergent structure (SX transition-table vs Erlang gen_server), NOT an extraction target. No gate-clearer.
• Date: 2026-06-07 (radar loop, pass 9)
• Pass 9: commerce + content reached Phase 2 (content 162/162). Key find: content built its op log directly on persist/log (backend-injected, append+replay- to-seq) — the live reference exemplar for W4 (see W4). events MONTHLY RRULE, identity OAuth2 auth-code + PKCE, search boolean-filtered ranked. A1 still 6 adopters.
• Date: 2026-06-06 (radar loop, pass 8)
• Pass 8 — fleet expanded by 4 app-domain loops (the briefing's anticipated commerce/identity arrivals, auto-picked up by dynamic discovery). All early-stage, pre-Phase-2 → moving targets, none count toward any gate yet:
  • commerce (Phase 1: api/cart/catalog/price). Its "per-line audit" is a cost breakdown view (api.sx:44), not an append-only decision log → NOT a W4 consumer.
  • events (Phase 1: calendar.sx, RRULE expansion).
  • identity (early: session/token). Defers authZ to acl (token.sx:15) — reinforces W2's "delegate permit? to acl-on-sx" routing; identity = authN, acl = authZ.
  • content (just-started: block.sx). These are the future consumers W2/W3 are waiting on — re-check their per-viewer filters / pagination once each clears Phase 2. No new gate-clearer this pass.
• Pass 7: A1 jumped 4→6 adopters — acl + mod migrated to the shared conformance driver (first app-domain adopters; proves it generalizes past substrates). host-persist closed its blob-adapter blocker (durable storage adapter now landing → W4 migration path opening). search shipped proximity/NEAR; flow + persist quiescent.
• Pass 6: new worktree host-persist (active — building persist's durable host adapter); feed went quiescent (left tmux). acl shipped hardening (+25), fed-sx-m1 at Step 6c. mod loop independently wrote a shared-plumbing note (mod-on-sx.md, 538b8a53) corroborating W4/W5 — folded its claims + home disagreements into W1/W4/W5. No new gate-clearer (audit log still 2 consumers), but consumers are now API-stable.
• Pass 5: search (+highlight/snippet) and fed-sx-m1 (+follower_graph) moved; rest unchanged. Filename census: api×6, fed×3, then schema/rank/query/page/explain/ engine/batch/audit×2. Examined the ×6 api.sx → Rejected (shared name, divergent structure incl. implicit-vs-explicit-state contract). rank/batch/engine all ≤2 + substrate/domain-divergent → no new gate-clearer.
• Pass 4: no churn vs pass 3 (same worktrees/tmux/HEADs/adopters). Swept audit+explain surfaces: acl/mod share an append-only-log shape (→ sharpened W4 with persist/log API evidence) and a proof-explain shape (→ new W5, substrate-bound). No new gate-clearer.
• Pass 3 (earlier today): subsystem set + tmux + A1 adopters (4) all unchanged vs pass 2. Loops advanced: acl shipped Phase 4 federation; search shipped Phase 4 + pagination; feed shipped pagination/threading; mod at Ext 19 (capstone); persist did a worked acl-grants migration (W4). New shape found: offset/limit pagination → folded into W3.
• Subsystem set discovered: loop worktrees acl, erlang, fed-prims, fed-sx-m1, feed, flow, go, kernel, mod, ocaml, persist, radar, ruby, search, sx-vm-extensions; main-repo lib/* incl. merged feed + substrates (apl, common-lisp, datalog, erlang, forth, go, haskell, hyperscript, js, lua, minikanren, ocaml, prolog, scheme, smalltalk, tcl) + lib/guest. Actively looping (tmux): acl, fed-sx-m1, feed, flow, mod, persist, search (+ radar).
• New since pass 1: worktrees kernel (empty/unset — not yet a repo) and ocaml (lib/ocaml/baseline only). Both early-stage, pre–Phase 2 → out of proposal scope.
• Re-enumerate every pass; new loops (e.g. a future commerce/identity) auto-join.

---

Proposed (cleared the gate)

A1 · Adopt the shared conformance driver across subsystems

• Pattern: every subsystem hand-rolls a near-identical conformance.sh (epoch-load → eval → scoreboard emit) and an inline <x>-test name got expected pass/fail counter.
• Consumers (≥3, overwhelming): 15 lib/*/conformance.sh — apl, feed, datalog, flow, mod, lua, erlang, forth, go, common-lisp, haskell, js, ocaml, prolog, smalltalk, tcl.
• Home: lib/guest — the one legitimate exception (the shared driver lib/guest/conformance.sh + lib/guest/conformance.sx already exist; modes dict and counters).
• Status: IN PROGRESS — 6 adopters (pass 7). prolog (dict), haskell (counters), apl (dict), datalog (dict), and acl (dict) + mod (dict), newly migrated this pass — all 3-line exec shims into lib/guest/conformance.sh with a conformance.conf. acl + mod are the first app-domain adopters (not language substrates) — strong evidence the driver generalizes beyond the substrate layer, which was the open question. The apl migration earlier surfaced a latent bug: the old awk extractor under-counted pipeline (40 vs the real 152 assertions); true apl total is 562, not 450 — evidence that adopting the driver also improves correctness.
• Not a target (different harness shape): lua/conformance.sh is a Python runner (lib/lua/conformance.py) that walks real *.lua source files via lua-eval-ast and classifies pass/fail/timeout — it does not run SX deftest suites with a counter/dict scoreboard, so the shared driver does not fit. Excluded, not pending.
• Remaining hand-rolled candidates (~120–220 lines each): common-lisp, erlang, feed, forth, go, js, ocaml, smalltalk, tcl — each its OWN loop's migration when quiescent. (search + lua excluded: different harness shapes — search assembles a Haskell source string, lua walks real *.lua files.)
• Action: each remaining subsystem's OWN loop migrates when quiescent — add a conformance.conf (+ a test-harness.sx preload defining its counters) and replace conformance.sh with the 1-line exec shim (exec bash …/guest/conformance.sh …/conformance.conf "$@"). Recipe template: lib/haskell/conformance.conf (counters) or lib/prolog/conformance.conf (dict). Keep the bash lib/X/conformance.sh entry point so no loop is disrupted.
• Priority: HIGH (15 consumers, low risk, interface-preserving, additive).

---

Watching (real but not yet through the gate)

W1 · Federation scaffold (merge / ingest / backfill / trust-gate)

• FAILS the structural-identity gate (deep-dived 2026-06-06, all 4 read). Consumer count is met (4) but they are superficially similar, not structurally identical — the federated unit and merge op differ fundamentally:

  Subsystem (file)	Federated unit	Merge op	Trust gate	Injected transport
  feed (fed.sx:14,18,40)	activity streams	dedupe by (actor verb object)	none (visibility via permit? separately)	send-fn, fetch-fn
  search (fed.sx:8)	inverted indices	relabel DocId peer*1000+local + union posting lists	none	none (pure merge fn)
  mod (fed.sx:11-14,99)	moderation decisions	advisory-list vs applied-list; bind iff mod/trusted?	yes — runtime list mod/trusted? peer scope	mock outbox / fed-send!
  acl (federation.sx:43,56)	Datalog delegate facts	pull facts, gate by trust/level_covers rule, re-saturate	yes — Datalog rule at query time	transport dict

• The ONLY real commonality is the injection seam, not extractable code: every one says "the real transport is fed-sx's job; inject send-fn/fetch-fn/transport/ outbox and mock it in tests." That is an architectural convention the fleet already follows, and the trust gate (where present) is implemented two incompatible ways (runtime list vs declarative rule). No shared merge, no shared trust mechanism.

• Disposition: do NOT extract a shared "federation lib." When fed-sx ships its real transport, these 4 become its consumers (wiring send-fn/fetch-fn/transport to it) — that work belongs to each subsystem's loop + the fed-sx loop, not a cross-cutting extraction. Stop re-proposing on the shared name. Home: fed-sx.

• Narrower sub-claim (mod note, pass 6): mod asserts the fed trust/outbox shape specifically shares between mod+acl. Radar evidence above shows the trust gate mechanism diverges (mod runtime-list vs acl Datalog-rule); the outbox/propagation envelope may share, but that's 2 consumers (mod, acl) on different substrates → Watching. Resolve at the architecture-merge point if it survives to a 3rd consumer.

W2 · Per-viewer visibility / permission filter

• 2 shipped consumers, same shape — filter <injected-permit> <ranked/candidate stream>:
  • feed/lib/feed/acl.sx:27 feed/visible = (feed/filter stream (fn (a) (permit? viewer a))), capstone at :34 (stream → ACL → rank → top-N). permit? injected, sig (viewer activity)→bool.
  • search/lib/search/fed.sx:16 aclFilter permit docs = filter permit docs; topNTfIdfAcl n permit ts idx = take n (aclFilter permit (rankTfIdf ts idx)). permit injected, sig DocId→Bool (viewer baked in by caller).
• NOT a consumer: mod/lib/mod/policy.sx is moderation policy (reviewer actions), no per-viewer read filter. So mod won't be the 3rd.
• Missing: (a) only 2 consumers, need ≥3; (b) the two interfaces diverge — feed passes (viewer, item), search bakes the viewer in — so any shared form must pick a convention; (c) both already inject the predicate, and the filter body is literally one line (filter permit xs). Leaning toward: the predicate's home is acl-on-sx (permit?), and the one-line filter is too thin to extract.
• Home when ripe: delegate permit? to acl-on-sx; do NOT extract the filter. Re-check if a 3rd genuine per-viewer read filter ships (e.g. events/commerce).

W3 · Collection helpers (group-by, dedupe-by-key, stable top-N, distinct-order, offset/limit page)

• feed built all of these on APL primitives. search/commerce/events will want group-by / top-N.
• NEW (2026-06-06): offset/limit pagination shipped in 2 subsystems, identical shape take limit (drop offset xs):
  • feed/lib/feed/page.sx:9 feed/page (offset/limit window over a stream).
  • search/lib/search/page.sx:9 paginate off lim docs = take lim (drop off docs).
  • NOT a 3rd: persist/lib/persist/query.sx:5 has a since-cursor for incremental log consumption — resumable-stream semantics, not result windowing. Different shape.
  • feed also has cursor-by-:at recency pagination (page.sx:21-44); search has no cursor. So only the plain offset/limit window is shared, and it is a literal 1-liner.
• Missing: ≥3 stable consumers; AND every item here is collection math that belongs in the substrate (APL/Haskell already expose grade/sort/unique/take/drop), not a shared lib. A 1-line take/drop window is far below the extraction threshold. Watch; revisit only if a non-substrate subsystem needs the same windowing without take/drop.

W4 · In-memory store fakes → persist-on-sx

• Not an abstraction to extract — a migration target. Every subsystem fakes its store with a mutable list (feed/-log, flow store, mod audit, …).

• Owner: persist-on-sx (in progress). Tracked there, listed here for visibility.

• Concrete instance (file:line, found pass 4): the append-only decision/audit log. acl/lib/acl/audit.sx and mod/lib/mod/audit.sx are the SAME hand-rolled shape, and persist/lib/persist/log.sx (the persist log facet) already implements it durably:

  role	acl/audit.sx	mod/audit.sx	persist/log.sx (target)
  log var	acl-audit-log :9	mod/*audit-log* :10	backend stream
  monotonic seq	acl-audit-seq :10	mod/*audit-seq* :11	per-stream high-water :1
  append (auto-seq)	acl-audit-decide!	commit :32	persist/append :17
  count	acl-audit-count :51	mod/audit-count :44	persist/count :12
  read-all oldest-first	snapshot/tail :73	mod/audit-all :43	persist/read :29
  read seq≥from	—	by-seq	persist/read-from :31

  Both deliberately use a monotonic seq with no wall-clock (deterministic/testable) — identical to persist/log's design. Action when persist's host adapter lands: acl + mod loops swap their in-memory log for persist/log. 2 consumers today; not a new lib — the home already exists. Belongs to acl/mod loops × persist loop, not an extraction.

• Cross-loop corroboration (pass 6): the mod loop independently reached the same conclusion — mod/plans/mod-on-sx.md (commit 538b8a53): "mod-sx (Prolog) and acl-sx (Datalog) converged on the same module shape … only the audit log + fed trust/outbox shapes truly share; extract at the architecture-merge point, refactoring both consumers atomically, not unilaterally from a loop branch." Confirms the shape AND the do-not-extract-unilaterally stance.

• Home disagreement to resolve at merge: mod's note proposes lifting the audit-log primitives into lib/guest/. Radar routing disagrees: a durable append-only log is a persist-on-sx concern (the log facet already exists), not language-impl plumbing. Hold the line — lib/guest is lexer/parser/AST/HM/test-runner, not an event log.

• Migration is becoming concrete: new host-persist loop (worktree + tmux, pass 6) is building the durable-storage host adapter persist was blocked on — once it lands, acl/mod can actually swap to persist/log.

• LIVE REFERENCE EXEMPLAR (pass 9): content already does it right. content (Phase 2 complete, 162/162) built its op log directly on persist/log instead of faking it — content/lib/content/store.sx: backend injected via (persist/open) ("content knows nothing about which backend", :10); append op as event persist/append b (content/-stream doc-id) … (:20); read persist/read (:36); persist/last-seq (:47); version = replay op stream up to a seq (filter persist/event-seq ev <= seq, :61). "The op log is the source of truth … the materialised doc is a cache, never primary state." This proves the W4 target is real, not hypothetical: acl + mod's hand-rolled monotonic-seq logs should adopt exactly content's persist/log pattern. W4 now = 1 correct exemplar (content) + 2 fakes to migrate (acl, mod).

W5 · Proof-tree explanation over a logic-program derivation

• acl/lib/acl/explain.sx (reconstructs a canonical proof by goal-directed search over a saturated Datalog db) and mod/lib/mod/explain.sx (renders a Prolog-style proof tree goal-by-goal with proved/unproved marks + unification bindings) are the same idea.
• Missing / disposition: only 2 consumers, and they sit on different substrates (acl→lib/datalog, mod→lib/prolog). Proof reconstruction/rendering is logic-engine machinery → it belongs in each substrate (datalog/prolog), not a shared app lib. Watch; revisit only if a 3rd logic-backed subsystem reimplements proof explanation.
• Cross-loop note (pass 6): mod's note calls mod/proof-goals (re-query-each-goal) generic and proposes lifting it into lib/guest/. Radar caveat: proof-tree reconstruction is engine-agnostic logic machinery, but lib/guest is for lexer/parser/AST/HM/match/test-runner — a logic-engine proof helper is a poor fit there. If genuinely shared by ≥3 engines, a lib/logic-style substrate helper is the better home than lib/guest. Still 2 consumers → stays Watching either way.

---

W6 · Guarded lifecycle state machine (illegal transition = explicit error)

• Recurs as a design principle, NOT a shared structure (found pass 10):
  • mod/lib/mod/lifecycle.sx — pure SX: immutable case {:state :error :history …}, explicit transition table mod/lc-transitions (:31), illegal transition returns the case unchanged with :error set. States open→triaged→decided→appealed→final.
  • identity/lib/identity/membership.sx — an Erlang gen_server fragment (identity runs on erlang-on-sx): a receive loop with case find(...) of … {error, St} guards. States none→pending→active→lapsed→revoked.
• Both share the guideline ("invalid transitions are explicit errors, never silent no-ops") but implement it substrate-idiomatically — SX transition-table over immutable values vs an Erlang process loop with per-message case guards. Same W1/api.sx trap: shared idea, divergent structure.
• Disposition: not an extraction target — the FSM mechanism is ~10 substrate-specific lines; the value is in each domain's state graph, not the plumbing. At most a design guideline ("model lifecycle as a guarded FSM with explicit-error transitions"). Watch whether commerce-checkout / events-booking add their own — if so it confirms the guideline, still not a lib. Do not propose extracting a shared state-machine lib.

Rejected (considered, declined — do not re-propose)

• "Continuous auto-implementing abstractor loop." Rejected at design time: an agent writing across lib/<x>/** breaks the worktree isolation that makes the fleet safe, and is rewarded for manufacturing premature/wrong abstractions. The radar is read-only by design. (This file is the alternative.)
• Shared api.sx "public boundary" module (×6). Rejected pass 4-5: every subsystem has an api.sx (acl, feed, flow, mod, persist, search — a 100% filename match), but it is a naming convention for the public entry point, not a shared structure. They disagree on the most basic contract: acl/feed use implicit module state (acl/api.sx "implicit current db", feed/api.sx "single mutable log") while persist/api.sx threads an explicit backend as every call's first arg; flow's api builds a Scheme env, search's api concatenates a Haskell source string, mod's is a lifecycle state-machine façade (17 defs vs persist's 1). Same role, no common shape — the W1 coincidental-resemblance trap. Do not re-propose on the filename.
• Dumping app-domain plumbing into lib/guest. Rejected: lib/guest is for language-implementation plumbing. App patterns route to acl/fed-sx/persist/ substrate/host instead (see the routing rule in the briefing).