rose-ash/plans/abstractions.md

Abstraction Radar — backlog

Maintained by the read-only radar loop (see plans/agent-briefings/radar-loop.md). Detection only — implementation is a separate, coordinated step owned by the relevant subsystem loop, never by radar.

AHA gate to reach Proposed: ≥3 real consumers · all past Phase 2 & API-stable · structurally identical (file:line evidence) · a natural home (usually NOT lib/guest). Anything short → Watching (what's missing) or Rejected (why).

---

Last scan

• Date: 2026-06-06 (radar loop, pass 4)
• Pass 4: no churn vs pass 3 (same worktrees/tmux/HEADs/adopters). Swept audit+explain surfaces: acl/mod share an append-only-log shape (→ sharpened W4 with persist/log API evidence) and a proof-explain shape (→ new W5, substrate-bound). No new gate-clearer.
• Pass 3 (earlier today): subsystem set + tmux + A1 adopters (4) all unchanged vs pass 2. Loops advanced: acl shipped Phase 4 federation; search shipped Phase 4 + pagination; feed shipped pagination/threading; mod at Ext 19 (capstone); persist did a worked acl-grants migration (W4). New shape found: offset/limit pagination → folded into W3.
• Subsystem set discovered: loop worktrees acl, erlang, fed-prims, fed-sx-m1, feed, flow, go, kernel, mod, ocaml, persist, radar, ruby, search, sx-vm-extensions; main-repo lib/* incl. merged feed + substrates (apl, common-lisp, datalog, erlang, forth, go, haskell, hyperscript, js, lua, minikanren, ocaml, prolog, scheme, smalltalk, tcl) + lib/guest. Actively looping (tmux): acl, fed-sx-m1, feed, flow, mod, persist, search (+ radar).
• New since pass 1: worktrees kernel (empty/unset — not yet a repo) and ocaml (lib/ocaml/baseline only). Both early-stage, pre–Phase 2 → out of proposal scope.
• Re-enumerate every pass; new loops (e.g. a future commerce/identity) auto-join.

---

Proposed (cleared the gate)

A1 · Adopt the shared conformance driver across subsystems

• Pattern: every subsystem hand-rolls a near-identical conformance.sh (epoch-load → eval → scoreboard emit) and an inline <x>-test name got expected pass/fail counter.
• Consumers (≥3, overwhelming): 15 lib/*/conformance.sh — apl, feed, datalog, flow, mod, lua, erlang, forth, go, common-lisp, haskell, js, ocaml, prolog, smalltalk, tcl.
• Home: lib/guest — the one legitimate exception (the shared driver lib/guest/conformance.sh + lib/guest/conformance.sx already exist; modes dict and counters).
• Status: IN PROGRESS — 4 adopters. prolog (dict), haskell (counters), apl (dict), and datalog (dict), newly migrated this pass — all now 3-line exec shims into lib/guest/conformance.sh with a conformance.conf (lib/datalog/conformance.conf: 11 PRELOADS + 11 SUITES). The apl migration earlier surfaced a latent bug: the old awk extractor under-counted pipeline (40 vs the real 152 assertions); true apl total is 562, not 450 — evidence that adopting the driver also improves correctness.
• Not a target (different harness shape): lua/conformance.sh is a Python runner (lib/lua/conformance.py) that walks real *.lua source files via lua-eval-ast and classifies pass/fail/timeout — it does not run SX deftest suites with a counter/dict scoreboard, so the shared driver does not fit. Excluded, not pending.
• Remaining hand-rolled candidates (~120–220 lines each): common-lisp, erlang, feed, forth, go, js, ocaml, smalltalk, tcl — each its OWN loop's migration when quiescent.
• Action: each remaining subsystem's OWN loop migrates when quiescent — add a conformance.conf (+ a test-harness.sx preload defining its counters) and replace conformance.sh with the 1-line exec shim (exec bash …/guest/conformance.sh …/conformance.conf "$@"). Recipe template: lib/haskell/conformance.conf (counters) or lib/prolog/conformance.conf (dict). Keep the bash lib/X/conformance.sh entry point so no loop is disrupted.
• Priority: HIGH (15 consumers, low risk, interface-preserving, additive).

---

Watching (real but not yet through the gate)

W1 · Federation scaffold (merge / ingest / backfill / trust-gate)

• FAILS the structural-identity gate (deep-dived 2026-06-06, all 4 read). Consumer count is met (4) but they are superficially similar, not structurally identical — the federated unit and merge op differ fundamentally:

  Subsystem (file)	Federated unit	Merge op	Trust gate	Injected transport
  feed (fed.sx:14,18,40)	activity streams	dedupe by (actor verb object)	none (visibility via permit? separately)	send-fn, fetch-fn
  search (fed.sx:8)	inverted indices	relabel DocId peer*1000+local + union posting lists	none	none (pure merge fn)
  mod (fed.sx:11-14,99)	moderation decisions	advisory-list vs applied-list; bind iff mod/trusted?	yes — runtime list mod/trusted? peer scope	mock outbox / fed-send!
  acl (federation.sx:43,56)	Datalog delegate facts	pull facts, gate by trust/level_covers rule, re-saturate	yes — Datalog rule at query time	transport dict

• The ONLY real commonality is the injection seam, not extractable code: every one says "the real transport is fed-sx's job; inject send-fn/fetch-fn/transport/ outbox and mock it in tests." That is an architectural convention the fleet already follows, and the trust gate (where present) is implemented two incompatible ways (runtime list vs declarative rule). No shared merge, no shared trust mechanism.

• Disposition: do NOT extract a shared "federation lib." When fed-sx ships its real transport, these 4 become its consumers (wiring send-fn/fetch-fn/transport to it) — that work belongs to each subsystem's loop + the fed-sx loop, not a cross-cutting extraction. Stop re-proposing on the shared name. Home: fed-sx.

W2 · Per-viewer visibility / permission filter

• 2 shipped consumers, same shape — filter <injected-permit> <ranked/candidate stream>:
  • feed/lib/feed/acl.sx:27 feed/visible = (feed/filter stream (fn (a) (permit? viewer a))), capstone at :34 (stream → ACL → rank → top-N). permit? injected, sig (viewer activity)→bool.
  • search/lib/search/fed.sx:16 aclFilter permit docs = filter permit docs; topNTfIdfAcl n permit ts idx = take n (aclFilter permit (rankTfIdf ts idx)). permit injected, sig DocId→Bool (viewer baked in by caller).
• NOT a consumer: mod/lib/mod/policy.sx is moderation policy (reviewer actions), no per-viewer read filter. So mod won't be the 3rd.
• Missing: (a) only 2 consumers, need ≥3; (b) the two interfaces diverge — feed passes (viewer, item), search bakes the viewer in — so any shared form must pick a convention; (c) both already inject the predicate, and the filter body is literally one line (filter permit xs). Leaning toward: the predicate's home is acl-on-sx (permit?), and the one-line filter is too thin to extract.
• Home when ripe: delegate permit? to acl-on-sx; do NOT extract the filter. Re-check if a 3rd genuine per-viewer read filter ships (e.g. events/commerce).

W3 · Collection helpers (group-by, dedupe-by-key, stable top-N, distinct-order, offset/limit page)

• feed built all of these on APL primitives. search/commerce/events will want group-by / top-N.
• NEW (2026-06-06): offset/limit pagination shipped in 2 subsystems, identical shape take limit (drop offset xs):
  • feed/lib/feed/page.sx:9 feed/page (offset/limit window over a stream).
  • search/lib/search/page.sx:9 paginate off lim docs = take lim (drop off docs).
  • NOT a 3rd: persist/lib/persist/query.sx:5 has a since-cursor for incremental log consumption — resumable-stream semantics, not result windowing. Different shape.
  • feed also has cursor-by-:at recency pagination (page.sx:21-44); search has no cursor. So only the plain offset/limit window is shared, and it is a literal 1-liner.
• Missing: ≥3 stable consumers; AND every item here is collection math that belongs in the substrate (APL/Haskell already expose grade/sort/unique/take/drop), not a shared lib. A 1-line take/drop window is far below the extraction threshold. Watch; revisit only if a non-substrate subsystem needs the same windowing without take/drop.

W4 · In-memory store fakes → persist-on-sx

• Not an abstraction to extract — a migration target. Every subsystem fakes its store with a mutable list (feed/-log, flow store, mod audit, …).

• Owner: persist-on-sx (in progress). Tracked there, listed here for visibility.

• Concrete instance (file:line, found pass 4): the append-only decision/audit log. acl/lib/acl/audit.sx and mod/lib/mod/audit.sx are the SAME hand-rolled shape, and persist/lib/persist/log.sx (the persist log facet) already implements it durably:

  role	acl/audit.sx	mod/audit.sx	persist/log.sx (target)
  log var	acl-audit-log :9	mod/*audit-log* :10	backend stream
  monotonic seq	acl-audit-seq :10	mod/*audit-seq* :11	per-stream high-water :1
  append (auto-seq)	acl-audit-decide!	commit :32	persist/append :17
  count	acl-audit-count :51	mod/audit-count :44	persist/count :12
  read-all oldest-first	snapshot/tail :73	mod/audit-all :43	persist/read :29
  read seq≥from	—	by-seq	persist/read-from :31

  Both deliberately use a monotonic seq with no wall-clock (deterministic/testable) — identical to persist/log's design. Action when persist's host adapter lands: acl + mod loops swap their in-memory log for persist/log. 2 consumers today; not a new lib — the home already exists. Belongs to acl/mod loops × persist loop, not an extraction.

W5 · Proof-tree explanation over a logic-program derivation

• acl/lib/acl/explain.sx (reconstructs a canonical proof by goal-directed search over a saturated Datalog db) and mod/lib/mod/explain.sx (renders a Prolog-style proof tree goal-by-goal with proved/unproved marks + unification bindings) are the same idea.
• Missing / disposition: only 2 consumers, and they sit on different substrates (acl→lib/datalog, mod→lib/prolog). Proof reconstruction/rendering is logic-engine machinery → it belongs in each substrate (datalog/prolog), not a shared app lib. Watch; revisit only if a 3rd logic-backed subsystem reimplements proof explanation.

---

Rejected (considered, declined — do not re-propose)

• "Continuous auto-implementing abstractor loop." Rejected at design time: an agent writing across lib/<x>/** breaks the worktree isolation that makes the fleet safe, and is rewarded for manufacturing premature/wrong abstractions. The radar is read-only by design. (This file is the alternative.)
• Dumping app-domain plumbing into lib/guest. Rejected: lib/guest is for language-implementation plumbing. App patterns route to acl/fed-sx/persist/ substrate/host instead (see the routing rule in the briefing).