rose-ash/plans/abstractions.md

# Abstraction Radar — backlog

Maintained by the read-only `radar` loop (see `plans/agent-briefings/radar-loop.md`).
Detection only — implementation is a separate, coordinated step owned by the
relevant subsystem loop, never by radar.

**AHA gate to reach _Proposed_:** ≥3 real consumers · all past Phase 2 & API-stable ·
structurally identical (file:line evidence) · a natural home (usually NOT lib/guest).
Anything short → _Watching_ (what's missing) or _Rejected_ (why).

---

## Last scan

- **Date:** 2026-06-06 (radar loop, pass 6)
- **Pass 6:** new worktree **`host-persist`** (active — building persist's durable host
  adapter); `feed` went quiescent (left tmux). acl shipped hardening (+25), fed-sx-m1 at
  Step 6c. **mod loop independently wrote a shared-plumbing note** (`mod-on-sx.md`,
  538b8a53) corroborating W4/W5 — folded its claims + home disagreements into W1/W4/W5.
  No new gate-clearer (audit log still 2 consumers), but consumers are now API-stable.
- **Pass 5:** search (+highlight/snippet) and fed-sx-m1 (+follower_graph) moved; rest
  unchanged. Filename census: `api`×6, `fed`×3, then `schema/rank/query/page/explain/
  engine/batch/audit`×2. Examined the ×6 `api.sx` → Rejected (shared name, divergent
  structure incl. implicit-vs-explicit-state contract). rank/batch/engine all ≤2 +
  substrate/domain-divergent → no new gate-clearer.
- **Pass 4:** no churn vs pass 3 (same worktrees/tmux/HEADs/adopters). Swept audit+explain
  surfaces: acl/mod share an append-only-log shape (→ sharpened W4 with persist/log API
  evidence) and a proof-explain shape (→ new W5, substrate-bound). No new gate-clearer.
- **Pass 3 (earlier today):** subsystem set + tmux + A1 adopters (4) all unchanged vs pass 2. Loops
  advanced: acl shipped Phase 4 federation; search shipped Phase 4 + pagination; feed
  shipped pagination/threading; mod at Ext 19 (capstone); persist did a worked acl-grants
  migration (W4). New shape found: offset/limit pagination → folded into W3.
- **Subsystem set discovered:** loop worktrees `acl, erlang, fed-prims, fed-sx-m1,
  feed, flow, go, kernel, mod, ocaml, persist, radar, ruby, search,
  sx-vm-extensions`; main-repo `lib/*` incl. merged `feed` + substrates (`apl,
  common-lisp, datalog, erlang, forth, go, haskell, hyperscript, js, lua, minikanren,
  ocaml, prolog, scheme, smalltalk, tcl`) + `lib/guest`.
  Actively looping (tmux): `acl, fed-sx-m1, feed, flow, mod, persist, search`
  (+ radar).
- **New since pass 1:** worktrees `kernel` (empty/unset — not yet a repo) and `ocaml`
  (`lib/ocaml/baseline` only). Both early-stage, pre–Phase 2 → out of proposal scope.
- Re-enumerate every pass; new loops (e.g. a future `commerce`/`identity`) auto-join.

---

## Proposed (cleared the gate)

### A1 · Adopt the shared conformance driver across subsystems
- **Pattern:** every subsystem hand-rolls a near-identical `conformance.sh`
  (epoch-load → eval → scoreboard emit) and an inline `<x>-test name got expected`
  pass/fail counter.
- **Consumers (≥3, overwhelming):** 15 `lib/*/conformance.sh` — `apl, feed, datalog,
  flow, mod, lua, erlang, forth, go, common-lisp, haskell, js, ocaml, prolog,
  smalltalk, tcl`.
- **Home:** `lib/guest` — the one legitimate exception (the shared driver
  `lib/guest/conformance.sh` + `lib/guest/conformance.sx` already exist; modes
  `dict` and `counters`).
- **Status: IN PROGRESS — 4 adopters.** `prolog` (dict), `haskell` (counters),
  `apl` (dict), and **`datalog` (dict), newly migrated this pass** — all now 3-line
  exec shims into `lib/guest/conformance.sh` with a `conformance.conf`
  (`lib/datalog/conformance.conf`: 11 PRELOADS + 11 SUITES).
  The `apl` migration earlier *surfaced a latent bug*: the old awk extractor
  under-counted `pipeline` (40 vs the real 152 assertions); true apl total is **562**,
  not 450 — evidence that adopting the driver also improves correctness.
- **Not a target (different harness shape):** `lua/conformance.sh` is a Python runner
  (`lib/lua/conformance.py`) that walks real `*.lua` source files via `lua-eval-ast`
  and classifies pass/fail/timeout — it does not run SX `deftest` suites with a
  counter/dict scoreboard, so the shared driver does not fit. Excluded, not pending.
- **Remaining hand-rolled candidates (~120–220 lines each):** `common-lisp, erlang,
  feed, forth, go, js, ocaml, smalltalk, tcl` — each its OWN loop's migration when
  quiescent.
- **Action:** each remaining subsystem's OWN loop migrates when quiescent — add a
  `conformance.conf` (+ a `test-harness.sx` preload defining its counters) and
  replace `conformance.sh` with the 1-line exec shim
  (`exec bash …/guest/conformance.sh …/conformance.conf "$@"`). Recipe template:
  `lib/haskell/conformance.conf` (counters) or `lib/prolog/conformance.conf` (dict).
  Keep the `bash lib/X/conformance.sh` entry point so no loop is disrupted.
- **Priority: HIGH** (15 consumers, low risk, interface-preserving, additive).

---

## Watching (real but not yet through the gate)

### W1 · Federation scaffold (merge / ingest / backfill / trust-gate)
- **FAILS the structural-identity gate (deep-dived 2026-06-06, all 4 read).** Consumer
  count is met (4) but they are *superficially* similar, not structurally identical —
  the federated unit and merge op differ fundamentally:

  | Subsystem (file) | Federated unit | Merge op | Trust gate | Injected transport |
  |---|---|---|---|---|
  | feed (`fed.sx:14,18,40`) | activity streams | dedupe by `(actor verb object)` | none (visibility via `permit?` separately) | `send-fn`, `fetch-fn` |
  | search (`fed.sx:8`) | inverted indices | relabel DocId `peer*1000+local` + union posting lists | none | none (pure merge fn) |
  | mod (`fed.sx:11-14,99`) | moderation decisions | advisory-list vs applied-list; bind iff `mod/trusted?` | **yes — runtime list** `mod/trusted? peer scope` | mock outbox / `fed-send!` |
  | acl (`federation.sx:43,56`) | Datalog delegate facts | pull facts, gate by `trust`/`level_covers` rule, re-saturate | **yes — Datalog rule** at query time | `transport` dict |

- **The ONLY real commonality is the injection seam**, not extractable code: every one
  says "the real transport is `fed-sx`'s job; inject `send-fn`/`fetch-fn`/`transport`/
  outbox and mock it in tests." That is an architectural *convention the fleet already
  follows*, and the trust gate (where present) is implemented two incompatible ways
  (runtime list vs declarative rule). No shared merge, no shared trust mechanism.
- **Disposition:** do NOT extract a shared "federation lib." When `fed-sx` ships its
  real transport, these 4 become its *consumers* (wiring `send-fn`/`fetch-fn`/`transport`
  to it) — that work belongs to each subsystem's loop + the `fed-sx` loop, not a
  cross-cutting extraction. Stop re-proposing on the shared name. Home: `fed-sx`.
- **Narrower sub-claim (mod note, pass 6):** mod asserts the *fed trust/outbox* shape
  specifically shares between mod+acl. Radar evidence above shows the **trust gate**
  mechanism diverges (mod runtime-list vs acl Datalog-rule); the **outbox/propagation
  envelope** may share, but that's 2 consumers (mod, acl) on different substrates →
  Watching. Resolve at the architecture-merge point if it survives to a 3rd consumer.

### W2 · Per-viewer visibility / permission filter
- **2 shipped consumers, same shape** — `filter <injected-permit> <ranked/candidate stream>`:
  - `feed/lib/feed/acl.sx:27` `feed/visible = (feed/filter stream (fn (a) (permit? viewer a)))`,
    capstone at `:34` (stream → ACL → rank → top-N). `permit?` injected, sig `(viewer activity)→bool`.
  - `search/lib/search/fed.sx:16` `aclFilter permit docs = filter permit docs`;
    `topNTfIdfAcl n permit ts idx = take n (aclFilter permit (rankTfIdf ts idx))`.
    `permit` injected, sig `DocId→Bool` (viewer baked in by caller).
- **NOT a consumer:** `mod/lib/mod/policy.sx` is moderation policy (reviewer actions),
  no per-viewer read filter. So mod won't be the 3rd.
- **Missing:** (a) only 2 consumers, need ≥3; (b) the two interfaces *diverge* —
  feed passes `(viewer, item)`, search bakes the viewer in — so any shared form must
  pick a convention; (c) both already **inject** the predicate, and the filter body is
  literally one line (`filter permit xs`). Leaning toward: the predicate's home is
  `acl-on-sx` (`permit?`), and the one-line filter is too thin to extract.
- **Home when ripe:** delegate `permit?` to `acl-on-sx`; do NOT extract the filter.
  Re-check if a 3rd genuine per-viewer read filter ships (e.g. events/commerce).

### W3 · Collection helpers (group-by, dedupe-by-key, stable top-N, distinct-order, offset/limit page)
- feed built all of these on APL primitives. search/commerce/events will want
  group-by / top-N.
- **NEW (2026-06-06): offset/limit pagination shipped in 2 subsystems, identical shape**
  `take limit (drop offset xs)`:
  - `feed/lib/feed/page.sx:9` `feed/page` (offset/limit window over a stream).
  - `search/lib/search/page.sx:9` `paginate off lim docs = take lim (drop off docs)`.
  - NOT a 3rd: `persist/lib/persist/query.sx:5` has a *since-cursor* for incremental log
    consumption — resumable-stream semantics, not result windowing. Different shape.
  - feed *also* has cursor-by-`:at` recency pagination (`page.sx:21-44`); search has no
    cursor. So only the plain offset/limit window is shared, and it is a literal 1-liner.
- **Missing:** ≥3 stable consumers; AND every item here is collection math that belongs
  in the **substrate** (APL/Haskell already expose grade/sort/unique/take/drop), not a
  shared lib. A 1-line `take/drop` window is far below the extraction threshold. Watch;
  revisit only if a non-substrate subsystem needs the same windowing without take/drop.

### W4 · In-memory store fakes → `persist-on-sx`
- Not an abstraction to extract — a migration target. Every subsystem fakes its
  store with a mutable list (`feed/-log`, flow store, mod audit, …).
- **Owner:** `persist-on-sx` (in progress). Tracked there, listed here for visibility.
- **Concrete instance (file:line, found pass 4): the append-only decision/audit log.**
  `acl/lib/acl/audit.sx` and `mod/lib/mod/audit.sx` are the SAME hand-rolled shape, and
  `persist/lib/persist/log.sx` (the persist *log facet*) already implements it durably:

  | role | acl/audit.sx | mod/audit.sx | persist/log.sx (target) |
  |---|---|---|---|
  | log var | `acl-audit-log` :9 | `mod/*audit-log*` :10 | backend stream |
  | monotonic seq | `acl-audit-seq` :10 | `mod/*audit-seq*` :11 | per-stream high-water :1 |
  | append (auto-seq) | `acl-audit-decide!` | commit :32 | `persist/append` :17 |
  | count | `acl-audit-count` :51 | `mod/audit-count` :44 | `persist/count` :12 |
  | read-all oldest-first | snapshot/tail :73 | `mod/audit-all` :43 | `persist/read` :29 |
  | read seq≥from | — | by-seq | `persist/read-from` :31 |

  Both deliberately use a monotonic seq with **no wall-clock** (deterministic/testable) —
  identical to persist/log's design. Action when persist's host adapter lands: acl + mod
  loops swap their in-memory log for `persist/log`. 2 consumers today; not a new lib —
  the home already exists. Belongs to acl/mod loops × persist loop, not an extraction.
- **Cross-loop corroboration (pass 6):** the mod loop independently reached the same
  conclusion — `mod/plans/mod-on-sx.md` (commit 538b8a53): *"mod-sx (Prolog) and acl-sx
  (Datalog) converged on the same module shape … only the audit log + fed trust/outbox
  shapes truly share; extract at the architecture-merge point, refactoring both consumers
  atomically, not unilaterally from a loop branch."* Confirms the shape AND the
  do-not-extract-unilaterally stance.
- **Home disagreement to resolve at merge:** mod's note proposes lifting the audit-log
  primitives into **`lib/guest/`**. Radar routing disagrees: a durable append-only log is
  a **`persist-on-sx`** concern (the log facet already exists), not language-impl plumbing.
  Hold the line — `lib/guest` is lexer/parser/AST/HM/test-runner, not an event log.
- **Migration is becoming concrete:** new `host-persist` loop (worktree + tmux, pass 6)
  is building the durable-storage host adapter persist was blocked on — once it lands,
  acl/mod can actually swap to `persist/log`.

### W5 · Proof-tree explanation over a logic-program derivation
- `acl/lib/acl/explain.sx` (reconstructs a canonical proof by goal-directed search over a
  saturated Datalog db) and `mod/lib/mod/explain.sx` (renders a Prolog-style proof tree
  goal-by-goal with proved/unproved marks + unification bindings) are the same *idea*.
- **Missing / disposition:** only 2 consumers, and they sit on **different substrates**
  (acl→`lib/datalog`, mod→`lib/prolog`). Proof reconstruction/rendering is logic-engine
  machinery → it belongs in each **substrate** (datalog/prolog), not a shared app lib.
  Watch; revisit only if a 3rd logic-backed subsystem reimplements proof explanation.
- **Cross-loop note (pass 6):** mod's note calls `mod/proof-goals` (re-query-each-goal)
  generic and proposes lifting it into **`lib/guest/`**. Radar caveat: proof-tree
  reconstruction *is* engine-agnostic logic machinery, but `lib/guest` is for
  lexer/parser/AST/HM/match/test-runner — a logic-engine proof helper is a poor fit there.
  If genuinely shared by ≥3 engines, a `lib/logic`-style substrate helper is the better
  home than `lib/guest`. Still 2 consumers → stays Watching either way.

---

## Rejected (considered, declined — do not re-propose)

- **"Continuous auto-implementing abstractor loop."** Rejected at design time: an
  agent writing across `lib/<x>/**` breaks the worktree isolation that makes the
  fleet safe, and is rewarded for manufacturing premature/wrong abstractions. The
  radar is read-only by design. (This file is the alternative.)
- **Shared `api.sx` "public boundary" module (×6).** Rejected pass 4-5: every subsystem
  has an `api.sx` (acl, feed, flow, mod, persist, search — a 100% filename match), but it
  is a naming *convention for the public entry point*, not a shared structure. They
  disagree on the most basic contract: acl/feed use **implicit module state**
  (`acl/api.sx` "implicit current db", `feed/api.sx` "single mutable log") while
  `persist/api.sx` threads an **explicit backend as every call's first arg**; flow's api
  *builds a Scheme env*, search's api *concatenates a Haskell source string*, mod's is a
  *lifecycle state-machine façade* (17 defs vs persist's 1). Same role, no common shape —
  the W1 coincidental-resemblance trap. Do not re-propose on the filename.
- **Dumping app-domain plumbing into `lib/guest`.** Rejected: `lib/guest` is for
  language-implementation plumbing. App patterns route to acl/fed-sx/persist/
  substrate/host instead (see the routing rule in the briefing).