rose-ash/plans/abstractions.md

# Abstraction Radar — backlog

Maintained by the read-only `radar` loop (see `plans/agent-briefings/radar-loop.md`).
Detection only — implementation is a separate, coordinated step owned by the
relevant subsystem loop, never by radar.

**AHA gate to reach _Proposed_:** ≥3 real consumers · all past Phase 2 & API-stable ·
structurally identical (file:line evidence) · a natural home (usually NOT lib/guest).
Anything short → _Watching_ (what's missing) or _Rejected_ (why).

---

## Last scan

- **Date:** 2026-06-06 (radar loop, pass 2)
- **Subsystem set discovered:** loop worktrees `acl, erlang, fed-prims, fed-sx-m1,
  feed, flow, go, kernel, mod, ocaml, persist, radar, ruby, search,
  sx-vm-extensions`; main-repo `lib/*` incl. merged `feed` + substrates (`apl,
  common-lisp, datalog, erlang, forth, go, haskell, hyperscript, js, lua, minikanren,
  ocaml, prolog, scheme, smalltalk, tcl`) + `lib/guest`.
  Actively looping (tmux): `acl, fed-sx-m1, feed, flow, mod, persist, search`
  (+ radar).
- **New since pass 1:** worktrees `kernel` (empty/unset — not yet a repo) and `ocaml`
  (`lib/ocaml/baseline` only). Both early-stage, pre–Phase 2 → out of proposal scope.
- Re-enumerate every pass; new loops (e.g. a future `commerce`/`identity`) auto-join.

---

## Proposed (cleared the gate)

### A1 · Adopt the shared conformance driver across subsystems
- **Pattern:** every subsystem hand-rolls a near-identical `conformance.sh`
  (epoch-load → eval → scoreboard emit) and an inline `<x>-test name got expected`
  pass/fail counter.
- **Consumers (≥3, overwhelming):** 15 `lib/*/conformance.sh` — `apl, feed, datalog,
  flow, mod, lua, erlang, forth, go, common-lisp, haskell, js, ocaml, prolog,
  smalltalk, tcl`.
- **Home:** `lib/guest` — the one legitimate exception (the shared driver
  `lib/guest/conformance.sh` + `lib/guest/conformance.sx` already exist; modes
  `dict` and `counters`).
- **Status: IN PROGRESS — 4 adopters.** `prolog` (dict), `haskell` (counters),
  `apl` (dict), and **`datalog` (dict), newly migrated this pass** — all now 3-line
  exec shims into `lib/guest/conformance.sh` with a `conformance.conf`
  (`lib/datalog/conformance.conf`: 11 PRELOADS + 11 SUITES).
  The `apl` migration earlier *surfaced a latent bug*: the old awk extractor
  under-counted `pipeline` (40 vs the real 152 assertions); true apl total is **562**,
  not 450 — evidence that adopting the driver also improves correctness.
- **Not a target (different harness shape):** `lua/conformance.sh` is a Python runner
  (`lib/lua/conformance.py`) that walks real `*.lua` source files via `lua-eval-ast`
  and classifies pass/fail/timeout — it does not run SX `deftest` suites with a
  counter/dict scoreboard, so the shared driver does not fit. Excluded, not pending.
- **Remaining hand-rolled candidates (~120–220 lines each):** `common-lisp, erlang,
  feed, forth, go, js, ocaml, smalltalk, tcl` — each its OWN loop's migration when
  quiescent.
- **Action:** each remaining subsystem's OWN loop migrates when quiescent — add a
  `conformance.conf` (+ a `test-harness.sx` preload defining its counters) and
  replace `conformance.sh` with the 1-line exec shim
  (`exec bash …/guest/conformance.sh …/conformance.conf "$@"`). Recipe template:
  `lib/haskell/conformance.conf` (counters) or `lib/prolog/conformance.conf` (dict).
  Keep the `bash lib/X/conformance.sh` entry point so no loop is disrupted.
- **Priority: HIGH** (15 consumers, low risk, interface-preserving, additive).

---

## Watching (real but not yet through the gate)

### W1 · Federation scaffold (merge / ingest / backfill / trust-gate)
- **Consumer count NOW met (4):** `feed/lib/feed/fed.sx` (60L/7 defs),
  `search/lib/search/fed.sx` (16L/1), `mod/lib/mod/fed.sx` (145L/19),
  `acl/lib/acl/federation.sx` (61L/5).
- **BUT structural identity unverified — and the size spread (16→145 lines) is a red
  flag.** search/fed is just relabel+merge of an index; mod/fed is 19 defs;
  these may be *superficially* similar (all "federation") not structurally identical.
- **Next pass (highest-value thread):** read all 4 side by side, extract the common
  shape (if any) with file:line evidence — trust-gate? injected transport? merge vs
  ingest split? — before this can move to Proposed. Do NOT propose on the name alone.
- **Home when ripe:** `fed-sx` (NOT lib/guest).

### W2 · Per-viewer visibility / permission filter
- **2 shipped consumers, same shape** — `filter <injected-permit> <ranked/candidate stream>`:
  - `feed/lib/feed/acl.sx:27` `feed/visible = (feed/filter stream (fn (a) (permit? viewer a)))`,
    capstone at `:34` (stream → ACL → rank → top-N). `permit?` injected, sig `(viewer activity)→bool`.
  - `search/lib/search/fed.sx:16` `aclFilter permit docs = filter permit docs`;
    `topNTfIdfAcl n permit ts idx = take n (aclFilter permit (rankTfIdf ts idx))`.
    `permit` injected, sig `DocId→Bool` (viewer baked in by caller).
- **NOT a consumer:** `mod/lib/mod/policy.sx` is moderation policy (reviewer actions),
  no per-viewer read filter. So mod won't be the 3rd.
- **Missing:** (a) only 2 consumers, need ≥3; (b) the two interfaces *diverge* —
  feed passes `(viewer, item)`, search bakes the viewer in — so any shared form must
  pick a convention; (c) both already **inject** the predicate, and the filter body is
  literally one line (`filter permit xs`). Leaning toward: the predicate's home is
  `acl-on-sx` (`permit?`), and the one-line filter is too thin to extract.
- **Home when ripe:** delegate `permit?` to `acl-on-sx`; do NOT extract the filter.
  Re-check if a 3rd genuine per-viewer read filter ships (e.g. events/commerce).

### W3 · Collection helpers (group-by, dedupe-by-key, stable top-N, distinct-order)
- feed built all of these on APL primitives. search/commerce/events will want
  group-by / top-N.
- **Missing:** ≥3 stable consumers; and these likely belong in the **substrate**
  (APL/Haskell already expose grade/sort/unique) rather than a shared lib. Watch.

### W4 · In-memory store fakes → `persist-on-sx`
- Not an abstraction to extract — a migration target. Every subsystem fakes its
  store with a mutable list (`feed/-log`, flow store, mod audit, …).
- **Owner:** `persist-on-sx` (in progress). Tracked there, listed here for visibility.

---

## Rejected (considered, declined — do not re-propose)

- **"Continuous auto-implementing abstractor loop."** Rejected at design time: an
  agent writing across `lib/<x>/**` breaks the worktree isolation that makes the
  fleet safe, and is rewarded for manufacturing premature/wrong abstractions. The
  radar is read-only by design. (This file is the alternative.)
- **Dumping app-domain plumbing into `lib/guest`.** Rejected: `lib/guest` is for
  language-implementation plumbing. App patterns route to acl/fed-sx/persist/
  substrate/host instead (see the routing rule in the briefing).