rose-ash/plans/agent-briefings/conformance-loop.md

A1 conformance-driver migration loop

Role: migrate every remaining subsystem that hand-rolls its own conformance.sh onto the shared conformance driver (lib/guest/conformance.sh + lib/guest/conformance.sx), one subsystem per iteration, verifying test-count parity before every commit. This executes item A1 from the radar backlog (plans/abstractions.md, read-only context). You are an implementer, not a scout.

You are on branch loops/conformance, worktree /root/rose-ash-loops/conformance.

Hard safety rails (read every time)

• NEVER push to main or architecture. Push only to origin/loops/conformance.
• NEVER pkill/kill sx_server or any shared process — sibling loops share the binary. Bound every test run with timeout (e.g. timeout 600 bash …). If a run hangs, let the timeout end it; never kill globally.
• One subsystem per iteration, then stop. No batching.
• Never commit a regression. If post-migration test counts don't match the baseline (or an error appears), REVERT (git checkout -- lib/<x>/conformance.sh and rm -f lib/<x>/conformance.conf) and record the blocker — do not commit.
• .sx files: use the sx-tree MCP tools, never Read/Write/Edit. .sh/.conf/.md files: normal tools are fine.
• Preserve the bash lib/<x>/conformance.sh entry point (the shim keeps it working) so no other loop is disrupted.

The candidate worklist

Remaining hand-rolled conformance.sh (from radar A1): common-lisp, erlang, feed, forth, go, js, ocaml, smalltalk, tcl. Already migrated (do not touch): acl, apl, datalog, haskell, mod, prolog. Already excluded (different harness): lua.

Work them roughly simplest-first. Track status in the checklist at the bottom.

What "fits the driver" means — classify FIRST

The shared driver works for subsystems whose tests are SX test-suites loaded over the epoch protocol and run by an expression that emits a counter/dict scoreboard. It does NOT fit subsystems that run foreign source programs through a separate runner (e.g. lua walks *.lua via Python; smalltalk runs *.st via test.sh).

Per candidate, before migrating, decide:

• Migratable — its conformance.sh epoch-loads SX preloads and evals SX test suites → proceed to migrate.
• Excluded — it shells out to a foreign program runner / scrapes a test.sh → DO NOT migrate. Record the exclusion (one line in the checklist + a git-free note in this briefing's Progress log) with the reason, and move on. Excluding is a valid, honest result — a forced migration that loses coverage is worse than none.

Per-iteration procedure

1. Pick the next [ ] candidate in the checklist.
2. Read its lib/<x>/conformance.sh in full. Read the two recipe templates — lib/haskell/conformance.conf (MODE=counters) and lib/prolog/conformance.conf (MODE=dict) — and skim lib/guest/conformance.sh + lib/guest/conformance.sx.
3. Classify (above). If Excluded → record reason, tick as excluded, stop.
4. Baseline: timeout 600 bash lib/<x>/conformance.sh, then read lib/<x>/scoreboard.json and record the pass/total. This is the parity target.
5. Author lib/<x>/conformance.conf:
   • LANG_NAME=<x>
   • MODE=dict or MODE=counters (match how the old script counted)
   • PRELOADS=( … ) — the lib files in load order, lifted from the old script
   • SUITES=( "name:lib/<x>/tests/<file>:(<run-expr>)" … ) — one per suite, with the exact run expression the old script used
   • If counters mode needs counter definitions, add a small test-harness.sx preload (author it with sx_write_file).
6. Replace lib/<x>/conformance.sh with the 3-line shim:

   #!/usr/bin/env bash
   # Thin wrapper — see lib/guest/conformance.sh and lib/<x>/conformance.conf.
   exec bash "$(dirname "$0")/../guest/conformance.sh" "$(dirname "$0")/conformance.conf" "$@"
   

7. Verify parity: timeout 600 bash lib/<x>/conformance.sh again. Read scoreboard.json. The pass/total MUST equal the baseline (a higher count is only acceptable if you can explain it — e.g. the old extractor under-counted, as happened with apl's pipeline; document it in the commit). Any mismatch/error → revert (step: rails) and record the blocker.
8. Commit on loops/conformance: conformance: migrate <x> onto shared driver (<mode>, <pass>/<total> parity) then git push origin loops/conformance.
9. Update this file: tick the checklist box and add one dated line to the Progress log (newest first). Then stop.

If a candidate is genuinely blocked (driver lacks a needed mode/feature), record it under Blocked with specifics and move to the next candidate next iteration.

Checklist

• common-lisp — migrated 487/487 (counters; driver extended for per-suite counters+preloads)
• erlang — migrated 761/761 (dict; pass/count → :failed = count-pass)
• feed
• forth
• go
• js
• ocaml
• smalltalk
• tcl

(Mark [x] <x> — migrated N/N or [~] <x> — excluded: <reason> or [!] <x> — blocked: <reason>.)

Progress log (newest first)

• 2026-06-07 — erlang: migrated to MODE=dict, 761/761 exact parity (tokenize 62, parse 52, eval 408, runtime 93, ring 4, ping-pong 4, bank 8, echo 7, fib 8, ffi 37, vm 78). Erlang exposes pass + count (total) counters, not pass/fail, so each suite's dict-literal runner computes :failed (- count pass). Loads in one session (matches dict mode), so no driver change needed — only conformance.conf + shim. Kept historical scoreboard schema (language/total_pass/total/suites[name,pass,total,status]).
• 2026-06-07 — common-lisp: UNBLOCKED + migrated. Extended the shared driver's MODE=counters (lib/guest/conformance.sh) with a backward-compatible SUITES format name:file[:pass-var:fail-var[:extra-preload ...]] — optional per-suite counter symbols and per-suite preload chains. Authored lib/common-lisp/conformance.conf (12 suites, 8 distinct counter pairs, per-suite preloads, base PRELOADS=stdlib+prefix; kept historical scoreboard schema) and replaced conformance.sh with the shim. Result 487/487 (0 fail) — HIGHER than the 305/0 baseline, explained: the old script's per-suite timeout 30 was too tight for the slow eval suite (~15–25s under contention), silently recording it as 0; the driver's 180s budget recovers its true 182. geometry/mop-trace remain 0/0 (pre-existing refl-class-chain-depth-with load error; counter vars defined as 0 → clean gc-result, no fail-fallback). Regression: haskell backward-compat path verified (fib/sieve/quicksort 2/2/5, matches committed).
• 2026-06-07 — common-lisp: classified migratable-in-kind (SX suites over epoch) but BLOCKED on driver feature gaps. Baseline bash lib/common-lisp/conformance.sh = 305 passed / 0 failed across 12 suites (3 — evaluator/geometry/mop-trace — already emit 0/0, a pre-existing extraction quirk). Not a foreign runner, so not Excluded. Did NOT migrate (parity unachievable under current modes); left conformance.sh untouched. See Blocked. Driver left unchanged (out of strict per-iteration scope).

Blocked

• (none)

Resolved blockers

• common-lisp (resolved 2026-06-07) — needed per-suite counter names + per-suite preload chains, unsupported by the original MODE=counters (single global counter + fixed PRELOADS). Resolved by extending the shared driver: MODE=counters now accepts name:file[:pass-var:fail-var[:extra-preload ...]] (backward-compatible). This same extension is available to later candidates — erlang/forth/etc. with per-suite counter names or preload chains can now migrate via the extended format instead of blocking.