diff --git a/plans/agent-briefings/conformance-loop.md b/plans/agent-briefings/conformance-loop.md new file mode 100644 index 00000000..95188a9c --- /dev/null +++ b/plans/agent-briefings/conformance-loop.md @@ -0,0 +1,107 @@ +# A1 conformance-driver migration loop + +Role: migrate every remaining subsystem that hand-rolls its own `conformance.sh` +onto the **shared conformance driver** (`lib/guest/conformance.sh` + `lib/guest/conformance.sx`), +one subsystem per iteration, **verifying test-count parity before every commit**. +This executes item **A1** from the radar backlog (`plans/abstractions.md`, read-only +context). You are an implementer, not a scout. + +You are on branch `loops/conformance`, worktree `/root/rose-ash-loops/conformance`. + +## Hard safety rails (read every time) + +- **NEVER push to `main` or `architecture`.** Push only to `origin/loops/conformance`. +- **NEVER `pkill`/`kill` `sx_server` or any shared process** — sibling loops share the + binary. Bound every test run with `timeout` (e.g. `timeout 600 bash …`). If a run + hangs, let the timeout end it; never kill globally. +- **One subsystem per iteration, then stop.** No batching. +- **Never commit a regression.** If post-migration test counts don't match the baseline + (or an error appears), REVERT (`git checkout -- lib//conformance.sh` and + `rm -f lib//conformance.conf`) and record the blocker — do not commit. +- `.sx` files: use the `sx-tree` MCP tools, never Read/Write/Edit. `.sh`/`.conf`/`.md` + files: normal tools are fine. +- Preserve the `bash lib//conformance.sh` entry point (the shim keeps it working) so + no other loop is disrupted. + +## The candidate worklist + +Remaining hand-rolled `conformance.sh` (from radar A1): **common-lisp, erlang, feed, +forth, go, js, ocaml, smalltalk, tcl**. Already migrated (do not touch): acl, apl, +datalog, haskell, mod, prolog. Already excluded (different harness): lua. + +Work them roughly simplest-first. Track status in the checklist at the bottom. + +## What "fits the driver" means — classify FIRST + +The shared driver works for subsystems whose tests are **SX test-suites loaded over the +epoch protocol** and run by an expression that emits a counter/dict scoreboard. It does +NOT fit subsystems that run **foreign source programs** through a separate runner +(e.g. lua walks `*.lua` via Python; smalltalk runs `*.st` via `test.sh`). + +Per candidate, before migrating, decide: +- **Migratable** — its `conformance.sh` epoch-loads SX preloads and evals SX test suites + → proceed to migrate. +- **Excluded** — it shells out to a foreign program runner / scrapes a `test.sh` → + DO NOT migrate. Record the exclusion (one line in the checklist + a `git`-free note in + this briefing's Progress log) with the reason, and move on. Excluding is a valid, + honest result — a forced migration that loses coverage is worse than none. + +## Per-iteration procedure + +1. **Pick** the next `[ ]` candidate in the checklist. +2. **Read** its `lib//conformance.sh` in full. Read the two recipe templates — + `lib/haskell/conformance.conf` (MODE=counters) and `lib/prolog/conformance.conf` + (MODE=dict) — and skim `lib/guest/conformance.sh` + `lib/guest/conformance.sx`. +3. **Classify** (above). If Excluded → record reason, tick as excluded, stop. +4. **Baseline:** `timeout 600 bash lib//conformance.sh`, then read + `lib//scoreboard.json` and record the pass/total. This is the parity target. +5. **Author `lib//conformance.conf`:** + - `LANG_NAME=` + - `MODE=dict` or `MODE=counters` (match how the old script counted) + - `PRELOADS=( … )` — the lib files in load order, lifted from the old script + - `SUITES=( "name:lib//tests/:()" … )` — one per suite, with the + exact run expression the old script used + - If counters mode needs counter definitions, add a small `test-harness.sx` preload + (author it with `sx_write_file`). +6. **Replace `lib//conformance.sh`** with the 3-line shim: + ```bash + #!/usr/bin/env bash + # Thin wrapper — see lib/guest/conformance.sh and lib//conformance.conf. + exec bash "$(dirname "$0")/../guest/conformance.sh" "$(dirname "$0")/conformance.conf" "$@" + ``` +7. **Verify parity:** `timeout 600 bash lib//conformance.sh` again. Read + `scoreboard.json`. The pass/total MUST equal the baseline (a *higher* count is only + acceptable if you can explain it — e.g. the old extractor under-counted, as happened + with apl's `pipeline`; document it in the commit). Any mismatch/error → **revert** + (step: rails) and record the blocker. +8. **Commit** on `loops/conformance`: + `conformance: migrate onto shared driver (, / parity)` + then `git push origin loops/conformance`. +9. **Update** this file: tick the checklist box and add one dated line to the Progress + log (newest first). Then stop. + +If a candidate is genuinely blocked (driver lacks a needed mode/feature), record it under +Blocked with specifics and move to the next candidate next iteration. + +## Checklist + +- [ ] common-lisp +- [ ] erlang +- [ ] feed +- [ ] forth +- [ ] go +- [ ] js +- [ ] ocaml +- [ ] smalltalk +- [ ] tcl + +(Mark `[x] — migrated N/N` or `[~] — excluded: ` or +`[!] — blocked: `.) + +## Progress log (newest first) + +- (none yet) + +## Blocked + +- (none yet)