W14: F10 expected-failures baseline gate (test-only)

The OCaml suite's permanent ~273-failure band (in-progress hs-* + the
r7rs radix shadow) is normalized, so real regressions hide in red noise
(conformance.md F-10). A runner skip-list would rewrite the hs loops'
scoreboards mid-flight — instead, pin the band:

scripts/test-suite-baseline.sh runs the full suite and diffs its FAIL set
against spec/tests/known-failures.txt (273 entries, identity =
"suite > name", error text stripped). Red on a NEW failure (regression)
AND red on a vanished failure (fix landed — delete it from the baseline,
locking in the win). The band still prints as FAIL lines for the teams
working through it; nothing in the runner changes.

Bonus capture: 2 of the 273 have EMPTY suite labels (can-map-an-array,
string->number) — live evidence for C9, the next checklist item.

Validated end-to-end: GREEN on current tree (5800p/273f — 38 net passes
above dc7aa709's 5762 from this loop's added pins). Runtime ~12 min.

Test-only: no semantics edits, no push.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-04 04:10:55 +00:00
parent ca4ad404f1
commit 8ba68e0365
3 changed files with 350 additions and 1 deletions

View File

@@ -94,7 +94,10 @@ Pin each confirmed-and-fixed finding with a minimal repro. Add suites to
- [x] C3/C4/C5/C6/C7 — protocol-quirk ledger (pins current behavior,
bidirectional) + seeded 60-line fuzz-liveness property in
`scripts/test-protocol-gate.sh` (11/11)
- [ ] F10 — hs-upstream skip-list so browser-only FAILs mean something
- [x] F10 — expected-failures BASELINE GATE instead of a skip-list
(`scripts/test-suite-baseline.sh` + `spec/tests/known-failures.txt`,
273 pinned: 271 hs-* + 2 empty-suite-label entries → C9 evidence).
New failure OR vanished failure = red; hs loops' scoreboards untouched
- [ ] C9 — empty suite label
### F. Differential battery
@@ -102,6 +105,18 @@ Pin each confirmed-and-fixed finding with a minimal repro. Add suites to
## Progress log (newest first)
- 2026-07-04 — **F10 baseline gate (item E.2)**. Deliberately NOT a
skip-list: skip-listing the hs red band in the runner would rewrite the
hs loops' scoreboards mid-flight. Instead
`scripts/test-suite-baseline.sh` diffs the full suite's FAIL set against
checked-in `spec/tests/known-failures.txt` (273 entries: 271 hs-* + 2
with EMPTY suite labels — live C9 evidence, `can-map-an-array` "map with
block" and `string->number` 2-arg, the "r7rs radix shadow"). Red on a
NEW failure (regression) and red on a VANISHED failure (fix landed —
delete from baseline, locking in the win). Identity = "suite > name"
with error text stripped (messages churn). Current suite: 5800p/273f
(up 38 passes from dc7aa709's 5762 — sections AD added pins). Validated
end-to-end: GREEN, exit 0, ~12 min runtime. Test-only.
- 2026-07-04 — **C3C7 protocol fuzz suite (item E.1)**. All five findings
are still OPEN server-side (sx_server.ml fixes are host-runtime work),
so the suite pins CURRENT behavior as a bidirectional ledger — verified