W14: F10 expected-failures baseline gate (test-only)
The OCaml suite's permanent ~273-failure band (in-progress hs-* + the r7rs radix shadow) is normalized, so real regressions hide in red noise (conformance.md F-10). A runner skip-list would rewrite the hs loops' scoreboards mid-flight — instead, pin the band: scripts/test-suite-baseline.sh runs the full suite and diffs its FAIL set against spec/tests/known-failures.txt (273 entries, identity = "suite > name", error text stripped). Red on a NEW failure (regression) AND red on a vanished failure (fix landed — delete it from the baseline, locking in the win). The band still prints as FAIL lines for the teams working through it; nothing in the runner changes. Bonus capture: 2 of the 273 have EMPTY suite labels (can-map-an-array, string->number) — live evidence for C9, the next checklist item. Validated end-to-end: GREEN on current tree (5800p/273f — 38 net passes above dc7aa709's 5762 from this loop's added pins). Runtime ~12 min. Test-only: no semantics edits, no push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -94,7 +94,10 @@ Pin each confirmed-and-fixed finding with a minimal repro. Add suites to
|
||||
- [x] C3/C4/C5/C6/C7 — protocol-quirk ledger (pins current behavior,
|
||||
bidirectional) + seeded 60-line fuzz-liveness property in
|
||||
`scripts/test-protocol-gate.sh` (11/11)
|
||||
- [ ] F10 — hs-upstream skip-list so browser-only FAILs mean something
|
||||
- [x] F10 — expected-failures BASELINE GATE instead of a skip-list
|
||||
(`scripts/test-suite-baseline.sh` + `spec/tests/known-failures.txt`,
|
||||
273 pinned: 271 hs-* + 2 empty-suite-label entries → C9 evidence).
|
||||
New failure OR vanished failure = red; hs loops' scoreboards untouched
|
||||
- [ ] C9 — empty suite label
|
||||
|
||||
### F. Differential battery
|
||||
@@ -102,6 +105,18 @@ Pin each confirmed-and-fixed finding with a minimal repro. Add suites to
|
||||
|
||||
## Progress log (newest first)
|
||||
|
||||
- 2026-07-04 — **F10 baseline gate (item E.2)**. Deliberately NOT a
|
||||
skip-list: skip-listing the hs red band in the runner would rewrite the
|
||||
hs loops' scoreboards mid-flight. Instead
|
||||
`scripts/test-suite-baseline.sh` diffs the full suite's FAIL set against
|
||||
checked-in `spec/tests/known-failures.txt` (273 entries: 271 hs-* + 2
|
||||
with EMPTY suite labels — live C9 evidence, `can-map-an-array` "map with
|
||||
block" and `string->number` 2-arg, the "r7rs radix shadow"). Red on a
|
||||
NEW failure (regression) and red on a VANISHED failure (fix landed —
|
||||
delete from baseline, locking in the win). Identity = "suite > name"
|
||||
with error text stripped (messages churn). Current suite: 5800p/273f
|
||||
(up 38 passes from dc7aa709's 5762 — sections A–D added pins). Validated
|
||||
end-to-end: GREEN, exit 0, ~12 min runtime. Test-only.
|
||||
- 2026-07-04 — **C3–C7 protocol fuzz suite (item E.1)**. All five findings
|
||||
are still OPEN server-side (sx_server.ml fixes are host-runtime work),
|
||||
so the suite pins CURRENT behavior as a bidirectional ledger — verified
|
||||
|
||||
Reference in New Issue
Block a user