conformance: migrate common-lisp onto shared driver (counters, 487/487)
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 14m2s

Extend the shared driver's MODE=counters with a backward-compatible SUITES
format: name:file[:pass-var:fail-var[:extra-preload ...]]. Optional per-suite
counter symbols (override the global COUNTERS_PASS/COUNTERS_FAIL) and per-suite
preload chains (loaded after the global PRELOADS). Plain name:file entries are
unchanged — verified against haskell (fib/sieve/quicksort 2/2/5, matches
committed scoreboard).

common-lisp has 8 distinct per-suite counter pairs and a different preload
chain per suite, so it could not fit the single-counter/fixed-preload model;
the extended format expresses it directly. conformance.conf keeps the historical
scoreboard schema; conformance.sh becomes the 3-line shim.

Result 487/487 (0 fail) vs the old 305/0 baseline — higher and explained: the
old per-suite 'timeout 30' was too tight for the slow eval suite (~15-25s under
contention), silently recording it as 0; the driver's 180s budget recovers its
true 182. geometry/mop-trace stay 0/0 (pre-existing refl-class-chain-depth-with
load error; counter vars defined as 0 -> clean gc-result, no fail-fallback).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-07 09:55:44 +00:00
parent bfdd0fe65a
commit 2e7a08309c
6 changed files with 152 additions and 214 deletions

View File

@@ -21,10 +21,17 @@
# MODE=dict — "name:test-file:(runner-fn)"
# The runner expression is evaluated and is expected to
# return a dict with :passed/:failed/:total.
# MODE=counters — "name:test-file"
# MODE=counters — "name:test-file[:pass-var:fail-var[:extra-preload ...]]"
# Each suite is run in a fresh sx_server session: preloads
# are loaded, then the test file, then counters are read.
# The suite is treated as starting from counters (0, 0).
# Optional per-suite fields:
# pass-var/fail-var — counter symbols for this suite,
# overriding COUNTERS_PASS/COUNTERS_FAIL.
# extra-preload ... — space-separated .sx files loaded
# after the global PRELOADS (per-suite
# dependency chains).
# Plain "name:test-file" still works (uses the globals).
#
# Output:
# Writes $SCOREBOARD_DIR/scoreboard.json and $SCOREBOARD_DIR/scoreboard.md.
@@ -163,22 +170,39 @@ case "$MODE" in
fi
;;
counters)
if [ -z "$COUNTERS_PASS" ] || [ -z "$COUNTERS_FAIL" ]; then
echo "MODE=counters requires COUNTERS_PASS and COUNTERS_FAIL in $CONF" >&2
exit 2
fi
# Each suite must resolve to a pass/fail counter name — either per-suite
# (fields 3 & 4 of the SUITES entry) or via the global COUNTERS_PASS /
# COUNTERS_FAIL defaults. Validate up front so a misconfigured suite fails
# loudly instead of silently recording a 0/1.
for entry in "${SUITES[@]}"; do
IFS=: read -r name file <<< "$entry"
IFS=: read -r _sname _sfile _spass _sfail _spre <<< "$entry"
if [ -z "${_spass:-$COUNTERS_PASS}" ] || [ -z "${_sfail:-$COUNTERS_FAIL}" ]; then
echo "MODE=counters: suite '${_sname}' has no counter names and COUNTERS_PASS/COUNTERS_FAIL are unset in $CONF" >&2
exit 2
fi
done
for entry in "${SUITES[@]}"; do
# Format: name:file[:pass-var:fail-var[:extra-preload ...]]
# pass-var / fail-var — per-suite counter symbols (default: the global
# COUNTERS_PASS / COUNTERS_FAIL).
# extra-preload ... — space-separated .sx files loaded after the
# global PRELOADS and before the test file. Lets
# each suite bring its own dependency chain.
IFS=: read -r name file spass sfail spre <<< "$entry"
cpass="${spass:-$COUNTERS_PASS}"
cfail="${sfail:-$COUNTERS_FAIL}"
TMPFILE=$(mktemp)
{
printf '(epoch 1)\n'
for f in "${PRELOADS[@]}"; do printf '(load "%s")\n' "$f"; done
# shellcheck disable=SC2086 # deliberate word-split: per-suite preloads
for f in $spre; do printf '(load "%s")\n' "$f"; done
printf '(load "lib/guest/conformance.sx")\n'
printf '(epoch 2)\n'
printf '(load "%s")\n' "$file"
printf '(epoch 3)\n'
printf '(eval "(gc-counters-result \\"%s\\" 0 0 %s %s)")\n' \
"$name" "$COUNTERS_PASS" "$COUNTERS_FAIL"
"$name" "$cpass" "$cfail"
} > "$TMPFILE"
OUTPUT=$(timeout "$TIMEOUT_PER_SUITE" "$SX" < "$TMPFILE" 2>&1 || true)
rm -f "$TMPFILE"