conformance: migrate common-lisp onto shared driver (counters, 487/487)
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 14m2s
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 14m2s
Extend the shared driver's MODE=counters with a backward-compatible SUITES format: name:file[:pass-var:fail-var[:extra-preload ...]]. Optional per-suite counter symbols (override the global COUNTERS_PASS/COUNTERS_FAIL) and per-suite preload chains (loaded after the global PRELOADS). Plain name:file entries are unchanged — verified against haskell (fib/sieve/quicksort 2/2/5, matches committed scoreboard). common-lisp has 8 distinct per-suite counter pairs and a different preload chain per suite, so it could not fit the single-counter/fixed-preload model; the extended format expresses it directly. conformance.conf keeps the historical scoreboard schema; conformance.sh becomes the 3-line shim. Result 487/487 (0 fail) vs the old 305/0 baseline — higher and explained: the old per-suite 'timeout 30' was too tight for the slow eval suite (~15-25s under contention), silently recording it as 0; the driver's 180s budget recovers its true 182. geometry/mop-trace stay 0/0 (pre-existing refl-class-chain-depth-with load error; counter vars defined as 0 -> clean gc-result, no fail-fallback). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -21,10 +21,17 @@
|
||||
# MODE=dict — "name:test-file:(runner-fn)"
|
||||
# The runner expression is evaluated and is expected to
|
||||
# return a dict with :passed/:failed/:total.
|
||||
# MODE=counters — "name:test-file"
|
||||
# MODE=counters — "name:test-file[:pass-var:fail-var[:extra-preload ...]]"
|
||||
# Each suite is run in a fresh sx_server session: preloads
|
||||
# are loaded, then the test file, then counters are read.
|
||||
# The suite is treated as starting from counters (0, 0).
|
||||
# Optional per-suite fields:
|
||||
# pass-var/fail-var — counter symbols for this suite,
|
||||
# overriding COUNTERS_PASS/COUNTERS_FAIL.
|
||||
# extra-preload ... — space-separated .sx files loaded
|
||||
# after the global PRELOADS (per-suite
|
||||
# dependency chains).
|
||||
# Plain "name:test-file" still works (uses the globals).
|
||||
#
|
||||
# Output:
|
||||
# Writes $SCOREBOARD_DIR/scoreboard.json and $SCOREBOARD_DIR/scoreboard.md.
|
||||
@@ -163,22 +170,39 @@ case "$MODE" in
|
||||
fi
|
||||
;;
|
||||
counters)
|
||||
if [ -z "$COUNTERS_PASS" ] || [ -z "$COUNTERS_FAIL" ]; then
|
||||
echo "MODE=counters requires COUNTERS_PASS and COUNTERS_FAIL in $CONF" >&2
|
||||
exit 2
|
||||
fi
|
||||
# Each suite must resolve to a pass/fail counter name — either per-suite
|
||||
# (fields 3 & 4 of the SUITES entry) or via the global COUNTERS_PASS /
|
||||
# COUNTERS_FAIL defaults. Validate up front so a misconfigured suite fails
|
||||
# loudly instead of silently recording a 0/1.
|
||||
for entry in "${SUITES[@]}"; do
|
||||
IFS=: read -r name file <<< "$entry"
|
||||
IFS=: read -r _sname _sfile _spass _sfail _spre <<< "$entry"
|
||||
if [ -z "${_spass:-$COUNTERS_PASS}" ] || [ -z "${_sfail:-$COUNTERS_FAIL}" ]; then
|
||||
echo "MODE=counters: suite '${_sname}' has no counter names and COUNTERS_PASS/COUNTERS_FAIL are unset in $CONF" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
for entry in "${SUITES[@]}"; do
|
||||
# Format: name:file[:pass-var:fail-var[:extra-preload ...]]
|
||||
# pass-var / fail-var — per-suite counter symbols (default: the global
|
||||
# COUNTERS_PASS / COUNTERS_FAIL).
|
||||
# extra-preload ... — space-separated .sx files loaded after the
|
||||
# global PRELOADS and before the test file. Lets
|
||||
# each suite bring its own dependency chain.
|
||||
IFS=: read -r name file spass sfail spre <<< "$entry"
|
||||
cpass="${spass:-$COUNTERS_PASS}"
|
||||
cfail="${sfail:-$COUNTERS_FAIL}"
|
||||
TMPFILE=$(mktemp)
|
||||
{
|
||||
printf '(epoch 1)\n'
|
||||
for f in "${PRELOADS[@]}"; do printf '(load "%s")\n' "$f"; done
|
||||
# shellcheck disable=SC2086 # deliberate word-split: per-suite preloads
|
||||
for f in $spre; do printf '(load "%s")\n' "$f"; done
|
||||
printf '(load "lib/guest/conformance.sx")\n'
|
||||
printf '(epoch 2)\n'
|
||||
printf '(load "%s")\n' "$file"
|
||||
printf '(epoch 3)\n'
|
||||
printf '(eval "(gc-counters-result \\"%s\\" 0 0 %s %s)")\n' \
|
||||
"$name" "$COUNTERS_PASS" "$COUNTERS_FAIL"
|
||||
"$name" "$cpass" "$cfail"
|
||||
} > "$TMPFILE"
|
||||
OUTPUT=$(timeout "$TIMEOUT_PER_SUITE" "$SX" < "$TMPFILE" 2>&1 || true)
|
||||
rm -f "$TMPFILE"
|
||||
|
||||
Reference in New Issue
Block a user