#861 the model-QA arm of make check cannot fail, and re-uses stale results

[ananthsforge]

From Claude Code

Version: 46+ (master); the scripts are unchanged in 46
OS: Ubuntu 24.04.4, x86-64
Severity: test infrastructure reports PASS for results it has just printed as wrong

Summary

Of the 59 tests make -C tests check runs, the 13 model-QA cases (bsim3, bsim4, bsimsoi, hisim, hicum2, hisimhv1, hisimhv2) always pass. Two independent reasons:

1. The comparator has no failing exit path. tests/bin/compareSimulationResults.pl contains exactly three exit calls, all exit(0) in usage/info handling. It prints DIFFER lines and returns success regardless.
2. Results are re-used, not re-simulated. tests/bin/run_cmc_check:73 passes -r to runQaTests.pl, documented as "re-use previously simulated results if they exist (default is to resimulate, even if results exist)". After the first make check, those 13 tests never invoke ngspice again.

Evidence

From one full make -C tests check run (exit 0, "All N tests passed" for every group):

$ grep -c "DIFFER" check.log
515
$ grep -cE "DIFFER.max rel error is [1-9][0-9].[0-9]+%" check.log
189
$ grep -c "max rel error is 100%" check.log
9

Direct proof that the driver cannot fail — run it with a simulator that produces nothing:

$ check_cmc.sh ... /bin/false ...
(every comparison prints FAIL)
$ echo $?
0

Impact

make check cannot detect a model regression in any of those seven device models, and a maintainer reading the summary sees 59/59 green while 189 comparisons exceeded 1% relative error and nine were at 100%.

Suggested fix

Have compareSimulationResults.pl exit non-zero when any comparison exceeds its tolerance, and have run_cmc_check propagate that status. Separately, reconsider -r: re-using results across runs means a rebuild is never actually re-tested.