By Claude:
Test files answer the questions that a test asks, for example from asksign, askinteger or integrate, with bare statements after the input. These are read from the file only when a question comes up. So when a change to Maxima makes a test ask a question it did not ask before, or stop asking one, every later problem in the file is read out of step, and run_testsuite often aborts without printing a summary.
A test file rtest_more.mac whose second problem now asks a question:
(kill(all), 0);
0$
integrate(x^a, x);
x^(a+1)/(a+1)$
1 + 1;
2$
The question Is a equal to - 1? takes the expected result x^(a+1)/(a+1) as its answer. It is not a valid answer, so the question is asked again and takes the next statement, until the file is used up:
(%i1) testsuite_files:["rtest_more"]
(%i2) run_testsuite()
Running tests in rtest_more: Is a equal to - 1?
Is a equal to - 1?
Is a equal to - 1?
Is a equal to - 1?
batch: missing expected result in test script.
-- an error. To debug this try: debugmode(true);
No summary is printed, and the test files after this one never run.
The reverse happens when a test stops asking: its answer pos in asksign(x^2 + 1); pos; is taken for the expected result, the problem passes by accident, and every later problem is shifted by one statement, again until the run aborts with batch: missing expected result in test script.
Plain batch input has the first problem too. With maxima --batch-string='integrate(x^a, x); 2 + 2; 3 + 3;', the question takes both 2 + 2 and 3 + 3 as answers, and neither is ever evaluated.
The current behaviour is that
askand related functions re-ask the question when the answer is unacceptable. That, combined with thebatch_answers_from_fileflag, is causing what you see.I think the only good solution here is to make the
askquestions callmerrorinstead ofmtellwith its message and then abort the remaining tests. When there is an error, we should echo the question as part of the error message, so the user can easily figure out where it is coming from.For the second type of error, I don't see a good way to catch it without fundamentally re-designing how tests are handled. E.g. something like:
could be re-crafted like
where
shouldis defined to evaluate the first argument (and, if needed, supply answers to any questions) and then compare the result with the un-evaluated second argument. If a test has a superfluous answer supplied, then it would be marked as a failure with some warning emitted.I think
shouldcould be a boolean-valued special function, sowould work as expected in an
rtest*.macfile.Fixed by commit [f6fd0a], which introduces the new
with_answersand converts all existing tests to use it. See the manual entry for details.Related
Commit: [f6fd0a]