Menu ▾ ▴

#5795 run_testsuite: a test that asks more or fewer questions garbles the rest of its file

None
closed
nobody
5
6 days ago
2026-11-08
No

By Claude:

Test files answer the questions that a test asks, for example from asksign, askinteger or integrate, with bare statements after the input. These are read from the file only when a question comes up. So when a change to Maxima makes a test ask a question it did not ask before, or stop asking one, every later problem in the file is read out of step, and run_testsuite often aborts without printing a summary.

A test file rtest_more.mac whose second problem now asks a question:

(kill(all), 0);
0$

integrate(x^a, x);
x^(a+1)/(a+1)$

1 + 1;
2$

The question Is a equal to - 1? takes the expected result x^(a+1)/(a+1) as its answer. It is not a valid answer, so the question is asked again and takes the next statement, until the file is used up:

(%i1) testsuite_files:["rtest_more"]
(%i2) run_testsuite()
Running tests in rtest_more: Is a equal to - 1?

Is a equal to - 1?

Is a equal to - 1?

Is a equal to - 1?

batch: missing expected result in test script.
 -- an error. To debug this try: debugmode(true);

No summary is printed, and the test files after this one never run.

The reverse happens when a test stops asking: its answer pos in asksign(x^2 + 1); pos; is taken for the expected result, the problem passes by accident, and every later problem is shifted by one statement, again until the run aborts with batch: missing expected result in test script.

Plain batch input has the first problem too. With maxima --batch-string='integrate(x^a, x); 2 + 2; 3 + 3;', the question takes both 2 + 2 and 3 + 3 as answers, and neither is ever evaluated.

Discussion

  • Leo Butler

    Leo Butler - 2026-11-09

    The current behaviour is that ask and related functions re-ask the question when the answer is unacceptable. That, combined with the batch_answers_from_file flag, is causing what you see.

    I think the only good solution here is to make the ask questions call merror instead of mtell with its message and then abort the remaining tests. When there is an error, we should echo the question as part of the error message, so the user can easily figure out where it is coming from.

    For the second type of error, I don't see a good way to catch it without fundamentally re-designing how tests are handled. E.g. something like:

    asksign(x^2+1); pos;
    pos;
    

    could be re-crafted like

    should(asksign(x^2+1),pos,pos);
    

    where should is defined to evaluate the first argument (and, if needed, supply answers to any questions) and then compare the result with the un-evaluated second argument. If a test has a superfluous answer supplied, then it would be marked as a failure with some warning emitted.
    I think should could be a boolean-valued special function, so

    should(asksign(x^2+1),pos,pos);
    true $
    

    would work as expected in an rtest*.mac file.

     
  • David Scherfgen

    David Scherfgen - 6 days ago
    • status: open --> closed
     
  • David Scherfgen

    David Scherfgen - 6 days ago

    Fixed by commit [f6fd0a], which introduces the new with_answers and converts all existing tests to use it. See the manual entry for details.

     

    Related

    Commit: [f6fd0a]


Log in to post a comment.