Re: [Maxima-discuss] nonzero exit on error, was: Writing tests

[Leo B. <Leo...@um...>]

On Mon, Apr 07 2025, Raymond Toy <toy...@gm...> wrote:

> On 4/7/25 10:30 AM, Leo Butler wrote:
>
>> On Mon, Apr 07 2025, Robert Dodier<rob...@gm...> wrote:
>>
>>> On Mon, Apr 7, 2025 at 2:29 AM Michel Talon<ta...@lp...> wrote:
>>>
>>>> In fact to arrange that your scripts get killed with an exit error as soon as there is an error in them you can do the following, for sbcl.
>>> Along these lines, I seem to recall that there was in the
>>> not-too-distant past some discussion about getting Maxima to exit with
>>> a nonzero exit code when an error is encountered while processing
>>> noninteractively (via --batch at least, maybe other means). Did
>>> someone make some progress on that? I think it's a great idea, but I
>>> didn't work on it myself.
>> Yes, it is in current HEAD.
>>
>> On other hand, I think this user is asking for something a bit
>> different, since one wants the whole testsuite to finish and then Maxima
>> should exit with exit code equal to the number of (unexpected) errors in
>> the test file.
>>
>> The attached patch implements such a function. Here is how it works:
>>
>> $ ./maxima-local -p src/mload.lisp --no-init --batch-string='quit(batch_test("tests/rtest1.mac"));'
>
> There are about 14000 tests. Isn’t there a limit on the value of the
> exit code? I have a vague memory that 255 is usually the max.

Well, that is easy enough to remedy: use quit(1-kron_delta(..., 0)) 

Note that this is not a patch to `run_testsuite', but an extension of
batch(..., test).

>
> Is there really anything to be gained by returning the number of
> tests? Is it really meaningful to know that 10117 tests failed instead
> of 103?

Sure, many unit-test frameworks display stuff like:

Test XYZ: N passed, M expected failures, P unexpected failures out of Q tests.

Currently, Maxima prints stuff like this, but it does not provide
programmatic access to that data to post-process results. Maybe we
should, though.

Leo