#557 non-empty return from [runAllTests] - indicate status

[Don Porter]

Request from Larry McVoy posted to TCLCORE:

In our tree we would like to include a subset of the tcl tests in our
normal builds and have it fail the build if the tests do not pass.
This patch accomplishes that for a subset or for all tests.

===== library/tcltest/tcltest.tcl 1.105 vs edited =====
--- 1.105/library/tcltest/tcltest.tcl 2008-02-11 09:34:32 -08:00
+++ edited/library/tcltest/tcltest.tcl 2009-05-12 14:13:10 -07:00
@@ -2735,6 +2735,7 @@ proc tcltest::runAllTests { {shell ""} }

set timeCmd {clock format [clock seconds]}
puts [outputChannel] "Tests began at [eval $timeCmd]"
+ set exit_status 0

# Run each of the specified tests
foreach file [lsort [GetMatchingFiles]] {
@@ -2772,6 +2773,7 @@ proc tcltest::runAllTests { {shell ""} }
}
if {$Failed > 0} {
lappend failFiles $testFile
+ set exit_status 1
}
} elseif {[regexp [join {
{^Number of tests skipped }
@@ -2799,6 +2801,7 @@ proc tcltest::runAllTests { {shell ""} }
puts [outputChannel] "\nTests ended at [eval $timeCmd]"
cleanupTests 1
if {[info exists testFileFailures]} {
+ set exit_status 1
puts [outputChannel] "\nTest files exiting with errors: \n"
foreach file $testFileFailures {
puts [outputChannel] " [file tail $file]\n"
@@ -2818,7 +2821,7 @@ proc tcltest::runAllTests { {shell ""} }
puts [outputChannel] ""
puts [outputChannel] [string repeat ~ 44]
}
- return
+ return $exit_status
}

#####################################################################
===== tests/all.tcl 1.18 vs edited =====
--- 1.18/tests/all.tcl 2006-11-02 16:34:52 -08:00
+++ edited/tests/all.tcl 2009-05-12 14:05:08 -07:00
@@ -16,4 +16,4 @@ package require Tcl 8.5
package require tcltest 2.2
namespace import tcltest::*
configure {*}$argv -testdir [file dir [info script]]
-runAllTests
+exit [runAllTests]

[Jan Nijtmans]

Patch looks fine to me. Seconded.

[Don Porter]

I agree it's a step in the right direction.

I logged it here for a chance to tweak it
just a big before adopting it.

[Erik Leunissen]

I agree with the need for [runAllTests] returning information regarding succes/failure. I supply a patch that improves upon the previous patch as follows:

Besides failed tests, the new patch additionally takes into account any test file failures. It differentiates between the two types of failure with the sign of the numeric result.

(I didn't find the option "attach a file" to this issue tracker. Therefore, like Larry McVoy, I just copy the text of the patch into this comment:)

--- tcltest.tcl.orig 2013-01-19 14:01:28.000000000 +0100
+++ tcltest.tcl 2013-01-19 14:08:58.000000000 +0100
@@ -2833,6 +2833,21 @@
}
}

+ # Prepare a result, indicating whether all is well
+ # (i.e. no test file failures and no failed tests).
+ #
+ # The result is a single integer, where:
+ # - a negative number indicates the number of test file failures
+ # (ignoring the number of failed tests).
+ # - a positive number indicates the number of failed tests,
+ # given zero test file failures.
+ # - zero indicates no failures at all
+ #
+ if {![info exists testFileFailures] ||
+ ![set result [llength $testFileFailures]]} {
+ set result $numTests(Failed)
+ }
+
# cleanup
puts [outputChannel] "\nTests ended at [eval $timeCmd]"
cleanupTests 1
@@ -2856,7 +2871,7 @@
puts [outputChannel] ""
puts [outputChannel] [string repeat ~ 44]
}
- return
+ return $result
}

#####################################################################

[Erik Leunissen]

Hmm (dissatisfaction): The copy/paste doesn't preserve the exact diff information. If someone tells me how/where to post the patch *file*, Ill be happy to do so.

Additional info: the supplied patch was made against tcltest.tcl 2.3.5 (latest version I found in the Fossil repo today).

[Erik Leunissen]

Oh, and of course true negative values are achieved by using a minus sign. Duh!

--- tcltest.tcl.orig 2013-01-19 14:01:28.000000000 +0100
+++ tcltest.tcl 2013-01-19 17:10:04.000000000 +0100
@@ -2833,6 +2833,21 @@
}
}

+ # Prepare a result, indicating whether all is well
+ # (i.e. no test file failures and no failed tests).
+ #
+ # The result is a single integer, where:
+ # - a negative number indicates the number of test file failures
+ # (ignoring the number of failed tests).
+ # - a positive number indicates the number of failed tests,
+ # given zero test file failures.
+ # - zero indicates no failures at all
+ #
+ if {![info exists testFileFailures] ||
+ ![set result -[llength $testFileFailures]]} {
+ set result $numTests(Failed)
+ }
+
# cleanup
puts [outputChannel] "\nTests ended at [eval $timeCmd]"
cleanupTests 1
@@ -2856,7 +2871,7 @@
puts [outputChannel] ""
puts [outputChannel] [string repeat ~ 44]
}
- return
+ return $result
}

#####################################################################