#4 Number of tests succeeded reported incorrectly by framework

open
nobody
None
5
2012-12-11
2006-08-03
mgrollins
No

The last line of output "XXX test(s) succeeded." will
count as successful a test that failed an assert
statement. The shUnit framework will increment the
number of successful tests unles the last assert
statement in a test function fails.

This is because in shuRunOneTest, the test function is
run, and then SHU_TOTAL_NR_SUCCEEDED is incremented
based on the variable SHU_TEST_SUCCEDED being equal to
SHU_TRUE. SHU_TEST_SUCCEDED, however, is written on
every shuAssert and shuDeny, so it only reflects the
status of the last assert statement within a test
function. From bobbynations' comment on bug 1185475,
it appers that you do expect/allow multiple asserts
within a single test function.

SHU_STR_FAILED will be the empty string if no asserts
failed during the execution of a single test function,
as it is initialized empty in the for loop of shuStart
before calling shuRunOneTest.
I believe testing SHU_STR_FAILED empty before
incrementing SHU_TOTAL_NR_SUCCEEDED would be a good
fix, as shown below:

if type ${strTestToRun} > /dev/null 2>&1
then
${strTestToRun}
# if [ ${SHU_TEST_SUCCEEDED} -eq ${SHU_TRUE} ]
if [ "${SHU_STR_FAILED}X" = X ]
then
SHU_TOTAL_NR_SUCCEEDED=`expr
$SHU_TOTAL_NR_SUCCEEDED + 1`
fi

This change does make shuUnitTest report that only 7
tests succeeded, because of the intentional failures in
the first test function.

Discussion

  • Logged In: NO

    This can be corrected by the following patch :

    ===================================================================
    --- shUnit (revision 85)
    +++ shUnit (working copy)
    @@ -49,7 +49,7 @@

    shuRunOneTest() {
    strTestToRun=${1}
    - SHU_TEST_SUCCEEDED=${SHU_FALSE}
    + SHU_TEST_SUCCEEDED=${SHU_TRUE}

    if type shuSetUp > /dev/null 2>&1
    then
    @@ -88,6 +88,7 @@
    fi
    }

    +SHU_RETURN_VALUE=0
    shuStart() {
    strInitFunction="${1}"

    @@ -107,6 +108,7 @@
    shuTestNbr=`expr ${shuTestNbr} + 1`
    printf " Test %i: %s " ${shuTestNbr} "${STR_TEST}"
    shuRunOneTest ${STR_TEST}
    + [ ${SHU_TEST_SUCCEEDED} = ${SHU_FALSE} ] && SHU_RETURN_VALUE=1
    printf "\n"

    if test -n "${SHU_STR_FAILED}"
    @@ -152,7 +154,6 @@
    shuRegisterFailedTest "${strMessage}"
    else
    printf "."
    - SHU_TEST_SUCCEEDED=${SHU_TRUE}
    fi
    }

    @@ -166,7 +167,6 @@
    shuRegisterFailedTest "${strMessage}"
    else
    printf "."
    - SHU_TEST_SUCCEEDED=${SHU_TRUE}
    fi
    }

     
  • Logged In: NO

    This can be corrected by the following patch :

    ===================================================================
    --- shUnit (revision 85)
    +++ shUnit (working copy)
    @@ -49,7 +49,7 @@

    shuRunOneTest() {
    strTestToRun=${1}
    - SHU_TEST_SUCCEEDED=${SHU_FALSE}
    + SHU_TEST_SUCCEEDED=${SHU_TRUE}

    if type shuSetUp > /dev/null 2>&1
    then
    @@ -88,6 +88,7 @@
    fi
    }

    +SHU_RETURN_VALUE=0
    shuStart() {
    strInitFunction="${1}"

    @@ -107,6 +108,7 @@
    shuTestNbr=`expr ${shuTestNbr} + 1`
    printf " Test %i: %s " ${shuTestNbr} "${STR_TEST}"
    shuRunOneTest ${STR_TEST}
    + [ ${SHU_TEST_SUCCEEDED} = ${SHU_FALSE} ] && SHU_RETURN_VALUE=1
    printf "\n"

    if test -n "${SHU_STR_FAILED}"
    @@ -152,7 +154,6 @@
    shuRegisterFailedTest "${strMessage}"
    else
    printf "."
    - SHU_TEST_SUCCEEDED=${SHU_TRUE}
    fi
    }

    @@ -166,7 +167,6 @@
    shuRegisterFailedTest "${strMessage}"
    else
    printf "."
    - SHU_TEST_SUCCEEDED=${SHU_TRUE}
    fi
    }