#20 Smolder doesn't count unrun tests as failed

open
nobody
UI (2)
5
2009-03-03
2009-03-03
Pete Krawczyk
No

Given the following TAP, Smolder says my test was 50% successful, when it should only be 33.33% successful:

1..3
ok 1
not ok 2

Discussion

  • This isn't really so much of a Smolder problem as a TAP interpretation problem. Does your example mean 1 of 2 tests failed and you gave a bad plan? Or does it mean that your plan was right but you didn't actually follow it?

    I'll post the TAP list to see what the consensus is for that and then change (or not) Smolder accordingly.

     
  • Pete Krawczyk
    Pete Krawczyk
    2009-03-04

    The plan is actually correct - there were 3 tests that were supposed to be run; test 3 wasn't run, in this case because test 2's failure caused the test to die.

     
  • Pete Krawczyk
    Pete Krawczyk
    2009-03-04

    Also, prove treats the unrun test as failed, as well:
    $ (echo '1..2'; echo 'ok 1') | prove -v --exec='cat -' /dev/null
    /dev/null....
    1..2
    ok 1
    Failed 1/2 subtests

    Test Summary Report
    -------------------
    /dev/null (Wstat: 0 Tests: 1 Failed: 0)
    Parse errors: Bad plan. You planned 2 tests but ran 1.
    Files=1, Tests=1, 0 wallclock secs ( 0.03 usr + 0.01 sys = 0.04 CPU)
    Result: FAIL
    $