Let's suppose I have a dozen of test cases, some of them fairly long to run. When debugging, I usually start to write a test triggering the issue, then step into the code. I find two things annoying there :
1- Tests are registered from bottom to top. The few other test frameworks I know either register tests manually or register them from top to bottom. This behaviour is important because I often write tests like code: more specific/inner layers first, and I like issues to be reported in this order. It could be easily changed by patching the add() method of test suites. What do you think about this ?
2- I know that manual test registration bugs you, but again when debugging, it allows run tests to be selected (at compile time). This is useful when the test suite contains time consuming tests. How do you handle this case ? I see several solutions:
a- Comment all tests you want to skip: I do not really like commenting tons and tons of code like this.
b- Write tests in separate functions, and call the functions from the test cases. Comment unused test cases like in (a) but they are likely to be one liners this time. Maybe, but it looks like test registration again :-).
Do you think (b) is the way to go and should be advertised as such ? Or do you have another solution ?
Thank you for any ideas, and for unit--.
Patrick
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1, test cases are registered and run with an undefined order. for those general/outer layers, personally, i prefer to provide some mock objects that emulate the needed inner objects.
it's just unit test, rather than integrated test -- any layers should be tested independently. when i find it impossible, i know it might be the time to improve the design
2, it is a feature to be added
but i'm still considering what is an advisable interface
it might have the mechanism to:
* filter unneeded(e.g. time consuming) units
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1- Actually, tests do not depend one of each other. What I meant by layer order is that in a layered class/whatever hierarchy, I usually write inner classes first. Then seeing "inner" tests failing before "outer" ones in the test report clearly tells me not to look at outer layers for errors, at least not until inner layers runs correctly. And to ease report decyphering, "inner" tests usually executes before "outer" ones. It is not about test configurations or setup, just about tests reporting order. But I can leave without it.
2- First time I wanted to select tests, I started looking at unit-- implementation to see whether I could register tests manually. Then I checked command line options, looking for something like python unittest interface :
"""
Usage: test_python.py [options] [test] [...]
Options:
-h, --help Show this message
-v, --verbose Verbose output
-q, --quiet Minimal output
Examples:
test_python.py - run default set of tests
test_python.py MyTestSuite - run suite 'MyTestSuite'
test_python.py MyTestCase.testSomething - run MyTestCase.testSomething
test_python.py MyTestCase - run all 'test*' test methods
in MyTestCase
"""
What do you think about this? Maybe it is missing some 'do not run this/these tests' option form?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
Let's suppose I have a dozen of test cases, some of them fairly long to run. When debugging, I usually start to write a test triggering the issue, then step into the code. I find two things annoying there :
1- Tests are registered from bottom to top. The few other test frameworks I know either register tests manually or register them from top to bottom. This behaviour is important because I often write tests like code: more specific/inner layers first, and I like issues to be reported in this order. It could be easily changed by patching the add() method of test suites. What do you think about this ?
2- I know that manual test registration bugs you, but again when debugging, it allows run tests to be selected (at compile time). This is useful when the test suite contains time consuming tests. How do you handle this case ? I see several solutions:
a- Comment all tests you want to skip: I do not really like commenting tons and tons of code like this.
b- Write tests in separate functions, and call the functions from the test cases. Comment unused test cases like in (a) but they are likely to be one liners this time. Maybe, but it looks like test registration again :-).
Do you think (b) is the way to go and should be advertised as such ? Or do you have another solution ?
Thank you for any ideas, and for unit--.
Patrick
Hi,
Thanks for your post, here is my consideration:
1, test cases are registered and run with an undefined order. for those general/outer layers, personally, i prefer to provide some mock objects that emulate the needed inner objects.
it's just unit test, rather than integrated test -- any layers should be tested independently. when i find it impossible, i know it might be the time to improve the design
2, it is a feature to be added
but i'm still considering what is an advisable interface
it might have the mechanism to:
* filter unneeded(e.g. time consuming) units
1- Actually, tests do not depend one of each other. What I meant by layer order is that in a layered class/whatever hierarchy, I usually write inner classes first. Then seeing "inner" tests failing before "outer" ones in the test report clearly tells me not to look at outer layers for errors, at least not until inner layers runs correctly. And to ease report decyphering, "inner" tests usually executes before "outer" ones. It is not about test configurations or setup, just about tests reporting order. But I can leave without it.
2- First time I wanted to select tests, I started looking at unit-- implementation to see whether I could register tests manually. Then I checked command line options, looking for something like python unittest interface :
"""
Usage: test_python.py [options] [test] [...]
Options:
-h, --help Show this message
-v, --verbose Verbose output
-q, --quiet Minimal output
Examples:
test_python.py - run default set of tests
test_python.py MyTestSuite - run suite 'MyTestSuite'
test_python.py MyTestCase.testSomething - run MyTestCase.testSomething
test_python.py MyTestCase - run all 'test*' test methods
in MyTestCase
"""
What do you think about this? Maybe it is missing some 'do not run this/these tests' option form?
hello,
i'm thinking about the next step of unit--
1- ordering the report might be a interesting feature, i may try to implement it or similar
2- filtering is cool, but unit--.cpp does not have this feature. i'm considering to implement a new test runner which can do it.
Thanks for your advices :-)