Harry Mangalam wrote:
> shouldn't all of these shell scripts be integrated into the regression
> tests / benchmark code? Is there a reason to have them spread out in shell
> scripts?
I think Rorik was trying to isolate the network-dependent parts
of the regression tests so that a failure on these would not cause
'make test' to fail. I can't remember the exact reasoning.
If the regression test discovers an unforeseen problem in NCO
functionality, a true bug that gives bad answers, then the 'make test'
should set off an alarm so the user knows not to run 'make install'.
However, lack of network connection is not an NCO bug.
Thus the exit value of 'make test' should be success if only the
network-dependent stuff fails.
When you put this all together, it may be that the network stuff
was broken out so that its exit value would not cause the autotools
Makefile to exit with error due to network stuff.
I think we can obsolete test1.sh and test2.sh by incorporating them
into the regression test in nco_bm.pl and still meet these design
principles. The regression test summary just needs to be clear
about the distinction between network-dependent (including firewall)
failures and true NCO failures.
Thanks,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Those scripts date from back when there wasn't any real testing, so I'm sure they should be canned now. It is true that I wrote them in an attempt to only run when needed, i.e. DODS-enabled, and was also part of my autotools-learning process. Things are way beyond that now.
rorik
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Harry Mangalam wrote:
> shouldn't all of these shell scripts be integrated into the regression
> tests / benchmark code? Is there a reason to have them spread out in shell
> scripts?
I think Rorik was trying to isolate the network-dependent parts
of the regression tests so that a failure on these would not cause
'make test' to fail. I can't remember the exact reasoning.
If the regression test discovers an unforeseen problem in NCO
functionality, a true bug that gives bad answers, then the 'make test'
should set off an alarm so the user knows not to run 'make install'.
However, lack of network connection is not an NCO bug.
Thus the exit value of 'make test' should be success if only the
network-dependent stuff fails.
When you put this all together, it may be that the network stuff
was broken out so that its exit value would not cause the autotools
Makefile to exit with error due to network stuff.
I think we can obsolete test1.sh and test2.sh by incorporating them
into the regression test in nco_bm.pl and still meet these design
principles. The regression test summary just needs to be clear
about the distinction between network-dependent (including firewall)
failures and true NCO failures.
Thanks,
Charlie
Those scripts date from back when there wasn't any real testing, so I'm sure they should be canned now. It is true that I wrote them in an attempt to only run when needed, i.e. DODS-enabled, and was also part of my autotools-learning process. Things are way beyond that now.
rorik
thanks, Rorik,
I'll merge the functionality into the new regression./benchmarking script and remove them from cvs when it's done.
harry