If a SQLUnit test suite is implemented, I may like to know not only what stored procedures are not returning the correct results but what stored procedures are running slower (e.g., the stored procedure was badly modified). It is clear that testing against elapse time is a hard thing to do since the time may depend on the hardware, network, and so on. However, it may be a good idea to bless the expected results with an expected time and when the execution goes above a certain threshold of that time, we can flag the test with a performance test failure. Note that this is in addition to the normal result checking.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It should be simple to do, since I am capturing the times in any case. I can add two optional attributes to the test tag:
expected-time - the expected time in milliseconds
tolerance - the +/- range tolerance in milliseconds
If these values are available, it will check and report a "performance error". If tolerance is not available, it will assume a default of 0 (or a hardcoded default of 10%).
-sujit
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I added some code to specify the expected time and a tolerance for a test. The code is checked into CVS now. It will be packaged into the next release, but thats going to take a while since the ordering feature that you requested will need me to do some heavy refactoring, so if you need this feature urgently, you should just get it from CVS.
-sujit
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If a SQLUnit test suite is implemented, I may like to know not only what stored procedures are not returning the correct results but what stored procedures are running slower (e.g., the stored procedure was badly modified). It is clear that testing against elapse time is a hard thing to do since the time may depend on the hardware, network, and so on. However, it may be a good idea to bless the expected results with an expected time and when the execution goes above a certain threshold of that time, we can flag the test with a performance test failure. Note that this is in addition to the normal result checking.
It should be simple to do, since I am capturing the times in any case. I can add two optional attributes to the test tag:
expected-time - the expected time in milliseconds
tolerance - the +/- range tolerance in milliseconds
If these values are available, it will check and report a "performance error". If tolerance is not available, it will assume a default of 0 (or a hardcoded default of 10%).
-sujit
Hi Jorge,
I added some code to specify the expected time and a tolerance for a test. The code is checked into CVS now. It will be packaged into the next release, but thats going to take a while since the ordering feature that you requested will need me to do some heavy refactoring, so if you need this feature urgently, you should just get it from CVS.
-sujit