Menu
▾
▴
#1 Add overall summary for test suite
I'd like to see an overall result displayed when a test
suite is run, so that you can tell at a glance if all of the
tests passed, or one or more failed. Currently, when
running a test suite, you must check for success or
failure individually for each test package.
Related
Feature Request Archive (Use GitHub now): #7
News: 2015/02/new-release-v230
The OUnit front-end gives you a nice big green tick if everything passes (or a big red cross if one or more fail).
This question was obviously asked a long time ago, so I don't know if OUnit existed then!
It'd be good if we could get the actual output from running a test suite added to this Ticket for reference so we can see what the current output looks like...
I think it is useful. I'd like to try implement this. In fact, I already have a brief implementation for Output reporter, using utoutput.extract and utoutput.replace.
Attachements:
testsuite_current_output.txt - actual output from Output Reporter.
testsuite_output_with_overall.txt - output of mine brief implementation. Difference is "suite success" of "suite failure" in the beginning.
That looks good. As well as having the
SUITE SUCCESSin large block caps, could you also put it as a normal line underneath it, the same as it does for the normal tests? (I've attached a modified version of your attachment showing what I mean at line 12).What would you show if there were one or more failures and/or errors? Would you list the package names?
(By the way, I've added you as a developer on this project and assigned the ticket to you)
Last edit: Paul Walker 2014-04-06
Changes description:
- Added new method to reporter interface
before_suite_results, called only for suite execution.- Added
before_suite_resultsimplementation for OUTPUT reporter. Display overall banner and suite execution statistics.- File reporter also display suite overall banner and execution statistics, as it uses OUTPUT reporter's method.
- Added
before_suite_resultsimplementation for HTML reporter. Display overall banner. (Without statistics for now)- Fixed bug: missed call of 'utreport.open' As result, HTML reporter generated file without opening '<HTML>' tag, '<HEAD>' and '<TITLE>' sections and opening '<BODY>' tag.
- Fixed bug: missed call of
utreport.closefor suite execution. As result, FILE reporter and HTML reporter couldn't close file.Couple things to go:
- Bug to fix: HTML reporter always show package name from last test execution. I will create a bug ticket.
- Enhancement: suite execution statistics should be collected outside of OUTPUT reporter. This will make it accessible for others reporters.
- To think: Current implementation uses ut_package.last_run_id to show packages results. It means, that we suppose that all last runs were done during current suite execution. Normally it is true.
- Enhancement: when we call utreport.show without parameter, OUTPUT reporter shows package name. when we call utreport.show passing run_id, OUTPUT reporter shows
RUNIDinstead of package name. I think reporter should always show package name or package name + RUNID.- Enchancement: implement order for package results. Same as in ticket #33
Attachements:
source.patch - patch
outputreporter_success.log, outputreporter_failure.log - result log for output reporter
filereporter_success.log, filereporter_failure.log - result log for file reporter
htmlreporter_success.html, htmlreporter_failure.html - result log for html reporter
Last edit: Svyatoslav Chatchenko 2014-04-18
I found a case when it isn't true. It happens when testing package cannot be found. Error
PLS-00201: identifier '...' must be declaredI'm gonna think about it and fix it.
I've made a branch in SVN and applied the "source.patch" from above as [r174]
Related
Commit: [r174]
I did some changes.
Chagelog
1. There was a bug that I mentioned in previous post. Steps to reproduce: run suite that will succeed; delete ut_teardown from test package; run suite again. As result you will get "success'. The reason for that is when ut_teardown raise exception there is no update of ut_package. Fix in
utplsqlpackage.2. Moved statistic compute from outputreporter to
utresult.get_suite_stats. Now it can be used by other reporters.3. Some enhancements in
utoutputreporter:3.1. Always shows packagename and runID. To implement this I added function
name_from_last_run_idtoutpackage.3.2. Always shows owner of package. It is very usefull as suite can have packages from different owners or even packages with same name from different owners (
ut_package.add(... owner_in ),ut_package.owner). To implement this i added functionowner_from_last_run_idtoutpackage.4. Now
showsuiteshows individual packages results in same order as packages were executed.Attachements:
patch_to_revision_175.patch - patch
result_success.log - example of log produced by outputreporter
result_failure.log - example of log produced by outputreporter
Other:
1. As i already mentioned, current implementation is based on usage of
ut_package.last_run_id. Well, i think that normally it is totally fine. So i don't plan change this.2. Similar changes as I made to outputreporter can be done to other reporters. But it should be done in others tickets. Reason for that is to reduce scope of changes to release it earlier.
Paul, should I do something else on this ticket or for now my work here is done?
Pardon, this is a little bit changed implementation of feature-requests-7.
Should I leave it here, or extract from this patch and attach in feature-requests-7 ?
Thanks for your work on this. Leave it with me for now and I'll have a look over it.
Will this fully address [feature-requests:#7]?
Related
Feature Request Archive (Use GitHub now):
#7Yes.
Related
Feature Request Archive (Use GitHub now):
#7I've updated the documentation to reflect the new
before_suite_resultsprocedure. Committed to the branch as [r181]Related
Commit: [r181]
Merged into trunk in [r182]
Branch deleted in [r183]
Related
Commit: [r182]
Commit: [r183]