#45 status of XML output and integration with Jenkins
Hi all,
I'm curious about the status of the xml output for pFUnit and if it can be enhanced. I wasn't able to find much documenting the expected behavior for this feature. Currently, it seems that the output consists of test names but no information about the status of the individual tests. Test suites seem to have summary information about number of failures, but the information doesn't seem to go any finer than that for the individual testcases.
Is this the expected behavior currently? If so, are there plans to add more capability for reporting? Maybe an example output snippet in the documentation would be beneficial.
I am using Jenkins to manage building and testing and am interested in working the pFUnit output into a published format through Jenkins.
Nathan
Related
It sounds like some other changes may have broken the functionality. This was externally contributed code that we do not use in house - other than to make certain that the unit tests pass. I know that there was another externally contributed change to that module in the last few months, but had assumed that those contributing developers would have complained if the merge did not do anything useful.
So, I don’t think this is the expected behavior, but will need to ask the other contributors. Ditto for contributed documentation.
I probably cannot do much on this until ~ 2 weeks from now due to upcoming travel.
On Jul 28, 2015, at 12:36 PM, Nathan Wukie nwukie@users.sf.net wrote:
Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
Head ASTG,Code 606
NASA GSFC
MS 610.8 B33-C128
Greenbelt, MD 20771
301-286-4635
Related
Feature Requests: #45
Thanks for the update Tom. I'll check back in after a few weeks. Safe travels.
OK - started looking at this now. I went into a project and purposefully made a test that failed. I see that the xml output then includes some additional info for that test:
<teststatus name="Test_Formatter_mod_suite.test_fail">
<testcase name="Test_Formatter_mod_suite.test_fail">
<failure message="Location: <span>[Test_Formatter.pf:251]</span>, expected 1 but found: 2; difference: |1|. ">
</failure></testcase>
</teststatus>
Is your concern that there should be a more explicit message for tests that succeed?
I have not given much thought to this issue, and thus must be guided in terms of what behavior would be useful. (Now trying to go back to find out who contributed the existing code ...)
Cheers,
I contributed the XmlPrinter in Feb/Mar 2014, with the intent that the XML output could be read by Jenkins. However, it seems that a commit on Jun 5 2014 added some more output (for status updates while the tests are running), which I think is noncompliant with the JUnit XML specification (and Jenkins). I've suggested to remove this output in merge request 16, which should make it work with Jenkins again. This intermediate output could rather be written to a separate file if necessary.
As for documentation, I realise now that it is scarce (or nonexistent). The options that I use are:
-xml filename.xml
Names the filename to which XML should be written
-name suitename
Gives a name to the suite, which lets Jenkins separate between XML outputs (if you run more than one set/suite of tests)
I recommend running the tests with the -robust flag, to tackle run-time errors properly.
Tom: I can write a short explanation of the XML feature in the doxygen file, if that's the appropriate way to document it?
Last edit: Halvor Lund 2015-08-18
Halvor,
Thanks. Somehow I missed merge request #16 in my sweep through open issues this morning.
And feel free to add documentation in doxygen file.
Cheers,
On Aug 17, 2015, at 4:54 PM, Halvor Lund halvorlu@users.sf.net wrote:
Thomas Clune, Ph. D. Thomas.L.Clune@nasa.gov
Head ASTG,Code 606
NASA GSFC
MS 610.8 B33-C128
Greenbelt, MD 20771
301-286-4635
Related
Feature Requests: #45
Tom and Halvor,
Thanks for following up on this.
Tom:
Not having much experience with xml output myself and what might be standard output, I don't know that I have a concrete definition of what I expect from the xml output. My main desire was to be able to report test results in the Jenkins environment. My initial thought was that each test would report pass/fail/skip along with some additional metrics like execution time. It sounds like the fix proposed by Halvor get things in line with the JUnit format and playing nice again with Jenkins so I will give that a whirl when it gets rolled in.
Halvor:
Thanks for including some info on usage. I think just a short blurb in the documentation on usage along with an example of what one would expect to see reported would be quite useful.
Nathan