How to make a detailed test report

2004-01-30
2013-04-22
  • Ekart Laszlo

    Ekart Laszlo - 2004-01-30

    What I am seeking is make a detailed test report.
    If I do this for example:

    // Create the event manager and test controller
        CPPUNIT_NS::TestResult controller;

        // Add a listener that collects test result
        CPPUNIT_NS::TestResultCollector result;
        controller.addListener( &result );

        CppUnit::TestRunner runner;
        runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() );
        runner.run( controller , "" );

        std::ofstream file( "results.xml" );
        CPPUNIT_NS::XmlOutputter xml( &result, file );
        xml.write();
        file.close();

    Then this code will generate to me (for example) a a folowing test report:

    <?xml version="1.0" encoding='ISO-8859-1' standalone='yes' ?>
    <TestRun>
      <FailedTests>
        <FailedTest id="2">
          <Name>CFirstTester::secondFunction</Name>
          <FailureType>Assertion</FailureType>
          <Location>
            <File>D:\CppUnitTesting\FirstAcceptable\FirstTester.cpp</File>
            <Line>40</Line>
          </Location>
          <Message>assertion failed
    - Expression: false
    </Message>
        </FailedTest>
      </FailedTests>
      <SuccessfulTests>
        <Test id="1">
          <Name>CFirstTester::firstTest</Name>
        </Test>
        <Test id="3">
          <Name>CFirstTester::thirdFunction</Name>
        </Test>
      </SuccessfulTests>
      <Statistics>
        <Tests>3</Tests>
        <FailuresTotal>1</FailuresTotal>
        <Errors>0</Errors>
        <Failures>1</Failures>
      </Statistics>
    </TestRun>

    Now what I want is that the report might look like this:

      <FailedTests>
        <FailedTest id="2">
          <Name>CFirstTester::secondFunction</Name>
          <Identification>Requirement Key That is Tested</Identification>
          <InputAction>The Input Action</InputAction>
          <ExpectedOutcome>Expected Outcome</ExpectedOutcome>
          <RealOutcome>The Real Outcome</RealOutcome>
          <Result>Fail</Result>
          <FailureType>Assertion</FailureType>
          <Location>
    <File>D:\CppUnitTesting\FirstAcceptable\FirstTester.cpp</File>
            <Line>40</Line>
          </Location>
          <Message>assertion failed
    - Expression: false
    </Message>
        </FailedTest>

    And in case of succes to:

        <Test id="1">
          <Name>CFirstTester::firstTest</Name>
          <Identification>Requirement Key That is Tested</Identification>
          <InputAction>The Input Action</InputAction>
          <ExpectedOutcome>Expected Outcome</ExpectedOutcome>
          <RealOutcome>The Real Outcome</RealOutcome>
          <Result>Pass</Result>
        </Test>

    Also a Preconditons and Postcondition fields would be also usefull before the test.

    Now I would like to do this the most easy way for example in the test function:

    void CFirstTester::firstTest()
    {
           CPPUNIT_Identification("Requirement Key That is Tested");
          CPPUNIT_InputAction("The Input Action");         
          CPPUNIT_ExpectedOutcome("Expected Outcome");

          ... Code ...

          if ( test )
          {
               CPPUNIT_RealOutcome("The Real Outcome")
               CPPUNIT_Result("Pass");
          }
          else
          {
               CPPUNIT_RealOutcome("The Real Outcome")
               CPPUNIT_Result("Fail");
               CPPUNIT_ASSERT( test );
    }

    An easier version commenting would be also good for example instead of
    CPPUNIT_RealOutcome("The Real Outcome");
    would be
    CPPUNIT_COMMENT("RealOutcome","The Real Outcome");

    I have ideas how to make this but for it I would need to rewrite big portions from CppUnit.
    Now I am not a CppUnit expert, I have just started playing around with him. My question is there is anybody out there who knows how to achive what I want with the most minimal modification to CppUnit and most minimal effort?

     
    • Andrey Melnikov

      Andrey Melnikov - 2004-01-30

      Look at Mock Objects for C++  http://mockpp.sourceforge.net/ and its expectation classes.

       
    • Ekart Laszlo

      Ekart Laszlo - 2004-02-02

      I have looked at mock objects. That is a different library for creating mock objects. What I want is to to use cppunit and to create testing reports that can be viewed by an auditor. In other words I want to trace into the report file.

       
    • Baptiste Lepilleur

      It look like you're trying to do functional testing rather than unit testing. You might want to give a look at testing framework designed for this, such as http://www.codesourcery.com/qm/qmtest.

      Anyway, this can be done in cppunit by using your own singleton TestListener. The listener track the test being executed, and you call some methods on it (in your test) to set the 'expected' and 'actual' data associated with the current test.

      You can then add a XMLOutputterHook to add your data in the xml result (see the clocker plug-in for an example).

      Baptiste.

       
    • Ekart Laszlo

      Ekart Laszlo - 2004-03-17

      I had time to complete a part of the solution. It looks workable. However the singleton listener a hack and it is not nice. It would be nice change if in case of success there would be a TestFailure similar structure that could take in messages. The Test class code throw exception in case of error and this how the TestFailure is added. Looking from design style I do not think that throwing exceptions in case of success is appropiate. But we need to reach from client test function CppUnit internals. We would need architectural changes for this. So this remains open question.

      I will present the full solution here, it might help other people. Just need to write a new XMLOutputter. I do not like that this one puts the failed ones in front. I would like that the test plans would be in order. (Just need to find free time)

      Thanks for responses.

       
    • Ekart Laszlo

      Ekart Laszlo - 2004-03-25

      Finished the full solution to the problem.
      If someone needs it than send me an e-mail.
      The code is not the best quality writed in big
      hurry but it ok.

      Thanks for responses again.

       
    • Anonymous - 2004-07-19

      hi Ekart,
      I would love to have a look at your solution.
      Can you mail me the same ?

      Thanks
      Raj

       
    • daraujo

      daraujo - 2004-11-23

      I also would like to have a look at your solution, pls send me an email. Thanks in advance!

       
    • vivid

      vivid - 2004-11-29

      I would also like to see your solution :D thanks

       
    • Ekart Laszlo

      Ekart Laszlo - 2005-02-04

      Hi again almoust after I year.
      Well When I have finised the solution I have uploaded the source code to the feature request:

      https://sourceforge.net/tracker/index.php?func=detail&aid=923069&group_id=11795&atid=361795

      It is not exactly the best quality but it is ok (I wrote in 2 days and I did not had any knowledge of CppUnit before that)

      Here is an xls for the xml output, I forgot to put on thet package if I remember:

      -----------------------
      <?xml version="1.0" encoding="ISO-8859-1"?>

      <xsl:stylesheet version="1.0"
      xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

      <xsl:template match="/">
          <html>
          <body>
       
          <table border="1" summary="TheErrorReport">

              <!-- Main Record Begin -->
                     
              <xsl:for-each select="TestRun/TestCases/Test">
             
              <tr bgcolor="#0000FF">
                  <th>Test Id</th>
                  <th>Name</th>
                  <th>Requirement Key</th>
                  <th>Input Action</th>
                  <th>Expected Outcome</th>
                  <th>Real Outcome</th>
                  <th>Result</th>
          </tr>
             
          <tr>
                  <td> <xsl:value-of select="@id"/> </td>       
                  <td> <xsl:value-of select="Name"/> </td>
                 
                  <td>
                  <xsl:for-each select="ReqKey">
                      <xsl:value-of select="."/> <BR/>
                  </xsl:for-each>
                  </td>
                 
                  <td> <xsl:value-of select="InputAction"/> </td>
                  <td> <xsl:value-of select="ExpectedOutcome"/> </td>
                  <td> <xsl:value-of select="RealOutcome"/> </td>
                  <td> <xsl:value-of select="Result"/> </td>
          </tr>
             
              <!-- Main Record End  -->       
             
          <!-- Failed Information BEGIN -->
             
              <xsl:if test="FailedInfo">
             
          <tr bgcolor="#FF0000">
              <th COLSPAN="7"> Failed Info </th>
          </tr>
         
          <tr>
                      <th> FailureType </th>
                      <th COLSPAN="3"> File </th>
                      <th> Line </th>
                      <th COLSPAN="2"> Message </th>
              </tr>
             
              <tr>
                      <td> <xsl:value-of select="FailedInfo/FailureType"/> </td>
                      <td COLSPAN="3"> <xsl:value-of select="FailedInfo/Location/File"/> </td>
                      <td> <xsl:value-of select="FailedInfo/Location/Line"/> </td>
                      <td COLSPAN="2"> <xsl:value-of select="FailedInfo/Message"/> </td>
              </tr>
             
              </xsl:if>
             
              <!-- Additonal info Begin -->
             
              <xsl:if test="TRACE">
                     
          <tr bgcolor="#00FF00">
                  <th COLSPAN="7"> TRACE </th>
          </tr>
         
              <xsl:for-each select="TRACE">
          <tr>
              <td COLSPAN="7"> <xsl:value-of select="."/> </td>
          </tr>
              </xsl:for-each>
             
              </xsl:if>
             
          <!-- Additional info End -->
                     
              </xsl:for-each>

          </table>
         
          <BR/>
          <BR/>

          <!-- Summary Begin -->
          <table summary="Summary" width="25%" border="1">
                  <tr>
                          <th COLSPAN="2">Statistics</th>
                  </tr>
                  <tr>
                          <th>Tests</th>
                          <td> <xsl:value-of select="TestRun/Statistics/Tests"/> </td>
                  </tr>
                  <tr>
                          <th>FailuresTotal</th>
                          <td> <xsl:value-of select="TestRun/Statistics/FailuresTotal"/> </td>
                  </tr>
                  <tr>
                          <th>Errors</th>
                          <td> <xsl:value-of select="TestRun/Statistics/Errors"/> </td>
                  </tr>
                  <tr>
                          <th>Failures</th>
                          <td> <xsl:value-of select="TestRun/Statistics/Failures"/> </td>
                  </tr>
          </table>
          <!-- Summary End -->

          </body>
          </html>
      </xsl:template></xsl:stylesheet>
      -------------------------

       

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks