Creating custom CPPUnit suite from a file

Help
Lizozom
2007-11-28
2013-04-22
  • Lizozom
    Lizozom
    2007-11-28

    Hello,

    I'm writing a CPPUnit test application that uses the standard CPPUnit macros and normally runs a mass of tests. All the tests are normally placed (heirarchaly) at a suite called mSuite and are ran using a single runner called mRunner.

    I would like to allow a user to run a list of tests from a text file that looks like this:

    TestFixture1::TestMethod1
    TestFixture1::TestMethod2
    TestFixture2::TestMethod2
    ....

    I would like to avoid creating the huge hash required for mapping the text to fixture and method names, so I'm trying to find a better solution.

    I tried resolving the path of the test using resolveTestPath and adding it to another suite called mCustomSuite, but since the suite and runner assume ownership over the tests, the program crashed when the destructor is called (The destructors ofcourse are virtual and it is impossible to override their behaviour).

    I also tried making a copy of the test I resolved from the path, but CPPUnit doesn't allow any copy constructors and the members are private, hence inheritance won't help here.

    I'm wondering is there a nice and structurized way to do what I want: To execute a custom list of tests.

    Thanks in advance for your help,
    Liz

     
    • Hi there Liz,
           I wanted to do the same thing: build a dynamic test suite by reading it from an .ini file, but on the way of getting there stumbled onto, more or less, the "ownership" problem.
           In my case I build 2 test suites:the first contains all existing tests, the second contains only the tests that I select from the first suite; and I run only the second suite. The problem is that I cannot "clean up" - both suites will try do delete the same pointers and this will crash the program.
         
           Did you have any luck in solving your problem?

      Regards,
           Dragos

      P.S.
           The mfc test runner seems to provide the functionality of creating a dynamic test - or at least running just a particular test from a suite - I have just began analyzing it - maybe we can find a solution inside it.

       
    • Niels Dekker
      Niels Dekker
      2008-10-06

      Hi Liz and Dragos,

      I think I have solved your problem... at least for my project (and hopefully for my collegues at LKEB <www.lkeb.nl> as well).  I wrote a TestSelection class, having two data members: a pointer to a Test created by TestFactoryRegistry, and a TestSuite containing /references/ to the selected Test objects.  So this TestSuite data member does /not/ own the tests created by the registry!

      Each reference to a selected Test is held by another class that I wrote: TestReference. This class is derived from Test; it forwards each public virtual function from its base class to its data member, the reference.  Except for the destructor!

      The TestSuite of my TestSelection class "owns" the TestReference objects, but when a TestReference is destructed, it won't destruct the originally selected Test object.

      TestSelection reads the selection from an XML file. I think XML is preferable to a simple plain text file, because of the hierarchical nature of CppUnit's Test class: a Test may own one or more other Test objects, that all may own multiple Test objects as well, recursively. Each XML tag <Test> has one attribute: the name of the test it represents.

      Please let me know if you're still interested...

      PS The TestSelection class was developed with help from my collegues at LKEB, as well as from members of the ACCU mailing list.

       
      • Hi there Niels,
               What you are describing sounds great :) I sure am interested in seeing your approach. Could you give more details about this TestReference class?  Can you release the code under a GPL licence? (or one of it's flavours)

               Looking forward to more information :)
                  Cheers,
                      Dragos

         
    • Niels Dekker
      Niels Dekker
      2008-10-18

      Hi Dragos,

      Thanks for being interested  :-)  The TestReference class that I wrote is very simple. Still I think it would be useful to have it added to CppUnit.  Here's approximately what it looks like:

        class TestReference: public Test
        {
        public:
          // Constructor.
          explicit TestReference(Test& arg)
          :
          m_test(arg) {}

          // Destructor does NOT destruct the Test object it referenced.
          ~TestReference() {}

        private:

          // Its only data member.
          Test& m_test;

          // Overriding all virtual functions of Test:

          void run(TestResult* result)
          {
            m_test.run(result);
          }

          int countTestCases() const
          {
            return m_test.countTestCases();
          }

          int getChildTestCount() const
          {
            return m_test.getChildTestCount();
          }

          Test* getChildTestAt(int index) const
          {
            return m_test.getChildTestAt(index);
          }

          std::string getName() const
          {
            return m_test.getName();
          }

          bool findTestPath(const std::string& name, TestPath& path) const
          {
            return m_test.findTestPath(name, path);
          }

          bool findTestPath(const Test* test, CppUnit::TestPath& path) const
          {
            return m_test.findTestPath(test, path);
          }

          Test* findTest( const std::string& name ) const
          {
            return m_test.findTest(name);
          }

          TestPath resolveTestPath(const std::string& path) const
          {
            return m_test.resolveTestPath(path);
          }
         
          Test* doGetChildTestAt(int) const
          {
            CPPUNIT_FAIL("Cannot access protected member of m_test!");
            return 0;
          }
         
          void checkIsValidIndex(int)
          {
            CPPUNIT_FAIL("Cannot access protected member of m_test!");
          }
        };

      Is it clear to you?

      Kind regards,
      --
      Niels Dekker
      http://www.xs4all.nl/~nd/dekkerware
      Scientific programmer at LKEB, Leiden University Medical Center

       
      • Hi Niels,
            I will try to build something based on your instructions as soon as I have some time (I'm a bit swapped right now :) )
            I'll get back to you with how it goes or with other questions if I get stuck :)

           Thanks,
               Dragos

         
    • Niels Dekker
      Niels Dekker
      2008-10-22

      Note that the TestReference class I just posted looks a bit similar to CppUnit::TestDecorator.  But TestDecorator does actually delete the m_test it holds, so it can't be used to solve our "ownership problem".

      The documentation of CppUnit::TestDecorator suggests otherwise, but I think it's outdated. So I just reported:
        "[ 2185407 ] TestDecorator does assume ownership!"
      http://sourceforge.net/forum/forum.php?thread_id=1880533&forum_id=37108

       
    • Niels Dekker
      Niels Dekker
      2008-10-22

      Oops, the correct link to my little report ("TestDecorator does assume ownership!")is:
      https://sourceforge.net/tracker/index.php?func=detail&aid=2185407&group_id=11795&atid=111795

       
    • Niels Dekker
      Niels Dekker
      2008-10-26

      Lizozom wrote on 2007-11-28:
      > I would like to allow a user to run a list of tests
      > from a text file that looks like this:

      > TestFixture1::TestMethod1
      > TestFixture1::TestMethod2
      > TestFixture2::TestMethod2

      When using the TestReference class that I posted here before (2008-10-18), you can write a function to add references of the tests listed in such a text file to your custom test suite, as follows:

      void addTestsFromTextFileToTestSuite(
          Test& allTests, // All tests from the registry.
          const std::string& fileName, // Input file name.
          TestSuite& testSuite) // Custom test suite.
      {
          std::ifstream fileStream( fileName.c_str() );
          std::string testName;

          while ( getline(fileStream, testName) )
          {
              Test* foundTest = allTests.findTest(testName);
              testSuite.addTest(new TestReference(*foundTest));
          }
      }

      Note that the foundTest objects are owned by allTests, while the created TestReference objects are owned by testSuite. This works well because TestReference does not delete foundTest.

      So do you think I've solved your problem?
      Anyway, I still prefer to have the names of the selected tests in an XML file...

       
    • Hi Niels,
         
          I had some yesterday to look at the problem. addTestsFromTextFileToTestSuite is working just great (using the TestReference class). Thanks.

         Another solution would be to have a ModifiedTestSuite that is exactly the same as TestSuite except in one aspect : will not delete the tests in the suite

      void
      ModifiedTestSuite::deleteContents()
      {
        //int childCount = getChildTestCount();
        //for ( int index =0; index < childCount; ++index )
        //  delete getChildTestAt( index );

        m_tests.clear();
      }

         Using this class addTestsFromTextFileToTestSuite would become

      void addTestsFromTextFileToTestSuite(
      Test& allTests, // All tests from the registry.
      const std::string& fileName, // Input file name.
      ModifiedTestSuite& testSuite) // Custom test suite.
      {
      std::ifstream fileStream( fileName.c_str() );
      std::string testName;

      while ( getline(fileStream, testName) )
      {
      Test* foundTest = allTests.findTest(testName);
      modifiedTestSuite.addTest(foundTest));
      }
      }

        - I tested it and it works - no memory leaks. What's your opinion on this approach?

        

       
    • Hi all,
          I have now a first version of the program that parses an XML file and then creates a custom test suite (also includes generating the xml) - let me know if you are interested.

       
      • Niels Dekker
        Niels Dekker
        2008-11-13

        Dragos Andronic wrote: "I have now a first version of the program that parses an XML file and then creates a custom test suite (also includes generating the xml) - let me know if you are interested."

        Of course, I'm interested  :-)  Especially because I'm curious to know in what aspects your implementation differs from mine.  (As used here at LKEB.)

        BTW, indeed, I think that your approach of having a modified (non-owning) version of TestSuite would also solve the ownership problem.

         
    • [long message warning :) ]

      The XML structure will be something this:

      <?xml version="1.0"?>
      <TestConfiguration>
         
          <Registry name="FirstRegistry" testSuiteName="TestSuiteName">
              <test name="Test1" addToRunner="true"/>
              <test name="Test2" addToRunner="true"/>
              <test name="Test3" addToRunner="true"/>
             
          </Registry>
         
          <Registry name="SecondRegistry" testSuiteName="TestSuiteName">
              <test name="Test1" addToRunner="true"/>
              <test name="Test2" addToRunner="true"/>
              <test name="Test3" addToRunner="true"/>
             
          </Registry>
         
          ...
      </TestConfiguration>

      If we modify the addToRunner attribute to something else than "true" the test will not be executed

      Now for the code: ( I stripped down to code to the important parts)

      #define CHECKHR(hr)\ {\     if( hr != S_OK)\     return false;\ }

      bool prepareSuiteToRun(/*in*/CString& pathToXMLFile,/*out*/TestSuite*& allTestSuite,/*out*/ ModifiedTestSuite*& suiteToRun)
      {
          CComPtr<IXMLDOMDocument> pXMLDoc;
          HRESULT hr = S_FALSE;
          hr = pXMLDoc.CoCreateInstance(CLSID_DOMDocument);
          CHECKHR(hr);
          VARIANT_BOOL bSuccess = false;
          CComVariant nameOfXML(pathToXMLFile);
          hr = pXMLDoc->load(nameOfXML,&bSuccess);
          CHECKHR(hr);
         
          /*
           *    position ourselves to the TestConfiguration node
           */
          CComPtr<IXMLDOMNode> pXMLTestConfigurationNode;
          hr = pXMLDoc->selectSingleNode(CComBSTR("TestConfiguration"),&pXMLTestConfigurationNode);
          CHECKHR(hr);
         
          /*
           *    iterate through the registries
           */
          CComPtr<IXMLDOMNodeList> pXMLListOfRegistries;
          hr = pXMLTestConfigurationNode->get_childNodes(&pXMLListOfRegistries);
          CHECKHR(hr);

          long nrOfRegistries = -1;
          hr = pXMLListOfRegistries->get_length(&nrOfRegistries);
          CHECKHR(hr);
          if(nrOfRegistries <= 0)
          {
              return false;
          }

          /*
           *    for each registry
           */
          for(int regIndex = 0 ; regIndex < nrOfRegistries; ++regIndex)
          {
              CComPtr<IXMLDOMNode> registry;
              hr = pXMLListOfRegistries->get_item(regIndex,&registry);
              CHECKHR(hr);
              CComPtr<IXMLDOMNamedNodeMap> registryAttributes;
              hr = registry->get_attributes(&registryAttributes);
              CHECKHR(hr);
              CComPtr<IXMLDOMNode> registryNameNode;
              hr = registryAttributes->getNamedItem(CComBSTR("name"),&registryNameNode);
              CHECKHR(hr);
              CComVariant registryName;
              hr = registryNameNode->get_nodeValue(&registryName);
              CHECKHR(hr);
              CppUnit::TestFactoryRegistry::getRegistry(string(CStringA(registryName))).addTestToSuite(allTestSuite);

              /*
               *    we only have one suite per registry
               */
              CComPtr<IXMLDOMNode> testSuiteNameNode;
              hr = registryAttributes->getNamedItem(CComBSTR("testSuiteName"),&testSuiteNameNode);
              CHECKHR(hr);
              CComVariant testSuiteName;
              hr = testSuiteNameNode->get_nodeValue(&testSuiteName);

              CComPtr<IXMLDOMNodeList> pXMLListOfTests;
              hr = registry->get_childNodes(&pXMLListOfTests);
              CHECKHR(hr);
             
              long nrOfTests = -1;
              hr = pXMLListOfTests->get_length(&nrOfTests);
              CHECKHR(hr);
              if(nrOfTests > 0)
              {
                  /*
                   *    for each test in the suite
                   */
                  for (int testIndex = 0; testIndex < nrOfTests ; ++testIndex)
                  {
                      CComPtr<IXMLDOMNode> testNode;
                      hr = pXMLListOfTests->get_item(testIndex,&testNode);
                      CHECKHR(hr);
                      CComPtr<IXMLDOMNamedNodeMap> testAttributes;
                      hr = testNode->get_attributes(&testAttributes);
                      CHECKHR(hr);
                      CComPtr<IXMLDOMNode> testNameNode;
                      hr = testAttributes->getNamedItem(CComBSTR("name"),&testNameNode);
                      CHECKHR(hr);
                      CComVariant testName;
                      hr = testNameNode->get_nodeValue(&testName);
                      CHECKHR(hr);

                      CComPtr<IXMLDOMNode> addToRunnerNode;
                      hr = testAttributes->getNamedItem(CComBSTR("addToRunner"),&addToRunnerNode);
                      CHECKHR(hr);
                      CComVariant addToRunnerValue;
                      hr = addToRunnerNode->get_nodeValue(&addToRunnerValue);
                      CHECKHR(hr);

                      CStringA addIfTrue(addToRunnerValue);
                      if(addIfTrue.CompareNoCase("true") == 0)
                      {
                          CStringA completeTestName(testSuiteName);
                          completeTestName = completeTestName + "::" + CStringA(testName);
                          string testNameString(completeTestName);
                          Test* foundTest = allTestSuite->findTest(testNameString);
                          if(foundTest)
                          {
                              suiteToRun->addTest(foundTest);
                          }
                      }
                  }
              }
          }
          return true;
      }

      void AddWhiteSpaceToNode( CComPtr<IXMLDOMDocument>& pDom,CComBSTR bstrWs,CComPtr<IXMLDOMElement>& pNode )
      {
          HRESULT hr;
          CComPtr<IXMLDOMText> pws = 0;
          CComPtr<IXMLDOMNode> pBuf = 0;
          hr = pDom->createTextNode(bstrWs,&pws);
          hr = pNode->appendChild(pws,&pBuf);
      }

      bool createXML(const CString& pathToXML,const vector<string> listOfRegistries)
      {
          CComPtr<IXMLDOMDocument> pXMLDoc;
          HRESULT hr = S_FALSE;
          hr = pXMLDoc.CoCreateInstance(CLSID_DOMDocument);
          CHECKHR(hr);
       
          //the processing instructions <?xml version="1.0"?>
          CComPtr<IXMLDOMProcessingInstruction> pi= 0;
          CComBSTR xml("xml");
          CComBSTR xmlVersion("version = '1.0'");
          hr = pXMLDoc->createProcessingInstruction(xml,xmlVersion,&pi);
          CComPtr<IXMLDOMNode> pTmpNode = 0;
          hr =  pXMLDoc->appendChild(pi,&pTmpNode);
          if(pTmpNode)
          {
              pTmpNode.Release();
              pTmpNode = 0;
          }
          CComBSTR bstr_wsn(L"\n");
          CComBSTR bstr_wsnt(L"\n\t");
          CComBSTR bstr_wsntt(L"\n\t\t");

          //the TestConfiguration category
          CComPtr<IXMLDOMElement> pCategoryInfo = 0;
          hr = pXMLDoc->createElement(CComBSTR("TestConfiguration"),&pCategoryInfo);
          hr = pXMLDoc->appendChild(pCategoryInfo,&pTmpNode);
          if(pTmpNode)
          {
              pTmpNode.Release();
              pTmpNode = 0;
          }

          //add new line and tab for formatting
          AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pCategoryInfo);

          for (unsigned int indexOfRegistry = 0 ; indexOfRegistry < listOfRegistries.size(); ++indexOfRegistry )
          {
              AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pCategoryInfo);
              TestSuite* temporarySuite = new CppUnit::TestSuite("Temporary Suite");
              CppUnit::TestFactoryRegistry::getRegistry(listOfRegistries[indexOfRegistry]).addTestToSuite(temporarySuite);
              CComPtr<IXMLDOMElement> pRegistryInfo;
              hr = pXMLDoc->createElement(CComBSTR("Registry"),&pRegistryInfo);
              CHECKHR(hr);
              pRegistryInfo->setAttribute(CComBSTR("name"),CComVariant(listOfRegistries[indexOfRegistry].c_str()));
              //get suite name
              if(temporarySuite->getChildTestCount() !=1)
              {
                  return false;
              }
              string suiteName = temporarySuite->getChildTestAt(0)->getName();
              pRegistryInfo->setAttribute(CComBSTR("testSuiteName"),CComVariant((temporarySuite->getChildTestAt(0)->getName()).c_str()));
              pCategoryInfo->appendChild(pRegistryInfo,&pTmpNode);
              if(pTmpNode)
              {
                  pTmpNode.Release();
                  pTmpNode = 0;
              }

             for(int nrOfTests = 0; nrOfTests < temporarySuite->getChildTestAt(0)->getChildTestCount(); ++nrOfTests)
             {
               //indentation for better formatting
                  AddWhiteSpaceToNode(pXMLDoc,bstr_wsntt,pRegistryInfo);
                 CComPtr<IXMLDOMElement> pTestInfo;
                 hr = pXMLDoc->createElement(CComBSTR("test"),&pTestInfo);
                 CHECKHR(hr);
                 CStringA testNameString = (temporarySuite->getChildTestAt(0)->getChildTestAt(nrOfTests)->getName()).c_str();
                 int tokenPos = testNameString.Find("::");
                 pTestInfo->setAttribute(CComBSTR("name"),CComVariant(testNameString.Right(testNameString.GetLength() - tokenPos - 2)));
                 pTestInfo->setAttribute(CComBSTR("addToRunner"),CComVariant("true"));
                 pRegistryInfo->appendChild(pTestInfo,&pTmpNode);
                 if(pTmpNode)
                 {
                     pTmpNode.Release();
                     pTmpNode = 0;
                 }
             }
             
              AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pRegistryInfo);
              AddWhiteSpaceToNode(pXMLDoc,bstr_wsnt,pCategoryInfo);
              delete temporarySuite;
              temporarySuite = 0;
          }
       
          AddWhiteSpaceToNode(pXMLDoc,bstr_wsn,pCategoryInfo);
          hr = pXMLDoc->save(CComVariant(pathToXML));
          CHECKHR(hr);
          return true;
      };

      int _tmain(int argc, _TCHAR* argv[])
      {
          //with a switch you can use this exe just to generate the xml
         
          ....
          CoInitializeEx(0, COINIT_MULTITHREADED);
          vector<string> listOfRegistries;
          listOfRegistries.push_back(string("FirstModuleTest"));
          listOfRegistries.push_back(string("SecondModuleTest"));
         
          CppUnit::TestSuite *allTestsSuite = new CppUnit::TestSuite("All tests");
          CppUnit::ModifiedTestSuite* suiteToRun = new CppUnit::ModifiedTestSuite( "Tests to run");
          if(!prepareSuiteToRun(pathToXml,allTestsSuite,suiteToRun))
          {
              return -1;
          }
          CppUnit::TextUi::TestRunner runner;
            runner.addTest(suiteToRun );
            runner.setOutputter(new CppUnit::XmlOutputter( &runner.result(),std::cerr));
            bool wasSucessful = runner.run();
         
          if(allTestsSuite)
          {
              delete allTestsSuite;
              allTestsSuite = 0;
          }
          CoUninitialize();
          return wasSucessful ? 0 : 1;
      }

          There are some limitations atm (like the fact that only one suite per registry is allowed - and I don't check the default registry - only named ones) specific to the tests I have - but those could be easily solved.

          If you have other questions/suggestions for improvement let me know :)

      Cheers,
            Dragos